A Non-Strict JSON UDF (JSMN)

[Decibel]

On 1/13/2018 at 10:06 AM, Jos said:

Added JsonDump() and updated/attached a new Zipfile in first post

 

Jos

In _Json_TokenDump, could you put a Local in front of $cObjPath so it doesn't cause a warning when we require that variables be declared?

expandcollapse popup

Func _Json_TokenDump(ByRef $Json, $Ptr, ByRef $Next, $ObjPath = "")
    If $Next = -1 Then Return Null

    Local $Token = DllStructCreate("int;int;int;int", $Ptr + ($Next * 20))
    Local $Type = DllStructGetData($Token, 1)
    Local $Start = DllStructGetData($Token, 2)
    Local $End = DllStructGetData($Token, 3)
    Local $Size = DllStructGetData($Token, 4)
    $Next += 1

    If $Type = 0 And $Start = 0 And $End = 0 And $Size = 0 Then ; Null Item
        $Next = -1
        Return Null
    EndIf

    Switch $Type
        Case 0 ; Json_PRIMITIVE
            Local $Primitive = StringMid($Json, $Start + 1, $End - $Start)
            Switch $Primitive
                Case "true"
                    Return True
                Case "false"
                    Return False
                Case "null"
                    Return Null
                Case Else
                    If StringRegExp($Primitive, "^[+\-0-9]") Then
                        Return Number($Primitive)
                    Else
                        Return Json_StringDecode($Primitive)
                    EndIf
            EndSwitch

        Case 1 ; Json_OBJECT
            For $i = 0 To $Size - 1 Step 2
                Local $Key = _Json_TokenDump($Json, $Ptr, $Next)
                Local $cObjPath = $ObjPath & "." & $Key                                     ;<--add Local here
                Local $Value = _Json_TokenDump($Json, $Ptr, $Next, $ObjPath & "." & $Key)
                If Not ($Value = False) Then
                    If Not IsString($Key) Then
                        $Key = Json_Encode($Key)
                    EndIf
                    ; show the key and its value
                    ConsoleWrite("+-> " & $cObjPath & '  =' & $Value & @CRLF)
                EndIf
            Next
            Return False
        Case 2 ; Json_ARRAY
            Local $sObjPath = $ObjPath
            Local $Array[$Size]
            For $i = 0 To $Size - 1
                $sObjPath = $ObjPath & "[" & $i & "]"
                $Value = _Json_TokenDump($Json, $Ptr, $Next, $sObjPath)
                If $Value <> False Then
                    ; show the key and its value
                    ConsoleWrite("+=> " & $sObjPath & "=>" & $Value & @CRLF)
                EndIf
            Next
            $ObjPath = $sObjPath
            Return False

        Case 3 ; Json_STRING
            Local $LastKey = Json_StringDecode(StringMid($Json, $Start + 1, $End - $Start))
            Return $LastKey
    EndSwitch
EndFunc   ;==>_Json_TokenDump

 

[zone97]

First, thank you for creating this UDF.. Hopefully, it will work for what I need it for. Second.. I know nothing about json files.. Having to learn for a project. I have a json with data similar to the following.

{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}

What I need to do is read the ID# and then take the largest # and add 1, then add a new section, with in this case id: 104.. name: Example 4 etc. Lastly write the json back to disk. Could I do this with this UDF?

[Jos]

That is why I made the json_dump() udf to help you with the logic of how the retrieve the appropriate field value:

#include <json.au3>
$data = '{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}'
Global $object = Json_Decode($data)
ConsoleWrite("!== Json_TokenDump ============================================================================================================" & @CRLF)
Json_Dump($data)

Start there and post in case you still have questions.

Jos

[zone97]

Yes, I played with the

Json_Dump($data)

and it produces and an output, but cannot figure out how to correlate that into variables I can manipulate.  Maybe I'm just blind? I mean I can single out items with this.

ConsoleWrite(Json_Get($Data1, '["items"][0]["id"]') & @LF)

but i'm looking for more of maybe $target = $Data1[items][$x][id] kinda thing so I can loop $x from 0 to 10 and get each id#? if that makes any sense? or is all the data from the decoded json stored in array? that then I can just push data into and convert the array back the json data.

Edited April 23, 2018 by zone97

[Jos]

Shouldn't be too hard, just look closely at the produced output in the dump! 
something like this you mean?   

#include <json.au3>
$data = '{"items":[{"id":100,"name":"Example 1","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":101,"name":"Example 2","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]},{"id":103,"name":"Example 3","hash":null,"folder":null,"download":null,"version":null,"website":null,"files":[]}]}'
Global $object = Json_Decode($data)
Local $i = 0
While 1
    $id = json_get($object, '.items[' & $i & '].id')
    If @error Then ExitLoop
    ConsoleWrite('id:' & $id & @CRLF)
    $i += 1
WEnd

Jos

Edited April 23, 2018 by Jos

[zone97]

That's it.. Thank you.. Helps make that data clearer.. I was in the process of attempting something similar with do/until but this works...

Side request. is the Json_put only designed to do one element at a time, or is there a preferred method to add a section?  Such as needing to add this..

{
      "id",104
      "name","Example 4"
      "hash",null
      "folder",null
      "download",null
      "version",null
      "website",null
      "files",[]
    }

 

Edited April 23, 2018 by zone97

[zone97]

Answered my own question from above..

 

Edited April 23, 2018 by zone97

[zone97]

Is parsing json data normally very slow? loading the file in...

Local $file_json_raw = FileRead($ini_file_path & "\data.json")
Local $file_json_data = Json_Decode($file_json_raw)

takes about 6 seconds.

Parsing the data into an array,

While 1 ; list of all servers
        $server_id = json_get($file_json_data, '.servers[' & $a & '].id')
        If ($server_id == "") Then ExitLoop
        $server_name = json_get($file_json_data, '.servers[' & $a & '].name')
        $server_name_override = json_get($file_json_data, '.servers[' & $a & '].nameoverride')
        If ($server_name_override <> "") Then $server_name = $server_name_override
    WEnd

takes about 30 seconds.

The json file is about 900mb, the server section consist of about 2100 entries.  Is there a quicker way to be handing the data?

[TheDcoder]

Well 6 seconds is reasonable for parsing a 900 MB JSON file. And you might want to consider using Json_ObjPut and Json_ObjGet directly since they are faster than using Json_Get/Put

[Jos]

6 minutes ago, zone97 said:

Is parsing json data normally very slow?

Fact or opinion? 

Parsing a 900Mb JSON encoded file doesn't sound as a common thing to me as I am used to using JSON in REST calls used for API's, which in general aren't that big.
Maybe you need you see whether you need to change your approach and use RegEx?

Jos

Edited April 27, 2018 by Jos

[zone97]

7 minutes ago, TheDcoder said:

Well 6 seconds is reasonable for parsing a 900 MB JSON file. And you might want to consider using Json_ObjPut and Json_ObjGet directly since they are faster than using Json_Get/Put

I'm very new to using this UDF.. how do they differ? does json_objGet, create a 2d array of the values of a section? (because that is ultimately what the goal is for me.)

[TheDcoder]

@zone97 No, it is a wrapper for the Scripting.Dictionary object or simply a dictionary variable. You can access the values inside a dictionary by using the Json_ObjGet and Json_ObjPut functions, you should see the examples and read the code for those functions so you could understand better

[TheXman]

I have found a couple of small issues with the Json_Dump() logic.  One issue is related to comparing different data types.  The other issue is probably just an oversight.

If you run the following example with the latest version of the json UDF, you will get the following console output:

#include "json.original.au3" ; <== Modify this line to point to your copy of json.udf
;~ #include "json.modified.au3" ; <== Modify this line to point to your copy of json.udf


example()

Func example()

    #cs - PrettyPrint of JSON const below
    {
        "null_item": null,
        "zero_primitive_item": 0,
        "false_item": false,
        "true_item": true,
        "string_item": "This is a string",
        "array_item": [0,1,2]
    }
    #ce
    
    Const $JSON_STRING = '{"null_item": null, "zero_primitive_item": 0,"false_item": false,"true_item": true,"string_item": "This is a string","array_item": [0,1,2]}'
    
    Json_Dump($JSON_STRING)

EndFunc

+-> .null_item  =
+-> .true_item  =True
+-> .string_item  =This is a string
+=> .array_item[1]=>1
+=> .array_item[2]=>2

As you can see, items that have a primitive value of 0 or false, do not get written out at all.  This was because of comparing different data types.  Also notice that values that are null do not display "null", they simply display no value.  Since it is a dump, I would assume that you would want to see all values. 

I have made changes to 5 lines of code in the UDF.  When run with my modifications, you get the following output.  All lines get displayed with their correct literal values.

+-> .null_item  =null
+-> .zero_primative_item  =0
+-> .false_item  =false
+-> .true_item  =true
+-> .string_item  =This is a string
+=> .array_item[0]=>0
+=> .array_item[1]=>1
+=> .array_item[2]=>2

I tried to leave the existing code as close to original as possible.  In other words, instead of rewriting the functions, I made the smallest modifications possible in order to get the correct result. 

I have attached the original and the modified versions of the UDF so that you can easily do a DIFF to see the changes. Each of the modified lines is also denoted by a comment that says ";XC - Changed line" to make them easy to FIND.

 

I just notice that I did make a few more changes to the UDF.  I added an extra function, "Json_ObjGetItems(ByRef $Object)".  Because I use very strict directives, I made sure that all variables were declared before being used so I would stop getting warnings during compilation.

 

Hopefully my modifications pass muster and can be added to the UDF.

 

Json.modified.au3

Json.original.au3

Edited September 11, 2018 by TheXman
Added that I made a few additional changes to the UDF and why.

[Jos]

Will have a look at what you have done in a week or 2. Remind me when you don't have a reply by the end of the month.

[TheXman]

On 9/12/2018 at 8:00 AM, Jos said:

Will have a look at what you have done in a week or 2. Remind me when you don't have a reply by the end of the month.

@Jos

As requested, this is an end of the month reminder. 

Because I have posted a version that fixes the issues that I identified, this reminder is more for those who don't keep up with this topic.

Being a Chief Technology Officer, I can empathize with the demands on one's time.  Any attention towards making this UDF as robust and reliable as possible, I'm certain, will be greatly appreciated by the many users of this very useful UDF, including me.

Thanks

[Jos]

Thanks for the reminder! It was needed. Will have a look at it tomorrow to see what you have done and let you know.

Stay tuned,
Jos

[Jos]

@TheXman, I see that there are quit some differences between your  "Json.original.au3" file and the json.au3 in the current zip file in the first post. Did you use another version for starters?

I do see the issue you report and will look at the fix you propose for that.

Jos

Edited October 1, 2018 by Jos

[Jos]

Updated first post with merge of the lines needed to fix the reported issue. All other stuff skipped for now. 

Thanks,
Jos

[TheXman]

@Jos Thanks for applying the fixes related to the json_dump() output.

 

3 hours ago, Jos said:

I see that there are quit some differences between your  "" file and the json.au3 in the current zip file in the first post.

The rest of the changes were not bug-related so they aren't as important.  One of the additional changes was to add a helper function, Json_ObjGetItems().  Json_ObjGetKeys() already existed so it seemed only logical to have a Json_ObjGetItems() too.  The other changes are just the declaration of variables before their use.  Unfortunately, I'm one of those old school programmers that strives to have a "clean" compile, meaning no warnings.  Since I always stipulate that all vars must exist before their use, when compiling scripts with the JSON UDF, it would throw multiple warnings about already declared vars and undeclared vars.  So I went through and made sure that all variables were declared only once prior to use.  Now the JSON UDF compiles "cleanly" with the strictest compilation directives, "#AutoIt3Wrapper_AU3Check_Parameters=-w 3 -w 4 -w 5 -w 6 -d".

Again, your efforts towards maintaining the JSON UDF, and AutoIt in general, are greatly appreciated.

Edited October 1, 2018 by TheXman
Added that I went through and fixed all undeclared and already declared warnings from AU3Check

[Jos]

Ah ok, well this is originally not my code and really only added the JSON_DUMP() function to it as I thought it would be very useful to others.

I am fine when we also clean up the code further and make an update for that. I honestly think these UDF's should be added to the standard AutoIt3 distribution but that means somebody needs to build the helpfilepages and examples.

Jos