Jump to content

JSON UDF in pure AutoIt


AspirinJunkie
 Share

Recommended Posts

I was already excited about it, when you @AspirinJunkie talked about it in the German forum. The "give it a try" reasons are promising and understandable. I wil have a deeper look soon. Thanks for your effort and well defined examples (description etc.) 👍 .

Best regards
Sven

Stay innovative!

Spoiler

🌍 Au3Forums

🎲 AutoIt (en) Cheat Sheet

📊 AutoIt limits/defaults

💎 Code Katas: [...] (comming soon)

🎭 Collection of GitHub users with AutoIt projects

🐞 False-Positives

🔮 Me on GitHub

💬 Opinion about new forum sub category

📑 UDF wiki list

✂ VSCode-AutoItSnippets

📑 WebDriver FAQs

👨‍🏫 WebDriver Tutorial (coming soon)

Link to comment
Share on other sites

Congratulations !!

file          size          JSMN     Pure AutoIt
;~ ------------------------------------------------------------------------
;~                   sample4.json    451 bytes      2.64  ms        0.77  ms
;~                      test.json      1,46 KB      3.93  ms        2.48  ms
;~                       USD.json      2,12 KB     10.68  ms        6.42  ms
;~  110-pcp-ytd-12-1895-2016.json      5,43 KB     54.20  ms       13.42  ms
;~          earth_meteorites.json       240 KB   1339.18  ms      546.57  ms
;~                     Test2.json      2,04 MB   4180.48  ms     2172.16  ms

 

I know that I know nothing

Link to comment
Share on other sites

To be fair, I have attached a file where the ratio is the other way around.
The reason here is that there is massive use of JSON unicode char escapes in the file.
The escapes resolution seems to be implemented more efficiently in jsmn.

To be honest it is still a big mystery to me how a parser written in pure AutoIt and executed by interpreter can be faster (even if it were only in a few cases) than one in machine code.
On the other hand, it must be said that jsmn does not peddle great speed, but rather the small code size and the portability of the code.
These promises are simply kept by jsmn.

Aerosmith.json

Link to comment
Share on other sites

4 hours ago, ioa747 said:

can you give me please an example How to assign sample4.json to an array

You would have to describe in more detail what you mean by array, because behind the attribute "people" there is already an array - more precisely an array of maps.
You would have to describe the exact shape if you want something else.

If you want to have a 2D array from the map array, then you have to do yourself:
 

Spoiler
#include <JSON.au3>
#include <Array.au3>

; read and parse the file into a AutoIt-structure:
$sString = FileRead("sample4.json")
$vData = _JSON_Parse($sString)

; the array in the attribute "people":
Global $aMapArray = $vData.people
; the list of used attributes inside a dataset:
Global $aAttrKeys = MapKeys($aMapArray[0])
; create and set size of result array:
Global $aResultArray[UBound($aMapArray)][UBound($aAttrKeys)]

; process the datasets in the map-array "people" and fill the result-array
Global $iC = 0
For $mDataset In $aMapArray
    For $i = 0 To UBound($aAttrKeys) - 1
        $aResultArray[$iC][$i] = $mDataset[$aAttrKeys[$i]]
    Next
    $iC += 1
Next

; present the result-array
_ArrayDisplay($aResultArray, "People", "", 64, Default, _ArrayToString($aAttrKeys))

; convert the array into json-syntax (you can overwrite the sample4.json with this)
ConsoleWrite(_JSON_Generate($aResultArray) & @CRLF)

 


Edit:
For such special cases as creating a 2D array from an array-of-maps, you can also write yourself a reusable function.
Curiously, I have written such a function for exactly this case (from a not yet published UDF):

Spoiler
#include <JSON.au3>
#include <Array.au3>

; read and parse the file into a AutoIt-structure:
Global $sString = FileRead("sample4.json")
Global $vData = _JSON_Parse($sString)

; convert the map-array into a 2D-array
Global $aHeader, $aResult = _td_MapArrayTo2DArray($vData.people, $aHeader)

_ArrayDisplay($aResult, "People", "", 64, Default, _ArrayToString($aHeader))



; #FUNCTION# ======================================================================================
; Name ..........: _td_MapArrayTo2DArray()
; Description ...: converts a map-array (a 1D-array with maps as values) into 2 2D-array where the colums = keys
; Syntax ........: _td_MapArrayTo2DArray($aMapArray, ByRef $aHeader)
; Parameters ....: $aMapArray     - the input map-array (1D-array with maps as values)
;                  $aHeader       - variable which get's overwritten with the header values (=attribute names and column id)
; Return values .: Success:       2D-array
;                  Failure: Null
;                     @error = 1: aMapArray is not a array
;                     @error = 2: array-value is not a map (@extend = index of wrong value)
; Author ........: AspirinJunkie
; Last changed ..: 2022-09-26
; Version .......: 0.5
; =================================================================================================
Func _td_MapArrayTo2DArray($aMapArray, ByRef $aHeader)
    If UBound($aMapArray, 0) <> 1 Then Return SetError(1, UBound($aMapArray, 0) <> 1, Null)

    Local $mHeaders[], $aResult[1][1]
    Local $mMap, $sKey, $sAttribute

    Local $iRow = 0, $nAttribs = 0
    For $mMap In $aMapArray
        If Not IsMap($mMap) Then Return SetError(2, $iRow, Null)

        For $sKey In MapKeys($mMap)
            If Not MapExists($mHeaders, $sKey) Then
                $mHeaders[$sKey] = $nAttribs
                $nAttribs += 1
                If UBound($aResult, 2) < $nAttribs Then Redim $aResult[UBound($aResult, 1)][UBound($aResult, 2) * 2]
            EndIf

            If UBound($aResult, 1) <= $iRow Then Redim $aResult[UBound($aResult, 1) * 2][UBound($aResult, 2)]

            $aResult[$iRow][$mHeaders[$sKey]] = $mMap[$sKey]
        Next
        $iRow += 1
    Next

    Redim $aResult[$iRow][$nAttribs]

    Dim $aHeader[UBound($mHeaders)]
    For $sAttribute In MapKeys($mHeaders)
        $aHeader[$mHeaders[$sAttribute]] = $sAttribute
    Next

    Return $aResult
EndFunc   ;==>_td_MapArrayTo2DArray

 

 

Edited by AspirinJunkie
Link to comment
Share on other sites

how put array back to .json? The result is different from sample4.json

; https://www.autoitscript.com/forum/topic/209502-json-udf-in-pure-autoit/?do=findComment&comment=1512686

#include <JSON.au3>
#include <Array.au3>

; read and parse the file into a AutoIt-structure:
$sString = FileRead("sample4.json")
$vData = _JSON_Parse($sString)

; the array in the attribute "people":
Global $aMapArray = $vData.people
; the list of used attributes inside a dataset:
Global $aAttrKeys = MapKeys($aMapArray[0])
; create and set size of result array:
Global $aResultArray[UBound($aMapArray)][UBound($aAttrKeys)]

; process the datasets in the map-array "people" and fill the result-array
Global $iC = 0
For $mDataset In $aMapArray
    For $i = 0 To UBound($aAttrKeys) - 1
        $aResultArray[$iC][$i] = $mDataset[$aAttrKeys[$i]]
    Next
    $iC += 1
Next

; present the result-array
_ArrayDisplay($aResultArray, "People", "", 64, Default, _ArrayToString($aAttrKeys))

; convert the array into json-syntax (you can overwrite the sample4.json with this)
ConsoleWrite(_JSON_Generate($aResultArray) & @CRLF)

$MyFile = @ScriptDir & "\sample5.json"
_SetFile($MyFile, _JSON_Generate($aResultArray))

;----------------------------------------------------------------------------------------
Func _SetFile($sFile, $sData, $iFormat = 266) ; FileWrite Alternative
    Local $hFileOpen = FileOpen($sFile, $iFormat)
    If $hFileOpen = -1 Then
        Return SetError(1, 0, "")
    EndIf
    Local $msg = FileWrite($hFileOpen, $sData)
    FileClose($hFileOpen)
    Return $msg
EndFunc   ;==>_SetFile
;----------------------------------------------------------------------------------------

 

I know that I know nothing

Link to comment
Share on other sites

7 minutes ago, ioa747 said:

how put array back to .json? The result is different from sample4.json

Of course it is different. You wanted to create an array from the data in sample4.json and overwrite the file with it (your words).
Before there was no classic array in the sample4.json but an array with maps (=objects) as values.
If you want to have it in the original structure again, then you should not transform it, of course, but leave it as it is.

Or you make an array of maps out of your 2D array again.
Anyway - tell us what exactly you want to do - what is the concrete goal?

Link to comment
Share on other sites

Sorry for the poor explanation.
What I want is to take the elements from the .json file into an array, modify them, and finally save them back to the .json  file for next using, as I would do with an .ini file.

Sorry for my poor English too

Τhank you for your time and patience!

I know that I know nothing

Link to comment
Share on other sites

50 minutes ago, ioa747 said:

What I want is to take the elements from the .json file into an array, modify them, and finally save them back to the .json  file for next using, as I would do with an .ini file.


Is there any particular reason why you absolutely need to convert this to an array in between?
You can edit the data directly without wildly converting:

Spoiler
#include <JSON.au3>

; read and parse the file into a AutoIt-structure:
$sString = FileRead("sample4.json")
$vData = _JSON_Parse($sString)

; change the name of all people:
For $i = 0 To UBound($vData.people) - 1
    _JSON_addChangeDelete($vData, "people[" & $i & "].firstName", "Jon")
    _JSON_addChangeDelete($vData, "people[" & $i & "].lastName", "Doe " & $i + 1)
Next

; convert the data structure into json-syntax:
ConsoleWrite(_JSON_Generate($vData) & @CRLF)

 


Otherwise you would first have to convert the map array into a 2D array, then make your changes there and then convert the 2D array into a map array again.
This is feasible and I have also a function for the reverse direction, but for me it seems artificially cumbersome:

Spoiler
#include <JSON.au3>
#include <Array.au3>

; read and parse the file into a AutoIt-structure:
Global $sString = FileRead("sample4.json")
Global $vData = _JSON_Parse($sString)

; convert the map-array into a 2D-array
Global $aHeader, $a2D = _td_MapArrayTo2DArray($vData.people, $aHeader)

; change some values inside the 2D-Array:
For $i = 0 To UBound($a2D) - 1
    $a2D[$i][0] = "Jon"
    $a2D[$i][1] = "Doe " & $i + 1
Next

; re-convert the 2D-array into a array of maps:
$aMapArray = _td_ArrToMaps($a2D, $aHeader)

; overwrite the array inside the data structure:
$vdata["people"] = $aMapArray

;  convert the structure into json-syntax:
ConsoleWrite(_JSON_Generate($vData) & @CRLF)


; #FUNCTION# ======================================================================================
; Name ..........: _td_MapArrayTo2DArray()
; Description ...: converts a map-array (a 1D-array with maps as values) into 2 2D-array where the colums = keys
; Syntax ........: _td_MapArrayTo2DArray($aMapArray, ByRef $aHeader)
; Parameters ....: $aMapArray     - the input map-array (1D-array with maps as values)
;                  $aHeader       - variable which get's overwritten with the header values (=attribute names and column id)
; Return values .: Success:       2D-array
;                  Failure: Null
;                     @error = 1: aMapArray is not a array
;                     @error = 2: array-value is not a map (@extend = index of wrong value)
; Author ........: AspirinJunkie
; Last changed ..: 2022-09-26
; Version .......: 0.5
; =================================================================================================
Func _td_MapArrayTo2DArray($aMapArray, ByRef $aHeader)
    If UBound($aMapArray, 0) <> 1 Then Return SetError(1, UBound($aMapArray, 0) <> 1, Null)

    Local $mHeaders[], $aResult[1][1]
    Local $mMap, $sKey, $sAttribute

    Local $iRow = 0, $nAttribs = 0
    For $mMap In $aMapArray
        If Not IsMap($mMap) Then Return SetError(2, $iRow, Null)

        For $sKey In MapKeys($mMap)
            If Not MapExists($mHeaders, $sKey) Then
                $mHeaders[$sKey] = $nAttribs
                $nAttribs += 1
                If UBound($aResult, 2) < $nAttribs Then Redim $aResult[UBound($aResult, 1)][UBound($aResult, 2) * 2]
            EndIf

            If UBound($aResult, 1) <= $iRow Then Redim $aResult[UBound($aResult, 1) * 2][UBound($aResult, 2)]

            $aResult[$iRow][$mHeaders[$sKey]] = $mMap[$sKey]
        Next
        $iRow += 1
    Next

    Redim $aResult[$iRow][$nAttribs]

    Dim $aHeader[UBound($mHeaders)]
    For $sAttribute In MapKeys($mHeaders)
        $aHeader[$mHeaders[$sAttribute]] = $sAttribute
    Next

    Return $aResult
EndFunc   ;==>_td_MapArrayTo2DArray


; #FUNCTION# ======================================================================================
; Name ..........: _td_ArrToMaps()
; Description ...: converts a 2D-Array (rows=records, columns=values) into a set of key-value maps (every record = key-value map)
; Syntax ........: _Array2Maps(ByRef $aArray, [$vHeader = Default, [$bHeader = False]])
; Parameters ....: $aArray        - the input array
;                  $vHeader       - Default: the header elements (attribute names) are taken from the first row
;                                   String: Semicolon separated entrys for the header (=attribute names), must have at least as many elements as there are columns
;                                   Array: the header values as an 1D-array (number of elements must be number of columns in $aArray)
;                  $bHeader       - True: header exists in the first row, if $bHeader = String then this header is skipped and the $vHeader is taken
;                                   False = no header row exists - $vHeader must be a string
; Return values .: Success:       1D-array with record-objects of type Map
;                  Failure: False
;                     @error = 1: no attribute names given because $vHeader = Default and $bHeader = False (no header in first row)
;                     @error = 2: $aArray is not a 2D-Array (@extended = num of dimensions of $aArray)
;                     @error = 3: error when processing $vHeader
;                     @error = 4: Less unique attribute names given than attributes themselves exists (n(header elements) < n(columns))
;                     @error = 5: $vHeader is an array with wrong dimensions
; Author ........: AspirinJunkie
; Last changed ..: 2020-09-26
; Version .......: 0.5
; =================================================================================================
Func _td_ArrToMaps(ByRef $aArray, $vHeader = Default, $bHeader = True)
    If $vHeader = Default And $bHeader = False Then Return SetError(1, 0, False)
    If UBound($aArray, 0) <> 2 Then Return SetError(2, UBound($aArray), False)

    ; prepare the header values
    If $vHeader = Default Then ; header values from first row
        $bHeader = True ; field must be written somewhere

        Local $aHeader[UBound($aArray, 2)]
        For $iI = 0 To UBound($aHeader) - 1
            $aHeader[$iI] = $aArray[0][$iI]
        Next

    ElseIf IsArray($vHeader) Then ; header from array
        If UBound($vHeader, 0) <> 1 Or UBound($vHeader, 1) <> UBound($aArray, 2) Then Return SetError(5, UBound($aArray, 2), False)
        Local $aHeader = $vHeader
        $bHeader = False

    Else ; header from string
        Local $aHeader = StringRegExp($vHeader, '\h*\K("(?>[^"]+|"")*"|[^";]++)\h*', 3)
        If @error Then Return SetError(3, @error, False)
        $bHeader = False
    EndIf

    ; process the header values
    For $iI = 0 To UBound($aHeader) - 1
        If StringRegExp($aHeader[$iI], '"(?> [^"]+ | "" )*"') Then $aHeader[$iI] = StringReplace(StringRegExpReplace($aHeader[$iI], '(?s)^\h*"(.*)"\h*$', "\1"), '""', '"', 0, 1)
    Next
    $aHeader = _ArrayUnique($aHeader, 0, 0, 1, 0)
    If UBound($aHeader) < UBound($aArray, 2) Then Return SetError(4, UBound($aHeader), False)

    ; prepare return Array
    Local $aRet[UBound($aArray) - ($bHeader ? 1 : 0)]

    For $iI = ($bHeader ? 1 : 0) To UBound($aArray) - 1
        Local $mMap[]
        For $iJ = 0 To UBound($aArray, 2) - 1
            $mMap[$aHeader[$iJ]] = $aArray[$iI][$iJ]
        Next
        $aRet[$iI - ($bHeader ? 1 : 0)] = $mMap
    Next

    Return $aRet
EndFunc   ;==>_td_ArrToMaps

 

 

50 minutes ago, ioa747 said:

Sorry for my poor English too

I'm not a native speaker either - your English sounds sufficiently good to me.

Link to comment
Share on other sites

Great work @AspirinJunkie! Wish we had something like this sooner :D

I don't like the JSMN-based UDF because it embeds the library as a binary blob in code, so it's great to have a native AutoIt implementation. And implementing objects with maps is a nice touch too!

On 1/18/2023 at 11:18 PM, AspirinJunkie said:

To be honest it is still a big mystery to me how a parser written in pure AutoIt and executed by interpreter can be faster (even if it were only in a few cases) than one in machine code.

I think it boils down to having the need to manually copy the parsed data from native data structures to AutoIt datatypes, a native AutoIt implementation would entirely side-step the issue as there's no need to convert parsed values :)

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

8 hours ago, TheDcoder said:

Wish we had something like this sooner :D

We had - I had already posted the UDF on autoit.de in 2017.
Also in this forum I had posted the old version here.

Now I have just made an extra thread for it so that there is a central point of contact for this.
But thank you for your feedback!
If it can be used effectively and helps some people, then it's already worth it.

9 hours ago, TheDcoder said:

I think it boils down to having the need to manually copy the parsed data from native data structures to AutoIt datatypes, a native AutoIt implementation would entirely side-step the issue as there's no need to convert parsed values

The basic approach of JSMN is that this lib does not parse the whole string, but it only sets breakpoints with the info where an element is located in the string.
The final parsing of the elements happens only when the element is accessed.

I had briefly skimmed the JSMN UDF and if I have interpreted it correctly (can also be wrong - have not examined it intensively), then each token is already parsed in the Json_Decode, no matter whether the element is used or not.
This makes the performance advantage of jsmn invalid, but the result is pretty much the same as with my UDF - a nested structure of AutoIt data types (with the difference that dictionaries are used instead of maps for objects).
So of course my UDF is not faster than the compiled jsmn-code but only faster than the processing around it (and also not in all cases).

Edited by AspirinJunkie
Link to comment
Share on other sites

  • 3 weeks later...

Hi AspirinJunkie,

Thank you for your Json-Udf. I use it for evaluating my Discogs queries.
Attached is a recursive dump script for my maps.

#include "Json.au3"
#include <String.au3>

Local $sJson = FileRead("..\Discogs-Abfragen\The Beatles - Lovely Rita.json")
Local $mDiscogsQuery = _JSON_Parse($sJson)

Global $iIndent = 0, $sIndent = "    "
ConsoleWrite("Map $mDiscogsQuery" & @CRLF)
_DumpMap($mDiscogsQuery)

Func _DumpMap(ByRef $mMap)
    For $Key In MapKeys($mMap)
        _WriteElement($Key, $mMap[$Key])
    Next
EndFunc

Func _DumpArray(ByRef $aArray, $Key)
    For $i = 0 To UBound($aArray) - 1
        _WriteElement($Key & "[" & $i & "]", $aArray[$i])
    Next
EndFunc

Func _WriteElement($Key, ByRef $vValue)
    $iIndent += 1
    Local $sInd = _StringRepeat($sIndent, $iIndent)
    Local $sType = VarGetType($vValue)
    If $sType = "String" Then $vValue = StringReplace($vValue, @LF, @CRLF & $sInd & $sIndent) ; Indent for multi-line strings
    If $sType = "Keyword" And IsKeyword($vValue) = 2 Then $vValue = "NULL" ; NULL
    ConsoleWrite($sInd & $Key & " (" & $sType & "): " & $vValue & @CRLF)

    If $sType = "Map" Then _DumpMap($vValue)
    If $sType = "Array" Then _DumpArray($vValue, $Key)
    $iIndent -= 1
EndFunc

MapDump.png.5d87bb4a76d5b61ff525d47fa5c3c5de.png

Paul

 

Link to comment
Share on other sites

  • 1 month later...

Just a very, very little thing...

I think description of this funktion is wrong -copy Past error ;)

; #FUNCTION# ======================================================================================
; Name ..........: _JSON_Generate
; Description ...: convert a JSON-formatted string into a nested structure of AutoIt-datatypes

Must be like in the funktion list description...

- converts a nested AutoIt data structure into a JSON structured string.

Link to comment
Share on other sites

I noticed this myself a while ago and have already adjusted it in my local master version.
But for an extra upload it was then still too low for me.
But sure - I have uploaded the silent update even.

By the way, the parameter description in the function was also garbage.

Such errors are typical for me, since I am not a native english speaker and therefore such things do not jump directly into my eye.

Link to comment
Share on other sites

  • 3 weeks later...

Hello,

I'm only getting started with JSON. Can this UDF handle GeoJSON, or should I look at the other UDFs?

Here's a sample:

{
  "type": "FeatureCollection",
  "timestamp": "2023-04-15T05:50:27Z",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "@id": "way/156798372",
        "name": "Location1"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          1.5978915,
          43.9694576
        ]
      },
      "id": "way/156798372"
    },
    {
      "type": "Feature",
      "properties": {
        "@id": "way/231992599",
        "name": "Location2"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          1.8692822,
          44.0008908
        ]
      },
      "id": "way/231992599"
    }
  ]
}

Thank you.

Link to comment
Share on other sites

57 minutes ago, littlebigman said:

Hello,

I'm only getting started with JSON. Can this UDF handle GeoJSON, or should I look at the other UDFs?

Here's a sample:

{
  "type": "FeatureCollection",
  "timestamp": "2023-04-15T05:50:27Z",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "@id": "way/156798372",
        "name": "Location1"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          1.5978915,
          43.9694576
        ]
      },
      "id": "way/156798372"
    },
    {
      "type": "Feature",
      "properties": {
        "@id": "way/231992599",
        "name": "Location2"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          1.8692822,
          44.0008908
        ]
      },
      "id": "way/231992599"
    }
  ]
}

Thank you.

is that what you mean?

#include <JSON.au3>
$test = '{"type":"FeatureCollection","timestamp":"2023-04-15T05:50:27Z","features":[{"type":"Feature","properties":{"@id":"way/156798372","name":"Location1"},"geometry":{"type":"Point","coordinates":[1.5978915,43.9694576]},"id":"way/156798372"},{"type":"Feature","properties":{"@id":"way/231992599","name":"Location2"},"geometry":{"type":"Point","coordinates":[1.8692822,44.0008908]},"id":"way/231992599"}]}'
$JSON = _JSON_Parse($test)
$features = _JSON_Get($JSON, 'features')
For $I = 0 To UBound($features) - 1
    $coordinates = _JSON_Get($JSON, 'features.['&$I&'].geometry.coordinates')
    $name = _JSON_Get($JSON, 'features.['&$I&'].properties.name')
    For $c = 0 To UBound($coordinates) - 1
        $coordinates = _JSON_Get($JSON, 'features.['&$I&'].geometry.coordinates.['&$c&']')
        MsgBox(0,$name,$coordinates)
    Next
Next

 

 

Edited by mccree
Link to comment
Share on other sites

1 hour ago, littlebigman said:

Can this UDF handle GeoJSON

As long it's valid JSON any parser worth its salt would be able to handle it, the internal structure doesn't matter.

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...