jq UDF - A Powerful & Flexible JSON Processor v1.7.4

This UDF brings the power and flexibility of jq to AutoIt scripts.  jq is an open-source, powerful, and flexible command-line based JSON processor.  As it says on their website, jq is like 'sed' for JSON.  jq can be used for the simplest of tasks like retrieving JSON objects and values (parsing), to very advanced JSON processing using its numerous built-in functions and conditional processing.  Its built-in functions can handle math, selection, conditional processing, mapping, object and array manipulation, flattening, reduction, grouping, and much more.  You can even create your own jq functions.  You can learn more about jq and even play with it in real-time, using jq's online jq playground, all on their website.

Here and some helpful links to get you more familiar with jq, what can be done with it, its built-in functions, and its syntax.

• jq Home Page: https://jqlang.github.io/jq/
• jq Manual: https://jqlang.github.io/jq/manual/
• jqWiki (FAQ, Cookbook, Advanced Topics) https://github.com/jqlang/jq/wiki

jq is a single 32 or 64 bit executable that has no other dependencies.  Just like using the SQLite UDF, the only requirement to use this UDF is that the jq executable reside in a location in which the UDF can execute it.  The latest win32 & win64 versions have been included in the UDF download.  You can always get newer versions from the jq website.

 

jq at a high level

Like 'sed', jq reads JSON in, either through STDIN or one or more files, processes it thru one or more "filters", and outputs the results.  You can, optionally, supply "options" that affect how it reads the input, where it gets its "filters", and how it writes its output.  It looks a little like this:

JSON ---> jq processor (using supplied filters and options) ---> Output

So in jq lingo, you basically use "Filters" to tell jq what you want it to do.  So in the UDF file, that is why the main functions ( _jqExec() and _jqExecFile() ) refer to filters and options.  Please make note that jq works with relatively strict JSON.  This means that all JSON read must be conform to the standard.  Luckily, jq is pretty good at identifying where a format error exists in non standard JSON.

The jq UDF

There are 2 main funtions in the UDF file, _jqExec and jqExecFile.  With these 2 functions, you can pretty much do anything that jq can do.  The only difference between to two functions is whether the JSON is supplied by a string or a file.  The 2 primary functions simply call the jq executable with the supplied information, after properly formatting the parameters.  There are additional functions in the UDF to easily pretty-print your json, compact-print your json, dump the json data with its associated paths, and see if specific JSON keys exist, but they all just execute the _jqExec or _jqExecFile function with the proper filter.  There are also a couple of extra functions to display what version of the UDF and jq executable you are currently using.  There are also a couple of functions to enable and disable logging of jq information for debugging purposes.  Most of the jq UDF file functions return an @error if unsuccessful.  Some also include @extended info.  Please see the actual function headers for more information on their usage and return values.

The 2 primary functions below just format your jq request and pass it on the jq executable.  The functions will also properly escape double quotes (") that are used in the filter.  For most simple tasks, you just need to supply the JSON source and a filter.

_jqExec($sJson, $sFilter, $sOptions = Default, $sWorkingDir = Default)

Or

_jqExecFile($sJsonFile, $sFilter, $sOptions = Default, $sWorkingDir = Default)

Using jq in your script

As stated earlier, the jq executable must reside somewhere where the script can locate and execute it.  The _jqInit() function always has to be executed before any jq processing occurs.  _jqInit() merely locates the executable or uses the supplied path.  It also clears any previous debug log.  The jq UDF folder contains a jq example script that has several examples to how to do some of the most common JSON processing tasks.  Here are a few examples to get you started:

How to pretty-print some JSON

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '{"fruits":[{"Apple":{"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}'
$sCmdOutput = _jqPrettyPrintJson($sJson)
ConsoleWrite(@CRLF & "Pretty-Print JSON" & @CRLF & $sCmdOutput & @CRLF)

How to compact-print some JSON

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '{ "fruits" : [{"Apple"   : {"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}'
$sCmdOutput = _jqCompactPrintJson($sJson)
ConsoleWrite(@CRLF & "Compact-Print JSON" & @CRLF & $sCmdOutput & @CRLF)

Dump JSON data (paths and values)

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '{ "fruits" : [{"Apple"   : {"color":"Red","season":"Fall"}}, {"Banana":{"color":"Yellow","season":"Summer"}}]}'
$sCmdOutput = _jqDump($sJson)
ConsoleWrite(@CRLF & "Dump JSON paths and values" & @CRLF & $sCmdOutput & @CRLF)

How to GET JSON values

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '{"Apple"   : {"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}'
$sFilter    = '.Banana.color'
$sCmdOutput = _jqExec($sJson, $sFilter)

ConsoleWrite("Get color of banana" & @CRLF)
ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF)
ConsoleWrite("Filter : " & $sFilter & @CRLF)
ConsoleWrite("Output : " & $sCmdOutput & @CRLF)

or

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '{"Apple"   : {"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}'
$sFilter    = 'getpath(["Banana", "color"])'
$sCmdOutput = _jqExec($sJson, $sFilter)

ConsoleWrite("Get color of banana" & @CRLF)
ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF)
ConsoleWrite("Filter : " & $sFilter & @CRLF)
ConsoleWrite("Output : " & $sCmdOutput & @CRLF)

 

Check for the existence of a key

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '{"Apple":{"color":"Red","season":"Fall"}, "Banana":{"color":"Yellow","season":"Summer"}}'
$sFilter    = '.Banana | has("color")'
$sCmdOutput = _jqExec($sJson, $sFilter)

ConsoleWrite("Check for existence of color key within Banana object" & @CRLF)
ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF)
ConsoleWrite("Filter : " & $sFilter & @CRLF)
ConsoleWrite("Output : " & $sCmdOutput & @CRLF)

Count of how many Items in an object

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '{"Apple":{"color":"Red"}, "Banana":{"color":"Yellow","season":"Summer"}}'
$sFilter    = '.Banana | length'
$sCmdOutput = _jqExec($sJson, $sFilter)

ConsoleWrite("How many items in the Banana object" & @CRLF)
ConsoleWrite("Input: : " & _jqCompactPrintJson($sJson) & @CRLF)
ConsoleWrite("Filter : " & $sFilter & @CRLF)
ConsoleWrite("Output : " & $sCmdOutput & @CRLF)

How to PUT/Create/Modify JSON

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sInput     = ""
$sFilter    = 'setpath(["Apple","color"];"Red") | setpath(["Banana","color"];"Yellow") | setpath(["Banana","season"];"Summer")'
$sOptions   = '-n' ;required if no input supplied
$sCmdOutput = _jqExec($sInput, $sFilter, $sOptions)

ConsoleWrite("Update/Create JSON" & @CRLF)
ConsoleWrite("Filter : " & $sFilter & @CRLF)
ConsoleWrite("Output : " & @CRLF & $sCmdOutput & @CRLF)

List all of the fruits (top-level keys)

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '{"Apple":{"color":"Red"}, "Banana":{"color":"Yellow","season":"Summer"}}'
$sFilter    = 'keys | .[]'
$sCmdOutput = _jqExec($sJson, $sFilter)

ConsoleWrite("List all top-level keys (fruits)" & @CRLF)
ConsoleWrite("Input  : " & $sJson & @CRLF)
ConsoleWrite("Filter : " & $sFilter & @CRLF)
ConsoleWrite("Output : " & @CRLF & $sCmdOutput & @CRLF)

Calculate the sum of all of the objects' price * qty

#include "jq.au3"

;Initialize jq environment
_jqInit()
If @error Then Exit ConsoleWrite("ERROR: Unable to initialize jq - @error = " & @error & @CRLF)

$sJson      = '[{"id":1,"price":20.00,"qty":10},{"id":2,"price":15.00,"qty":20.25},{"id":3,"price":10.50,"qty":30}]'
$sFilter    = 'map(.price * .qty) | add'
$sCmdOutput = _jqExec($sJson, $sFilter)

ConsoleWrite("Calculate the sum of all of the objects' price * qty" & @CRLF)
ConsoleWrite("Input  : " & $sJson & @CRLF)
ConsoleWrite("Filter : " & $sFilter & @CRLF)
ConsoleWrite("Output : " & $sCmdOutput & @CRLF)

The examples above, and the ones in the example files, merely scratch the surface of what jq can do.  It may look intimidating at first but it really isn't that bad once you start playing with it.  You can find several more examples and play around with them or create your own using jqPlayground.

If you have any questions regarding the UDF, or how to perform a certain task using jq, I'll try my best to answer them.  Since jq has been around for a while now, there's also several jq-related questions and answers on StackOverflow.