How to get number of sections in JSON file

[ur]

I have a github repository and when I run the below command it will retrieve a json file.

curl --user "user:password" https://api.github.com/repos/ukreddy-erwin/Build-war/commits > Desktop/uday.html

The output json file will be of the below format.

[
  {
    "sha": "525ddee5f25628f89c04980d7f683e9cd8f56b7a",
    "commit": {
      "author": {
        "name": "ukreddy",
        "email": "ukreddy@erwin.com",
        "date": "2016-12-26T12:27:14Z"
      },
      "committer": {
        "name": "ukreddy",
        "email": "ukreddy@erwin.com",
        "date": "2016-12-26T12:27:14Z"
      },
      "message": "changed gwt and gxt folder structure to build.xml",
      "tree": {
        "sha": "8d4f2cbca93efec18be8a17f25f78ad168e65347",
        "url": "https://api.github.com/repos/ukreddy-erwin/Build-war/git/trees/8d4f2cbca93efec18be8a17f25f78ad168e65347"
      },
      "url": "https://api.github.com/repos/ukreddy-erwin/Build-war/git/commits/525ddee5f25628f89c04980d7f683e9cd8f56b7a",
      "comment_count": 0
    },
    "url": "https://api.github.com/repos/ukreddy-erwin/Build-war/commits/525ddee5f25628f89c04980d7f683e9cd8f56b7a",
    "html_url": "https://github.com/ukreddy-erwin/Build-war/commit/525ddee5f25628f89c04980d7f683e9cd8f56b7a",
    "comments_url": "https://api.github.com/repos/ukreddy-erwin/Build-war/commits/525ddee5f25628f89c04980d7f683e9cd8f56b7a/comments",
    "author": {
      "login": "ukreddy-erwin",
      "id": 21240709,
      "avatar_url": "https://avatars.githubusercontent.com/u/21240709?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/ukreddy-erwin",
      "html_url": "https://github.com/ukreddy-erwin",
      "followers_url": "https://api.github.com/users/ukreddy-erwin/followers",
      "following_url": "https://api.github.com/users/ukreddy-erwin/following{/other_user}",
      "gists_url": "https://api.github.com/users/ukreddy-erwin/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/ukreddy-erwin/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/ukreddy-erwin/subscriptions",
      "organizations_url": "https://api.github.com/users/ukreddy-erwin/orgs",
      "repos_url": "https://api.github.com/users/ukreddy-erwin/repos",
      "events_url": "https://api.github.com/users/ukreddy-erwin/events{/privacy}",
      "received_events_url": "https://api.github.com/users/ukreddy-erwin/received_events",
      "type": "User",
      "site_admin": false
    },
    "committer": {
      "login": "ukreddy-erwin",
      "id": 21240709,
      "avatar_url": "https://avatars.githubusercontent.com/u/21240709?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/ukreddy-erwin",
      "html_url": "https://github.com/ukreddy-erwin",
      "followers_url": "https://api.github.com/users/ukreddy-erwin/followers",
      "following_url": "https://api.github.com/users/ukreddy-erwin/following{/other_user}",
      "gists_url": "https://api.github.com/users/ukreddy-erwin/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/ukreddy-erwin/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/ukreddy-erwin/subscriptions",
      "organizations_url": "https://api.github.com/users/ukreddy-erwin/orgs",
      "repos_url": "https://api.github.com/users/ukreddy-erwin/repos",
      "events_url": "https://api.github.com/users/ukreddy-erwin/events{/privacy}",
      "received_events_url": "https://api.github.com/users/ukreddy-erwin/received_events",
      "type": "User",
      "site_admin": false
    },
    "parents": [
      {
        "sha": "db4d61c5bbef555e6cc9f2e852aaf9d75772475c",
        "url": "https://api.github.com/repos/ukreddy-erwin/Build-war/commits/db4d61c5bbef555e6cc9f2e852aaf9d75772475c",
        "html_url": "https://github.com/ukreddy-erwin/Build-war/commit/db4d61c5bbef555e6cc9f2e852aaf9d75772475c"
      }
    ]
  }
]

So, I need to parse above file and get the number of opening and closing brackets {} .

So that would be the number of commits in the repository.

Any suggestion on how to parse this file of json format.

[ur]

This is the equivalent GET command of GitHub API.

GET /repos/:owner/:repo/commits

 

Edited December 28, 2016 by ur

[careca]

Quote

get the number of opening and closing brackets {}

StringInStr can find those, and retrieves the position.

[ur]

No I just need the count...Like number of sets...Is there any UDF for Json Files?

 

Edited December 28, 2016 by ur

[mLipok]

_Example()
Func _Example()
    Local $sJSON = ClipGet()
    StringReplace($sJSON,'{','')
    ConsoleWrite("! { count=" & @extended & @CRLF)
    StringReplace($sJSON,'}','')
    ConsoleWrite("! } count=" & @extended & @CRLF)
EndFunc

 

[ur]

May be this is wrong now.
But I got below input from support team that.

 

WHen we run the curl command differently as below.

we can get the total count of commits by looking at the Link header when making a request with only 1 commit per page.
For example, here's a request to the Linguist repository:
curl -I https://api.github.com/repos/github/linguist/commits?per_page=1.

Looking at our Link header, you can see the last page value:
Link: <https://api.github.com/repositories/1725199/commits?per_page=1&page=2>; rel="next", <https://api.github.com/repositories/1725199/commits?per_page=1&page=5454>; rel="last"
Since the last page is 5454, this means that there are a total of 5454 commits in that repository
 

How to get this number by parsing the output.

[ur]

This is the command we can run for example.

curl -I https://api.github.com/repos/github/linguist/commits?per_page=1

And the command output is as below.

HTTP/1.1 200 OK
Server: GitHub.com
Date: Thu, 29 Dec 2016 10:17:35 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 3448
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 57
X-RateLimit-Reset: 1483009802
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept
ETag: "9120c8c80a4342a0b46cad7086358638"
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
X-GitHub-Media-Type: github.v3; format=json
Link: <https://api.github.com/repositories/1725199/commits?per_page=1&page=2>; rel="next", <https://api.github.com/repositories/1725199/commits?per_page=1&page=5454>; rel="last"
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Content-Security-Policy: default-src 'none'
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
Vary: Accept-Encoding
X-Served-By: edf23fdc48375d9066b698b8d98062e9
X-GitHub-Request-Id: 673CD4B2:772D:DC0C69D:5864E2BF

So, if we observe the Link: line above, the number 5454 is visible, how to read this number from the output.?

 

Can anyone suggest that.

[careca]

If it's only that output maybe you can do a regex that uses that '&page=' '>;' and get what's in the middle.

[ur]

I didn't get chance to work on regular expressions before.

Can you share me any references.

[mLipok]

3 hours ago, ur said:

May be this is wrong now.
But I got below input from support team that.

 

WHen we run the curl command differently as below.

we can get the total count of commits by looking at the Link header when making a request with only 1 commit per page.
For example, here's a request to the Linguist repository:
curl -I https://api.github.com/repos/github/linguist/commits?per_page=1.

Looking at our Link header, you can see the last page value:
Link: <https://api.github.com/repositories/1725199/commits?per_page=1&page=2>; rel="next", <https://api.github.com/repositories/1725199/commits?per_page=1&page=5454>; rel="last"
Since the last page is 5454, this means that there are a total of 5454 commits in that repository
 

How to get this number by parsing the output.

This look like in TeamViewer API (look in my signature).
When github have to many information to send to you, then github is splitting this data into pages.
You should get each page and parse them in a loop.

btw.
I think you should try to use WinHTTP instead CURL, but for now I want to ask how you use CURL with AutoIt ?

 

Edited December 29, 2016 by mLipok

[JohnQSmith]

I'm guessing you're using CURL because you have either Cygwin or are using Windows 10's Subsystem for Linux. Here's a quick solution using that...

curl --silent -I https://api.github.com/repos/github/linguist/commits?per_page=1 | grep "^Link" | sed -e 's/.*page=\([0-9]*\)>; rel=\"last.*/\1/'

 

Edited December 29, 2016 by JohnQSmith
Trying to remove syntax highlighting

[JohnQSmith]

Here it is using WinHTTP as suggested by @mLipok

$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
$oHTTP.Open("GET", "http://api.github.com/repos/github/linguist/commits?per_page=1")
$oHTTP.Send()
$header = $oHTTP.Getresponseheader("Link")
$numCommits = StringRegExpReplace( $header, '.*page=([0-9]+)>; rel="las.*', '$1' )
ConsoleWrite($numCommits)

 

[ur]

This is my solution for this....using curl command

expandcollapse popup

;#include "ProcessEx.au3"
#include "ProcessEx_AddedNewEntryForLogging.au3"
#include <Array.au3>
#include <WinAPIFiles.au3>

Func NoChangesRequired($sBuildCommand = 'curl -I --user "userid:password" https://api.github.com/repos/erwin-inc/erwin-main/commits?per_page=1',$commitfile = @TempDir&"\git_erwin_commit.txt")

$valueRead = ProcessGitCommitCount($sBuildCommand)

If not FileExists($commitfile) Then
    FileWrite($commitfile,$valueRead)
    ;MsgBox(0,"","Changes are there.")
    return False
ELse
     Local $hFileOpen = FileOpen($commitfile, $FO_READ)
     If $hFileOpen = -1 Then
        ;MsgBox($MB_SYSTEMMODAL, "", "An error occurred when reading the file.")
        Return False
     EndIf

     ; Read the fist line of the file using the handle returned by FileOpen.
    Local $sFileRead = FileReadLine($hFileOpen, 1)

    ; Close the handle returned by FileOpen.
    FileClose($hFileOpen)
    If Number(StringStripWS($sFileRead,$STR_STRIPLEADING + $STR_STRIPTRAILING))= Number($valueRead) Then
        ;MsgBox(0,"","No change in commit")
        Return True
    Else
        ;MsgBox(0,"","Changes are there")
        FileDelete($commitfile)
        FileWrite($commitfile,$valueRead)
        Return False
    EndIf

EndIf

EndFunc

Func ProcessGitCommitCount($sBuildCommand)
$sOutput = _Process_RunCommand($PROCESS_RUNWAIT, $PROCESS_COMMAND & """" &$sBuildCommand &"""") ; Record the output
$Ret = @extended ; Note the exit code
;MsgBox(0,$Ret,$sOutput)
Local $strTemp = StringSplit($sOutput, @CRLF,$STR_ENTIRESPLIT)
;_ArrayDisplay($strTemp)


$u = ""
For $vElement In $strTemp
        ;$sString = $sString & $vElement & @CRLF

        If StringInStr($vElement,"Link:") then
            $u = $vElement
            ExitLoop
        EndIf

Next

$u = StringStripWS($u,$STR_STRIPLEADING + $STR_STRIPTRAILING)
If StringLen($u)=0 Then
    ;MsgBox(0,"","No output from GitHub")
    return "No output"
ElseIf StringInStr($u,"Link:") Then
    return ProcessHeader($u)
Else
    ;MsgBox(0,"","Some random unnecessary output")
    return "Some random unnecessary output"
EndIf
EndFunc

;to remove any empty strings in array and reduce its size
Func RemoveEmptyArrayValues(ByRef $arrLines)
  $intCount = 1
  While $intCount < UBound($arrLines)
         $arrLines[$intCount] = StringStripWS($arrLines[$intCount],$STR_STRIPLEADING + $STR_STRIPTRAILING)
         If StringLen($arrLines[$intCount])=0 Then
            _ArrayDelete($arrLines, $intCount)
            $intCount = $intCount - 1
         EndIf
         $intCount = $intCount + 1
   WEnd
EndFunc

Func RemoveOtherData(ByRef $arrLines)
  $intCount = 1
  While $intCount < UBound($arrLines)
         $arrLines[$intCount] = StringStripWS($arrLines[$intCount],$STR_STRIPLEADING + $STR_STRIPTRAILING)
         If StringInStr($arrLines[$intCount],">;") Then
             $arrLines[$intCount] = StringTrimRight($arrLines[$intCount], 2)
         Else
            _ArrayDelete($arrLines, $intCount)
            $intCount = $intCount - 1
         EndIf
         $intCount = $intCount + 1
   WEnd
EndFunc

Func ProcessHeader($header)
     $splittedArray = StringSplit ( $header, "per_page=1&page=" )
     ;_ArrayDisplay($splittedArray)
     RemoveEmptyArrayValues($splittedArray)
     ;_ArrayDisplay($splittedArray)
     RemoveOtherData($splittedArray)
     ;_ArrayDisplay($splittedArray)
     ;MsgBox(0,"",$splittedArray[UBound($splittedArray)-1])
     return $splittedArray[UBound($splittedArray)-1]
EndFunc

 

Edited January 2, 2017 by ur

[ur]

Original UDF and the modified one.

 

Original UDF link.

 

ProcessEx.au3

ProcessEx_AddedNewEntryForLogging.au3

[ur]

On 12/29/2016 at 7:37 PM, mLipok said:

This look like in TeamViewer API (look in my signature).
When github have to many information to send to you, then github is splitting this data into pages.
You should get each page and parse them in a loop.

btw.
I think you should try to use WinHTTP instead CURL, but for now I want to ask how you use CURL with AutoIt ?

 

Can you help me how to use WinHTTP so that I can use them in later scripts.

And I used CURL by the UDF mentioned above