Danp2 Posted November 7, 2022 Author Posted November 7, 2022 1 hour ago, TheXman said: I may be able to help flatten your learning curve by answering questions you may have or by possibly providing stripped down examples taken from my scripts Thanks for the offer for continued help. Can you share some basic details on your implementation? Did you communicate with the sgcWebSocketClient app via TCP? Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 7, 2022 Posted November 7, 2022 (edited) Yes, I communicate with the sgcWebSocket client through its, optional, internal TCP server. I needed to look back at some of my old scripts to refresh my memory as to why. I recalled that trying to either read from stdout or write to stdin caused AutoIt to crash, I can't remember which. I don't think that the console apps were designed to have external processes share stdout or stdin. That's probably why the developers of the console apps added an optional, internal TCP server to each their console apps. The basic framework that my scripts use: Open the sgcWebSocket console client using the optional TCP server, on localhost, listening on a specified port. Open a TCP socket to the client's console app Send websocket commands to a server, through the TCP socket, to the websocket client Listen for websocket responses, sent back to the client, using the TCP socket Enqueue those websocket responses using an internal response queue Process the one or more responses from my internal response queue Rinse and repeat the send/listen/enqueue/process loop as needed When finished send a close message to the websocket server close & shut down the tcp session That is the basic framework for both my send/receive and subscription models. Below is a current log from sample WebSocket client subscription script that I wrote last year. It's nice to know that it still works. The script was indirectly inspired by a link in this post by @jugador. The topic was related to decoding protobuf packets that were coming from a Yahoo Finance WebSocket server. As you can see, the script could handle decoding and parsing multiple protobuf messages per second from the WebSocket server. Sample Log: expandcollapse popup2022-11-10 12:17:11 Websocket client console window opened 2022-11-10 12:17:11 TCP connection to Websocket client established 2022-11-10 12:17:11 Connecting to WebSocket server (wss://streamer.finance.yahoo.com) 2022-11-10 12:17:12 Event = connected 2022-11-10 12:17:12 Subscribing to tickers (AMZN DOW WMT) 2022-11-10 12:17:12 Ticker message format: Ticker|Price|Change% 2022-11-10 12:17:13 Event = message Message = DOW|49.4541|1.77410126 2022-11-10 12:17:14 Event = message Message = WMT|141.16|1.69000244 2022-11-10 12:17:14 Event = message Message = AMZN|96.46|10.32 2022-11-10 12:17:15 Event = message Message = DOW|49.4541|1.77410126 2022-11-10 12:17:16 Event = message Message = AMZN|96.44|10.3000031 2022-11-10 12:17:17 Event = message Message = WMT|141.16|1.69000244 2022-11-10 12:17:17 Event = message Message = DOW|49.4541|1.77410126 2022-11-10 12:17:17 Event = message Message = AMZN|96.44|10.3000031 2022-11-10 12:17:18 Event = message Message = WMT|141.16|1.69000244 2022-11-10 12:17:18 Event = message Message = AMZN|96.41|10.2700043 2022-11-10 12:17:19 Event = message Message = WMT|141.169403|1.69940186 2022-11-10 12:17:19 Event = message Message = AMZN|96.415|10.2750015 2022-11-10 12:17:20 Event = message Message = AMZN|96.41|10.2700043 2022-11-10 12:17:21 Event = message Message = WMT|141.165|1.69499207 2022-11-10 12:17:21 Event = message Message = DOW|49.4541|1.77410126 2022-11-10 12:17:22 Event = message Message = DOW|49.4541|1.77410126 2022-11-10 12:17:22 Event = message Message = WMT|141.165|1.69499207 2022-11-10 12:17:22 Event = message Message = AMZN|96.44|10.3000031 2022-11-10 12:17:23 Event = message Message = WMT|141.165|1.69499207 2022-11-10 12:17:24 Event = message Message = DOW|49.4541|1.77410126 2022-11-10 12:17:24 Event = message Message = AMZN|96.4299|10.2899017 2022-11-10 12:17:24 Event = message Message = WMT|141.18|1.70999146 2022-11-10 12:17:25 Event = message Message = AMZN|96.4114|10.2714 2022-11-10 12:17:27 Event = message Message = AMZN|96.4225|10.2825012 2022-11-10 12:17:27 Event = message Message = WMT|141.18|1.70999146 2022-11-10 12:17:28 Event = message Message = DOW|49.445|1.76499939 2022-11-10 12:17:29 Event = message Message = AMZN|96.39|10.25 2022-11-10 12:17:29 Event = message Message = WMT|141.18|1.70999146 2022-11-10 12:17:29 Event = message Message = AMZN|96.3889|10.2489014 2022-11-10 12:17:30 Event = message Message = WMT|141.18|1.70999146 2022-11-10 12:17:30 Event = message Message = DOW|49.445|1.76499939 2022-11-10 12:17:31 Event = message Message = DOW|49.445|1.76499939 2022-11-10 12:17:31 Event = message Message = WMT|141.18|1.70999146 2022-11-10 12:17:32 Event = message Message = AMZN|96.39|10.25 2022-11-10 12:17:32 Event = message Message = WMT|141.16|1.69000244 2022-11-10 12:17:32 Stop monitoring requested by user 2022-11-10 12:17:32 Event = message Message = AMZN|96.39|10.25 2022-11-10 12:17:33 Event = message Message = AMZN|96.38|10.2399979 2022-11-10 12:17:33 Event = message Message = WMT|141.16|1.69000244 2022-11-10 12:17:33 Event = disconnected (Code: 0) 2022-11-10 12:17:33 TCPShutdown requested 2022-11-10 12:18:14 Websocket client window closed Edited July 9, 2023 by TheXman Added a copy of a sample log showing the basic framework in action. Danp2 1 CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Danp2 Posted November 7, 2022 Author Posted November 7, 2022 (edited) @TheXmanAre you able to connect to a websocket on localhost using this tool? I've tried using "ws://localhost:9222/" and "ws://127.0.0.1:9222/", and each time the request fails. Here's one of the requests -- {"message":"open", "params":{"url": "ws://localhost:9222/session/e4f94a1e-39a8-45b8-bce3-a2a26a52f2b8"}} {"event": "error", "description": "Error Decoding Header: Switching Protocols [HTTP/1.1 400 Bad Request]"} {"event": "disconnected", "code": 0} Any ideas / suggestions? Edit: This may be why my prior foray with sgcWebSocketClient was short lived 🤔 Edited November 7, 2022 by Danp2 Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 7, 2022 Posted November 7, 2022 In the gym, so reply is short. I will respond when I return home and have a chance to see if I can reproduce error and see why it may be failing. CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
TheXman Posted November 7, 2022 Posted November 7, 2022 (edited) I haven't had any issues connecting to secure and non-secure websocket servers, especially an error saying that it couldn't decode the header. That error message is very odd. Did you enter your "open" command directly into a console window or through a script? If it was using a script, can you post the script? I have never gotten the "Error Decoding Header..." message in any of my testing with the sgcWebSocket client. If I use the sgcWebSocket server listening on 9222 and use the sgcWebSocket client to connect to it, I am unable to reproduce the error. So it doesn't have anything to do with being on localhost and or port 9222. It looks more like an issue with how it was sent. Are you able to successfully connect to a public websocket echo server like: {"message":"open", "params":{"url": "wss://echo.websocket.events"}} or {"message":"open", "params":{"url": "wss://ws.postman-echo.com/raw"}} Edit: Note: To make secure connections (wss://...) using the free "Indy" version of the sgcWebSocket client, you must make sure you have the OpenSSL DLL's (libeay32.dll & ssleay32.dll) in the sgcWebSocketClient.exe app's directory or in the PATH. There are 32 & 64 versions of the OpenSSL DLL's (Unfortunately, both sets are named the same. Yes, the 64 bit DLL's are named libeay32.dll & ssleay32.dll). If you don't have a copy, they are included in the sgcWebSockets .NET Community demo package. They are in the lib\openssl folder. Edited November 7, 2022 by TheXman Added not about make secure client connections CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Danp2 Posted November 7, 2022 Author Posted November 7, 2022 (edited) @TheXmanI was trying to connect to the browser (Firefox in this case) via a websocket. Whenever you start a webdriver session with the correct settings, it will return a string like this -- {"value":{"sessionId":"e4f94a1e-39a8-45b8-bce3-a2a26a52f2b8","capabilities":{"acceptInsecureCerts":true,"browserName":"firefox","browserVersion":"106.0.5","moz:accessibilityChecks":false,"moz:buildID":"20221104133228","moz:debuggerAddress":"127.0.0.1:9222","moz:geckodriverVersion":"0.32.0","moz:headless":false,"moz:platformVersion":"10.0","moz:processID":16004,"moz:profile":"C:\\Users\\danpo\\AppData\\Local\\Temp\\rust_mozprofilekYywNT","moz:shutdownTimeout":60000,"moz:useNonSpecCompliantPointerOrigin":false,"moz:webdriverClick":true,"moz:windowless":false,"pageLoadStrategy":"normal","platformName":"windows","proxy":{},"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webSocketUrl":"ws://127.0.0.1:9222/session/e4f94a1e-39a8-45b8-bce3-a2a26a52f2b8"}}} The URL I was using as the target comes from the webSocketUrl portion at the end, and I can connect to this websocket target using Postman, iola, and websocat. The "open" command was entered directly in the console. Yes, I am able to connect to other public non-secure WS servers. I do get the following error if I try to connect to a secure WS - {"event": "error", "description": "Could not load SSL library. ***PATH*** ***VERSION*** ***METHODS*** "Failed to load libeay32.dll.""} I haven't looked into this one, but I'm guessing that it wouldn't be difficult to fix. Edited November 7, 2022 by Danp2 Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 7, 2022 Posted November 7, 2022 (edited) 9 minutes ago, Danp2 said: I haven't looked into this one, but I'm guessing that it wouldn't be difficult to fix. I added a note about secure client connections in my previous post. I must have submitted it right before you hit enter. Unfortunately, I don't know anything about the new Webdriver BiDi Protocols. Hopefully, it's something that may be resolved by adding a setting to the session config. If it were me, I would probably start by sniffing the sgc and postman websocket request conversations to see what the differences are. Edited November 7, 2022 by TheXman CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
TheXman Posted November 7, 2022 Posted November 7, 2022 (edited) Have you tried using the tcp:// form of the open message? {"message":"open", "params":{"url": "tcp://localhost:9222/session/e4f94a1e-39a8-45b8-bce3-a2a26a52f2b8"}} Edited November 7, 2022 by TheXman CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Danp2 Posted November 7, 2022 Author Posted November 7, 2022 4 minutes ago, TheXman said: Have you tried using the TCP form of the connections command? I just did, and it appears to successfully connect. I haven't yet tried sending any additional requests to see if they work as expected when connected in this mode. Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 7, 2022 Posted November 7, 2022 (edited) 13 minutes ago, Danp2 said: I just did, and it appears to successfully connect. Nice!!! Postman and the others probably automatically fall back to a more legacy (tcp://) connection handshake sequence if the websocket (ws://) handshake fails. Obviously, the free sgcWebSocket client isn't that smart. I just ran a test. I think the tcp:// connection is pure TCP connection. It will not work with the websocket server. Edited November 7, 2022 by TheXman CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Danp2 Posted November 7, 2022 Author Posted November 7, 2022 A simple request would look like this -- {"id":0, "method":"session.status", "params":{}} and the response would be something like {"id":0,"result":{"ready":false,"message":"Session already started"}} I tried sending this with sgcWebSocketClient -- {"message":"write", "params":{"id":0, "method":"session.status", "params":{}}} but no response was received. I will also be testing with Chrome and Edge to see if they behave any differently. Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 7, 2022 Posted November 7, 2022 (edited) I updated my previous post to say that I don't think the tcp:// connection will work. It appears to be pure tcp, no websocket protocol support. Edited November 11, 2022 by TheXman CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Danp2 Posted November 7, 2022 Author Posted November 7, 2022 Some good news --- I was able to successfully connect to Chrome, send a command and receive a response. {"message":"open", "params":{"url":"ws://127.0.0.1:9515/session/7deb92b544692b14fababe9dd78a2680"}} {"event": "connected"} {"message":"write", "text":{"id":0, "method":"session.status", "params":{}}} {"event": "message", "text": "{"id":0,"result":{"message":"ready","ready":true}}"} {"message":"close"} {"event": "disconnected", "code": 0} Thanks for you guidance with this. I will continue investigating the issue with FF. Maybe there's a way to adjust how their websockets work. Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 7, 2022 Posted November 7, 2022 (edited) Awesome! You're welcome. I was doing a little research and saw this Mozilla ticket about Webdriver websocket connections and how they added validation of the Origin & Host request headers. I then did a quick sniff of a non-secure sgcWebSocketClient open message to see if the Host & Origin headers were set correctly, and they were. GET / HTTP/1.1 Host: echo.websocket.events Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: bqpkyi65bouSV8Kqun8pNg== Origin: echo.websocket.events Sec-WebSocket-Version: 13 HTTP/1.1 101 Switching Protocols Connection: Upgrade Upgrade: websocket Sec-Websocket-Accept: HOcE1/MjyW+5fE7PkLD6AbUS3Bs= Via: 1.1 vegur It would be interesting to see what the connection conversation with the Firefox Webdriver websocket server looks like and how it differs from the Chrome conversation. Edited November 7, 2022 by TheXman Danp2 1 CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Danp2 Posted November 8, 2022 Author Posted November 8, 2022 Here are the full details of the failing request to FF -- expandcollapse popupFrame 457: 195 bytes on wire (1560 bits), 195 bytes captured (1560 bits) on interface \Device\NPF_Loopback, id 0 Section number: 1 Interface id: 0 (\Device\NPF_Loopback) Encapsulation type: NULL/Loopback (15) Arrival Time: Nov 7, 2022 19:09:33.752941000 Central Standard Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1667869773.752941000 seconds [Time delta from previous captured frame: 0.000023000 seconds] [Time delta from previous displayed frame: 0.001845000 seconds] [Time since reference or first frame: 18.164051000 seconds] Frame Number: 457 Frame Length: 195 bytes (1560 bits) Capture Length: 195 bytes (1560 bits) [Frame is marked: True] [Frame is ignored: False] [Protocols in frame: null:ip:tcp:http:data-text-lines] [Coloring Rule Name: HTTP] [Coloring Rule String: http || tcp.port == 80 || http2] Null/Loopback Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 191 Identification: 0x2918 (10520) 010. .... = Flags: 0x2, Don't fragment ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 128 Protocol: TCP (6) Header Checksum: 0x0000 [validation disabled] [Header checksum status: Unverified] Source Address: 127.0.0.1 Destination Address: 127.0.0.1 Transmission Control Protocol, Src Port: 9222, Dst Port: 56835, Seq: 1, Ack: 217, Len: 151 Source Port: 9222 Destination Port: 56835 [Stream index: 5] [Conversation completeness: Complete, WITH_DATA (31)] [TCP Segment Len: 151] Sequence Number: 1 (relative sequence number) Sequence Number (raw): 555817769 [Next Sequence Number: 152 (relative sequence number)] Acknowledgment Number: 217 (relative ack number) Acknowledgment number (raw): 2973920663 0101 .... = Header Length: 20 bytes (5) Flags: 0x018 (PSH, ACK) Window: 8441 [Calculated window size: 2160896] [Window size scaling factor: 256] Checksum: 0x0b33 [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [SEQ/ACK analysis] TCP payload (151 bytes) Hypertext Transfer Protocol HTTP/1.1 400 Bad Request\r\n [Expert Info (Chat/Sequence): HTTP/1.1 400 Bad Request\r\n] Response Version: HTTP/1.1 Status Code: 400 [Status Code Description: Bad Request] Response Phrase: Bad Request Server: httpd.js\r\n Content-Type: text/plain\r\n Content-Length: 59\r\n \r\n [HTTP response 1/1] [Time since request: 0.001871000 seconds] [Request in frame: 435] [Request URI: http://127.0.0.1:9222/session/afbb4b0c-aed8-45db-b16e-baba587e7369] File Data: 59 bytes Line-based text data: text/plain (1 lines) The handshake request has incorrect Origin header 127.0.0.1 TheXman 1 Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 8, 2022 Posted November 8, 2022 It would help if you include the request also, not just the response. Looking at just the response, it appears that the request was invalid because the URI was: [Request in frame: 435] [Request URI: http://127.0.0.1:9222/session/afbb4b0c-aed8-45db-b16e-baba587e7369] Shouldn't that have been ws://localhost:9222/session/afbb4b0c-aed8-45db-b16e-baba587e7369? CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Danp2 Posted November 8, 2022 Author Posted November 8, 2022 I was just looking into that. Here's the request -- expandcollapse popupFrame 435: 260 bytes on wire (2080 bits), 260 bytes captured (2080 bits) on interface \Device\NPF_Loopback, id 0 Null/Loopback Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1 Transmission Control Protocol, Src Port: 56835, Dst Port: 9222, Seq: 1, Ack: 1, Len: 216 Source Port: 56835 Destination Port: 9222 [Stream index: 5] [Conversation completeness: Complete, WITH_DATA (31)] [TCP Segment Len: 216] Sequence Number: 1 (relative sequence number) Sequence Number (raw): 2973920447 [Next Sequence Number: 217 (relative sequence number)] Acknowledgment Number: 1 (relative ack number) Acknowledgment number (raw): 555817769 0101 .... = Header Length: 20 bytes (5) Flags: 0x018 (PSH, ACK) Window: 8442 [Calculated window size: 2161152] [Window size scaling factor: 256] Checksum: 0xfffa [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [SEQ/ACK analysis] TCP payload (216 bytes) Hypertext Transfer Protocol GET /session/afbb4b0c-aed8-45db-b16e-baba587e7369 HTTP/1.1\r\n Host: 127.0.0.1:9222\r\n Upgrade: websocket\r\n Connection: Upgrade\r\n Sec-WebSocket-Key: Fz/9gz4uVaJHZU8885/ryQ==\r\n Origin: 127.0.0.1\r\n Sec-WebSocket-Version: 13\r\n \r\n [Full request URI: http://127.0.0.1:9222/session/afbb4b0c-aed8-45db-b16e-baba587e7369] [HTTP request 1/1] [Response in frame: 457] I believe http is correct here because this is the request where the upgrade from http --> websocket is taking place. Here's an open issue that I believe touches on this issue. TheXman 1 Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 8, 2022 Posted November 8, 2022 (edited) Yes, the request looks fine. That open issue does seem to be related. Nice find! 👍 That would explain why the same open message works in Chrome but not in FF. I wonder why some of your other websocket clients worked with FF, maybe they don't send an Origin header? That's why I asked for a capture from one of them also -- just to see the differences. Edited November 8, 2022 by TheXman CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Danp2 Posted November 12, 2022 Author Posted November 12, 2022 (edited) @TheXmanThis is an example of what is being returned -- {"event": "message", "text": "{"id":1,"result":{"children":[],"context":"5345B590C795D8DA358D3E1CFC4E61A9","parent":null,"url":"about:blank"}}"} Only the opening bracket is returned when using this UDF to process the JSON. Here's a short reproducer -- #include "JSON.au3" ; https://www.autoitscript.com/forum/topic/148114-a-non-strict-json-udf-jsmn $sJSON = '{"event": "message", "text": "{"id":1,"result":{"children":[],"context":"5345B590C795D8DA358D3E1CFC4E61A9","parent":null,"url":"about:blank"}}"}' Json_Dump($sJSON) $oJSON = Json_Decode($sJSON) $oJSON2 = Json_ObjGet($oJSON, 'text') $sJSON2 = Json_Encode($oJSON2) ;### Debug CONSOLE ↓↓↓ ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $sJSON2 = ' & $sJSON2 & @CRLF & '>Error code: ' & @error & @CRLF) Local $sKey = '[text]' $sText = Json_Get($oJSON, $sKey) ;### Debug CONSOLE ↓↓↓ ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $sText = ' & $sText & @CRLF & '>Error code: ' & @error & @CRLF) The output looks like this -- +-> .event =message +-> .text ={ +-> .id" =1 +-> .result.context =5345B590C795D8DA358D3E1CFC4E61A9 +-> .result.parent =Null +-> .result.url =about:blank @@ Debug(14) : $sJSON2 = "{" >Error code: 0 @@ Debug(19) : $sText = { >Error code: 0 It works as expected if you remove the double quotes surrounding the value of the "text" node. Does this match your experience? If so, how did you resolve this issue? Edit: There may be a better way to handle this, but I used StringRegExpReplace to remove the extra double quotes in the text node. Edited November 12, 2022 by Danp2 Latest Webdriver UDF Release Webdriver Wiki FAQs
TheXman Posted November 12, 2022 Posted November 12, 2022 (edited) Yes, it matches my experience. As you've noticed, the sgcWebSocketClient will always wrap the text of its message events in double quotes ("). In your example, the text happens to be JSON. When I expected a JSON response from a websocket server, which of course is not always the case, I simply stripped the leading and trailing double quotes from the text before processing it. If I were using json.au3, or some other tool that couldn't handle the double quotes itself, then I would simply do it by using StringReplace or a much more accurate StringRegExpReplace. Spoiler #include <Constants.au3> #include <MyIncludes\json\json.au3> $sJSON = '{"event": "message", "text": "{"id":1,"result":{"children":[],"context":"5345B590C795D8DA358D3E1CFC4E61A9","parent":null,"url":"about:blank"}}"}' ;Remove leading/trainling double quotes from JSON text $sJSON = StringReplace($sJSON, '"{', '{') $sJSON = StringReplace($sJSON, '}"', '}') ConsoleWrite("JSON = " & $sJSON & @CRLF & @CRLF) ConsoleWrite("Dumped JSON" & @CRLF) Json_Dump($sJSON) ConsoleWrite(@CRLF) ;Decode JSON objects $oJSON = Json_Decode($sJSON) $oText = Json_Get($oJSON, '.text') ConsoleWrite("Pretty-printed full JSON" & @CRLF) ConsoleWrite(Json_Encode($oJSON, $JSON_PRETTY_PRINT) & @CRLF) ConsoleWrite("Pretty-printed .text object" & @CRLF) ConsoleWrite(Json_Encode($oText, $JSON_PRETTY_PRINT) & @CRLF) ConsoleWrite("Dump of .text object" & @CRLF) Json_Dump(Json_Encode($oText)) Output: expandcollapse popupJSON = {"event": "message", "text": {"id":1,"result":{"children":[],"context":"5345B590C795D8DA358D3E1CFC4E61A9","parent":null,"url":"about:blank"}}} Dumped JSON +-> .event =message +-> .text.id =1 +-> .text.result.context =5345B590C795D8DA358D3E1CFC4E61A9 +-> .text.result.parent =Null +-> .text.result.url =about:blank Pretty-printed full JSON { "event": "message", "text": { "id": 1, "result": { "children": [], "context": "5345B590C795D8DA358D3E1CFC4E61A9", "parent": null, "url": "about:blank" } } } Pretty-printed .text object { "id": 1, "result": { "children": [], "context": "5345B590C795D8DA358D3E1CFC4E61A9", "parent": null, "url": "about:blank" } } Dump of .text object +-> .id =1 +-> .result.context =5345B590C795D8DA358D3E1CFC4E61A9 +-> .result.parent =Null +-> .result.url =about:blank For the record, I do not use the json.au3 UDF to process JSON. I only use it when I need to parse a few specific values because jsmn (the parsing engine used by json.au3) is strictly a parser and is very fast and efficient at doing that task. As it has been shown, the json.au3 UDF is not always the best tool to use when more advance processing of JSON is required. Further more, it has issues with certain types of valid keys -- which I have offered fixes when those issues have been identified. Side Note: I don't know if you noticed, but in the 2nd post on this page I recently added a log file from a recent execution of an old script that I wrote using the sgcWebSocketClient. I posted it only to show that the websocket client, depending on the script that implements it, can efficiently handle multiple protobuf responses per second. I also added the log to show the basic subscription framework that I used in action -- one in which I queue the responses and process them from my queue. That was necessary in this particular subscription because I had to assume that I was receiving more than one response from the server at any given time -- at least from the server that I was working with in the script. Which is another reason why I don't use json.au3. It does not correctly handle multiple JSON responses without breaking them up first. Which is, again, because it is basically a JSON parser, not a processor. Edited November 12, 2022 by TheXman CryptoNG UDF: Cryptography API: Next Gen jq UDF: Powerful and Flexible JSON Processor | jqPlayground: An Interactive JSON Processor Xml2Json UDF: Transform XML to JSON | HttpApi UDF: HTTP Server API | Roku Remote: Example Script About Me How To Ask Good Questions On Technical And Scientific Forums (Detailed) | How to Ask Good Technical Questions (Brief) "Any fool can know. The point is to understand." -Albert Einstein "If you think you're a big fish, it's probably because you only swim in small ponds." ~TheXman
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now