[UDF] Simple TCP

[matwachich]

Here i am again!

Today, i got a new UDF i want to share!

It's a clone of my but without POO, only classic functions!

It's faster, easier, and the client is more performant, cause it's now very simple to handle mutiple clients per script.

Here is v 1.0 (deprecated) (63 downloads)

Update v 1.2

Here is the new version.

Change Log:

Legend:
-------
+: Addition, -: Remouved/Deprecated, *: Modified, !: Bug correction
Change Log:
-----------
=== Version 1.2 === (24/11/2011)
*: Now UTF-8 String are correctly processed (thanks jchd!)
=== Version 1.1.1 === (21/11/2011)
+: Added 2 helper functions (see documentation)
!: Another bug corrected in _TCPServer_SocketList. Thanks Tetdoss (again!)
=== Version 1.1 === (20/11/2011)
+: ZLib data compression, optional (disabled by default)
!: Client's properties were not reset when a client disconnects
!: Incorrect _TCPServer_SocketList. Now works well (thanks Tetdoss)
=== Version 1.0 === (11/11/2011)
First public version

Archive:

SimpleTCP_1.2.zip

Edited November 24, 2011 by matwachich

[davidkim]

thanks matwachich

[matwachich]

Update

[autoall]

very nice job!

[denchu]

Hi. Very good job.

Is there a function to flush buffer of client after read received data? I plan to use your udf in my scripts that already have a file transfer routine. I transfer big binary files(about several gigobytes). This amount of data cannot be transfered by one time and even can't be held in memory. I transfer it by chunks. Script read the chunk from file and send it by native tcpsend function, in many cases script don't send the chunk by one tcpsend command and split it on "sub"chunks. There is part of my script:

if not BinaryLen($sBuffer[$x]) then $sBuffer[$x] = FileRead($f_fopen[$x], $chunk_size)
$f_Byte = TCPsend($aSocket[$x], $sBuffer[$x]) ; TCPSend returns the number of bytes sent
If @error  Then
    ConsoleWrite('!Lost Connection')
    FileClose($f_fopen[$x])
else
     $sBuffer[$x] = BinaryMid($sBuffer[$x], $f_Byte+1)
     $f_size[$x]-=$f_byte
     if not $f_size[$x] then
         ConsoleWrite('+Upload Complited')
         FileClose($f_fopen[$x])
     endif
endif

Is it possible to receive data by chunks with your UDF ? Read data from buffer then flush buffer save to disk and read again.

And a second question. How the clinet.connect func works, it sending some keepalive packets or just reconnect socket if needed or i need to connect manualy every time when i needs to send data after some time(10min for example).

[matwachich]

For the 2nd question, you don't need to connect every time you want to send data, you only need to reconnect when _TCPClient_IsConnected returns 0.

For the first question, for now, NO. But i think it's simply modifiable, but the problem: i don't have any time these days to focus on this (exams!) sorry.

You will have to study the function _TCPxxx_Process to see how it works

[denchu]

When i try to send binary file data over _TCPServer_Send without compression and encoding i have string on other end. I inserted some debugging command(consolewrite(isbinary($var)) in all used scripts and found that binary data become string after calling TCPRecv function in _TCPClient.au3. I suggests a bug in TCPRecv function that recognize data as string. So i alweys use cryptpass in my script.

Next problem i have is if i sends file in chunks, i have data lost on other end. For example i transfer file of size 400kb by chunks of 122880bytes. By consolewrite i know that server sends 4 chunks but client receives only 2 - first and last. If i try to transfer bigger file i have 3 of 6 chunks received etc. It is code of my send function.

local $aSocket=_TCPServer_Socketlist()
for $x=1 to $aSocket[0][0]
      if _TCPServer_ClientPropertyGet($aSocket[$x][0],0)='SendingBinaryFileData' then 
           _ConsoleWrite('SendingBinaryFileData')
           _SendChunk($aSocket[$x][0])
      endif
Next

func _SendChunk($f_Socket)
local $f_fopen=_TCPServer_ClientPropertyGet($f_Socket,2);get handle of file to send
local $f_size=_TCPServer_ClientPropertyGet($f_Socket,4);get file size to send
_ConsoleWrite('Need to send '&$f_size)
_Consolewrite("Read next chunk")
local $Buffer=FileRead($f_fopen, $chunk_size)
  $f_size-=BinaryLen($Buffer)
_CONSOLEWRite("isbinary="&isbinary($Buffer))
$f_Byte=_TCPServer_Send($f_Socket, $Buffer) ; it returns amount of data sent in crypted state with chr(2) and chr(3) - useless info
If @error Then
       _ConsoleWrite('! Lost Connection! error='&@Error, 1)
         FileClose($f_fopen)
       _TCPServer_ClientPropertySet($f_Socket, 0,'')
        return -1
elseif $f_size<=0 then
       _ConsoleWrite('+ Upload Complited', 0)
       FileClose($f_fopen)
      _TCPServer_ClientPropertySet($f_Socket, 0,'')
EndIf
_TCPServer_ClientPropertySet($f_Socket,4,$f_size)
return 1
endfunc

[denchu]

Next problem i have is if i sends file in chunks, i have data lost on other end. For example i transfer file of size 400kb by chunks of 122880bytes. By consolewrite i know that server sends 4 chunks but client receives only 2 - first and last. If i try to transfer bigger file i have 3 of 6 chunks received etc. It is code of my send function.

I have solved second problem! Just a little patch to your code. You flush the buffer every time when __Check_Buffer() founds new data between chr(2) and chr(3). At first i changed chr(2) and chr(3) to chr(2)&chr(2)&chr(2)&chr(2) because of chr(2) is often placed in binary files. And at main i replace buffer flushing to shrink buffer to last border of last packet. It's a part of your code(TCPClient.au3) after line 455.

; Check buffer ok
$tmp = __TCPClient_CheckBuffer($iClient)
If IsArray($tmp) Then
  For $elem In $tmp
   If $__TCPClient_Clients[$iClient][$__TCPc_CB_RECV] Then _
    Call($__TCPClient_Clients[$iClient][$__TCPc_CB_RECV], $iClient, $elem)
  Next
  ; ---
  $__TCPClient_Clients[$iClient][$__TCPc_BUFFER] = StringRegExpReplace($__TCPClient_Clients[$iClient][$__TCPc_BUFFER],'.*'&Chr(3)&Chr(3)&Chr(3)&Chr(3),'') ;it was ""(flush) here
EndIf
EndFunc
Func __TCPClient_CheckBuffer($iClient)
If $__TCPClient_Clients[$iClient][$__TCPc_BUFFER] = "" Then Return 0
; ---
$bet = _StringBetween($__TCPClient_Clients[$iClient][$__TCPc_BUFFER],Chr(2)&Chr(2)&Chr(2)&Chr(2),Chr(3)&Chr(3)&Chr(3)&Chr(3))
If IsArray($bet) Then
  For $i = 0 To UBound($bet) - 1
   $bet[$i] = __TCPClient_dCry($iClient, $bet[$i])
  Next
  Return $bet
Else
  Return 0
EndIf
EndFunc

Another way to solve this issue is sending some confirmation packet from client every chunk and send next chunk only after recieving that packet on server.

P.S.: Taking data from buffer by some marker(chr(2)) it's a bad idea because there is no guarantee that your marker will never included in transfered data. Better way is make some packet header, for example chr(2)&$Packetdatasize&chr(2). When script checks the buffer it find header, proof that packet is longer that $PacketDataSize+header_lenght and then get $Packetdatasize of data from buffer.

Edited May 24, 2012 by denchu

[denchu]

I rewrote several functions in your UDF, so you may use my changes to improve your UDF:

expandcollapse popup

Func _TCPServer_Send($iSocket, $Data)
If Not _TCPServer_IsStarted() Then Return SetError(-1, 0, 0)
If Not __TCPServer_IsSocketValid($iSocket) Then Return SetError(1, 0, 0)
; ---
;Consolewrite('Data to send is '&VarGetType($Data)&@crlf)
$Data=__TCPServer_Cry($Data)
local $DataType=isbinary($data)
Local $DataSize=Binarylen($DATA)
local $Header=chr(2)&chr(3)&chr(2)&Hex($DataType,2)&Hex($DataSize,16)&chr(3)&chr(2)&chr(3)
local $f_byte=TcpSend($__TCPServer_Sockets[$iSocket][0],$Header)
if @error then
  return 0
endif
if $f_byte<>binarylen($Header) then
  Seterror(1,$f_byte,0)
  return 0
endif
local $failtimer=timerinit()
;Consolewrite('After servercrypt '&VarGetType($Data)&@crlf)
while binaryLen($Data)
  if $DataType then $Data=Binary($Data)
  $f_byte= TcpSend($__TCPServer_Sockets[$iSocket][0],$DATA)
  if @error then
   return 0
  endif
  if $f_byte > 0 then
   $failtimer=timerinit()
   $Data=BinaryMid($Data,$f_byte+1)
  Elseif TimerDiff($failtimer)>1000 then
   seterror(2,BinaryLen($Data),0)
   return 0
  endif
  local $tmprec=TCPRecv($__TCPServer_Sockets[$iSocket][0],1)
  if $tmprec=chr(2) or ($tmprec=chr(3) and binaryLen($Data)>0) then
   seterror(3,$tmprec,0)
   return 0
  endif
  if $tmprec=chr(3) and not binarylen($Data) then return 1
  $failtimer=timerinit()
  while $tmprec<>chr(3)
   if TimerDiff($failtimer)>=$__TCPSrv_Info_TIMEOUT Then ; Timed-out!
    If $__TCPSrv_Info_CB_TIMEDOUT <> "" Then _
     Call($__TCPSrv_Info_CB_TIMEDOUT, $iSocket, $__TCPServer_Sockets[$iSocket][1], BinaryLen($__TCPServer_Sockets[$iSocket][2]))
    seterror(4,BinaryLen($Data),0)
    return 0
   endif
   if $tmprec=chr(2) then
    seterror(5,BinaryLen($Data),0)
    return 0
   endif
   $tmprec=TCPRecv($__TCPServer_Sockets[$iSocket][0],1)
  wend
wend
return 1
EndFunc

expandcollapse popup

Func _TCPServer_Process()
If Not _TCPServer_IsStarted() Then Return SetError(-1, 0, 0)
; ---
Local $hNewSocket, $recv, $tmp
; ---
; Check new connexions
$hNewSocket = TCPAccept($__TCPSrv_Info_SOCKET)
If $hNewSocket <> -1 Then
  __TCPServer_StoreNewClient($hNewSocket)
EndIf
; ---
; Process Connected clients
For $i = 1 To $__TCPServer_Sockets[0][0]
  If $__TCPServer_Sockets[$i][0] = -1 Then ContinueLoop
  ; ---
  $recv = TCPRecv($__TCPServer_Sockets[$i][0], 24)
  ; ---
  ; Check disconnection
  If @error Then
   If $__TCPSrv_Info_CB_LOST <> "" Then _
    Call($__TCPSrv_Info_CB_LOST, $i, $__TCPServer_Sockets[$i][1])
   ; ---
   TCPCloseSocket($__TCPServer_Sockets[$i][0])
   __TCPServer_ResetSocket($i)
  EndIf
  ; ---
  ; Check Receiving
  If $recv Then ;Header received
   $__TCPServer_Sockets[$i][3] = TimerInit()
   if stringleft($recv,3)<>chr(2)&chr(3)&chr(2) or stringright($recv,3)<>chr(3)&chr(2)&chr(3) then
    SetError(3,'Bad Header')
    Consolewrite('Bad Header received')
    TCPRecv($__TCPServer_Sockets[$i][0], 999999999)
    continueloop
   endif
   local $Header=StringMid($recv,4,18)
   Local $DataSize=Dec(StringRight($Header,16))
   Local $DataType=Dec(StringLeft($Header,2))
   ConsoleWrite('Header='&$HEader&' Datasize='&$DataSize&'bytes datatype='&$DataType&@crlf)
   $__TCPServer_Sockets[$i][2]=''
   if $DataType<>0 and $DataType<>1 then
    SetError(1,$DataType)
    Consolewrite('Bad datatype in header')
    TCPRecv($__TCPServer_Sockets[$i][0], $DataSize)
    continueloop
   endif
   #region recieving data
   While BinaryLen($__TCPServer_Sockets[$i][2])<$DataSize
    $Recv=TCPRecv($__TCPServer_Sockets[$i][0], $DataSize-BinaryLen($__TCPServer_Sockets[$i][2]),$DataType)
    if $recv then
     $__TCPServer_Sockets[$i][2]&=$recv
     If $__TCPSrv_Info_CB_RECEIVING <> "" Then _
       Call($__TCPSrv_Info_CB_RECEIVING, $i, $__TCPServer_Sockets[$i][1], BinaryLen($__TCPServer_Sockets[$i][2]))
     $__TCPServer_Sockets[$i][3]=TimerInit()
    elseif timerdiff($__TCPServer_Sockets[$i][3])>= $__TCPSrv_Info_TIMEOUT then
     If $__TCPSrv_Info_CB_TIMEDOUT <> "" Then _
       Call($__TCPSrv_Info_CB_TIMEDOUT, $i, $__TCPServer_Sockets[$i][1], BinaryLen($__TCPServer_Sockets[$i][2]))
     ; ---
     TCPSend($__TCPServer_Sockets[$i][0],chr(2))
     SetError(2,BinaryLen($__TCPServer_Sockets[$i][2]))
     $__TCPServer_Sockets[$i][2] = ""
     exitloop
    endif
    ;Consolewrite('+Received '&BinaryLen($__TCPServer_Sockets[$i][2])&' of '&$DataSize&'bytes'&@crlf)
   wend
   ;Consolewrite('Calling _TCP_RECV function, data is '& __TCPServer_dCry($__TCPServer_Sockets[$i][2]) &@crlf)
   if BinaryLen($__TCPServer_Sockets[$i][2])=$DataSize then
    TCPSend($__TCPServer_Sockets[$i][0],chr(3))
   else
    TCPSend($__TCPServer_Sockets[$i][0],chr(2))
   endif
   If $__TCPSrv_Info_CB_RECV <> "" Then _
      Call($__TCPSrv_Info_CB_RECV, $i, $__TCPServer_Sockets[$i][1], __TCPServer_dCry($__TCPServer_Sockets[$i][2]))
   $__TCPServer_Sockets[$i][2]=''
   #endregion
   ; ---
  EndIf
  ; ---
  ; Check timeout
  If $__TCPServer_Sockets[$i][2] Then
   If TimerDiff($__TCPServer_Sockets[$i][3]) >= $__TCPSrv_Info_TIMEOUT Then ; Timed-out!
    If $__TCPSrv_Info_CB_TIMEDOUT <> "" Then _
     Call($__TCPSrv_Info_CB_TIMEDOUT, $i, $__TCPServer_Sockets[$i][1], BinaryLen($__TCPServer_Sockets[$i][2]))
    ; ---
    $__TCPServer_Sockets[$i][2] = ""
   EndIf
  EndIf
  ; ---
Next
; ---
Return 1
EndFunc

Send and recieve functions have no "string compare" commands and does not stuck on big data chunks. Now script make packet with 24bytes of header with information about data length and data type(bynary/string). Searching for magic sequence in all amount of data is not needed anymore. I have no problems with > 100Mb files, transfer speed is about 5Mb/s. Plain functions that i wrote in the past have much speed but have no stability of this.

Func __TCPServer_Cry($Data)

; ---
Local $bin = 1
If Not IsBinary($Data) Then
  $bin = 0
  $Data = StringToBinary($Data, 4)
EndIf
; ---
If $__TCPSrv_Info_CRYKEY Then $Data = _Crypt_EncryptData($Data, $__TCPSrv_Info_CRYKEY, $CALG_USERKEY)
; ---
If $__TCPSrv_Info_COMPRESSION Then $Data = _ZLIB_Compress($Data, $__TCPSrv_Info_COMPRESSION)
; ---
If Not $bin Then $Data = StringTrimLeft($Data, 2)
; ---
Return $Data
EndFunc
Func __TCPServer_dCry($Data)
; ---
Local $bin = 0
If StringLeft($Data, 2) = "0x" Then
  $bin = 1
  $Data = Binary($Data)
Else
  $Data = Binary("0x" & $Data)
EndIf
; ---
If $__TCPSrv_Info_COMPRESSION Then $Data = _ZLIB_Uncompress($Data)
; ---
If $__TCPSrv_Info_CRYKEY Then $Data = _Crypt_DecryptData($Data, $__TCPSrv_Info_CRYKEY, $CALG_USERKEY)
If Not $bin Then $Data = BinaryToString($Data, 4)
; ---
Return $Data
EndFunc

crypt and decrypt functions now made zlib compression(if needed) and prepare string to transfer even if encryption is disabled.

[matwachich]

Good work!

Gonna check all this when i have time.

[matwachich]

I really like you ideas!

Can you post the complete server and client scripts? I wanna study them more.