Unwanted carriage return output in _FileWriteFromArray

[sanfrancrisko]

Hi,

I've written two scripts - a sender and a receiver - that sends a file across a TCP connection.

The sender reads the file, then splits it in to chunks and populates it in an array. It then creates a TCP connection to the the receiver and sends the chunks across.

The receiver populates the chunks in to an array on it's side of the network, then, once it has received all the chunks, writes the file to disk using the _FileWriteFromArray function.

All is working well, as expected, however, the _FileWriteFromArray function appears to prepend and tail the output file with a carriage return (hex: 0D 0A). Is there any way to disable this functionality? The data integrity needs to be 100% intact as the send and receive binaries will be md5'd and compared.

Thanks for your time,

Crisko

SEND SCRIPT:

expandcollapse popup

#include <File.au3>
#include <Array.au3>
FileDelete("C:\testlog.log")
hotkeyset("{Esc}","quit")
$destinationIPAddress = $CmdLine[1]
$destinationPort = $CmdLine[2]
$sendFile = FileRead ($CmdLine[3])
$fileLength = StringLen($sendFile)
$chunkSize = 0
;Local $fileArray
;_FileReadToArray($sendFile, $fileArray)
;MsgBox(0, "", $fileLength)
FileDelete("C:\testlog.log")
$Success = false
$i = 2
$remainder = 1
While $Success == False
If $remainder <> 0 Then
;logwrite("If statement executed")
$i = $i + 1
Else
;logwrite("Else statement executed")
$chunkSize = $i
If $chunkSize > 1023 AND $chunkSize < 8097 Then
$Success = true
ExitLoop
Else
$Success = False
$i = $i + 1
EndIf
EndIf
$remainder = Mod ($fileLength, $i)
;logwrite("Divider is: " & $i & " | Remainder is: " & $remainder)
WEnd
IniWrite("\\diskstation\public\xFPTool\config.ini", "System", "chunkSize", $chunkSize)
;MsgBox (0, "", $chunkSize, 5)
$chunkCount = $fileLength / $chunkSize
IniWrite("\\diskstation\public\xFPTool\config.ini", "System", "chunkCount", $chunkCount)
Local $avSendFileChunked[$chunkCount + 1]
$startDividePoint = 1
For $split = 1 to $chunkCount Step + 1
$endDividePoint = $startDividePoint + $chunkSize
$avSendFileChunked[$split] = StringMid($sendFile, $startDividePoint, $endDividePoint)
;logwrite("Split iteration: " & $split & " | Start Divide Point: " & $startDividePoint & " | End Divide Point: " & $endDividePoint & " | Array Count: " & $avSendFileChunked[$split])
$startDividePoint = $endDividePoint + 1
Next
TCPStartup()
Do
$ConnectedSocket = TCPConnect($destinationIPAddress, $destinationPort)
;Sleep(2500)
Until $ConnectedSocket <> -1
For $sendCount = 1 to $chunkCount Step + 1
;$sendData = StringtoBinary($avSendFileChunked[$sendCount])
$sendStatus = TCPSend($ConnectedSocket, $avSendFileChunked[$sendCount])
;logwrite("Send Iteration: " & $sendCount & " ||| Send Status: " & $sendStatus)
Next

TCPCloseSocket($ConnectedSocket)
TCPShutdown()
Func logwrite($data)
FileOpen("C:\testlog.log", 9)
FileWriteLine("C:\testlog.log", $data)
FileClose("C:\testlog.log")
EndFunc
Func quit()
Exit
EndFunc

Exit

RECEIVE SCRIPT:

#include <String.au3>
#include <File.au3>
FileDelete("C:\testlog2.log")
hotkeyset("{Esc}","quit")
$localIPAddress = @IPAddress1
$listeningPort = $CmdLine[1]
$xFPSenderModule = "C:\xFP Sender\xFP_Sender.exe"
$sendFile = $CmdLine[2]
$remoteIPAddress = $CmdLine[3]
$chunkSize = IniRead("\\diskstation\public\xFPTool\config.ini", "System", "chunkSize", "")
$chunkCount = IniRead("\\diskstation\public\xFPTool\config.ini", "System", "chunkCount", "")
Local $avChunkedFile[$chunkCount + 1]
TCPStartup()
$mainSocket = TCPListen($localIPAddress, $listeningPort)
If $mainSocket = -1 Then Exit
$connectedSocket = -1
Do
$connectedSocket = TCPAccept($mainSocket)
Until $connectedSocket <> -1
For $i = 1 to $chunkCount Step +1
$avChunkedFile[$i] = TCPRecv($connectedSocket, $chunkSize)
;logwrite($avChunkedFile[$i])
Next
_FileWriteFromArray("C:\output.bin", $avChunkedFile)
TCPCloseSocket($connectedSocket)
TCPShutdown()
Func logwrite($data)
FileOpen("C:\testlog2.log", 9)
FileWriteLine("C:\testlog2.log", $data)
FileClose("C:\testlog2.log")
EndFunc
Func quit()
Exit
EndFunc
Exit

[BrewManNH]

You could avoid using the _FileWriteFromArray function and write your own. The function does append a @CRLF to the end of every line sent so your last line is going to have an extra tagged onto it. You could alway have your receiver program delete that last CRLF after it's written though, which might be easier. Another option might be to have the receiveer save it to a string instead of an array and then just use FileWrite to write the string to a file.

[Shaggi]

Doesn't seem so. You can just copy the function and delete the @crlf entries like so:

expandcollapse popup

Func _FileWriteFromArray2($File, $a_Array, $i_Base = 0, $i_UBound = 0, $s_Delim = "|")
; Check if we have a valid array as input
If Not IsArray($a_Array) Then Return SetError(2, 0, 0)
Local $iDims = UBound($a_Array, 0)
If $iDims > 2 Then Return SetError(4, 0, 0)

; determine last entry
Local $last = UBound($a_Array) - 1
If $i_UBound < 1 Or $i_UBound > $last Then $i_UBound = $last
If $i_Base < 0 Or $i_Base > $last Then $i_Base = 0

; Open output file for overwrite by default, or use input file handle if passed
Local $hFile
If IsString($File) Then
$hFile = FileOpen($File, $FO_OVERWRITE)
Else
$hFile = $File
EndIf
If $hFile = -1 Then Return SetError(1, 0, 0)

; Write array data to file
Local $ErrorSav = 0
Switch $iDims
Case 1
For $x = $i_Base To $i_UBound
If FileWrite($hFile, $a_Array[$x]) = 0 Then
$ErrorSav = 3
ExitLoop
EndIf
Next
Case 2
Local $s_Temp
For $x = $i_Base To $i_UBound
$s_Temp = $a_Array[$x][0]
For $y = 1 To $iDims
$s_Temp &= $s_Delim & $a_Array[$x][$y]
Next
If FileWrite($hFile, $s_Temp) = 0 Then
$ErrorSav = 3
ExitLoop
EndIf
Next
EndSwitch

; Close file only if specified by a string path
If IsString($File) Then FileClose($hFile)

; Return results
If $ErrorSav Then Return SetError($ErrorSav, 0, 0)
Return 1
EndFunc ;==>_FileWriteFromArray

Edited August 3, 2012 by Shaggi

[sanfrancrisko]

Thanks for the replies, BrewMan and Shaggi

You could avoid using the _FileWriteFromArray function and write your own. The function does append a @CRLF to the end of every line sent so your last line is going to have an extra tagged onto it. You could alway have your receiver program delete that last CRLF after it's written though, which might be easier. Another option might be to have the receiveer save it to a string instead of an array and then just use FileWrite to write the string to a file.

Good suggestions - I was thinking of just post-processing the file to remove the @CRLFs, but I'll try the string idea. One other question though - you say that the _FileWritefromArray function will add a line ending after every array entry? I haven't noticed that. Just the start and end of the file. Or do you mean, if the file had those line endings to begin with? In that case, I need to preserve them.

Thanks for the code Shaggi - just to confirm I understand it correctly, that is removing line endings by only closing the file if it's on the last array value? (Sorry, you're speaking to a non-programming newb, I'm afraid! Little bit of scripting is about all I do!)

[BrewManNH]

This is taken from the _FileWriteFromArray UDF which shows that it's doing a FileWrite of each element of the array, and tacking on a @CRLF at the end of it.

For $X = $i_Base To $i_UBound
    If FileWrite($hFile, $a_Array[$X] & @CRLF) = 0 Then
     $ErrorSav = 3
     ExitLoop
    EndIf
   Next

If it's writing a 2D array to a file it will create a string of each "row" of the array with a delimiter between each "column" and then will write the string to the file, with a @CRLF after it.

[sanfrancrisko]

Thanks for the clarification - that makes sense.

In the end, I used the FileOpen() and FileWrite() command to achieve what I needed and did away with the array.

Thanks again to you both!

[Spiff59]

Since @CRLF is really saying "start a new line" and not "this is the end of this line", it might make sense to add a parameter to _FileWriteFromArray() to suppress the final @CRLF.

Edit: Or maybe not even a parameter, just make _FileWriteFromArray() suppress the final @CRLF by default (script breaker?)

Edited August 8, 2012 by Spiff59

[guinness]

I would say for consistency the @CRLF should be at the end of a file and anyway it can be done with a simple function. Seems silly introducing a new parameter when most would just go with this >>

ConsoleWrite(_StripLineFeed(@ScriptFullPath) & @CRLF)

Func _StripLineFeed($sFilePath)
    Return StringRegExpReplace(FileRead($sFilePath), 'v*$', '') ; Strip vertical space at the end of a file.
EndFunc   ;==>_StripLineFeed

[Spiff59]

I would say for consistency the @CRLF should be at the end of a file and anyway it can be done with a simple function. Seems silly introducing a new parameter when most would just go with this >>

Yes, it's easy as pie to remove the trailing @CRLF after a file read, as in your example.

In the case of a write using _FileWriteFromArray(), if you have a requirement to not have a blank line at the end of the file, then the function basically becomes unusable.

As to consistency, _ArrayToString() works differently and does not append a trailing delimiter, so I guess an easy two-line, no-trailing-crlf, replacement for _FileWriteFromArray() could be:

$sArray = _ArrayToString($aArray, @CRLF)
FileWrite(@ScriptDir & "arraydump.txt", $sArray)

[guinness]

Yes, it's easy as pie to remove the trailing @CRLF after a file read, as in your example.

In the case of a write using _FileWriteFromArray(), if you have a requirement to not have a blank line at the end of the file, then the function basically becomes unusable.

As to consistency, _ArrayToString() works differently and does not append a trailing delimiter, so I guess an easy two-line, no-trailing-crlf, replacement for _FileWriteFromArray() could be:

$sArray = _ArrayToString($aArray, @CRLF)
FileWrite(@ScriptDir & "arraydump.txt", $sArray)

I would say the topic is solved with your solution of_ArrayToString and we agree to disagree on the trailing line feed in _FileWriteFromArray.

[BrewManNH]

Only problem with using _ArrayToString to replace the issue with the CRLF in _FileWriteFromArray is that _ArrayToString only works with 1D arrays, and _FileWriteFromArray works with 2D and 1D arrays.

[Spiff59]

There is that, that these 2 lines can only replace the 1D functionality of the _FileWriteFrom Array() function.

I'd forgotten you had already suggested the OP use an "array to string to filewrite" method way back in post #2.

I've basically rehashed your idea.