[Fixed] Fileread() has a problem reading 1GB/2GB pieces.

[Exit]

Edit: Bug fixed in Beta 3.3.15.4

Edit: This is a restriction in the string length of 2GB for Autoit strings.
More information in third post.

-----------------------------------------------------

When testing my file verification script, I noticed that 1GB blocks can be written but not read.
The maximum block size is 1073741822 bytes. With a block size of 1073741823 or more, Fileread() ends with @error = 0 and @extended shows the correct size, but there is an empty string in the receiving variable.
Here is a reproducer:

expandcollapse popup

#AutoIt3Wrapper_UseX64=y
#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w no4 -w 5 -w 6 -w 7

#include <String.au3>

Local $sTestDir = "C:\~Test~\"

Local $hFileHandle, $iLength, $s1GB, $sInitData, $sReadData

DirCreate($sTestDir)
FileDelete($sTestDir & "*.txt")
ShellExecute($sTestDir)

OnAutoItExitRegister(_Exit)
Func _Exit()
    MsgBox(64 + 262144, Default, "Press OK to exit.", 0)
    DirRemove($sTestDir, 1)
    Sleep(3000)
    WinClose("[TITLE:" & StringLeft($sTestDir, 3) & "; CLASS:CabinetWClass]")
EndFunc   ;==>_Exit

$iLength = 1024 * 1024 * 1024    ;  1 Gigabyte
$s1GB = _StringRepeat("-", $iLength)
$iLength -= 3

For $i = 1 To 2
    Beep(1000, 100)
    $iLength += 1
    $hFileHandle = FileOpen($sTestDir & $i & ".txt", 2)
    $sInitData = StringLeft($i & $s1GB, $iLength)
    FileWrite($hFileHandle, $sInitData)
    FileSetPos($hFileHandle, 0, 0)
    $sReadData = FileRead($hFileHandle)
    ConsoleWrite(@LF & "Fileread $i:" & $i & " Error: " & @error & " Extended: " & @extended & @LF & "    Datalen: " & StringLen($sInitData) & @LF & "ReadDatalen: " & StringLen($sReadData) & @LF & "ReadData(first 10 Bytes): >" & StringLeft($sReadData, 10) & "< " & @LF & "Stringcompare Result: " & (StringCompare($sInitData, $sReadData, 1) ? "NOT" : "") & " equal" & @LF & @LF)
    If StringCompare($sReadData, $sInitData, 1) Then Exit MsgBox(16 + 262144, Default, "Compare Error. I=" & $i, 0)
    FileClose($hFileHandle)
Next

and here is the console log:

Fileread $i:1 Error: 0 Extended: 1073741822
    Datalen: 1073741822
ReadDatalen: 1073741822
ReadData(first 10 Bytes): >1---------< 
Stringcompare Result:  equal


Fileread $i:2 Error: 0 Extended: 1073741823
    Datalen: 1073741823
ReadDatalen: 0
ReadData(first 10 Bytes): >< 
Stringcompare Result: NOT equal

I think Fileread() should return an error code in this case.

Or am I on the wrong track?

 

Edited June 12, 2021 by Exit
Edit: Bug fixed in Beta 3.3.15.4

[Musashi]

Maybe thread a-set-of-question-for-file  brings some light into the dark. Especially the posts from @Valik I consider as very informative.

[Exit]

@Musashi Thanks for the info.

I will investigate and report here soon. Stay tuned.  

---------------------------------------------------------------------------------

The maximum string length in Autoit is 2GB

Since each character in UTF8 is 2 bytes long, only 1GB strings can be processed.
When using ANSI, each character is only 1 byte long and a 2GB string can then be processed. 

You have to accept this limitation.
But that doesn't change the fact that it is probably wrong to set an error code of 0 under these circumstances and return an empty variable.

Is this a case for the bug tracker?
Please give your opinion.

Edited November 7, 2020 by Exit

[JockoDundee]

32bit version, 64bit version?  Both?

[jchd]

7 hours ago, Exit said:

The maximum string length in Autoit is 2GB

Since each character in UTF8 is 2 bytes long, only 1GB strings can be processed.
When using ANSI, each character is only 1 byte long and a 2GB string can then be processed. 

No: UTF8 uses 1 to 4 bytes per Unicode character. Whatever encoding a text source uses (file or not), native AutoIt strings uses UCS2 encoding (the first 64k Unicode codepoints, without surrogates) so every character in such a string uses one 16-bit encoding unit, limiting character count to 2Gi.

7 hours ago, Exit said:

Is this a case for the bug tracker?

My opinion is YES.

[Exit]

7 hours ago, JockoDundee said:

32bit version, 64bit version?  Both?

See first line of script:

#AutoIt3Wrapper_UseX64=y

 

[Exit]

Bug created in bug tracker :

https://www.autoitscript.com/trac/autoit/ticket/3789#ticket

[Exit]

Edit: Bug fixed in Beta 3.3.15.4

and working perfect.