Custom Query

OS = WinXP SP2 RAM = 2.00Gb AutoIt Ver = 3.2.11.12

I'm not sure if this is a bug or some limit of Window or AutoIt

Background: I'm working with some large text files 200Gb to 400Gb and found that any of the file functions that use FileRead() generate an error, even though Task Manager shows plenty of memory left, in the form of a message box Title = 'AutoIt' Text = 'Error allocating memory.'

I can work around this by processing the files line by line, but obviously this is much slower.

Observations: The help file give the theoretical limit 2147483647 characters but it would appear that it is impossible to get anywhere near this limit.

I created the scripts to try and find exactly what the limit is on my system. The results seem to show that the limit is some sort of global memory limit within AutoIt, rather than a limit for each string, as each time I double the number string variable filled the total memory used before the error allocating memory occurs is approximately the same.

Should AutoIt be able to make use of more of the memory available to it?

Script 1

#Include <String.au3>
Local $sString = _StringRepeat( "A", 1024)
Local $iCount = 0
Local $sTest1 =''

While 1
	$sTest1 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
WEnd

Script 2

#Include <String.au3>
Local $sString = _StringRepeat( "A", 1024)
Local $iCount = 0
Local $sTest1 =''
Local $sTest2 =''

While 1
	$sTest1 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
	$sTest2 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
WEnd

Script 3

#Include <String.au3>
Local $sString = _StringRepeat( "A", 1024)
Local $iCount = 0
Local $sTest1 =''
Local $sTest2 =''
Local $sTest3 =''
Local $sTest4 =''

While 1
	$sTest1 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
	$sTest2 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
	$sTest3 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
	$sTest4 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
WEnd

Maximum values of $iCount which equates to the size of the $sTextn Script 1 = 262142Kb in 1 string Script 2 = 262141Kb in 2 strings Script 3 = 262139Kb in 4 strings

This one's been around a while, and I think maybe I've fixed it. Or something. I'm sure this will help someone, since I've seen this problem go unanswered in the archived bug report forum!

I found this proposed change by looking at examples given for the Clipboard API in other programming languages, and the one *difference* that they all had in common was to lock and copy the memory when reading from the clipboard.

Func _ClipBoard_GetData_Carefully($iFormat = 1)
Local $hMemory, $tData

Local $hMemoryDest, $hLock

If Not _ClipBoard_IsFormatAvailable($iFormat) Then Return SetError(-1, 0, 0)
If Not _ClipBoard_Open(0) Then Return SetError(-2, 0, 0)
$hMemory = _ClipBoard_GetDataEx($iFormat)
_ClipBoard_Close()
If $hMemory = 0 Then Return SetError(-3, 0, 0)
Switch $iFormat
Case $CF_TEXT, $CF_OEMTEXT

;THIS WAS BAD: $tData = DllStructCreate("char Text[8192]", $hMemory)

$tData = DllStructCreate("char Text[8192]") ; do it this way so it is allocated...?
$hMemoryDest = DllStructGetPtr($tData)

; so let's lock that clipboard memory down and make our own for-sure copy.
$hLock = _MemGlobalLock($hMemory)
If $hLock = 0 Then Return SetError(-4, 0, 0)
; OK, $hLock should now be the pointer into the clipboard memory
_MemMoveMemory($hLock, $hMemoryDest, 8192)
_MemGlobalUnlock($hMemory)
; OK *now* we should have our own, good copy.

Return DllStructGetData($tData, "Text")
Case $CF_UNICODETEXT
Return _WinAPI_WideCharToMultiByte($tData)
Case Else
Return $hMemory
EndSwitch
EndFunc ;==>_ClipBoard_GetData_Carefully

Manual states that success returns 1, But its always returning zero.

looking at the _ArraySort code I see only 'errorlevel returns' Seems that a final 'Return 1' is missing here.