Memory allocation error when lots of ram still available.

[anonymous]

OS = WinXP SP2
RAM = 2.00Gb
AutoIt Ver = 3.2.11.12

I'm not sure if this is a bug or some limit of Window or AutoIt

Background:
I'm working with some large text files 200Gb to 400Gb and found that any of the file functions that use FileRead() generate an error, even though Task Manager shows plenty of memory left, in the form of a message box
Title = 'AutoIt'
Text = 'Error allocating memory.'

I can work around this by processing the files line by line, but obviously this is much slower.

Observations:
The help file give the theoretical limit 2147483647 characters but it would appear that it is impossible to get anywhere near this limit.

I created the scripts to try and find exactly what the limit is on my system. The results seem to show that the limit is some sort of global memory limit within AutoIt, rather than a limit for each string, as each time I double the number string variable filled the total memory used before the error allocating memory occurs is approximately the same.

Should AutoIt be able to make use of more of the memory available to it?

Script 1

#Include <String.au3>
Local $sString = _StringRepeat( "A", 1024)
Local $iCount = 0
Local $sTest1 =''

While 1
	$sTest1 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
WEnd

Script 2

#Include <String.au3>
Local $sString = _StringRepeat( "A", 1024)
Local $iCount = 0
Local $sTest1 =''
Local $sTest2 =''

While 1
	$sTest1 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
	$sTest2 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
WEnd

Script 3

#Include <String.au3>
Local $sString = _StringRepeat( "A", 1024)
Local $iCount = 0
Local $sTest1 =''
Local $sTest2 =''
Local $sTest3 =''
Local $sTest4 =''

While 1
	$sTest1 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
	$sTest2 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
	$sTest3 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
	$sTest4 &= $sString
	$iCount += 1	
	ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $iCount = ' & $iCount & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
WEnd

Maximum values of $iCount which equates to the size of the $sTextn
Script 1 = 262142Kb in 1 string
Script 2 = 262141Kb in 2 strings
Script 3 = 262139Kb in 4 strings

[Bowmore]

Forgot to add my name to the original.

[anonymous]

The maximum amount of characters you can have in a variable is 4095. It is most likely due to this, since you are dealing with very very very large files. I had the same problem when I set my length high in TCPRecv ()

[anonymous]

[quote]Script 1 = 262142Kb in 1 string
Script 2 = 262141Kb in 2 strings
Script 3 = 262139Kb in 4 strings quote

You're in fact over the "theoretical limit" - if those values are correct.
262141000 bytes > 2147483647 bytes

[Valik]

Replying to anonymous:

The maximum amount of characters you can have in a variable is 4095. It is most likely due to this, since you are dealing with very very very large files. I had the same problem when I set my length high in TCPRecv ()

Oh really?

Local $var
For $i = 1 To 10000
	$var &= "a"
Next
MsgBox(4096, "", StringLen($var) & " is more than 4095.")

The "problem" is two-fold. First, the practical limit is going to be less than half of 2^{31 when you try to do concatenation. The string is going to have to grow so there has to be room for 2 copies of the string *and* the extra space due to natural growth. Also, 2GB is the limit of all memory in the entire process (on 32-bit Windows). Things like DLLs and the executable eat into this space (not much when talking 2GB but still). There is also space reserved for the stack. And put quite simply,}

The main "problem", however, is we aren't requesting a large heap. It's unlikely the default heap is going to be ~2GB large. We would have to explicitly request a big-ass heap and then use that heap when allocating things.

[Jpm]

perhaps another pb as the limit today is something as 256Mb

[Jon]

I've made some tweaks that "calm down" how much large strings leave themselves room to grow for which will help _a little_. But either way this is an OS / memory issue - the AutoIt code actually has the limit on a string set to 4GB but we hit performance and OS memory problems loooong before we get near that.

Some things to be aware of:

• On a 2GB system under x86, the biggest amount of memory a single process can allocate is going to be in the 1.5GB range.
• We are running in unicode mode, so each character actually takes up TWO bytes
• Strings leave themselves a little room to grow (rc4=double, rc5 it will be between double and 5% depending on the size). This is for performance reasons.
• As the string is passed around AutoIt or concatenated (or when the string is resizing itself to make more room) there will be 2 copies allocated for a short time.

An example string of 256MB charcters before I made the upcoming tweaks:

• Worst case is that the string has allocated double the room to grow which instantly makes it occupy 512MB chars.
• Unicode, so we double the size... 1GB

That's before we've made any copies or passed it around AutoIt. Now we try to add another character to that string and it has no room, so it has to allocate another string of bigger size so that we can move the original string into it. Blam. We don't have another 1GB to spare so we get an error.

Upcoming changes mean that strings of that size will only give themselves 5% to grow, so worst case for the example string would be 256 * 2 * 1.05 = 537MB which is a bit better. But we'll still get problems after another few hundred MB...

Going to close this as a "wontfix" - there's nothing we can do really. If AutoIt gives a "out of memory" error then it's not lieing...it's run out of memory :)

[Bowmore]

Replying to Jon:
Thanks for taking the time for the detailed explanations Jon. very much appreciated.

Bowmore.