Latest Beta

[-Ultima-]

- Testing new Visual Studio 2008 compiler options (100KB larger exe, 20% speed increase)

- Array handling rewrite

- DLLStruct handling rewrite

Hm, would it be the new compiler, or the array handling that caused the massive memory usage increase? Or maybe a little bit of both? It seems that AutoIt is now using around 4x more memory in an application I'm writing with it (same code, recompiled), as can be seen here:

(I make extensive use of arrays -- nested and all -- to store the bdecoded data)

An additional minor point... Did the DLLStruct handling rewrite cause a minor slowdown as well? When I fill the treeview, I use DLL structs and calls entirely (I can't make use of the internal treeview item creation functions because of various limitations), and as can be seen, it's around 5% slower now.

I'm still testing to try to figure out where the memory usage is going (it might even be that the treeview is now taking more memory as well -- I don't know), but I thought I'd ask now for something more official.

__________

Edit: Hm, looks like it's a little bit of everything...

You'll notice that the private bytes graph has an increase in slope when transitioning from bdecoding to treeview filling in the new betas (indicated by the red circle), while the slope actually decreases in the transition for the stable builds. The treeview now takes up a large marjority of the memory usage. Where it used to take around 30-40% of the memory, it now takes around 50-60%. In absolute terms, that's a jump from around 15MB to 120MB for the treeview (very rough estimate, but estimate nonetheless).

By the time the decoding has finished in the beta (right before filling the treeview), AutoIt is using around 100MB of private bytes (which is already almost double the total it was using in the stable). Crazy

Edited June 15, 2008 by -Ultima-

[Jon]

If anything, the memory use should be a lot less assuming everything is working correctly. Need some easy to use non-gui examples to troubleshoot further.

[-Ultima-]

Global $avArray[1] ;= ["0"]
For $i = 1 To 2000
    ReDim $avArray[$i+1]
    $avArray[$i] = $i ;& $avArray[$i-1]
Next

While 1
    Sleep(1000)
WEnd

Here's a simple one. It doesn't directly reflect how my application is working, but you'll notice that the latest beta ends up taking almost 3x the amount of memory just to perform this (~58.9MB vs ~21.7MB).

Excuse the silly-looking ReDims

__________

Edit: Hm, after a bit more testing, it appears this may not only be an issue with arrays (if at all). It might be an issue with the string optimizations you made for the latest beta -- the problem does not occur with 3.2.13.0, but does with 3.2.13.1. That might also explain why the treeview was affected very greatly (lots of strings are getting sent over to the treeview, after all).

With 3.2.13.0, AutoIt is using sane amounts of memory again (consistently using around 300-400KB more than 3.2.12.1 -- not a problem at all).

__________

Edit: Heh, I'm confused (yet again). Removing the string concatenation, I still get bloated memory usage in 3.2.13.1. I've edited the test script to reflect that. With the modified example, 3.2.13.1 is using a bit more than 11x as much memory as 3.2.12.1 (~43.2MB vs ~3.8MB). At this point, all I can say is that there's something funky going on

Edited June 15, 2008 by -Ultima-

[Valik]

Jon, COW optimization memory leak, maybe?

[Jon]

It's definately string related, if I rollback our string class to a previous version (yay modular programming) the memory use drops back to normal. I thought I'd checked the copy-on-write optimization pretty well but I must have missed something. I originally thought it might be redim/array rewrite problems but I can see memory use on another non-array related script.

[Jon]

Found it. A single line...

[Jon]

http://www.autoitscript.com/autoit3/files/beta/autoit

3.2.13.2 (15th June, 2008) (Beta)

AutoIt:

- Changed: @ProcessorArch changed to @OSArch as it was misleading.

- Changed: Size limits removed from RegRead() and RegWrite() (previously 64KB for some registry types).

- Changed: RegRead() and RegWrite() no longer use hex strings for REG_BINARY types - native binary datatypes are enforced.

- Fixed #380: @OSVersion for Windows XP 64-bit Edition.

- Fixed: Increased memory use since last beta.

UDFs:

- Fixed #371: _GDIPlus_Startup return value (wraithdu)

- Fixed #368: _ArrayToClip return value

[-Ultima-]

Nice, thanks for the quick fix. Some observations...

1. This build appears to be slower than the previous 2 betas... Results of tests with my application are as follows:

CODE

3.2.12.1

[15:36:02] [bDecode] Start: "D:\resume.dat"

[15:36:15] [bDecode] End (12.651 seconds)

[15:36:15] [TreeView Fill] Start

[15:36:27] [TreeView Fill] End (11.536 seconds)

3.2.13.0

[15:36:42] [bDecode] Start: "D:\resume.dat"

[15:36:53] [bDecode] End (11.505 seconds)

[15:36:53] [TreeView Fill] Start

[15:37:06] [TreeView Fill] End (11.282 seconds)

3.2.13.1

[15:37:18] [bDecode] Start: "D:\resume.dat"

[15:37:30] [bDecode] End (11.304 seconds)

[15:37:30] [TreeView Fill] Start

[15:37:43] [TreeView Fill] End (12.059 seconds)

3.2.13.2

[15:37:57] [bDecode] Start: "D:\resume.dat"

[15:38:10] [bDecode] End (12.319 seconds)

[15:38:10] [TreeView Fill] Start

[15:38:23] [TreeView Fill] End (12.479 seconds)

I understand my application isn't some kind of super benchmarking utility for AutoIt, and that without the source, the numbers are of somewhat limited use, but it's just a frame of reference that I have off hand to give -- I'll see about whipping up some real test scripts.

2. Using the test script I posted earlier (with string concatenation), this build takes ~4MB more RAM than the stable. I'm not trying to be a RAM whore here, just wondering if this is normal, since you did say that everything is supposed to be taking less RAM now It definitely completes the process much more quickly, though -- ~2 seconds (3.2.13.2) vs ~10 seconds (3.2.12.1):

Global $avArray[1] = ["0"]
Global $iTime = TimerInit()
For $i = 1 To 2000
    ReDim $avArray[$i+1]
    $avArray[$i] = $i & $avArray[$i-1]
Next
MsgBox(0, @AutoItVersion, TimerDiff($iTime)/1000)

3. This build is almost 100KB smaller than the previous 2... Expected? I ask only because it wasn't noted in the changelog.

Edited June 15, 2008 by -Ultima-

[Jon]

Crap, I forgot to recompile with the compiler optimizations on. Oh well, never mind.

[Jon]

I just recompiled and uploaded (same version number).

The memory difference is hard to predict. We've implemented a copy-on-write optimization ( http://en.wikipedia.org/wiki/Copy-on-write ) so each string now allocates an additional 4 bytes per unique string to use as a reference counter. However strings that are copied and only ever read are not actually copied - they just reference the initial string. This means that copying massive strings that you only read is essentially "free" as they are not duplicated. Once one of the copies is written to then the real copy is initiated. So mem comparisons should be in the same ballpark as previous versions for many tasks and much better (and a lot faster) for others especially with strings that are bigger than a few chars.

[SlimShady]

Sounds great! More scripting pleasure

[Jon]

http://www.autoitscript.com/autoit3/files/beta/autoit

3.2.13.3 (22nd June, 2008) (Beta)

AutoIt:

- Fixed #381: DirCreate() was trashing input variables (old bug brought to light by recent optimizations).

UDFs:

- Fixed #388: _GUICtrlToolbar_SetButtonSize Example

- Fixed #400: SQLite.dll.au3 @ProcessorArch replaced with @OSArch

- Fixed #390: _viPrintf replaced return type

- Added: _Timer_GetIdleTime, _WinAPI_GetWindowPlacement, _WinAPI_SetWindowPlacement (PsaltyDS)

[igorm]

Since you are dropping support for Win9x can we expect in one of next betas removing limitation of IniReadSection?

Cheers muttley

[Valik]

Less than 30 posts ago in this thread that was asked and answered (Posts #502 and #503).

[igorm]

OK, thanks for response. Anyway, it would be nice to rewrite IniReadSection, since in my opinion it's really needed. Or at least to extend limit to Win XP.

Cheers muttley

[Valik]

OK, thanks for response. Anyway, it would be nice to rewrite IniReadSection, since in my opinion it's really needed. Or at least to extend limit to Win XP.

Cheers muttley

Yes, yes. Everything is always important to somebody. So you ask us to pay special attention to you and your needs. Who will be next? And who after that? And when do we get to work on things we want to if we go down that road?

Here's a general comment directed at the community: Don't tell us what's important. We really don't care about your opinion of whether or not something is "critical". Believe it or not, it doesn't cause us to prioritize things. If it does affect prioritization, it has an inverse effect. On a related note, don't set the blocking flag on tickets when creating them. People accuse me of having a huge ego but as far as I'm concerned it takes one giant headed buffoon to think they can actually determine what blocks a release in our software. And it takes only a slightly smaller headed one to attempt to request prioritization. For those of you who resemble that remark and are about to reply chastising me or rebuking my claims, stop and think for a second. Who are you to us, really? Just another user amongst thousands. So why should we care about what you find important compared to everyone else?

[Jon]

http://www.autoitscript.com/autoit3/files/beta/autoit/

3.2.13.4 (19th July, 2008) (Beta)

AutoIt:

- Fixed #346: FileOpenDialog/FileSaveDialog filter length limit.

- Fixed #387: DllClose() and DllCallbackFree() crashing when an invalid handle used.

UDFs:

- Added #334: _SQLite 3.5.9 x86 and X64 version.

- Fixed #422: _GDIPlus_GraphicsSetSmoothingMode $iSmooth accepts only 0 - 4, doc updated

- Added: Missing FrameConstants.au3

- Added: Excel UDFs

- Updated: GuiComboBox Documentation

[Valik]

Jon missed the Aut2Exe change (learntocopypaste?). Fixed a reported bug where the compression level when selected via the GUI menu was not honored.

[Jon]

Oh, that was in the changelog under 3.2.13.3...oops.

[Valik]

Hmm. Maybe that was the first fix for this beta and I put it in the wrong place as a result?