Finding combinations and remove line

[water]

I'm glad a working and fast solution has beend found.

As the big input file is still read in one go I would suggest to run this script after rebooting the PC so memory isn't too fragmented.

[Spiff59]

spiff59, water, et al,

Slick idea! It would be pretty cool if one could process the entire file with a single SRER call per delete mask.

You seem to be the first to report no memory error using Fileread(). I'm guessing a 64-bit machine with 8GB RAM?

The regular expression does need some more work though, as it only partially deletes a line, leaving behind the 6th unmatched number pair and the @CRLF.

Edit: On closer look, the SRER always leaves the @CRLF behind, but only leaves the 6th unmatched number behind when it's the first or last pair on the data line.

Edited January 25, 2012 by Spiff59

[kylomas]

spiff59,

Yes, 64 bit 8GB RAM however the major difference is that I am not trying to split the file into an array, as you pointed out. I know that my regex's need work ( see my sig). That is something that the origionator of this thread can work out.

kylomas

[AdmiralAlkex]

This memory thing was discussed a few years ago, probably in the developers forum (can't find the thread). I remember Valik and I did some shouting at each other there.

@Spiff59

Take a look at Task Manager while running your script

You will notice some things.

Your ToolTip() only shows half the real value. Strings in AutoIt are Unicode so they take two bytes.

Peak Working Set (similar (but not the same) as the Peak Memory Usage in XP) is quite a bit higher than the current, as extra memory is needed for the copying.

I hope this sheds some of your doubts, as you are using (almost) three times the memory (albeit temporary) of what you write in the ToolTip()!!

My results on x64:

x64 on a fresh boot goes one step higher (I only have 6 GB RAM):

[water]

Thanks for explaining!

BTW: Funy avatar

[jaberwacky]

OK guys, I just woke up so please excuse any major fail which may follow.

1,583,704K?????????????????????????????????

The Fvck?

[AdmiralAlkex]

OK guys, I just woke up so please excuse any major fail which may follow.

1,583,704K?????????????????????????????????

The Fvck?

Yes? K = kilo = 1000. Bytes. So that's 1,5 GB.

[Spiff59]

The example from kylomas, scanning the entire input file with a single SRER statement, has so much of a speed advantage over single-record processing that I thought I'd see if I could work the bugs out and get it working properly. The following modified script is the result:

#include <File.au3>
Global $aKeys, $count

$timer    = TimerInit()
_FileReadToArray(@ScriptDir & "DeletePatterns.txt", $aKeys)
If $aKeys[$aKeys[0]] = "" Then $aKeys[0] -= 1
For $i = 1 To $aKeys[0]
    $aKeys[$i] = "(?m)^.*(" & StringReplace($aKeys[$i], " ", ").+(") & ").*rn"
Next

$sFile = FileRead(@ScriptDir & "DataFile.txt")
For $i = 1 To $aKeys[0]
    $sFile = StringRegExpReplace($sFile, $aKeys[$i], "")
    $count += @extended
Next
FileDelete(@ScriptDir & "DataFile_new.txt")
FileWrite(@ScriptDir & "DataFile_new.txt",$sFile)
MsgBox(0,"", $count & " deletions" & @CRLF & Round(TimerDiff($timer)/1000,2) & ' seconds')

I think I may have to scan through my batch scripts and see if any of those can take advantage of using StringRegExpReplace()

[jaberwacky]

I was thinking it was 1,583,704 * 1000!

[kylomas]

spiff59,

Very nice! Stopped working on it as the thread originator has not responded in some days, nor did I see any attempt at code from him/her...

Just ran this with several instances of IE open (forum and streaming a radio broadcast), Live Mail open, and Scite running with no memory problems.

I suspect that because this is valid

$x = stringsplit($x,@crlf,3)

memory usage is multiplied to allow interim copies of the target variable/array.

kylomas

Edited January 27, 2012 by kylomas

[AdmiralAlkex]

I was thinking it was 1,583,704 * 1000!

Isn't it?

[jaberwacky]

Oh, well I dunno. Wait, bytes or bits?