My script runs too slow - any alternative?

[rossnixon]

My script works. It strips off the first 9 lines of each ASCII file (*.mif) in a folder.
But it is incredibly slow on large files. I previously used http://www.ecobyte.com/replacetext/ - but it won't work on Win 7 64-bit
Does anyone know of a way to speed up my code, or should I look for another program?

 

strip9.au3

[iamtheky]

#Include <File.au3>
Dim $text

$FileList = _FileListToArray(@ScriptDir, "*.MIF", 1)
If @error = 4 Then
    MsgBox(0, "", "No MIF Files Found.")
    Exit
EndIf

For $i = 1 To $FileList[0]
    ProcessFile($FileList[$i])
Next
MsgBox(0, "", "Updated " & $i-1 & " MIF files.")

Func ProcessFile($filefromlist)

    $MaxCharsPerLine = 256
    $sFile = FileRead($filefromlist , $MaxCharsPerLine * 9)
    $aFile = stringsplit($sFile , @LF , 2)
    $hFile = FileOpen($filefromlist , 2)

      For $j = 0 To 8
         FileWriteLine($hFile , $aFile[$j] & @LF)
      Next

EndFunc   ;==>ProcessFile

 

[Zedna]

Without posting your code you will get no help ...

EDIT: try this

Func ProcessFile($filefromlist)
    $text = FileRead($filefromlist)
    $i = StringInStr($text, @CRLF, 0, 9) ; position of 9th CRLF
    $text = StringMid($text, $i+2) ; skip first 9 lines, +2 to skip found 9th CRLF
      ;Write the new MIF file.
      $file = FileOpen($filefromlist,2)
      FileWrite($file, $text)
      FileClose($file)
EndFunc   ;==>ProcessFile

 

Edited April 30, 2015 by Zedna

[iamtheky]

zed, its attached...

ross, you can just use code tags and paste it in, that is how its usually done

Edited April 30, 2015 by boththose

[Danp2]

Maybe load the file to an array (_FileReadToArray), edit the array, and then rewrite it to disk using _FileWriteFromArray?

[rossnixon]

Without posting your code you will get no help ...

EDIT: try this

Func ProcessFile($filefromlist)
    $text = FileRead($filefromlist)
    $i = StringInStr($text, @CRLF, 0, 9) ; position of 9th CRLF
    $text = StringMid($text, $i+2) ; skip first 9 lines, +2 to skip found 9th CRLF
      ;Write the new MIF file.
      $file = FileOpen($filefromlist,2)
      FileWrite($file, $text)
      FileClose($file)
EndFunc   ;==>ProcessFile

 

​Great! And runs fast!
I only had to change $i to $j, as I was already using $i
Thanks very much Zedna! :-)

[iamtheky]

ahh i was backward, since you are already including file this would also do it

 

 

#Include <File.au3>
Dim $text

$FileList = _FileListToArray(@ScriptDir, "*.MIF", 1)
If @error = 4 Then
    MsgBox(0, "", "No MIF Files Found.")
    Exit
EndIf

For $i = 1 To $FileList[0]
    ProcessFile($FileList[$i])
Next
MsgBox(0, "", "Updated " & $i-1 & " MIF files.")

Func ProcessFile($filefromlist)

      for $i = 1 to 9
      _FileWriteToLine($filefromlist , $i , "" , 1)
      next

EndFunc   ;==>ProcessFile

 

 

Edited April 30, 2015 by boththose
back to functional

[jguinch]

Here is a regexpreplace way, which could be faster :

Func ProcessFile($sFilename)
    Local $sContent = FileRead($sFilename)
    If @error Then Return SetError(1, 0, 0)
    
    $sContent = StringRegExpReplace($sContent, "^(?:\N*\R){1,9}", "")
    
    Local $hFileW = FileOpen($sFilename, 2)
    If $hFileW = -1 Then Return SetError(1, 0, 0)
    
    FileWrite($hFileW, $sContent)
    FileClose($hFileW)
    Return 1
EndFunc

 

[Zedna]

@jquinch

StringRegExpReplace() in this case on such big files/texts will be definitely much slower than one simple StringInStr() + StringMid()

EDIT: Anyway thanks for nice example for RegExp pattern ...

Edited April 30, 2015 by Zedna

[jguinch]

Yes, you're right.  I did not see you code. 

[UEZ]

Try this:

Func FileStripFirstNLines($sFile, $iLines = 9)
    Local $hFile = FileOpen($sFile)
    If @error Then Return SetError(1, 0, 0)
    Local $sResult, $i
    For $i = 1 To $iLines
        $sResult &= FileReadLine($hFile, $i) & @CRLF
    Next
    FileClose($hFile)
    Return $sResult
EndFunc

For a 166 MB file it takes around 0.81 milli seconds to read first 9 lines.

[Zedna]

Try this:

For a 166 MB file it takes around 0.81 milli seconds to read first 9 lines.

​

But he needs to SKIP first 9 lines.

[UEZ]

Then I misunderstood op.

 

Sorry.

[UEZ]

Ok, here another version:

Func FileCutFirstNLines($sFile, $iLines = 9)
    Local $hFile = FileOpen($sFile)
    If @error Then Return SetError(1, 0, 0)
    Local $sResult, $i
    For $i = 1 To $iLines
        $sResult &= FileReadLine($hFile, $i) & @CRLF
    Next
    FileClose($hFile)
    Local $iLength = StringLen($sResult), $iFilesize = FileGetSize($sFile), $nBytes, $iDelta = $iFilesize - $iLength
    Local $tBuffer = DllStructCreate("byte[" & $iDelta & "]")
    $hFile = _WinAPI_CreateFile($sFile, 2, 2)
    _WinAPI_SetFilePointer($hFile, $iLength)
    _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer), $iDelta, $nBytes)
    _WinAPI_CloseHandle($hFile)
    $hFile = FileOpen(StringTrimRight($sFile, 4) & "_new." & StringRight($sFile, 3) , 2)
    FileWrite($hFile, DllStructGetData($tBuffer, 1))
    FileClose($hFile)
    $tBuffer = 0
    Return 1
EndFunc

Takes around 1.7 seconds to create the new file (~170 MB).

 

The higher the $iLines value the slower the runtime.

Edited April 30, 2015 by UEZ