Fast delete multiple lines from text file

[stamat]

I have to automate going through a text file and deleting quite a lot of lines.

I use _FileReadToArray() to read the file into array. But deleting hundreds of elements from the array one by one using _ArrayDelete() takes long. Do you know any more time efficient methods?

[water]

Welcome to AutoIt and the forum!

How large in megabytes is your file?

Edited November 15, 2012 by water

[stamat]

thanks water!

the file is a couple of MBs. but over 30k in lines.

Since I usually have to delete a range of lines (array elements) e.g. from 2000 to 3000 I thought I may use an array split function and then concat the new arrays. But there is no such array split function in AutoIt. Any suggestions?

[water]

You delete the lines and then write the array back to disk?

[stamat]

exactly

[water]

This reads the file into an array, ignores some records and writes the rest to an output file.

Global $aInput[1]
_FileReadToArray("C:tempInput.txt")
$hOutput = FileOpen("C:tempOutput.txt", 1)
For $iIndex = 1 to $aInput[0]
  If $aInput[$iIndex] <> "..." Then FileWrite($hOutput, $aInput[$iIndex])
Next
FileClose($hOutput)

[stamat]

This also uses a loop to go through each individual index. This is slow. I need a function which splits an array. E.g.

$avArray[0] = "JPM"
$avArray[1] = "Holger"
$avArray[2] = "Jon"
$avArray[3] = "Larry"
$avArray[4] = "Jeremy"
$avArray[5] = "Valik"
$avArray[6] = "Cyberslug"
$avArray[7] = "Nutster"
$avArray[8] = "JdeB"
$avArray[9] = "Tylo"

Let's say I need to get rid of lines 3 through 7. I could split the array at positions 3 and 7. Then concatenate the first and last arrays. This will be faster than iterating through all the elements. Is this possible? Or is there a better way? Thanks.

[water]

Looping through an array is quite fast. The following example fills an array with 30000 elements and checks each element in less then 1/2 second.

What's "slow" is reading and writing a file.

Global $aArray[30000]
Global $iTimer = TimerInit()
For $i = 0 To UBound($aArray) - 1
    $aArray[$i] = Random(1,100000, 1)
Next
ConsoleWrite(TimerDiff($iTimer) & @LF)
Global $bFlag
$iTimer = TimerInit()
ConsoleWrite(UBound($aArray) - 1 & @LF)
For $i = 0 To UBound($aArray) - 1
    If $aArray[$i] > 1000 Then $bFlag = True
Next
ConsoleWrite(TimerDiff($iTimer) & @LF)

[water]

BTW: How often do you need to process the file (every 5 minutes, daily, once)? And how fast do you need it?

[stamat]

True, looping through an array is fast, but the _ArrayDelete() function is slow. What I need is an "array splice" function. Thanks for the help, water. I have a 5 post per day limit so I won't be able to answer till tomorrow.

[water]

stamat,

Melba23 has lifted your 5 posts limit for the first 24 hours so we can go on discussing your problem.

[AZJIO]

$sText = ''
For $i = 1 To 20
    $sText &= $i & ' Line' & @CRLF
Next
; MsgBox(0, 'Сообщение', $sText)
$iStartingLine = 10
$iEndingLine = 15

$iStartingLine -= 1
; $sText = FileRead(@ScriptDir&'file.txt')
$iPos1 = StringInStr($sText, @CRLF, 1, $iStartingLine)
$iPos2 = StringInStr($sText, @CRLF, 1, $iEndingLine - $iStartingLine, $iPos1 + 1)
; MsgBox(0, 'Сообщение', $iPos1 &@CRLF& $iPos2)
$sText = StringLeft($sText, $iPos1) & StringTrimLeft($sText, $iPos2)
MsgBox(0, 'Сообщение', $sText)

Edited May 8, 2013 by AZJIO

[water]

I modified my example from post #6 so it "deletes" records 3 to 7:

Global $aInput[1]
_FileReadToArray("C:tempInput.txt")
$hOutput = FileOpen("C:tempOutput.txt", 1)
For $iIndex = 1 to $aInput[0]
    If $iIndex < 3 Or $iIndex > 7 Then FileWrite($hOutput, $aInput[$iIndex])
Next
FileClose($hOutput)

BTW:

As you can see my example does not call _ArrayDelete so it should be quite fast.

Edited November 15, 2012 by water

[AZJIO]

$sText = ''
For $i = 1 To 20
    $sText &= $i & ' Line' & @CRLF
Next
; $sText ='1 Line'
; MsgBox(0, 'Preview', $sText)
_StringDelete($sText, 2, 18)
MsgBox(0, 'After', $sText)
; MsgBox(0, 'After, @error=' & @error, $sText)

Func _StringDelete(ByRef $sText, $iStart, $iEnd)
    If $iStart > $iEnd Then
        Local $tmp = $iStart
        $iStart = $iEnd
        $iEnd = $tmp
    EndIf
    Local $iPosStart, $iPosEnd
    $iStart -= 1
    If $iStart < 1 Then
        $iPosStart = 0
        $iStart = 0
    Else
        $iPosStart = StringInStr($sText, @CRLF, 1, $iStart)
        If Not $iPosStart Then Return
    EndIf
    $iPosEnd = StringInStr($sText, @CRLF, 1, $iEnd - $iStart, $iPosStart + 1)
    If $iPosEnd Then
        $sText = StringLeft($sText, $iPosStart) & StringTrimLeft($sText, $iPosEnd)
    Else
        $sText = StringLeft($sText, $iPosStart)
    EndIf
EndFunc

Edited May 8, 2013 by AZJIO

[stamat]

water, I tried your code again. And it is fast - 60k rows written in 1.3 sec. So instead of deleting array elements and writing the array back to disk, I will write each line individually using the file write stream. Thanks!! I will try to complete the code asap but I'm sure this solves my problem.

AZJIO, your code works with strings and not arrays. So to use it I will have to convert the array to string which will make it hard for me to work with. Thanks for the input.

[water]

Glad to hear it's working for you

That's perfect to end the day. Now it's time for bed!

[stamat]

Thanks again. See you around on the forums

[AZJIO]

Read the array necessary? Read how the string far faster.

You can even so, but checking that the index did not exceed the size of the array

Global $aInput[1]
_FileReadToArray("C:tempInput.txt")
$hOutput = FileOpen("C:tempOutput.txt", 1)
For $iIndex = 1 to 3
    FileWrite($hOutput, $aInput[$iIndex])
Next
For $iIndex = 7 to $aInput[0]
    FileWrite($hOutput, $aInput[$iIndex])
Next
FileClose($hOutput)

Edited May 8, 2013 by AZJIO

[AZJIO]

Probably you didn't understand me

$sText = FileRead(@ScriptDir&'file.txt')
_StringDelete($sText, 2, 18)

Edited May 8, 2013 by AZJIO