Delete Duplicate lines in a text file

[JimC]

Is there a way to delete duplicate lines in a text file ? I am having problem figuring out how to do that. Any help would be greatly appreciated.

[GEOSoft]

Is there a way to delete duplicate lines in a text file ? I am having problem figuring out how to do that. Any help would be greatly appreciated.

it will be a long Process but it can be done. I wouldn't try it on large files though.

_FileReadToArray()

Then

_ArraySearch()

FileWriteLine()

This means that you have to to test each element of the array to see if another element contains the same text.

If it's a known text string then it gets simpler

_FileReadToArray()

Then

_ArrayFindAll()

That would allow you to use _ArrayDelete() on the duplicate elements before doing the FileWriteLine()

[weaponx]

Here is a quick and dirty way:

#include <file.au3>
#include <array.au3>

Dim $oldArray, $newArray[1]
Dim $inFile = "in.txt"
Dim $outFile = "out.txt"

_FileReadToArray($inFile, $oldArray)

For $X = 1 to $oldArray[0]

    $result = _ArraySearch($newArray,$oldArray[$X])

    If $result < 0 Then 
        $newArray[0] += 1
        _ArrayAdd($newArray, $oldArray[$X])
    EndIf
Next

FileDelete($outFile)

For $X = 1 to $newArray[0]
    FileWriteLine($outFile,$newArray[$X])
Next

;Currently there is a bug in this function, it will add an extra space at the top of the file
;_FileWriteFromArray("out.txt",$newArray, 1)

[randallc]

Hi,

Quicker as;

$arrdupes0=_ArrayDupes( $arrItems0,  1,  0); _ArrayDupes(ByRef $arrItemsF, $iDelete = 0, $iDetails = 0)oÝ÷ Ù«­¢+ØìÉÉåÕÁÍà¹ÔÌ(ì}ÉÉåÕÁÌ¡    åIÀÌØíÉÉ%ѵͰÀÌØí¥±ÑôÀ°ÀÌØí¥Ñ¥±ÌôÀ¤(¥¹±Õµ½¹(¥¹±Õ±ÐíÉÉä¹ÔÌÐì(¥¹±ÕÅÕ½ÐíÉÉåÕÁÌà¹ÔÌÅÕ½Ðì)1½°ÀÌØí¥9Õ´ôÄÀÀÀ)1½°ÀÌØíÉÉ%ѵÍlÀÌØí¥9Õµt)½ÈÀÌØí¤ôÀQ¼ÀÌØí¥9Õ´´Ä($ÀÌØíÉÉ%ѵÍlÀÌØí¥tôÅÕ½Ðí%Ñ´ÅÕ½ÐìµÀìÀÌØí¤)9áÐ)½ÈÀÌØí¤ôÀQ¼ÀÌØí¥9Õ´´Ä´àÀ($ÀÌØíÉÉ%ѵÍlÀÌØí¤¬ÈÁtôÅÕ½Ðí%Ñ´ÅÕ½ÐìµÀìÀÌØí¤)9áÐ)½ÈÀÌØí¤ôÀQ¼ÀÌØí¥9Õ´´Ä´àÀ($ÀÌØíÉÉ%ѵÍlÀÌØí¤¬ÔÁtôÅÕ½Ðí%Ñ´ÅÕ½ÐìµÀìÀÌØí¤)9áÐ(íø}ÉÉå¥ÍÁ±ä ÀÌØíÉÉ%ѵ̰ÅÕ½ÐìÀÌØíÉÉ%ѵÌÅÕ½Ðì¤(ÀÌØíÉÉ%ѵÌÀôÀÌØíÉÉ%ѵÌ(ÀÌØíÉÉ%ѵÌÄôÀÌØíÉÉ%ѵÌ(ÀÌØíÉÉ%ѵÌÈôÀÌØíÉÉ%ѵÌ(ÀÌØíÉÉÕÁÌÀõ}ÉÉåÕÁÌ ÀÌØíÉÉ%ѵÌÀ°Ä°À¤ì}ÉÉåÕÁÌ¡    åIÀÌØíÉÉ%ѵͰÀÌØí¥±ÑôÀ°ÀÌØí¥Ñ¥±ÌôÀ¤)}ÉÉå¥ÍÁ±ä ÀÌØíÉÉÕÁÌÀ°ÅÕ½ÐìÀÌØíÉÉÕÁÌÀìÀÌØíÉÉ%ѵÌÀìÅÕ½ÐìµÀíU   ½Õ¹ ÀÌØíÉÉ%ѵÌÀ¤´Ä¤(ÀÌØíÉÉÕÁÌÄõ}ÉÉåÕÁÌ ÀÌØíÉÉ%ѵÌİİĤ)}ÉÉå¥ÍÁ±ä ÀÌØíÉÉÕÁÌÄ°ÅÕ½ÐìÀÌØíÉÉÕÁÌÄìÀÌØíÉÉ%ѵÌÄìÅÕ½ÐìµÀíU   ½Õ¹ ÀÌØíÉÉ%ѵÌĤ´Ä¤

Best, Randall

ArrayDupes8.zip

Edited February 22, 2008 by randallc