delete duplicate lines from text file or array

[dickjones007]

i've managed to get links from my text file to display into array like this:

#include <file.au3>
#include <Array.au3>
 
Dim $text
If Not _FileReadToArray(@ScriptDir & "\AnchorLinks.txt", $text) Then
MsgBox(4096, "Error", " Error reading text file to Array error:" & @error)
Exit
EndIf
Dim $link[UBound($text)]
$link[0] = $text[0]
For $x = 1 To $text[0]
$array = StringSplit($text[$x], ",")
$link[$x] = $array[1]
Next
_ArrayDisplay($link, "Link list")

so what i need now is to remove all the duplicate lines either from text file before getting it into array or from array itself.

does anyone know how i might do that?

Edited November 4, 2011 by dickjones

[dickjones007]

one more thing that troubles me

when i get the array, the first line in it is "[0] 24" which is not in my text file

i assume it is number of lines detected

how can i remove that line?

Edited November 4, 2011 by dickjones

[BrewManNH]

Look at the functions _ArrayUnique for the first question, and also _ArrayDelete for deleting the first element of the array if you feel you really need to delete it.

Edited November 4, 2011 by BrewManNH

[BitByteBit]

This should do the trick. You will however loose the order of your links, is that a problem?

#include<array.au3>
#include<file.au3>
Dim $aFile
_FileReadToArray(@DesktopDir & '\Codes2.txt', $aFile)
_ArrayDelete($aFile, 0)
_ArraySort($aFile)
For $i = UBound($aFile) - 1 To 1 Step -1
   If $aFile[$i] = $aFile[$i - 1] Then _ArrayDelete($aFile, $i)
Next
_ArrayDisplay($aFile)

[Malkey]

Without the sort version.

#include<Array.au3>
#include<File.au3>

Local $aFile, $aArray

_FileReadToArray(@ScriptDir & "\AnchorLinks.txt", $aFile)

$aArray = _ArrayUnique($aFile, 1, 1)
_ArrayDelete($aArray, 0)

_ArrayDisplay($aArray)

[dickjones007]

_arrayunique and _arraydelete works great

thnx so much for this

now i can go on with my code