Remove Duplicates

[Kaeft]

OK so I'm going to attempt to read through a word file and remove all of the duplicate words.

the words are like this:

games
yahoo
hotmail
lyrics
hotmail.com
yahoo.com
jokes
ebay
maps
games
warez
napster
www.hotmail.com
song+lyrics
www.yahoo.com
dictionary

Problem is there are 900 words. I am thinking that I should do a _FileCountLine, accompanied by a While loop, a StringReplace (to delete the line), and then _FileWriteToLine. Make it take the first line and then check every line in the file for the first line and delete the same. Then take the second line, and the third, and fourth, etc. Maybe through it into a function, idk....

That would take forever... Do you guys know of a quicker way?

Edited July 17, 2010 by Kaeft

[Kaeft]

Ok so I just did a quick sort through an array and then write back to the file and it seems there is a duplicate, triplicate, or quadruplicate of every single word. That will be handy for testing I suppose. Anyone have an idea yet?

[omikron48]

The algorithm you want is fairly simple. You just need to think through it.

If the array you start with is already sorted, you can just loop through the whole array and write the first element into a new file, then keep skipping the next element if it is the same as the element you just wrote.

[Kaeft]

I am still learning all the different functions of autoit, but I'm going to assume that the function I'm looking for is in the array section. Good to know. I'll tell you how it goes... tomorrow, lol it's getting late.

[picaxe]

Or use _ArrayUnique