File Search/Indexing Engine

[randallc]

Hi,

Compiles OK for me!

1. _ArraySearch(): in "Array.au3"

2. _IsPressed(): in "misc.au3"

Is your "include" directory not set?

(Also maybe include "Array2.au3" upsets this if you do not have it; You need it, so i have now changed the zip upload and "include" call to current @scriptdir; see next post too)

Best, randall

PS which beta do you have?

Edited January 26, 2006 by randallc

[randallc]

PS

Did you mean the .au3 file searchengine would not run at all [Alt-F5], or just not compile [Alt-F7]?

sqlite.au3

There should not be such a script in your directory, and none of my files should call it;

[it is a "rem" in one file only]

Perhaps you could reload the zip file into an empty diectory and extract it clean (was it the one file from my last signature link that you used?)

Here it is again, from the link site;

ZIP of library SQLiteEXE files

Best, Randall

PS[Edit - I have later uploaded fresh too ]

Compiled;

SearchEngine.zipCheck file versions?

;SqliteExe.au3 v 1_34

;ExcelCom.au3 Version: 2_81

;SearchEngine.au3 1.5.1.2.Rand15

;SQL_View.au3 ;version 1_48

;Array2.au3 ; version _1_21

Edited January 26, 2006 by randallc

[The Kandie Man]

Yup, got it to work. Very neat. Did you notice my total reconstruction of the indexing engine?

[randallc]

yes,

1. INDEX

I think you have sped up your index by 7X, and you could do more with DOS, probably to get it (nearly?) as fast as mine;

Make 2 simultaneous text files as directory, then text process them both, same line number each [check for error by line count being the same before processing? - you may have to check in the file process that any mismatch means a file has disappeared or been created in the time; so you don't need to parse or array at all from file attributes, names, sizes etc from hard drive (the time-consumers!);

Try this to see the DOS command and the 2 files... I'll leave it to you.

;dos34.au3

#include<File.au3>

$n1TimerQuery = TimerInit()

$drive=FileGetShortName("z:")

_FileCreate(@ScriptDir&"\AnsFullPath.txt")

_FileCreate(@ScriptDir&"\AnsInfo.txt")

$s_AnswerFile=FileGetShortName(@ScriptDir&"\AnsFullPath.txt")

$s_AnswerFile2=FileGetShortName(@ScriptDir&"\AnsInfo.txt")

$sCommand= " dir " & $drive & "\* /s/b /On /a-d-h-s>" & $s_AnswerFile& _

"|dir " & $drive & "\* /s /On /a-d-h-s| FIND "&'"M "'&"> " & $s_AnswerFile2 ; only lines with "AM" or "PM" [find "M"]ConsoleWrite($sCommand&@CRLF)

RunWait (@ComSpec & " /c " & $sCommand, @WorkingDir, @SW_HIDE)

ConsoleWrite('@@ Debug(681) : ' & "$sDbTableIndexSearch" & _

"=" & Round(TimerDiff($n1TimerQuery), 2) & " MSEC." & @LF ) ;### Debug Console

$n1TimerQuery = TimerInit()

$Count1=_FileCountLines($s_AnswerFile)

$Count2=_FileCountLines($s_AnswerFile2)

2. SEARCH

You may even have the search slower than before with your new system, but this should solve it, as all the info will already be parsed?...

Let me know what you think.

3. ListView.

I can add the _SQL_GuiView for more than 4000 in result once you are ready; let me know..

Best, Randall

[EDIT -note I had to think to change in the post with DOS script the find "M" to find "M " with at least 2 spaces; safer ? 3 or 4]

Edited January 27, 2006 by randallc

[randallc]

Best, Randall

[EDIT -note I had to think to change in the post with DOS script the find "M" to find "M " [after the PM or AM] with at least 2 spaces; safer ? 3 or 4]

[randallc]

I made a file just to show the index works with 2DOS files [to myself as much as anyone!]; another 5X as fast, presumably just by avoiding the array handling?...

The script overall still has problems....

Best, randall

Edited January 27, 2006 by randallc

[The Kandie Man]

Very impressive. I am still trying to figure out what you did to replace my arrays. It is much faster though.

[randallc]

hi,

1. Sorted the file by dos.

2. Read the file line by line, parsed each line, wrote parts of each line to the 2 files , master and filepath as needed and was originally in your array entry.

Uses hardly any memory, no large "fileread", and no large arrays.

Best, Randall

[EDIT - I was worried about your index missing duplicate files in different directories ("if $s_NameFile <> $s_CurrentFile Then") but I have edited it out as I see it works; I just do not understand that line]

[btw2 , In your search display, you have the same problem I have with sorting the files by columns for date ,time, and size ; they are being sorted as strings at present, i think, and not in appropriate order; needs work (and there will be a prob for international order for date? )]

Edited January 28, 2006 by randallc

[randallc]

Hi,

See my latest version...

OK -

It certainly means that my way of using these commands has put a lot of people at risk;

Filegetshortname Not Working Completely

I'll put a warning here and at the main sites I've posted code, and withdraw the code till I have a chance to correct it.

Best, Randall

Changes to the SQL version are;

1. Indexing takes longer than just acquiring the full path, as it does the dos command three times and obtains file size, time modified, and time created all at the beginning rather than later in search[and indexes on each line for fast search and sort]. (Do we really need date of creation?)

2. Still, index of 400GB, 600,000 files over six drives takes 15 or 18 minutes only

3. Search with the result overall drives, of only say 100 files, still takes less than five seconds (quicker over just 1 Disk Drive). Search with huge result, say 200,000 files, May take one-minute

4. Each database is saved with its own computer name attached, so it is possible to look at the index of other computers if you have stored a database.

5. Wildcards can be used in search terms, based on SQL wildcards (also *. ?)

6. Sort quickly, even on huge database

7. Can view the entire database of disk files, rather than just search results, if preferred

8. Double-click on file, brings up in Explorer

9. Spreadsheet view on the second tab if outlook and Excel available

10. Spreadsheet transfer of the entire database (maximum 272,000 lines) on third tablet[(cf p2) below to come..]

11. Simple output to text file if desired.

Done

2. Tidy the table list

3. Option to delete old backup tables

4. Vacuum database? -- not sure what this means, but I think the database is increasing in size too much

5. Option to turn off the spreadsheet view which slows the display a little

Still to do.

1. Incorporate buttons in main GUI?

6. Loop transfer to OWC spreadsheet, via Excel, then new sheets in the spreadsheet , for huge database

7. Enable searching on directory names as well?

8. Option for "ALL" vs "FIXED" drives

9. Change time from file to table.

Other thoughts?

Edited April 23, 2006 by randallc

[dhack]

Randallc,

just scanned through this thread,.. if i'm not mistaken, you could use the Date of Creation to 'update' the database instead of completely refreshing it..

if you left the program always running,.. you could do a quickscan of the current working folder every now and again.. and update based on the DOC.... wait.. maybe i'm thinking modification date.. if it's modified after the most recent update.. i dunno, maybe you've already considered that..

perhaps i'm way off base, but i was thinking of making a similar program a few months back.. for--- oh yeah.. for managing files.. finding dupes/etc.. which wouldn't be a bad addition..

anyway, gluck!

Damon

Hi,

See my latest version...

ZIP of library SQLiteEXE files

Still incomplete; any suggestions?

RandallZIP of library SQLiteEXE files

[randallc]

Hi,

thanks for your input.

you could use the Date of Creation to 'update' the database instead of completely refreshing it..

I think that would be no quicker than doing a full DOS dir anyway..?

if you left the program always running

Not sure i know how to implement that...

managing files

Also suggested by @Kandiman... sounds like an idea; though it double-clicks to explorer anyway, and I'm not sure I could be fussed about batching them...

finding dupes

Really already does that with the "search"?...

Thanks, Randall

[randallc]

Hi,

Here is a straight AutoIt version, no SQL, modifying @Kandiman's engine, using DOS more extensively.

The GUI still maxes out at 4000.

1. "Indexing" 60X faster, as no "indexing", just DOS list.

2. Shows folders matching search as well; once only for each.

3. Then "Searching" 20X faster as uses a DOS search of file list, then parses with AutoIt.

4. Keeps a record by computer name; this could be modified to open on other computers as well (as already in SQL version)

Best, Randall

[EDIT version 0_9; fix error]

Edited May 9, 2006 by randallc

[spyrorocks]

You should make a easy index reader for it. It saves the index file in a special fomat that the reader can read then it gives you a visual look at all the files and stuff. That would make it more useful in my option.

[randallc]

Hi,

Thanks;

"You should make a easy index reader for it"

Can you give me an example?...

Would it still allow sorting by date, size etc?

Best, Randall

[randallc]

Changes;

1. The GUI NO LONGER maxes out at 4000 [first 4000 faster "create"; thereafter "add" so no maximum]

2. "Indexing" in 2 versions now; fastest is a little slower (3 x ) at searching; but not much

3. "Searchinging" in 2 versions now; fastest is fastest (3 x ) at searching; but 8x slower with original index, depending which way you want to run it.

4. Still Shows folders matching search as well; once only for each.

5. "Searching" faster as uses a DOS search of file list, now parses with VBS call!

6. Keeps a record by computer name [and index type]; this could be modified to open on other computers as well (as already in SQL version)

7. I still prefer my SQL version as display as fast as you want [select number of GUI lines you want tyo show] and rapid, reliable sorting [Also shows entire file list and sorts rapidly on drives; not just current search list]

Best, Randall

[ptrex]

@randallc

Hi,

Could you do a speed comparison between your SQL version and my SQL Logparser version.

Are you still using SQLite ?

LogParserFS_ExecBatch_DataGrid.au3

I am interested to see the result.

[randallc]

@ptrex,

Hi,

LogParser is blistering fast for what it does, but it is hard to compare apples with apples here.

The short answer is "about the same", and I am still using SQLIteExe, and very pleased with it thanks.

The long answer is started in attached word table;

I'll try to put together scipts to compare similar functionality by using LogParser when I have time.

Best, Randall

[ptrex]

Hi Randall,

thanks for the feedback and comparison !!

I can see that they have about the same speed. So there only the difference in application.

I thouhgt yours would need an index to be build and maintained isn't it.

While the logParser runs out of the box at any time.

Anyhow good to know your still in SQL

See you later.

[kcd-clan]

woops

Edited June 28, 2006 by kcd-clan