Web crawler script?

[bboysza]

Before I start dev on a webcrawler script, I thought I should ask if any has already written one.

What I need to do specifically, is crawl a site, and dump all hrefs and arefs to a log file. Basically, logging the source for each image. Also, I would need to then follow any hrefs (or other jscript or asp links) to other pages and then do the same thing there.

I don't need it written for me, just looking for some ideas for the architecture.

Thanks,

Ben

[SmOke_N]

Before I start dev on a webcrawler script, I thought I should ask if any has already written one.

What I need to do specifically, is crawl a site, and dump all hrefs and arefs to a log file. Basically, logging the source for each image. Also, I would need to then follow any hrefs (or other jscript or asp links) to other pages and then do the same thing there.

I don't need it written for me, just looking for some ideas for the architecture.

Thanks,

Ben

Searched the word Spider... and poof: http://www.autoitscript.com/forum/index.ph...26entry125826

[Valuater]

Might take a look at this too!

http://www.autoitscript.com/forum/index.ph...68&hl=Valuater#

8)

Edited February 28, 2006 by Valuater