Posted By
Csabo
on 2002-10-24
|  Not me
Here's a document on what ROBOTS.TXT is: http://www.robotstxt.org/wc/norobots.html. Lots of spiders (e.g. search engines) use it. BTW my crawler only crawls your old link ("http://www.stormpages.com/plus4/downloads.htm"), I haven't added the new one yet, but I should!  |
