 |
Categories |
|
 |
|
| robots.spider(tm)
|
Hits: 376
|
|
Description: robots.spider(tm) will read a robots.txt file and compare it with the pages on the site.
Search engines spider the site. If there are no references to a directory or file, it will not be indexed. robots.txt files should not refer to unreferenced content because it allows other viewers to identify areas of your site that you don't want indexed. The read depth limits the number of pages scanned.
|
| Resource Specification |
| Platform(s): |
linux |
| Date Added: |
Mar 7, 2008 |
| Last Updated: |
Mar 7, 2008 |
| Author: |
Betsy A. Gamrat |
| License Information |
|
License # 1: |
| |
License Type: |
GPL
|
Price:
| Free |
Average Visitor Rating:
0.00 (out of 5)
Number of Ratings: Votes |
Average Member Rating:
Not yet reviewed by any member....
You can be the FIRST one to write a review for
robots.spider(tm).
|
| Other Links by This Member |
|
|
|
|
Notes
