robots.spider(tm)
robots.spider(tm) will read a robots.txt file and compare it with the pages on the site. Search engines spider the site. If there are no references to a directory or file, it will not be indexed. robots.txt files should not refer to unreferenced content because it allows other viewers to identify areas of your site that you don't want indexed. The read depth limits the number of pages scanned.
- Visit Site:
- robots.spider(tm)
- Version:
- 1.0
- Filed in:
-
PHP / Scripts & Programs / Development Tools / Utilities
- Platforms:
-
- Databases:
-
- Date Added:
- Mar 7, 2008
- Last Updated:
- Mar 7, 2008
License and Pricing Information
GPL
Price: $0.00 USD
| Publisher site visits: | 495 |
|---|---|
| Average rating: | 0.00 |
| Total ratings: | 0 |
Publisher Info
- Author:
- Betsy A. Gamrat
