Site Indexing

Fastget is the system to organize a large set of information with care and accuracy without expending enormous time and resources. FASTGET is the system to browse a large amount of information in a systematic and direct manner.

Wordindex is a full text indexing suite developed using perl as the backend and PHP as the web based search utility. Any language can be used to search as long as it has access to MySQL databases. Wordindex is capable of indexing huge amounts of data, one production system has indexed over 14GB of textual, PDF, and compressed text files. Searches on that system are still less than a second on a modest server. Wordindex is clusterable, the indexing process which can take a very long time to complete on a huge dataset (meaning ~10G+) can be run over a couple nodes to spread out the load.

Blackwidow spider engine is an internal web spidering utility that will index your entire web server and create an ASCII output file of urls. Explores subdirectory structures. URL database is 100% compatible with exploit submission wizard and WAHOO.