Douglas Thrift's Search Engine is an indexing search engine for use on small websites such as personal or small business sites. It is designed to be very similar to Google for end users and its output is customizable. For indexing, it supports both the Robots Exclusion Protocol and the Robots META Tag as specified at http://www.robotstxt.org/wc/exclusion.html.