— The Internet Archive Crawler Team
Thanks for helping us build a web history – or for choosing to stay out of it. Either way, robots.txt is your voice.
IA_Support_Team Date: October 12, 2005 Applies to: Webmasters, site owners, and robot enthusiasts Hey everyone, By now, many of you have noticed our bot, ia_archiver , visiting your sites. We’re the Internet Archive (archive.org), and we’re building the Wayback Machine – a historical record of the web.
