# Robots.txt file to disallow all bots but googles and archive.org # User-agent: googlebot Disallow: User-agent: ia_archiver Disallow: User-agent: * Disallow: /