# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # robots.txt for http://www.spaarinformatie.nl/ # Autodiscover sitemap by spiders Sitemap: http://www.spaarinformatie.nl/sitemap.xml User-agent: ia_archiver Disallow: / User-Agent: MJ12bot Disallow: