# Crawlers must be able to read the noindex meta/header. # Disallowing all crawling could hide noindex and leave URL-only search listings. User-agent: * Allow: /