User-agent: * Disallow: # ads.txt must remain crawlable for ad inventory verification (IAB ads.txt). Allow: /ads.txt