# Everything here is public conference deadline data. All crawlers are welcome. User-agent: * Allow: / # AI assistant crawlers, named explicitly. # # These are already covered by the wildcard above; they are listed so the intent # is on the record and a future tightening of the wildcard does not silently cut # the site off from AI retrieval. Note that in robots.txt a named user-agent group # REPLACES the wildcard group for that crawler rather than adding to it, so each # group below repeats its own Allow rule. # # Two different jobs are represented here, and both matter: # - retrieval/search bots fetch pages to answer a user's question right now # - training bots collect pages that may inform future model weights # OpenAI: OAI-SearchBot powers ChatGPT search results, ChatGPT-User handles # user-initiated page fetches, GPTBot collects training data. User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: GPTBot Allow: / # Anthropic User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: ClaudeBot Allow: / # Perplexity User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Google. Googlebot itself is covered by the wildcard; Google-Extended is the # separate control for Gemini and AI Overviews grounding. User-agent: Google-Extended Allow: / # Microsoft / Bing. Bing's index is what ChatGPT search leans on most. User-agent: bingbot Allow: / # Apple User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # Others User-agent: Amazonbot Allow: / User-agent: meta-externalagent Allow: / User-agent: DuckAssistBot Allow: / # Common Crawl, which feeds a number of downstream training corpora. User-agent: CCBot Allow: / Sitemap: https://mldeadlines.com/sitemap.xml