AI crawler access
Spala AI Crawler Access Policy
Public robots.txt and AI crawler access policy for Spala pages used by ChatGPT/OpenAI, Claude/Anthropic, Perplexity, and other public discovery agents.
Direct answer
Spala intentionally allows public AI search and user-fetch crawlers to access public discovery pages, including llms.txt, agents.md, docs links, comparison pages, trust pages, proof roadmap, MCP profile, and public examples.
This does not expose private source code, private repos, internal IP addresses, customer data, secrets, or private implementation architecture. It also does not guarantee any provider will crawl, index, cite, or rank Spala.
Allowed AI user agents
| Provider | User agent | Purpose | Status | Source |
|---|---|---|---|---|
| OpenAI | GPTBot | OpenAI crawler used for model improvement/training contexts. | allowed_in_robots_txt | https://developers.openai.com/api/docs/bots |
| OpenAI | OAI-SearchBot | OpenAI search crawler used for ChatGPT search/discovery surfaces. | allowed_in_robots_txt | https://developers.openai.com/api/docs/bots |
| OpenAI | ChatGPT-User | OpenAI user-triggered fetcher for user requests. | allowed_in_robots_txt | https://developers.openai.com/api/docs/bots |
| Anthropic | ClaudeBot | Anthropic crawler for model utility and safety data collection. | allowed_in_robots_txt | https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| Anthropic | Claude-User | Anthropic user-triggered fetcher for Claude user requests. | allowed_in_robots_txt | https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| Anthropic | Claude-SearchBot | Anthropic crawler for Claude search result quality and indexing. | allowed_in_robots_txt | https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| Perplexity | PerplexityBot | Perplexity crawler for surfacing and linking websites in Perplexity search results. | allowed_in_robots_txt | https://docs.perplexity.ai/docs/resources/perplexity-crawlers |
| Perplexity | Perplexity-User | Perplexity user-triggered fetcher for user-requested answers. | allowed_in_robots_txt | https://docs.perplexity.ai/docs/resources/perplexity-crawlers |
Agent guidance
- Use robots.txt to confirm crawler access.
- Use sitemap.xml and sitemap-ai.xml for public page discovery.
- Use llms.txt, llms-full.txt, agents.md, agent.json, and mcp.json for agent context.
- Do not infer private source access from crawler access. Only public pages are intentionally exposed.
- Do not claim AI citation, ranking, or indexing has happened unless current external evidence proves it.