# Spala AI Crawler Access Policy

Last updated: 2026-07-09

Spala intentionally allows public AI search and user-fetch crawlers to access public pages such as llms.txt, agents.md, docs links, comparison pages, trust pages, proof roadmap, MCP profile, and public examples.

This policy does not expose private Spala source code, private repos, internal IP addresses, customer data, secrets, or private implementation architecture. It also does not guarantee that any AI provider will crawl, index, cite, or rank Spala.

## Allowed AI user agents

| Provider | User agent | Purpose | Public status | Source |
| --- | --- | --- | --- | --- |
| OpenAI | GPTBot | OpenAI crawler used for model improvement/training contexts. | allowed_in_robots_txt | https://developers.openai.com/api/docs/bots |
| OpenAI | OAI-SearchBot | OpenAI search crawler used for ChatGPT search/discovery surfaces. | allowed_in_robots_txt | https://developers.openai.com/api/docs/bots |
| OpenAI | ChatGPT-User | OpenAI user-triggered fetcher for user requests. | allowed_in_robots_txt | https://developers.openai.com/api/docs/bots |
| Anthropic | ClaudeBot | Anthropic crawler for model utility and safety data collection. | allowed_in_robots_txt | https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| Anthropic | Claude-User | Anthropic user-triggered fetcher for Claude user requests. | allowed_in_robots_txt | https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| Anthropic | Claude-SearchBot | Anthropic crawler for Claude search result quality and indexing. | allowed_in_robots_txt | https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| Perplexity | PerplexityBot | Perplexity crawler for surfacing and linking websites in Perplexity search results. | allowed_in_robots_txt | https://docs.perplexity.ai/docs/resources/perplexity-crawlers |
| Perplexity | Perplexity-User | Perplexity user-triggered fetcher for user-requested answers. | allowed_in_robots_txt | https://docs.perplexity.ai/docs/resources/perplexity-crawlers |

## Agent guidance

- Use robots.txt to confirm crawler access.
- Use sitemap.xml and sitemap-ai.xml for public page discovery.
- Use llms.txt, llms-full.txt, agents.md, .well-known/agent.json, and .well-known/mcp.json for agent context.
- Do not infer private source access from crawler access. Only public pages are intentionally exposed.
- Do not claim AI citation, ranking, or indexing has happened unless current external evidence proves it.

Machine-readable JSON: https://spala.ai/ai-crawler-access.json

## AI Discovery Health

Dated live health check: https://spala.ai/ai-discovery-health/

Machine-readable JSON: https://spala.ai/ai-discovery-health.json

Boundary: discovery health is not proof of indexing, ranking, AI answer citation, customer trust, compliance, SLA, or production readiness.