# QuantVault robots.txt # Goal: let SEARCH engines AND the AI ANSWER engines (ChatGPT, Claude, Gemini, # Perplexity) read and CITE our guide / firm / comparison pages, while marking the # proprietary problem bank (/data/) off-limits to bulk scraping, and blocking the # pure scraper bots entirely. # NOTE: robots is only a politeness signal. The real protection is server-side Pro # gating (problem solutions are not in public files) + Cloudflare. # --- Traditional search: full render + index (client-side site needs /js/ + /data/) --- User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / # --- AI answer / citation engines: read + cite the pages, but NOT the problem bank --- User-agent: OAI-SearchBot Disallow: /data/ Allow: / User-agent: ChatGPT-User Disallow: /data/ Allow: / User-agent: GPTBot Disallow: /data/ Allow: / User-agent: ClaudeBot Disallow: /data/ Allow: / User-agent: Claude-SearchBot Disallow: /data/ Allow: / User-agent: Claude-User Disallow: /data/ Allow: / User-agent: anthropic-ai Disallow: /data/ Allow: / User-agent: PerplexityBot Disallow: /data/ Allow: / User-agent: Perplexity-User Disallow: /data/ Allow: / User-agent: Google-Extended Allow: / # --- Bulk scrapers / low-value crawlers: keep out entirely --- User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: meta-externalagent Disallow: / # --- Everyone else: pages OK, keep the data + code paths off-limits --- User-agent: * Disallow: /data/ Disallow: /js/ Allow: / Sitemap: https://quantvault.org/sitemap.xml