Skip to content

Instance · Settings

Robots.txt

Last updated 7 November 2026

The Robots.txt file guides search engine crawlers on how to interact with your application. It contains instructions that specify which URLs are accessible, which ones to avoid, or even how to block certain bots from crawling your site altogether.

It is important to note that the Robots.txt file is primarily utilized to prevent overwhelming your site with crawler requests and not as a method to exclude web pages from search engine results. To ensure optimal SEO results, adhere to best practices when creating your Robots.txt file, and ensure that the instructions in both your Robots.txt file and Sitemap complement each other harmoniously.

To make changes to your Robots.txt file:

  1. Navigate to Settings from the main menu.
  2. Select Robots.
  3. Press EDIT.
  4. Make your changes on the code editor.
  5. Press SAVE.

Your instance serves a robots.txt file only when it has a robots page. If the Robots screen shows no file yet, select EDIT, add your rules in the code editor and select SAVE. Save creates the file. Once the file has content, VIEW FILE opens it.

AI Crawlers

Insites serves your robots.txt exactly as you write it. It does not add rules of its own, so AI crawlers such as GPTBot (OpenAI), ClaudeBot (Anthropic) and Google-Extended may read your site unless your file says otherwise. For most marketing sites this is what you want: AI tools such as ChatGPT, Claude and Perplexity can then learn about your business and mention it in their answers.

If you do not want AI companies to train their models on your content, add rules like these to your robots.txt:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

These rules block training only. AI tools that fetch a page to answer a live question use different crawlers, such as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot. Leave those allowed if you want your site to appear in AI answers.

Note

Staging instances are always hidden from search engines, whatever their robots.txt says. See Staging vs. Production.

To learn more about robots.txt files, please refer to this Google article.