Robots.txt

The Robots.txt file guides search engine crawlers on how to interact with your application. It contains instructions that specify which URLs are accessible, which ones to avoid, or even how to block certain bots from crawling your site altogether.

It is important to note that the Robots.txt file is primarily utilized to prevent overwhelming your site with crawler requests and not as a method to exclude web pages from search engine results. To ensure optimal SEO results, adhere to best practices when creating your Robots.txt file, and ensure that the instructions in both your Robots.txt file and Sitemap complement each other harmoniously.

To make changes to your Robots.txt file:

1. Navigate to Settings from the main menu.
2. Select Robots.
3. Press EDIT.
4. Make your changes on the code editor.
5. Press SAVE.

Insites automatically creates a Robots.txt file upon creation of your Instance. In order to adhere to SEO best practices, the system prevents search engines from indexing staging Instances and production Instances that lack a custom domain. This decision is rooted in the understanding that search engines consider duplicate content a disadvantage, hence the need to avoid it.

To learn more about robots.txt files, please refer to this Google article.