Robots.txt

The Robots.txt file guides search engine crawlers on how to interact with your application. It contains instructions that specify which URLs are accessible, which ones to avoid, or even how to block certain bots from crawling your site altogether.

It is important to note that the Robots.txt file is primarily utilized to prevent overwhelming your site with crawler requests and not as a method to exclude web pages from search engine results. To ensure optimal SEO results, adhere to best practices when creating your Robots.txt file, and ensure that the instructions in both your Robots.txt file and Sitemap complement each other harmoniously.

To make changes to your Robots.txt file:

1. Navigate to Settings from the main menu.
2. Select Robots.
3. Press EDIT.
4. Make your changes on the code editor.
5. Press SAVE.

Your instance serves a robots.txt file only when it has a robots page. If the Robots screen shows no file yet, select EDIT, add your rules in the code editor and select SAVE. Save creates the file. Once the file has content, VIEW FILE opens it.

To learn more about robots.txt files, please refer to this Google article.