Generate robots.txt file for search engines
Generate a properly formatted robots.txt file for your website with our free generator.
Create a properly formatted robots.txt file to control how search engines crawl your website
# robots.txt generated by FreeDevKit
# 2026-03-17
User-agent: *
Disallow:
https://example.com/robots.txtA robots.txt file is a text file placed in a website's root directory that instructs search engine crawlers which pages they are allowed (or not allowed) to crawl. It is part of the Robots Exclusion Protocol and is one of the first files crawlers check when visiting a website. A properly configured robots.txt can prevent search engines from indexing admin pages, duplicate content, staging environments, and other areas you want to keep out of search results. It also specifies the location of your sitemap.
Select the user agents you want to configure (Googlebot, Bingbot, etc.).
Add Allow and Disallow rules for specific paths.
Specify your sitemap URL for search engine discovery.
Add a crawl-delay if needed to reduce server load.
Copy or download the generated robots.txt file.
robots.txt prevents crawling, but not necessarily indexing. Google may still index a URL if other pages link to it. To completely block a page from search results, use a "noindex" meta tag or X-Robots-Tag header.
Discover more free tools you might find useful