Build your robots.txt file
Start with a preset or create your own crawler rules. The generated file can be downloaded directly.
General settings
Crawler groups
Each group specifies a crawler and the paths it may or may not crawl.
Your robots.txt file
Review the generated text before downloading it.
https://yourdomain.com/robots.txt.
What is a robots.txt file?
A robots.txt file is a plain-text file that gives automated crawlers instructions about which parts of a website they are requested to access. Website owners commonly use it to manage crawler traffic and prevent crawling of selected URL paths.
User-agent
Identifies the crawler or group of crawlers to which the following rules apply.
Disallow
Specifies a URL path that the targeted crawler is requested not to access.
Allow
Specifies a URL path that can be accessed. It can be used to make exceptions within blocked paths.
Sitemap
Provides the absolute URL of an XML sitemap that search engines can use to discover website URLs.
Multiple groups
Different crawler groups can have different rules, depending on the crawler's user-agent token.
Comments
Lines beginning with the hash character can explain the purpose of rules for website administrators.
How to use the Robots.txt Generator
- Choose a template that is close to your website setup.
- Enter your website URL and sitemap location if applicable.
- Select or add crawler groups.
- Add the paths you want to allow or disallow.
- Generate and inspect the resulting robots.txt text.
- Download the file with the exact filename
robots.txt. - Upload it to your website's document root and test the public URL.
Common robots.txt examples
Allow all crawlers
User-agent: *
Disallow:An empty Disallow value does not block any paths for this group.
Block a private-looking directory from crawling
User-agent: *
Disallow: /internal/This requests that compliant crawlers do not crawl URLs under that path. It does not protect the directory from visitors or unauthorized access.
Add an XML sitemap
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlAllow a specific file within a blocked directory
User-agent: *
Disallow: /assets/
Allow: /assets/public-logo.pngThe interpretation of overlapping rules depends on the crawler. Google generally uses the most specific matching rule.
Important robots.txt best practices
Place the file in the website root
The robots.txt file should be available at the root of the relevant host,
such as https://example.com/robots.txt. A file in a
subdirectory does not control crawling for the entire domain.
Use the correct sitemap URL
Sitemap directives should use complete absolute URLs. Check that your sitemap is accessible and contains the canonical URLs you want search engines to discover.
Do not block important website resources unnecessarily
Blocking CSS, JavaScript, or image resources can prevent crawlers from seeing pages as users experience them. Avoid broad rules unless you understand their consequences.
Do not use robots.txt for confidential content
Robots.txt is public and its instructions are voluntary. Use server-side authentication, authorization, or other access controls to protect private documents, administration areas, and sensitive data.
Understand crawler support
Not all crawlers support the same directives. For example, Google does not support Crawl-delay as a robots.txt directive. Check the documentation for the crawler you intend to manage.
Frequently asked questions
1. What is robots.txt used for?
It provides crawling instructions to automated clients. Website owners commonly use it to manage crawler access to selected URL paths and reduce unnecessary crawler requests.
2. Where should I upload robots.txt?
Upload it to the root directory of the relevant website host. For example, it should be publicly accessible at https://example.com/robots.txt.
3. Does robots.txt prevent Google from indexing a page?
Not reliably. A blocked URL can still appear in search results if Google discovers it through links. Use noindex or access protection when you need to prevent indexing.
4. What does User-agent: * mean?
The asterisk represents the default group for crawlers that do not have a more specific matching user-agent group.
5. What is the difference between Allow and Disallow?
Disallow requests that a crawler should not access a matching path. Allow identifies a path that the crawler may access. Some crawlers use the most specific matching rule when rules overlap.
6. Can I have multiple robots.txt files?
A host has one robots.txt file at its root path. That file can contain multiple user-agent groups. Different subdomains have their own robots.txt locations.
7. Should I block my admin directory?
A disallow rule can request that compliant crawlers avoid crawling an admin path, but it does not secure it. Protect administrative pages with proper authentication and authorization.
8. Does Google support Crawl-delay?
Google does not support Crawl-delay in robots.txt. Other crawlers may interpret it differently. Consult the documentation of the crawler you are targeting.
9. How can I check my robots.txt file?
Visit your public robots.txt URL and inspect the text. You can also use the relevant search engine's current documentation and testing tools to review crawler behavior.
Related SoftRebate tools
Explore more tools for website development, SEO, and technical workflows.