How to Create a Robots.txt File (With Examples for WordPress)
To create a robots.txt file, make a plain text file named robots.txt, add rules such as User-agent and Disallow, and upload it to the root of your website, for example example.com/robots.txt. You can write it by hand or generate it with our Robots.txt Generator.
Table of Contents
What Is a Robots.txt File?
A robots.txt file tells search engine crawlers which parts of your site they may crawl. It is a request, not a security tool. Well-behaved crawlers follow it, but anyone can still open the file or visit a blocked URL.

Robots.txt Syntax
- User-agent: the crawler the rule applies to. An asterisk means all crawlers.
- Disallow: a path the crawler should not crawl.
- Allow: a path that is allowed, often used inside a blocked folder.
- Sitemap: the full URL of your XML sitemap.
User-agent: *
Disallow: /private/
Sitemap: https://www.example.com/sitemap.xml
How to Create a Robots.txt File
- Open a plain text editor and create a new file named
robots.txt. - Add a User-agent line. Use an asterisk to apply the rules to all crawlers.
- Add Disallow or Allow lines for the folders you do not want crawled.
- Add the full URL of your sitemap.
- Save the file as plain UTF-8 text and upload it to the root of your domain.
- Open yourdomain.com/robots.txt in a browser to confirm it loads, then check it in the robots.txt report in Google Search Console.
If you prefer not to write it by hand, the Robots.txt Generator builds the file for you.
Robots.txt Example for WordPress
A common WordPress setup blocks the admin area but still allows the file that many themes and plugins need:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap_index.xml
Replace the sitemap address with your own. If there is no physical file, WordPress serves a basic virtual one. SEO plugins such as Rank Math also let you edit robots.txt from their settings.
Common Robots.txt Mistakes
- Using it to hide pages from Google. A blocked page can still appear in results if other sites link to it. Use a noindex tag instead, and do not block that page, or Google cannot see the tag.
- Blocking everything. The line
Disallow: /blocks the whole site. It is often left over from a staging site. - Blocking CSS and JavaScript. Google needs these files to render your pages properly.
- Wrong location. The file must be at the root of the domain. A subdomain needs its own file.
- Ignoring case. Paths in robots.txt are case-sensitive.
- Listing secret URLs. The file is public, so never use it to hide private pages.
Robots.txt works alongside your page-level SEO. See also how long a meta description should be.
Frequently Asked Questions
Where do I put a robots.txt file?
In the root directory of your domain, so it loads at yourdomain.com/robots.txt.
Does robots.txt stop a page from being indexed?
No. It controls crawling, not indexing. To keep a page out of search results, use a noindex tag.
Is a robots.txt file required?
No. Without one, crawlers assume they can crawl your whole site. A file is useful when you want to keep crawlers away from certain areas.
How do I test my robots.txt file?
Open it in your browser to check that it loads, then use the robots.txt report in Google Search Console.
Final Thoughts
Creating a robots.txt file takes only a few minutes. Keep the rules simple, add your sitemap, avoid blocking important pages, and check the file after every change.
