WordPress robots.txt Generator
Build a clean, SEO-safe robots.txt for WordPress in seconds — sensible defaults, a sitemap line, and common crawl rules. Free, and generated entirely in your browser.
🔒 Built locally in your browser — nothing is sent to a server. Upload the file to your site's web root at /robots.txt.
What this does
This tool builds a robots.txt file tuned for WordPress. Pick a preset, toggle the rules you want, add your sitemap URL, and copy or download the result. It runs entirely in your browser — nothing is sent to a server.
How to use it
- Choose a preset — WordPress (recommended) for a live site, or Block everything for staging.
- Toggle any extra rules (block internal search, feeds, and so on).
- Paste your sitemap URL so crawlers can discover it.
- Copy the output or download
robots.txt, then upload it to your site’s web root.
What a good WordPress robots.txt includes
The default WordPress rule blocks the /wp-admin/ directory while keeping admin-ajax.php crawlable — many plugins rely on it for front-end requests, so blocking it can break functionality and confuse crawlers. Beyond that, it’s common to block internal search result pages (/?s=) to avoid thin, duplicate content, and to point crawlers at your XML sitemap with a Sitemap: line.
Keep robots.txt lean. It is a set of crawl directives, not an indexing or security control — anything truly private must be protected with authentication, not hidden here.
Common mistakes to avoid
- Using it to deindex pages. robots.txt blocks crawling, not indexing. Use a
noindextag for that. - Blocking CSS and JS. Google renders pages; blocking assets can hurt how your site is evaluated.
- Leaving “block everything” on a live site. This quietly removes you from search.
- Forgetting the sitemap line. It is the easiest way to help crawlers find every URL.