Robots.txt Guide for WordPress
Optimize your WordPress robots.txt — what to allow, what to block, and WooCommerce-specific rules.
Published:
Tags: robots.txt WordPress, WordPress SEO robots, WooCommerce robots.txt
Robots.txt Guide for WordPress WordPress's default virtual robots.txt is a minimal placeholder. A properly configured physical robots.txt reduces crawl budget waste, prevents indexing of low-value pages, and points Google to your sitemap. --- WordPress Default vs Custom Robots.txt Default virtual robots.txt (WordPress auto-generates this): This is fine for a brand-new site but misses many common improvements. To override it, create a physical file in the root of your WordPress installation (the same directory as ). Use the free Robots.txt Generator to build a complete file and upload it via FTP or File Manager. --- Recommended WordPress Robots.txt Adjust based on your specific setup. --- What Each Rule Does | Path | Why block it | |------|-------------| | | Admin interface — no organic…
Frequently Asked Questions
What is the default WordPress robots.txt?
WordPress generates a virtual robots.txt file if no physical one exists. It only disallows `/wp-admin/` (but allows the admin-ajax.php endpoint). For most sites, you should override this with a physical robots.txt at your domain root that includes sitemap references and blocks additional low-value URLs.
Should I block wp-admin from crawlers?
`/wp-admin/` is blocked in WordPress's default robots.txt and should stay blocked — there's no indexing value and it wastes crawl budget. Keep `Allow: /wp-admin/admin-ajax.php` since some themes and plugins use this for AJAX requests that affect the frontend.
How do I block WooCommerce cart pages?
Add `Disallow: /cart/` and `Disallow: /checkout/` to your robots.txt. These are transactional pages that generate no organic traffic value and create duplicate or thin content signals if indexed.
How do I add a sitemap to WordPress robots.txt?
Add `Sitemap: https://yourdomain.com/sitemap.xml` to your robots.txt. WordPress generates a sitemap at `/wp-sitemap.xml` natively since version 5.5. Yoast SEO generates one at `/sitemap_index.xml`. Use whichever your SEO plugin creates.
What WordPress plugins help with robots.txt?
Yoast SEO and Rank Math both provide robots.txt editors accessible from the admin dashboard. They let you modify robots.txt without FTP access. Yoast provides it under SEO → Tools → File Editor; Rank Math under Rank Math → General Settings → Edit robots.txt.
All articles · theproductguy.in