WordPress robots.txt: what it outputs and how to edit it safely

A wrong rule can keep Google, Bing and AI crawlers away from pages you want found, or leave pages you meant to hide indexed (in the search engine's library that results and AI answers are built from). This guide covers the default output as of October 2026, editing with Yoast SEO, Rank Math, All in One SEO or a file, safe AI-crawler rules, what Cloudflare adds, and testing. For the rest of the setup (sitemap, Search Console, Bing and IndexNow), start at our WordPress indexing hub, or see the indexing guides for other platforms if you run more than one stack.

What does the default WordPress robots.txt contain?

The default WordPress robots.txt has one group for all crawlers that disallows the admin area, allows the AJAX endpoint themes and plugins use, and lists the core XML sitemap. Current core code outputs this for a public site at example.com with pretty permalinks:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Where each part comes from, as of October 2026:

• The two rules come from [do_robots()](https://developer.wordpress.org/reference/functions/do_robots/), which the WordPress developer reference describes as the function that "Displays the default robots.txt file content." It builds the paths from your admin URL, fires the do_robotstxt action, then passes the text through the robots_txt filter (since WordPress 3.0).
• The Sitemap line is added by core's built-in sitemaps (WordPress 5.5 and later), which hook into robots_txt and append the index URL only when the site is public (WP_Sitemaps source). Without pretty permalinks the line points to /?sitemap=index instead of /wp-sitemap.xml (WP_Sitemaps_Index source).
• Only root installs get it. Core registers the robots.txt rewrite with the comment "robots.txt -- only if installed at the root" (WP_Rewrite source). Google says robots.txt must sit "in the top-level directory of a site" (robots.txt spec), so a WordPress install in a subfolder needs the file managed at the domain level.

Virtual or physical: which robots.txt does WordPress serve?

WordPress serves its virtual robots.txt only when no physical robots.txt file exists in the site root; a real file always wins. Yoast's help page puts it plainly: "WordPress generates a virtual robots.txt file if the site root does not contain a physical file" (Yoast). The reason is the server, not PHP: WordPress's standard Apache rules only pass a request to WordPress when "RewriteCond %{REQUEST_FILENAME} !-f" is true, meaning no real file matches (WordPress .htaccess docs).

That matters for plugins that edit the virtual file:

Rank Math, which edits "by creating a virtual file", says you'll "need to delete the actual robots.txt file (if any)" first (Rank Math). If a plugin's edits never reach the live file, look for an old physical file.

What does "Discourage search engines from indexing this site" do to robots.txt?

The "Discourage search engines from indexing this site" checkbox (Settings → Reading → Search engine visibility) now works through a robots meta tag, not robots.txt. WordPress's 5.3 announcement says the old Disallow: / line "has been removed for non-public websites in WordPress 5.3", and that discouraged sites instead "display an updated robots meta tag" (Make WordPress Core, September 2, 2019).

With the box ticked, as of October 2026:

• Every page's <head> gets <meta name='robots' content='noindex, nofollow' /> from core's wp_robots_noindex() (robots-template.php source).
• robots.txt still shows the /wp-admin/ rules but loses the Sitemap line, because core sitemaps switch off when the site isn't public.
• The dashboard's At a Glance box shows "Search engines discouraged".

The setting's own note is honest about its limits: "It is up to search engines to honor this request" (options-reading.php source). For development sites, the 2019 post recommends an X-Robots-Tag: noindex, nofollow HTTP header, since images and PDFs bypass WordPress.

How do you edit robots.txt in WordPress?

You edit robots.txt in WordPress through an SEO plugin's editor, a physical file in the site root, or the robots_txt filter in code. Pick one method; mixing a physical file with a plugin editor is how edits go missing. Menu paths below are from each plugin's own documentation as of October 2026.

Yoast warns "This menu will not appear if your WordPress install has disabled file editing," and its starting block drops the /wp-admin/ rule, so add that back if you want it. All in One SEO's guide (dated August 14, 2026) says WordPress leaves "no static file to be found on your server."

Without a plugin, developers can append rules with the core filter in a small must-use plugin. This snippet adds an AI-training opt-out and leaves discouraged staging sites alone; we ran it against core's do_robots() code on October 17, 2026:

add_filter( 'robots_txt', function ( $output, $public ) {
	if ( ! $public ) {
		return $output; // Leave a discouraged (staging) site alone.
	}
	$output .= "\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: Google-Extended\nDisallow: /\n";
	return $output;
}, 10, 2 );

Like plugin edits, the filter only changes the virtual file.

Does robots.txt keep a WordPress page out of Google?

No: robots.txt stops crawling (a bot's visit to read a page), not indexing. Google's robots.txt guide says it "is not a mechanism for keeping a web page out of Google" and that a blocked URL "can still appear in search results, but the search result won't have a description" (Google Search Central).

Search Console reports this case as "Indexed, though blocked by robots.txt". Google explains: "Google won't request and crawl the page, but we can still index it, using the information from the page that links to your blocked page" (Page indexing report help). Its fix is the opposite of what most people try first: "To ensure that a page is not indexed by Google, remove the robots.txt block and use a 'noindex' directive." Our Page indexing report guide covers the other statuses you may see next to it.

Which robots.txt rules are safe for AI crawlers?

The safe pattern for AI crawlers is to block the training crawlers by name and let search and user-requested fetchers follow your normal User-agent: * rules. Operators document separate tokens, so a training opt-out needn't remove you from AI search. Tokens below are from the operators' own pages as of October 2026; our AI crawlers directory covers more.

Sources: OpenAI crawlers, Anthropic, Google common crawlers.

A WordPress robots.txt that opts out of training and keeps everything else as default:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

Stacking several User-agent lines over one rule set is valid in Google's spec (robots.txt spec). One trap: the spec also says "User agent specific groups and global groups ( * ) are not combined." If you give OAI-SearchBot or any other bot its own group, copy the /wp-admin/ rules into it.

How does Cloudflare change your WordPress robots.txt?

Cloudflare can serve a managed robots.txt that sits in front of whatever WordPress outputs. As of October 2026, Cloudflare's docs say that if your origin returns a robots.txt with HTTP 200, "Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single response" (Cloudflare robots.txt setting). Because WordPress always answers with its virtual file, a WordPress site behind Cloudflare gets both.

Cloudflare's example block adds a User-Agent: * group with Content-signal: search=yes, ai-train=no, use=reference and Allow: /, then Disallow: / for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. It is "available on all plans."

To turn it on, Cloudflare's steps are: Security Settings → filter by Bot traffic → Configure AI bot policies → set Training to Disallow → turn on Enable Bot Preference Sync → Save.

Three things to know:

1. Your /wp-admin/ rule still holds for Google. Cloudflare's User-Agent: * group and WordPress's are merged, since Google says "If there are multiple groups in a robots.txt file that are relevant to a specific user agent, Google's crawlers internally merge the groups" (robots.txt spec), and the longer /wp-admin/ path is the more specific rule.
2. Search Console may flag it. Cloudflare warns that Search Console "may occasionally report Syntax not understood" for Content Signals lines.
3. robots.txt is a request. Cloudflare notes that compliance "is voluntary" and points to AI Crawl Control for actual blocking.

Which WordPress robots.txt mistakes cost you traffic?

The costly WordPress robots.txt mistakes block what Google needs to read a page, or try to deindex with the wrong tool:

• Disallowing /wp-content/. WordPress keeps themes, plugins and uploads under /wp-content/ (WordPress file permissions guide). Google advises that if missing resources "make the page harder for Google's crawler to understand the page, don't block them" (Google).
• Launching with "Discourage search engines" still ticked. A site copied from staging keeps the noindex meta tag on every page until the box is cleared. If your whole site is missing, start with why a website isn't showing up on Google.
• Using Disallow to remove pages. It creates "Indexed, though blocked by robots.txt" instead of removing them; use noindex.
• A forgotten physical file. An old robots.txt from a migration or past plugin overrides every plugin edit.
• Blocking your sitemap. Google's help says to "Be sure that your sitemap is not blocked by robots.txt" (Google).

If a robots.txt fix has just unblocked pages that matter, check which of them Google has indexed rather than waiting to find out.

How do you test your WordPress robots.txt?

Test your WordPress robots.txt in three places: the live file, Google's robots.txt report and Bing's tester.

1. Open https://example.com/robots.txt. Google suggests you "open a private browsing window (or equivalent)" for this (Google); you'll see what is really served after plugins and Cloudflare.
2. Check Search Console's robots.txt report, under Settings. Google says it "shows which robots.txt files Google found for the top 20 hosts on your site, the last time they were crawled, and any warnings or errors encountered" (robots.txt report help). It works for Domain properties and URL-prefix properties without a path; after an urgent fix, use the more-settings icon and choose Request a recrawl.
3. Allow for caching. Google "generally caches the contents of robots.txt file for up to 24 hours," and enforces "a robots.txt file size limit of 500 kibibytes (KiB)" (robots.txt spec).
4. Run Bing's robots.txt tester in Bing Webmaster Tools. Bing says it "operates as Bingbot and BingAdsBot would" and checks a URL against the file in its editor (Bing Webmaster Blog, September 4, 2020).
5. Check the pages, not just the file. Unblocked URLs still have to be recrawled and indexed; if they show as crawled, currently not indexed, the page itself is now the issue.

For step 5, FreshlyCited checks each URL against Google ("Seen by Google" or "Invisible"), pushes only the invisible ones, and re-checks on day 1, 3 and 7. For sites you verify, it also sends changes through IndexNow (an open "this page changed" notice that Bing and other participating engines accept; Google doesn't use it), free. Indexed pages are eligible for Google's AI Overviews and AI Mode, and Google still decides what it indexes. We don't submit to ChatGPT, Claude or Perplexity, and we don't claim to.

Fixed your robots.txt? Check your WordPress pages free: 10 free links to try, then 100 free checks a month, no card needed.