Indexed, Though Blocked by robots.txt: What It Means and How to Fix It

"Indexed, though blocked by robots.txt" means Google has the page in its index but your robots.txt file contains a Disallow rule that covers the page's URL. Google did not deindex the page because robots.txt does not control indexing. It controls crawling. If Google found the URL through links before the block was added, or if another page linked to it after the block was placed, the URL can stay indexed without Google ever re-crawling it. The fix depends on whether you want the page indexed or not.

What Is Actually Happening

Robots.txt tells Googlebot which URLs it should not crawl. It does not tell Google which URLs should not appear in search results. That distinction matters here.

According to Google Search Central's robots.txt documentation, a Disallow rule blocks the crawl of a URL but cannot remove it from the index. If Googlebot learned about the URL before the block was added, or if the URL appears in a sitemap or is linked to from an external site, Google can list the URL in search results even without crawling it. The result is a Search Console status of "Indexed, though blocked by robots.txt."

The page appears in the Page Indexing report under this status. It will likely show limited data in the Performance report because Google has not recently crawled it, so the content data Google holds is stale or absent.

There are two distinct scenarios:

You want the page blocked from crawl but indexed. This is an unusual but valid configuration. You might not want Googlebot to consume crawl budget on a low-value page while still letting it appear in search results. If this is intentional, there is nothing to fix.

The robots.txt block is unintentional. This is the more common case. A developer added a Disallow rule that was too broad, or a CMS updated the robots.txt file and swept in a path you need indexed. In this case, you have two choices: remove the Disallow rule so Google can crawl the page properly, or add a noindex directive so the page is both blocked from crawl and removed from the index.

How to Find Your robots.txt File

Your robots.txt file is served at the root of your domain: https://yourdomain.com/robots.txt. Open it in a browser or fetch it with curl. Every path listed under a User-agent block with a Disallow entry is blocked from crawling for that agent.

A minimal robots.txt looks like this:

User-agent: *
Disallow: /private/

Sitemap: https://yourdomain.com/sitemap.xml

This would block all bots from crawling any URL under /private/. A URL like /private/product-reviews would match the Disallow and appear in Search Console as "Indexed, though blocked by robots.txt" if it was already indexed.

To check whether a specific URL is blocked, use the robots.txt Tester in Search Console. It is under the Index menu. Enter the URL path and it will confirm whether the current robots.txt blocks it, and which rule is responsible.

What Does Disallow in robots.txt Mean

A Disallow directive tells the specified user-agent not to crawl any URL that starts with the given path. It is not a security control and it is not a noindex instruction.

The key behaviors:

• Disallow: / blocks all pages on the site for the specified user-agent
• Disallow: /blog/ blocks all URLs under the /blog/ path
• Disallow: /private blocks any URL that starts with /private (including /private-notes, /privacy, etc., because the match is a prefix, not a path segment)
• Disallow: with no path means nothing is blocked

Disallow rules do not cascade to other bots unless the user-agent is *. A rule under User-agent: Googlebot applies only to Googlebot. A rule under User-agent: * applies to all bots that have not been given a more specific rule.

The prefix matching behavior is a common source of unintentional blocks. Disallow: /app would block /application/, /apply, and any other path starting with /app, not just /app/.

Why is Google Not Indexing Your Site When robots.txt Is the Issue

If a page you want indexed is showing "Indexed, though blocked by robots.txt," Google has that page in its index but cannot crawl it to verify current content or confirm signals. Over time, Google may decide the page is too stale to rank confidently and demote it, even though it stays technically indexed.

There is a secondary consequence: if the page is blocked from crawl, Google cannot follow any links on it. Any internal links from that page do not pass link signals to other pages. A page blocked by robots.txt that also contains important internal links is doubly costly.

According to Google Search Central, the robots.txt file is the first thing Googlebot checks before crawling a URL. If the crawl is blocked, Googlebot does not read the page content, does not evaluate noindex tags, does not follow links. The only signal Google has is what it learned from external sources before the block.

How to Fix an Indexing Problem Caused by robots.txt

The resolution depends on the goal.

If You Want the Page Indexed and Crawled

Remove the Disallow rule that covers the URL. If the rule is intentional for other URLs but too broad, narrow it.

For example, if your robots.txt has:

Disallow: /resources

And you need /resources/case-studies to be crawled, change the rule to block only the paths you actually want blocked:

Disallow: /resources/admin
Disallow: /resources/drafts

After editing robots.txt, verify the fix using the robots.txt Tester in Search Console. Then use URL Inspection to request a re-crawl of the affected page. The status in the Page Indexing report should shift to "Indexed" once Google re-crawls and confirms the page.

Check the Page Indexing report and Index Coverage errors guide to audit the full scope of affected URLs before making robots.txt changes. It is easy to fix one URL without realizing the same rule blocks twenty others.

If You Want the Page Removed from the Index

Add a noindex directive to the page. Because robots.txt currently blocks Googlebot from crawling, you have to temporarily allow the crawl so Google can read the noindex tag, then re-block if desired.

The process:

1. Remove the Disallow rule for the page temporarily.
2. Add <meta name="robots" content="noindex"> to the page's <head>.
3. Wait for Google to re-crawl the page (use URL Inspection to request it).
4. Once the page drops from the index, you can re-add the Disallow rule or leave it without one.

Alternatively, use the Removals tool in Search Console to request a temporary six-month removal while you work on the permanent fix.

If the Block Is Intentional

No action is needed. The page will remain indexed but uncrawled. Monitor it in the Page Indexing report. If ranking performance matters for this page, consider whether a Disallow block is the right long-term approach, since an uncrawled page cannot reflect content updates.

Auditing All URLs With This Status

To see the full set of affected pages, go to Search Console, open Indexing > Pages, and click on the "Indexed, though blocked by robots.txt" row. The list shows all URLs with this status.

Export it to a spreadsheet. Group URLs by path prefix to identify whether the problem comes from a single broad Disallow rule or multiple specific ones. Fixing the root rule will often clear dozens of URLs at once.

For large sites with many affected URLs, the Index Boost tool surfaces which pages with indexing status issues have the strongest internal link signals, so you can prioritize the fix order by likely ranking impact.

Preventing the Problem

Two practices eliminate most unintentional robots.txt blocks:

Review robots.txt before deploying. Treat it like a configuration file: check it in source control, review changes in a diff, and test affected paths in the robots.txt Tester before deploying to production.

Audit after CMS updates. WordPress, Shopify, and similar platforms can regenerate robots.txt on plugin updates or settings changes. After any significant platform update, fetch your live robots.txt and compare it to the expected version.

If you use a staging environment with its own robots.txt, verify that the staging Disallow: / rule is not accidentally shipping to production. This is one of the most common sources of sudden mass deindexation.

Key Takeaways

• "Indexed, though blocked by robots.txt" means Google has the page indexed but your robots.txt Disallow rule is preventing re-crawls.
• Robots.txt controls crawling, not indexing. Removing a URL from the index requires a noindex directive, not a Disallow rule.
• The most common cause is a Disallow rule that is too broad, covering paths you need indexed.
• Use the robots.txt Tester in Search Console to identify which rule is causing the block.
• Fix the Disallow rule, verify with the tester, then request a re-crawl via URL Inspection.
• Audit the full list of affected URLs in the Page Indexing report before making changes.

Frequently Asked Questions

Does removing the Disallow rule immediately fix the status? Not immediately. After you remove or narrow the Disallow rule, use the URL Inspection tool to request a re-crawl. The Search Console status updates after Google successfully crawls the page. That can take a few days to a few weeks depending on crawl demand.

Can a page be blocked by robots.txt and still rank? Yes. A page blocked from crawl can appear in search results using link-based signals, but it cannot rank confidently for queries that depend on fresh content signals. Google cannot evaluate what is currently on the page if Googlebot cannot crawl it.

What is the difference between robots.txt Disallow and noindex? Disallow prevents crawling. Noindex prevents indexing. A noindex tag on a robots.txt-blocked page will not be read by Googlebot because it cannot crawl the page. To use noindex effectively, the page must be crawlable. Using both on the same URL creates the same situation this article describes.

How do I know which Disallow rule is covering a URL? Use the robots.txt Tester in Search Console. Enter the path, and the tester highlights which rule in your robots.txt is responsible for the block.

Will fixing the robots.txt issue move the page to fully indexed status? It moves it from "Indexed, though blocked by robots.txt" to crawlable. Whether Google then keeps it indexed depends on the content quality and whether it passes Google's indexing evaluation. Use the URL Inspection tool after the fix to check the latest crawl data.