Indexing guide
Crawled, currently not indexed: how to fix it
Google visited your page but chose not to add it to the index yet. Here is why, and what to do.
ReadBlocked by robots.txt: how to fix it
A rule in your robots.txt file stops Googlebot from crawling these URLs. Sometimes that is on purpose, sometimes it is a costly mistake.
If Google Search Console shows blocked by robots.txt for some of your pages, this guide explains what it means and how to fix it, step by step.
Quick answer
Blocked by robots.txt means a Disallow rule in your robots.txt stops Googlebot from crawling the URL. If the block is intentional, do nothing. If you want the page in Google, remove or narrow the rule. If you want it out of Google, unblock it and add a noindex tag, because robots.txt does not reliably stop indexing.
What does blocked by robots.txt mean?
The Search Console Page indexing report help says the page was blocked by your site’s robots.txt file. Google’s Google introduction to robots.txt adds that robots.txt manages crawling and is not a way to keep a page out of Google: a blocked URL can still be indexed without its content if other pages link to it.
What causes it?
These are the most common causes I see when auditing client sites:
- A Disallow rule that is broader than intended, such as Disallow: /blog
- A staging site robots.txt (Disallow: /) copied to the live site
- Blocking parameter, filter or search URLs on purpose
- Blocking CSS or JavaScript folders that pages need to render
- Rules added by a security or SEO plugin
How do you fix it?
First find the affected URLs in Search Console, then work through the steps in order and stop when the issue is solved. The pictures show each stage.
- Open yoursite.com/robots.txt and read every Disallow line
- Paste the rules and an affected URL into the free robots.txt tester to find the exact rule
- Decide per group: should this URL be in Google or not?
- For pages you want indexed, remove or narrow the rule. The most specific matching rule wins
- For pages you want out of Google, remove the block and add a noindex meta tag instead
- Never block CSS and JavaScript that pages need to render
- Click Validate fix in Search Console and request indexing for key pages
Quick reference table
| Your goal | Right tool | Why |
|---|---|---|
| Save crawl on useless URLs (filters, internal search) | robots.txt Disallow | Stops crawling, saves server load |
| Keep a page out of Google | noindex meta tag or X-Robots-Tag header | Reliably removes it from results |
| Keep private content private | Password protection | Crawlers and people cannot read it |
| Remove a page for good | 404 or 410 status | Google drops it after recrawling |
Example: one line hiding a whole blog
Example
A site moved from staging and its robots.txt said Disallow: /bl to block a test folder called /bl-test/. Because robots.txt rules match by prefix, the rule also blocked every URL under /blog/. Search Console showed 210 posts as blocked by robots.txt. Changing the rule to Disallow: /bl-test/ fixed it, and the posts were crawled again within days.
When is it a bigger problem?
It is urgent when important pages or the whole site are listed, which usually means a Disallow: / left over from development. Also check the related warning Indexed, though blocked by robots.txt: Google indexed the URL without reading it. Fix that one by choosing either noindex (and unblocking) or unblocking so Google can read the page. Google’s Google guide to blocking indexing with noindex explains that noindex only works when the page is not blocked by robots.txt.
Not sure which pages are affected or why? My technical SEO service starts with a free audit, and the free SEO checklist covers the basics.
Frequently asked questions
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing. To remove a page, unblock it and add a noindex tag, or delete it with a 404 or 410.
Is blocked by robots.txt always bad?
No. Blocking internal search, filters and admin URLs is normal. It is only bad when pages you want ranking are blocked.
How long after I fix robots.txt will Google crawl the page?
Google usually refreshes its cached copy of robots.txt within about a day, then crawls when it next schedules the URL. Requesting indexing helps for key pages.
Which rule wins if Allow and Disallow both match?
Google uses the most specific rule, meaning the longest matching path. If they are equal, the less restrictive Allow rule wins.
Should I block AI crawlers in robots.txt?
That is your choice. Blocking them can keep your content out of some AI tools, but also stops them from citing you as a source.
What is indexed, though blocked by robots.txt?
It is a warning that Google indexed a URL it could not crawl, usually because other pages link to it. Decide if it should be indexed and fix robots.txt or use noindex.
Sources
Facts checked on 8 October 2026 against these official pages. Read how I check facts in my editorial policy and disclaimer.
Related indexing guides
All guides
Indexing guide
Discovered, currently not indexed: causes and fixes
Google knows the page exists but has not crawled it yet. Here is how to change that.
Read
Indexing guide
Duplicate without user-selected canonical: how to fix it
Google found duplicate versions of a page and picked one itself, because you did not tell it which one.
ReadWant help with your SEO?
Send your website and I will reply with a free audit.