Robots.txt Is Blocking Google
Search & Indexing Errors
Two lines in a text file can hide an entire website from Google.
Search Console reports “Blocked by robots.txt” or “Indexed, though blocked by robots.txt”. Pages do not appear in search, or appear with no description — just a note that no information is available.
What it actually means
Robots.txt is a plain text file at the root of your domain containing instructions for crawlers. It is the first thing Google reads before it fetches anything else, and it is honored absolutely.
Which makes it the highest-leverage file on your site in the most dangerous sense. Two lines — User-agent: * and Disallow: / — tell every search engine to stay out entirely. It takes four seconds to write and can remove a business from search until someone notices.
The usual origin is a staging site. Development copies are blocked deliberately, then the site is launched and the block goes live with it.
What usually causes it
Most likely first.
- 1
A staging block that shipped to production
Disallow: / was correct on the development server and copied across at launch. The most common cause by a wide margin.
- 2
The WordPress “discourage search engines” setting
A single checkbox in Settings, Reading that blocks the whole site. Frequently ticked during a rebuild and forgotten.
- 3
An over-broad disallow rule
A rule meant to hide one folder written loosely enough to catch templates, scripts, or entire sections.
- 4
Blocking CSS and JavaScript
Google renders pages like a browser. Blocking assets means it sees a broken page and judges it accordingly.
- 5
A security plugin writing its own rules
Some add crawler restrictions that go further than intended.
How to fix it
Work through these in order. Take a backup before you change anything.
Step 1 — Read the file yourself
Visit yoursite.com/robots.txt. If you see Disallow: / under User-agent: *, that single line is hiding the whole site.
Step 2 — Check the WordPress reading setting
Settings, Reading — make sure “Discourage search engines from indexing this site” is unchecked. WordPress generates a blocking robots.txt while it is on.
Step 3 — Test in Search Console
Use the URL Inspection tool on a page that should rank. It states plainly whether robots.txt is the blocker, so you are not guessing.
Step 4 — Rewrite the rules narrowly
Allow everything by default and disallow only what genuinely should stay out — admin paths, internal search results, cart URLs. Never block CSS or JavaScript.
Step 5 — Request re-crawling
Submit the sitemap again and request indexing on key pages. Recovery is not instant, but it starts as soon as the block lifts.
When to stop and call someone
Call someone if pages stay missing after the block is removed, since robots.txt is often only one of several things suppressing a site, or if you are unsure which paths are safe to disallow — a rule written too broadly can undo the fix you just made.
Frequently asked
Visit yoursite.com/robots.txt and look for Disallow: / beneath User-agent: *, which blocks the entire site. Google Search Console's URL Inspection tool confirms it for any specific page by naming robots.txt as the reason it was not crawled.
Almost always because it was copied from a staging server, where blocking crawlers is correct, and shipped to production at launch. On WordPress the same effect comes from the “Discourage search engines” checkbox in Settings, Reading.
Crawling usually resumes within days, but full recovery of rankings can take weeks depending on the site's size and authority. Submitting your sitemap and requesting indexing for key pages in Search Console speeds up the first pass.
No. Google renders pages the way a browser does, so blocking stylesheets or scripts means it sees a broken layout and may judge the page as poor quality or not mobile-friendly. Allow all rendering assets.
Robots.txt asks crawlers not to fetch a page; noindex asks them not to list it. They conflict in a specific way: a page blocked by robots.txt can never be read, so a noindex tag inside it is never seen, and the URL can still appear in results without a description.
If you’d rather not
We do this work every week. These are the services that cover it.
- SEO & Google Visibility RecoveryWhen traffic falls off a cliff, guessing is expensive. We find why Google stopped showing your pages — and fix the cause, not the symptom.
- Search Engine OptimizationHelp the right searchers find the right page. We improve technical structure, service intent, content depth, internal links, schema, and answer-ready copy so search can create better leads.
- AI & GEO Search OptimizationMake your business easier for AI search to understand. We shape pages for organic rankings, answer engines, generative search, citations, schema, and national discovery.
See exactly what’s holding your website back.
Get a free audit of your site — speed, SEO, mobile, and security — with the fixes that matter most, delivered as a PDF to your inbox.