Technical SEO: The Checklist That Decides Whether Google Can Read Your Site

Four gates between your page and a searcher

Every page on your site has to pass four gates before it can earn a click, and technical SEO is the job of keeping each gate open. Content and links decide how well a page ranks. Technical work decides whether it is in the race at all. Our guide to how search engines work covers the journey in depth; here is the short version, with the failure that stops each step.

Most small business sites never meet an exotic technical problem. They meet the same dozen ordinary ones, usually introduced by a redesign, a plugin update or a well-meaning developer who copied the staging settings to the live server. The rest of this guide is that dozen, in the order they do damage. The SEO guide shows where technical work sits beside on-page and off-page SEO.

robots.txt and status codes: the doorman and the receptionist

The robots.txt file at the root of your domain tells crawlers which paths they may request. Google is blunt about its limits: robots.txt "is not a mechanism for keeping a web page out of Google." A blocked URL that other sites link to can still show up in results, just without a description. To keep a page out, Google recommends a noindex rule or a password, and the noindex only works if robots.txt lets Googlebot fetch the page and read it. Block the page and noindex it, and the noindex is never seen.

• Google reads only the first 500 KiB of a robots.txt file and ignores the rest.
• Each host, protocol and port needs its own file, so shop.example.com is not covered by the file on www.example.com.
• Google generally caches robots.txt for up to 24 hours, so a fix is not instant.
• Google ignores crawl-delay. Only user-agent, allow, disallow and sitemap lines count.
• A robots.txt that returns a 404 means no restrictions. One that returns a server error makes Google stop crawling the site while it retries.

Status codes are the second half of crawl control. Every response your server sends carries a three-digit verdict, and Google acts on it. The table is drawn from Google's page on HTTP status codes and network errors. If Google suddenly stopped seeing your site, start with robots.txt blocking Google.

One page, one address: canonicals, redirects and duplicate URLs

The same page often answers at several addresses: with and without www, over http and https, with tracking parameters, with a trailing slash and without. Google picks one as the canonical and folds the others into it. Its documentation says plainly that some duplicate content "is normal and it's not a violation of Google's spam policies." The cost is confusion, not a penalty: split signals, the wrong URL in results and reports that are hard to read.

You can steer the choice. Google ranks the signals by strength: a redirect is a strong signal, a rel="canonical" link is a strong signal, and listing a URL in your sitemap is a weak one. Google also prefers HTTPS pages over their http twins as canonical. It treats every one of these as a hint, so it can still overrule you when the signals disagree. Do not use robots.txt to handle duplicates, and do not use noindex to choose between versions, since noindex removes the page outright. The full method lives in our guide to canonical tags, and online stores with filter pages should also read the ecommerce SEO section.

Redirects carry the most weight and cause the most trouble. Google's crawlers follow up to 10 redirect hops, but every hop wastes time, and a loop sends visitors to a browser error. An instant meta refresh counts as permanent, a delayed one as temporary, and a JavaScript redirect is the last resort because Google only sees it if rendering works. Our redirects guide has the server and platform settings, and too many redirects fixes the loop.

Sitemaps that tell the truth

An XML sitemap is a list of the URLs you want found. Google calls submitting one "merely a hint": it helps discovery and says nothing about whether a page deserves indexing. The rules that matter fit on an index card.

1. 1.One sitemap holds at most 50,000 URLs or 50MB uncompressed. Bigger sites split it and list the parts in a sitemap index file.
2. 2.Use full, absolute URLs, and list only the canonical version of each page.
3. 3.Google ignores the priority and changefreq fields, so do not spend an hour tuning them.
4. 4.Google uses lastmod only when it is consistently accurate. A plugin that stamps every page with today's date teaches Google to ignore your dates.
5. 5.Point to the sitemap from robots.txt with a Sitemap: line and submit it in Search Console.

A sitemap full of redirected, noindexed or 404 URLs is a list of contradictions, and it makes the Sitemaps report useless as an alarm. If yours throws errors, XML sitemap errors walks through each one, and the XML sitemaps guide covers formats and generators.

Rendering: what Google sees after the scripts run, and what AI crawlers never see

Google processes JavaScript sites in three phases: crawling, rendering and indexing. A page can wait in the render queue "for a few seconds, but it can take longer than that," in Google's words. Its standing advice:"server-side or pre-rendering is still a great idea." Two failures account for most JavaScript SEO trouble. Links built as buttons with click handlers instead of real anchor tags with an href, which Google cannot follow. And content that only loads after a tap, a scroll or a cookie banner, which no crawler performs.

Google is not the only reader now. Vercel's December 2024 analysis found that the crawlers of OpenAI, Anthropic, Perplexity, Meta, ByteDance and Common Crawl did not execute JavaScript, while Google's and Apple's did. Text that appears only after scripts run may be invisible to the systems behind ChatGPT and Perplexity. Our guide on JavaScript and AI crawlers covers that side, and JavaScript SEO covers Google's.

Structured data belongs in the same delivery check. It must be in the HTML Google renders, valid in the Rich Results Test, and honest about what the page shows. Remember that Google stopped showing FAQ rich results on May 7, 2026, so FAQ markup still describes a page but no longer wins extra space. The schema markup guide lists the types that still earn rich results.

If the words are not in the HTML, assume a machine somewhere cannot read them.

Speed, phones and padlocks

Google says Core Web Vitals "are used by our ranking systems," and adds that good scores do not guarantee top rankings. Treat them as a tiebreaker and a conversion problem, since a slow page loses callers whatever its position. Google indexes and ranks the mobile version of your site, crawled with its smartphone agent, so a mobile layout that hides text, reviews or links hides them from Google too. HTTPS is table stakes: Google prefers HTTPS URLs as canonical, and its page experience questions ask whether pages are served securely.

Fixing speed is its own trade. Start with how to improve Core Web Vitals, the mobile SEO guide and, for a padlock warning, mixed content errors. Or hand it to our site speed and Core Web Vitals team.

A one-afternoon technical SEO audit with free Google tools

You do not need a paid crawler to find the problems that cost a small business the most. You need a verified Google Search Console property, a phone and about four hours. Write down what you find as you go; the list is the fix plan.

1. 1

   Read robots.txt yourself

   Open yoursite.com/robots.txt in a browser. Look for a lone "Disallow: /" or a blocked folder that holds real pages. Then check the robots.txt report under Settings in Search Console to confirm Google fetched the current version.

2. 2

   Count what Google kept

   In the Page indexing report, compare indexed pages with the number of pages you think matter. Switch the filter to All submitted pages to see how your sitemap fared.

3. 3

   Read the reasons, not the totals

   Open each not-indexed reason. "URL marked 'noindex'" on a service page is an emergency. "Alternate page with proper canonical tag" is usually Google doing its job.

4. 4

   Inspect your five most valuable URLs

   Run the home page, top services and contact page through URL Inspection. Compare the user-declared and Google-selected canonical, then run a live test and view the rendered HTML for your key text.

5. 5

   Check the sitemap report

   Every sitemap should say Success. Compare discovered pages with what you expect; a huge gap means the sitemap lists junk or misses pages.

6. 6

   Read Core Web Vitals and HTTPS

   Note which URL groups are Poor on mobile. Open the HTTPS report and note how many indexed URLs are still HTTP, and the reasons it gives.

7. 7

   Walk the site on a phone

   Logged out, on cellular data. Tap every menu link, open old URLs from your last site, and time the slowest page with a watch.

8. 8

   Check Manual actions and Security issues

   Both should be empty. If either is not, stop and fix that first, because nothing else matters until it is cleared.

If pages are missing entirely, our guide to a site not indexed by Google goes deeper. If you would rather have the afternoon back, our free website audit checks speed signals, SEO, mobile, security and content and emails a PDF with the fixes.

Log files, crawl budget and the "advanced SEO" invoice

Somewhere an agency is selling crawl budget optimization to a plumber with forty pages. Google's own crawl budget guide is written for large sites with a million or more unique pages that change weekly, sites with 10,000 or more pages that change daily, and sites where much of the site sits in "Discovered - currently not indexed." If your pages get crawled the day you publish them, Google says you can skip the guide. Its Crawl stats report, found under Settings, carries a similar label: it "is aimed at advanced users."

Server logs are still worth a look once a year, or after a strange drop. They show which URLs Googlebot really requests, how often, and what your server answered. Verify that the visitor is Googlebot before you trust the line: Google says to run a reverse DNS lookup that resolves to googlebot.com, google.com or googleusercontent.com and confirm it with a forward lookup, or match the IP against its published common-crawlers.json ranges. Plenty of scrapers wear a Googlebot costume.