Skip to content
Peacock Search
ServicesLocationsIndustriesVisibility ReviewAI AutomationSearch NotesAboutWhatsAppBook review
Home / Search Notes / How to Check If Google Can Crawl and Index Your Site

Search Notes

How to Check If Google Can Crawl and Index Your Site

May 11, 2026

Supports Technical SEO

Crawled, indexed and ranking are three different things. Check them in that order, and start with the pages that matter.

Quick answer

List your priority pages, inspect each one with URL Inspection in Search Console, then compare what you find with the Pages indexing report, your robots.txt, meta robots tags, canonicals and sitemap. Look for conflicts between those signals, because most indexing problems are mixed messages, not single faults.

Start with priority pages

Do not start by checking every URL. List the pages that matter most: service pages, category pages, best-performing articles and anything that supports enquiries or revenue. Check those first, then widen the review.

What to check

  • URL Inspection: is the URL on Google, which canonical did Google choose, when was it last crawled?
  • Pages report: which URLs are not indexed, and for which reason?
  • robots.txt: is anything important blocked?
  • Meta robots and headers: is there an accidental noindex?
  • Canonicals: does each page point to itself, unless there is a reason not to?
  • Status codes: do important URLs return 200, and do removed ones redirect or return 404 or 410?
  • Internal links: do priority pages have links from other relevant pages?
  • XML sitemap: are they listed, and is the sitemap clean?

What the not-indexed reasons usually mean

On my own site, Search Console showed 55 indexed pages and 47 not indexed. The reasons were mostly these:

  • Discovered, currently not indexed (33 pages): Google knows the URL exists but has not crawled it yet. Common causes are a new site, few internal links, or the page looking low priority.
  • Crawled, currently not indexed (10 pages): Google fetched the page and decided not to index it. Look at content quality, duplication and whether the page adds anything.
  • Page with redirect (4 pages): old URLs that now redirect. This is expected and not a problem.

Among the not-indexed pages were several of my service pages, such as `/services/` and `/process/`, and most of my Search Notes. That suggested the issue was less about blocking and more about how strongly the site signalled that these pages mattered.

Fixing it

Work through causes in order, from cheapest to most involved.

  • Blocked or noindexed: remove the block. Request indexing only after.
  • Wrong canonical: correct it so the page canonicalises to itself, or accept that Google has chosen the right one.
  • Weak internal links: add links from relevant pages, in body text, not only the footer. See how to find and fix orphan pages.
  • Thin or overlapping content: improve, merge or remove.
  • Sitemap clutter: keep it to canonical, indexable pages that return 200.

Requesting indexing does not override these. It only asks Google to look again.

XML sitemap checks worth doing

  • Submit one sitemap, or a sitemap index. During my own migration I found two sitemaps live at once, one of which listed two duplicate URLs.
  • List only pages that return 200, are indexable and canonicalise to themselves
  • Remove redirected, noindexed and removed URLs
  • Do not assume inclusion means indexation. A sitemap helps discovery. It does not make Google index a page.
  • Make sure the sitemap URL is in robots.txt and submitted in Search Console

Look for conflicting signals

A page might be in the sitemap but canonicalised elsewhere. It might be linked internally but blocked in robots.txt. It might be indexable but buried deep in the site. Fix the conflict, not each signal in isolation.

When to get support

If priority pages are not indexing and you cannot see why, the Technical SEO service covers crawl and index diagnosis, and the Visibility Review puts it in a wider prioritised plan.

FAQ

Can a page be crawled but not indexed?

Yes. Google can crawl a page and decide not to index it, especially if it is duplicate, thin or not clearly useful.

Should every page be indexed?

No. The aim is to index pages that deserve to appear in search and keep low-value URL patterns under control.

How long does indexing take?

It varies from days to weeks, and longer for new sites or pages with few links. Requesting indexing helps a little but does not guarantee a timescale.

Related Search Notes

  • How to Find and Fix Orphan Pages
  • Technical SEO Checklist for Large Websites
  • What Should Be Included in an SEO Audit?

Practical use

How to turn this into action

  • Check whether the issue affects important templates, not just isolated URLs.
  • Separate urgent fixes from useful clean-up so the work can be prioritised.
  • Collect examples before briefing developers, writers or stakeholders.
  • Link the recommendation back to visibility, traffic quality or launch risk.
See how this supports Technical SEO

Author

Written by Chris Payne

Chris Payne is the freelance SEO consultant behind Peacock Search. He has 8 years of SEO experience across ecommerce, publishing, UK and international websites, with a focus on technical SEO, content strategy, migrations and complex large-site audits.

More about Chris Payne
Peacock Search

Freelance SEO consultancy based in Romsey, specialising in audits, technical SEO, local SEO, content strategy and migration support.

Process SEO Audits Free SEO Audit Template Freelance SEO Consultant Agency Support AI Automation Sample Outputs Case Studies Privacy Cookies