Crawling, indexing and ranking
The three stages every page goes through before it can appear in results.
- Lesson
- 1/7
- Reading time
- 8 min
- Sections
- 5
- Through the course
- 14%
Search engines work in three stages. Understanding which stage a page is stuck at is the fastest way to diagnose why it isn't getting traffic.
Crawling
Crawlers discover URLs by following links and reading sitemaps, then request each page. A page nobody links to, or one blocked in robots.txt, may never be fetched. Crawlers have limited time per site, so slow servers, endless filter URLs and redirect chains waste it.
Indexing
After fetching, the engine parses the HTML, works out what the page is about, picks a canonical version among duplicates and decides whether to store it. A noindex robots tag or X-Robots-Tag header, an error status, or a canonical pointing elsewhere keeps a page out of the index.
Ranking
When someone searches, the engine selects indexed pages that match the query and orders them. Relevance to the query, the quality and depth of the content, links from other sites, page experience and the searcher's location and device all play a part.
Check your understanding
A page returns a 200 status, is linked from the menu and has a canonical pointing to a different URL. Which stage is it most likely stuck at?
Diagnose by stage
- Not crawled: check internal links, the sitemap and robots.txt.
- Crawled but not indexed: check status code, noindex, canonical and duplicate content.
- Indexed but not ranking: check intent match, content depth, links and competitors.
Work through the stages in order and stop where it fails.
- Is it linked from another page or listed in the sitemap? If not, it may never be crawled.
- Does it return 200, without noindex, with a self-referencing canonical? If not, it won't be indexed.
- Is it indexed but buried? Compare intent and depth with the pages ranking above it.
Lesson 1 of 7
Crawling, indexing and ranking