Crawlers and indexing: how Google reads your site

Even before deciding whether to show you, Google has to be able to find you and understand what's on your pages. If this step doesn't work, it's as if your site doesn't exist — even if it's online and looks great.

What a crawler is

A crawler (or "spider", "bot") is a program search engines use to explore the web. It jumps from link to link, reads the pages it encounters and takes note of them. Google's crawler is called Googlebot.

Think of a tireless librarian roaming the world's shelves, opening books, reading their contents and jotting down what each one is about. That "reading tour" is crawling.

What indexing is

After reading a page, the engine decides whether to add it to its enormous archive: the index. This step is indexing. Only indexed pages can appear in results: if a page isn't in the index, it can never show up in a search.

So the path is always the same: the crawler finds → the engine indexes → the page can rank. Skip the first step and nothing else gets started.

Why some sites "don't exist" for Google

It happens more often than you'd think. Some typical reasons:

• Isolated pages — no links pointing to them, so the crawler never finds them.
• Hard-to-read content — text hidden inside images or loaded in ways the bot struggles to interpret.
• Site too slow — if pages take too long to load, the crawler reads fewer of them.
• Unintentional blocks — settings that, without meaning to, tell the engine "don't look here".

What to do about it

The good news: a tidy, fast site with clearly written content is also a site that's easy for a crawler to read. The same things that make a site pleasant for people make it understandable for engines — and, increasingly, for AI assistants too. The next piece of the puzzle is understanding which words your customers use to search for you.