Indexing is how search engines store your pages so they can appear in results. How it works, why pages get skipped and how AI crawlers fit in.
Popular Posts
Latest Articles
16 views
What Is Indexing in SEO? How Google and AI Crawlers Index Your Pages
Last updated: Nov. 6, 2026
Summary
Indexing is the step where a search engine stores a page it has crawled so the page can appear in search results. If a page is not indexed, it cannot rank, no matter how good it is. Google indexes almost every site using its mobile version. In 2026, AI services such as ChatGPT search also run their own crawlers and indexes, so “being indexed” now means more than Google.
Crawling, indexing, ranking
| Step | What happens | Common blockers |
|---|---|---|
| Crawling | A bot finds and downloads the page | robots.txt blocks, no internal links, server errors |
| Indexing | The engine processes the content and decides whether to store it | noindex, duplicates, thin content |
| Ranking | An indexed page is chosen for a query | Competitors answer better |
The status people misread: “Crawled, currently not indexed” in Search Console means Google saw the page and chose not to store it. That is usually a content quality signal, not a technical bug.
Mobile-first indexing in one sentence
Google uses the mobile version of your page for indexing and ranking, so anything hidden or missing on mobile, including content, links and structured data, effectively does not exist for Google.
AI crawlers index too
OpenAI’s OAI-SearchBot, PerplexityBot and others crawl the web for their own search features. Blocking them does not affect your Google index, but it lowers your chances of appearing in their answers.
From our own migration: After moving stradiji.com to a new host in November 2026, Googlebot passed every test, yet a network-level rate limit returned 429 errors to a couple of AI crawlers. Search Console will never show you that. Test AI user agents directly after any hosting or CDN change.
“Is my page indexed?” 5 checks
1. Search Console > URL Inspection: "URL is on Google"? 2. Any noindex in the page source? 3. Does robots.txt block this path? 4. Does the canonical point to this URL? 5. Does at least one internal link point here?
All five clean and still not indexed? Request indexing in URL Inspection. It has a daily quota and does not guarantee indexing.
Frequently asked questions
What is indexing in SEO?
It is when a search engine stores a crawled page in its database so it can show the page in results.
How long does Google take to index a page?
Anywhere from hours to weeks. Well-linked, frequently updated sites get indexed faster.
Is indexing the same as ranking?
No. Indexing gets the page into the database. Ranking decides where it appears for a query.
Does blocking GPTBot remove me from Google?
No. Google uses its own crawlers. Blocking AI crawlers only affects those AI services.
The one thing to do this week
Open the Pages report in Search Console and review “Crawled, currently not indexed.” If an important page is there, improve the content before you request indexing again. To check how ready your site is for AI search, calculate your GEO score →
Sources
Google Search Central, Mobile-first indexing best practices: developers.google.com
Google Search Central, How Search works: developers.google.com
OpenAI, Overview of OpenAI crawlers: platform.openai.com
These May Also Interest You
The SEOs Diners Club · Weekly newsletter
Every Monday: what's actually moving in SEO and AI search
Mert Erkal breaks down one big shift each week, backed by real tests and data.
Free · 245 weekly issues since 2022 · Unsubscribe anytime
