Screaming Frog Semantic Similarity in 2026: What It Finds and How We Run It With Claude

How Screaming Frog’s semantic similarity works in 2026, the exact settings we use, and how we run crawls through Claude and MCP. Copy our free checklist.

Last updated: November 1, 2026

Summary

Screaming Frog’s semantic similarity feature turns every crawled page into a vector embedding, then flags pages whose meaning overlaps too much with another page (default threshold 0.95) and pages that drift away from the rest of the site (low relevance, default 0.4). It needs a paid license and an AI provider API key (OpenAI, Gemini or Ollama). Since version 24.0, released May 19, 2026, the SEO Spider also ships its own MCP server, so an assistant like Claude can start crawls and pull exports. We use that MCP every week, and this post shows the setup and the guardrails we run it with.

What changed since we first wrote this

We published the first version of this post in June 2025, the week Screaming Frog shipped version 22. It was a first look. A year and a half later, the feature has matured and so has the way we use it.

The tool kept moving. Here is the timeline, checked against the official release history:

The two additions that changed how we work are the embedding rules in version 23 and the MCP server in version 24.

How semantic similarity works, without the math

An embedding is a list of numbers that represents what a piece of text means. Two pages about the same idea land close to each other, even if they use different words. Screaming Frog sends each page’s visible text to your AI provider, stores the vector, and compares every page with every other page.

According to the official semantic similarity guide, you get three things:

• Semantically Similar (Content tab): the closest matching URL, a similarity score and how many similar URLs a page has. Anything above 0.95 is flagged by default.
• Low Relevance Content (Content tab): uses a Semantic Relevance Score that measures how close a page is to the site’s average content. The default cutoff is 0.4.
• Content Cluster Diagram (Visualisations menu): plots up to 10,000 pages in 2D, colored by cluster. Lonely dots on the edges are your likely outliers.

Since version 23 you can right-click a node and choose “Show Inlinks Within Cluster.” Underrated. It shows whether a page gets links from its closest neighbors in meaning, and a good migration trick sits next to it: embedding rules let you match staging pages only against live pages for redirect mapping.

One warning up front: a high similarity score is a reason to look, not a verdict. Two pages can be 0.96 similar and serve different intents, for example a pricing page and a feature page on the same product. Read both pages before you merge anything.

Why near-duplicates matter more when AI writes the answer

Google’s own documentation is plain about the old part of this. When a site has duplicate or very similar pages, Google chooses the canonical URL, and consolidating helps merge signals such as links into one preferred page. If you don’t choose, Google chooses for you.

Now add AI search. Google says the best practices for SEO remain relevant for AI Overviews and AI Mode, with no extra requirements. We read that as: the housekeeping that decides which page ranks also shapes which page can be cited.

Our view, and it is a view, not a measured result: three pages that say roughly the same thing split your links, your internal anchors and your clarity across three URLs. When an AI system pulls a source for an answer, you have made the choice harder instead of easier. One focused page with the full answer is a cleaner candidate. We wrote about which content patterns actually get cited in Why is AI ignoring you?.

Our rule: we don’t promise clients that consolidating pages will raise AI citations by any number. Nobody has published a study that ties a cosine score to citation rates. What we do say is that SEO is the foundation of GEO, and duplicate content is one of the oldest foundation cracks there is.

The settings checklist we use

Every menu path below comes from the official guide and the version 23 and 24 release posts. Copy it, paste it next to the Spider, and tick as you go. The last part is a prompt you can give Claude once the crawl is done.

SCREAMING FROG SEMANTIC SIMILARITY CRAWL: SETUP CHECKLIST (v24)

Before you start
[ ] Paid SEO Spider license (embeddings do not work in the free version)
[ ] API key for OpenAI, Gemini or Ollama (Gemini is the guide's suggestion;
    use a paid tier, the free tier rate limits are very low)

1. AI provider
[ ] Config > API Access > AI > pick provider
[ ] Account Information > Connect
[ ] Prompt Configuration > Add from Library > embeddings preset
    (Gemini: "Extract Semantic Embeddings from Page")
[ ] Optional: cog next to the model > Add Parameter, type Number
    OpenAI: dimensions = 256 | Gemini: outputDimensionality = 256

2. Store the page text
[ ] Config > Spider > Extraction > Store HTML
[ ] Config > Spider > Extraction > Store Rendered HTML (if JS rendering)

3. Remove boilerplate
[ ] Config > Content > Area > exclude nav, footer, cookie banner, sidebar

4. Embeddings
[ ] Config > Content > Embeddings > Enable Embedding functionality
[ ] Tick Semantic Similarity (default threshold 0.95)
[ ] Tick Low Relevance (default threshold 0.4)
[ ] Bilingual site? Add embedding rules so each language is compared
    with itself
[ ] Long pages? Advanced tab > Limit Page Content

5. Crawl and analyze
[ ] Crawl until both the crawl and API progress bars hit 100%
[ ] Crawl Analysis > Start (or Crawl Analysis > Configure >
    Auto-Analyse at End of Crawl)
[ ] AI tab: Prompt Request Status = Success for a sample of URLs

6. Review
[ ] Content tab > Semantically Similar / Low Relevance Content
[ ] Lower pane > Duplicate Details (all similar URLs per page)
[ ] Visualisations > Content Cluster Diagram
[ ] Bulk Export > Content > Semantically Similar

PROMPT FOR CLAUDE (after the crawl, via the SEO Spider MCP)
"Load the latest crawl of [domain]. Export Content > Semantically
Similar. Group the pairs by URL folder. For the top 20 pairs, show both
URLs, the score, each page's title and H1, and say whether they look
like the same search intent or different intents. Save the result to a
CSV in the allowed directory and tell me the file name. Do not
recommend merging anything. Flag anything you are unsure about."

How we use Screaming Frog + MCP + Claude

Screaming Frog runs on a Mac Mini in our office. Since version 24, the SEO Spider has an official MCP server, and we connect it to Claude. MCP (Model Context Protocol) is the plug that lets an assistant call a tool directly instead of us pasting CSV files into a chat. The server runs locally, needs a paid license and works in the default database storage mode.

Our last full crawl of stradiji.com ran on Oct. 14, 2026 and returned 9,468 URLs.

The weekly loop:

1. Claude starts or loads the crawl. The MCP exposes tools such as sf_crawl, sf_crawl_progress and sf_load_crawl. In STDIO mode Claude launches the Spider in the background; you have to ask it explicitly to load a saved crawl.
2. Claude pulls exports, not screenshots. sf_generate_bulk_export, sf_export_crawl and, for embeddings, sf_export_embeddings write files to an allowed base directory set under File > Settings > MCP Server.
3. We ask questions in plain English. Which redirect chains are new since last week? Which pages lost their H1? Which folders grew?
4. A human checks the numbers. Every count that goes into a report gets compared with the exported CSV.
5. The decision stays with us. Claude drafts the list. A person on our team decides what gets fixed.

Our rule: the model can misread a number in a table, so nothing goes to a client until someone has opened the export and matched the count. Screaming Frog itself writes that not every feature is supported in the MCP yet and that it is not a replacement for an experienced SEO.

Two more guardrails we keep:

• Script tools stay off unless needed. The Node.js tools (sf_run_node_js_script, sf_npm_install) are disabled by default. The official guide says to grant that permission only if you fully trust the model. We agree.
• One job per chat. Large exports blow through the context window. The guide recommends saving output to files and starting a new chat for each job. Sensible advice, and cheaper too.

This is the same pattern we use with Search Console. If you want the sister workflow, with prompts and the failure modes we hit, read AI agents for Search Console analysis. And if you are building your stack from scratch, our paid and free SEO tools list explains why Screaming Frog is in it.

How we would read an embeddings crawl on a bilingual site

A note on scope: the embeddings setup above describes what the tool does and how we configure it. Our weekly crawls on stradiji.com run without embeddings. On a site shaped like ours, these are the checks we would start with:

• Language pairs. stradiji.com publishes in Turkish and English, and the two versions are written separately for each reader. A multilingual embedding model could still place a Turkish post and its English sibling side by side. That is expected, not a problem, so we would use embedding rules to compare /tr/ pages with /tr/ pages and English with English.
• Glossary overlap. We have close neighbors such as vector search, semantic SEO and embeddings. Some overlap is natural. A 0.97 between two glossary entries would tell us one of them is not pulling its weight.
• Low relevance outliers. Fast news posts that no longer match the site’s focus.

Limits worth knowing before you spend API credits

• It costs money twice: the license and the API calls.
• Don’t mix models. Embeddings from different models or dimensions are not comparable. Pick one setup and keep it.
• Similarity is not intent. The score cannot tell a comparison page from a buying guide. You can.

Frequently asked questions

Does Screaming Frog semantic similarity work in the free version?

No. The embeddings features require a paid license plus an API key from OpenAI, Gemini or a local Ollama setup.

What is a good semantic similarity threshold?

Start with the default of 0.95. On a small site with few results, lower it gradually and review pairs by hand. There is no universal right number.

Does Screaming Frog have an official MCP server?

Yes. Version 24.0, released May 19, 2026, added the SEO Spider MCP server. It runs locally, works with Claude Desktop and LM Studio per the official guide, and requires a paid license.

Will merging similar pages improve my AI visibility?

It can make your site clearer to search engines and AI systems, but nobody can promise a citation lift. Treat it as SEO housekeeping that supports GEO, and measure before and after.

One thing to do this week

Open your last Screaming Frog crawl, go to Config > Content > Area and check whether your navigation and footer are excluded. It takes 15 minutes. If they are not, fix that before you spend a cent on embeddings, because every similarity score you get afterward depends on it.