Advanced Screaming Frog Crawling Techniques for E-Commerce Sites

Effective crawling techniques on e-commerce sites with Screaming Frog! URL exclusions, segments, and more.

Every SEO professional’s nightmare is moderating e-commerce sites with nearly endless pages. Fortunately, the Screaming Frog SEO Spider is invaluable for overcoming this challenge. But to fully utilize this powerful ally, you must explore advanced techniques beyond its essential functions.

There are excellent resources for detailed information, such as crawling large sites, memory, and database storage modes. Therefore, we will not cover these basics in this article. Instead, we’ll go over advanced crawling tips to help you master Screaming Frog and audit your e-commerce sites at scale.

This guide by Maria Camañes Forés shows you how to make the most of Screaming Frog on e-commerce sites.

Some methods that are important in terms of content and efficiency, especially on e-commerce sites, stand out:

1. URL Exclusion:

You can exclude unwanted sections so Screaming Frog can crawl the most critical content. For this, you can reduce the scan size by adding sections that should not be included in the site from the Configuration > Exclude section.

2. Creating Crawl Segments:

Screaming Frog’s “Segments” feature categorizes your site browsing. You can segment your site into different types, such as Product, Category, and Blog. You can start defining your segments from Configuration > Segments > Add. This way, you can see crawling issues specific to certain page templates.

3. Determining Out-of-Stock Products:

Screaming Frog’s “Special Sticker” feature lets you quickly identify out-of-stock products on your site. To add the URL of an out-of-stock product, go to Configure > Custom > Extraction > Add > Globe icon. Then, you can review the HTML of your “Out of Stock” message and create a custom sticker rule.

4. Scraping Category Page Content:

E-commerce sites often include content at the top or bottom of category pages. Screaming Frog can extract this text from each page using the “Custom Extraction” process. You can then transfer these texts to a chart and identify the sections that need to be optimized.

5. Finding Products That Don’t Receive Reviews:

Using unique stickers, you can find products that do not have reviews in scale. Using the web page extraction feature, you can find the star review section on product pages and extract the number of reviews. This way, you can quickly see products with few or no reviews.