Advanced Screaming Frog Crawling Techniques for E-Commerce Sites

Effective crawling techniques on e-commerce sites with Screaming Frog! URL exclusions, segments, and more.

Effective crawling techniques on e-commerce sites with Screaming Frog! URL exclusions, segments, and more.

E-commerce SEO can be a minefield. But with the right tools, it can be a fun journey. One of the most essential tools for e-commerce professionals is Screaming Frog. It’s a crawling beast that can find issues on your site that you didn’t even know existed. This guide will explore advanced Screaming Frog techniques for e-commerce sites. From finding and fixing crawl errors to site architecture and page speed optimisation, these techniques will help you get more visibility and functionality on your site and more traffic and sales.

Imagine discovering that search engines don’t index a big chunk of your product pages because of tiny, fixable problems. Or, consider the conversion rate increase if your site is user-friendly with no broken links or duplicate content to annoy your customers. You can make these kinds of improvements with Screaming Frog’s advanced features. Whether you’re an SEO pro or a newbie looking to level up, this guide will give you the tips and techniques to take your e-commerce site to the next level.

Some methods that are important in terms of content and efficiency, especially on e-commerce sites, stand out:

1. URL Exclusion:

You can exclude unwanted sections so Screaming Frog can crawl the most critical content. For this, you can reduce the scan size by adding sections that should not be included in the site from the Configuration > Exclude section.

2. Creating Crawl Segments:

Screaming Frog’s “Segments” feature categorizes your site browsing. You can segment your site into different types, such as Product, Category, and Blog. You can start defining your segments from Configuration > Segments > Add. This way, you can see crawling issues specific to certain page templates.

3. Determining Out-of-Stock Products:

Screaming Frog’s “Special Sticker” feature lets you quickly identify out-of-stock products on your site. To add the URL of an out-of-stock product, go to Configure > Custom > Extraction > Add > Globe icon. Then, you can review the HTML of your “Out of Stock” message and create a custom sticker rule.

4. Scraping Category Page Content:

E-commerce sites often include content at the top or bottom of category pages. Screaming Frog can extract this text from each page using the “Custom Extraction” process. You can then transfer these texts to a chart and identify the sections that need to be optimized.

5. Finding Products That Don’t Receive Reviews:

Using unique stickers, you can find products that do not have reviews in scale. Using the web page extraction feature, you can find the star review section on product pages and extract the number of reviews. This way, you can quickly see products with few or no reviews.

Introduction to Custom Extraction

Custom extraction is one of the most powerful features in Screaming Frog SEO Spider, enabling SEO professionals to extract specific data from web pages with precision. Whether you need to analyze product descriptions, meta tags, or other HTML elements, custom extraction lets you scrape data directly from your website’s code. This is especially valuable for e-commerce sites, where understanding the structure and content of product pages can reveal opportunities for SEO optimization. With custom extraction, you can pull data from HTML elements, meta tags, and even inline JavaScript, giving you a comprehensive view of your site’s SEO performance. By leveraging this feature, you can ensure your website’s data is structured for maximum visibility in search engines and make informed decisions to enhance your SEO strategy.

Extracting Data from HTML Elements

Extracting data from HTML elements is at the heart of custom extraction in Screaming Frog. By targeting specific elements—such as headings, paragraphs, or links—you can gather detailed insights about your web pages. Using XPath or CSS Path, you can pinpoint and extract data from any HTML element. For example, to extract all H1 headings on a page, you can use the XPath expression “` //h1

, or for H2 headings, ```
//h2

. If you want to extract all links, simply use “` //a

. This flexibility allows you to analyze the structure of your pages, check for missing or duplicate elements, and ensure your content is optimized for both users and search engines. Custom extraction makes it easy to collect and review this data at scale, helping you identify areas for improvement across your site.

Using CSS Path for Extraction

CSS Path offers an intuitive way to extract data from web pages using familiar CSS selectors. With custom extraction, you can use CSS Path to target HTML elements based on their classes, IDs, or other attributes. For instance, if you want to extract all product descriptions that use the class “` product-description

, you can use the CSS Path ```
.product-description

. This method is particularly user-friendly for those who are comfortable with CSS, and it allows for precise targeting of elements within complex page layouts. Whether you’re extracting product prices, reviews, or other data, CSS Path in Screaming Frog’s custom extraction feature streamlines the process, making it easier to gather the information you need for SEO analysis and optimization.

Extracting Meta Descriptions and JSON LD Format

Meta descriptions and structured data in JSON LD format are critical for SEO, as they help search engines understand your web pages and display rich results. With custom extraction, you can easily extract meta descriptions using the XPath expression “` //meta[@name=’description’]/@content

. This allows you to audit your meta tags across your site and ensure they are unique, relevant, and optimized. Additionally, extracting JSON LD format is straightforward with the XPath ```
//script[@type='application/ld+json']

, enabling you to review and validate your structured data implementation. By regularly extracting and analyzing these elements, you can enhance your site’s SEO performance and ensure your pages are properly represented in search engine results.

Integrating with Google Analytics and Google Sheet

To maximize the value of your extracted data, Screaming Frog allows you to export results to a CSV file, which can then be imported into Google Sheet for further analysis and visualization. This integration makes it easy to organize, filter, and share your findings with your team. Additionally, by combining custom extraction data with insights from Google Analytics, you can correlate on-page SEO factors with website traffic and user behavior. This holistic approach enables you to make data-driven decisions that improve your site’s SEO performance. Whether you’re tracking changes in product descriptions, monitoring meta tags, or analyzing structured data, integrating custom extraction with Google Analytics and Google Sheet provides a powerful workflow for ongoing SEO optimization.

Common Challenges and Troubleshooting

While custom extraction is a robust feature, it can present challenges—especially for those new to XPath or CSS Path. Extracting data from complex web pages, dealing with broken links, or handling dynamic content can sometimes lead to incomplete or inaccurate results. To overcome these issues, it’s important to familiarize yourself with the Screaming Frog SEO Spider documentation and experiment with different extraction rules. The Screaming Frog community forum is also a valuable resource for troubleshooting and sharing solutions with other users. By using the right tools and refining your extraction techniques, you can resolve common issues and ensure you’re capturing the data you need to optimize your e-commerce site’s SEO.

Screaming Frog Custom Extraction Summary

Using Screaming Frog for your e-commerce site is a game changer in online retail. Following the advanced crawling techniques in this guide will keep your site robust, user-friendly, and visible to search engines. Do site audits regularly, and you’ll find and fix issues and prevent problems that could impact your SEO and user experience.

Think of Screaming Frog as your digital dog. It’s always watching your site to keep it healthy and performing. The effort you put into these advanced techniques will pay off in search rankings, organic traffic and sales. As the e-commerce landscape changes, being ahead of the curve with tools like Screaming Frog will keep you competitive. Get into these techniques and watch your e-commerce site thrive in the digital jungle.