Keys to crawling a website with Screaming Frog

There are a lot of SEO tools in the market that, to a greater or lesser extent, provide information on the state of your website, but without a doubt, none compare with the level of depth with which it analyzes Screaming Frog.

This tracking tool It has a series of features that will allow you to do a in-depth SEO audit and position it better and better. SEO Spider was created especially to perform technical SEO audits of websites and quickly find valuable data by tracking them.

The tracking process can be a bit overwhelming given the amount of information that is collected, here we mention some of the most important aspects offered by the paid version of Screaming Frog for analyzing and optimizing a website.

How does Screaming Frog work?

Before we start talking about what Screaming Frog offers, it's important to explain how the tool works so that you understand where and how it extracts all the information.

This software simulates the work that the Google bot does when it crawls a web page, allowing us to know exactly how search engines see it and what errors it presents in order to solve them.

The tool is so powerful that it crawls all the code on your page, as well as images, external and internal links, content, and actually the entire site! The best part is that you can get all this information in just a few minutes.

The best features of Screaming Frog SEO Spider

Personalized tracking

One of the very attractive features of the paid version of SEO Spider is the customizing the crawl. This is the first step you should do.

You can choose whether or not you want to include images, external links or hreflang tags in your tracking, for example, if you have a website in several languages.

Tracking settings in Screaming Frog

The more elements you leave selected and the larger the site to be crawled, the software will take longer to analyze and, therefore, the computer will need more resources.

After that, you can enter what you want to track, whether it's a domain or a specific URL. It doesn't need to be your property, the great thing is that you can crawl any website!

Screaming Frog crawl bar

Say goodbye to unidentified response code errors

If you have a large website, this functionality will serve you a lot and save you a lot of headaches.

With tracking, you'll be able to know exactly what response codes each URL on your site returns that could be affecting the user experience. Remember that, if a website has several broken links, its quality will be reduced since it hinders the work of the search engine bot and will affect positioning.

Screaming Frog Response Code Report

In this section you can see the percentage of URLs that have a 2xx, 3xx, 4xx and 5xx code, you will identify which one it is and where the internal link comes from (inlink).

View your metadata as in Google results

The SERP Snippet section shows how each URL appears in Google results. The ideal is to complement this functionality with that of metadata, so you will know which meta titles and meta descriptions can be optimized.

Important! In the Metadata section, you will be able to know, for example, what titles you could improve to use all the space provided by Google, so you can provide more relevant information for the user and include keywords that help you position yourself better.

Screaming Frog allows you to edit the metadata in the SERP Snippet section, in order to do the visualization exercise before you implement it on your page, which is very positive to ensure that it looks perfect.

Screaming Frog SERP Snippet functionality for viewing metadata

This software calculates the available space in the same way that Google does: using the width of the pixels, instead of an exact number of characters. This is because there are some letters or characters that use more space, such as capital letters.

How many clicks do your users make to find what they're looking for?

Yes, you can also know. For all this and much more, this tool is our favorite for SEO!

This section will allow you to have an overview of the depth of the website and how the pages are linked internally.

Screaming Frog website depth report

The list above are the top 20 URLs that have the most unique internal links per page (or unique inlinks).

As an example, this means that page “A” links 5 times to page “B”, so you will have 5 inlinks and 1 unique inlink.

With this in mind, the graphic that accompanies the table reflects the number of clicks that must be made from the home page to reach the content. Here it is grouped by the number of URLs that make, for example, 3 clicks from the home.

Identify and say goodbye to HTTP, and greet all URLs with HTTPS

The premium version of SEO Spider allows you to crawl practically an unlimited number of URLs, and we mean practically because many times such large websites require a large storage space.

With this in mind, if you manage to crawl all the URLs of a website, you will be able to know exactly which pages are complying with the HTTPS security protocol and which are showing up as unsafe (HTTP).

Screaming Frog security report graphic

This is essential, since currently Internet security is the basis of browsing and users are counting on the fact that their data is being protected. In addition, not having the HTTPS certificate directly affects the user experience because browsers display messages announcing that the web is not secure and that they recommend not entering.

Automatically generate your XML Sitemap

Taking advantage of the fact that you've already done all the domain crawling, SEO Spider offers you the possibility to create the Sitemap with all pages that have a response code of 200, and of course, they were identified in the tracing.

Important! So that you don't have errors in the list of your pages, you should pay attention to the tracking configuration and also the Sitemap generator.

Track what you want with the include/exclude tool

Let's imagine that you have a very deep website with a lot of URLs, but you only want to crawl a subfolder or search for, for example, all the pages that have a certain word in their path.

With the Include feature of SEO Spider, you can tell the software which path the SEO spider will crawl and it will only focus on matching URLs. For example, you can only crawl pages that contain the word “/blog/” in their path.

As for the Exclude functionality, the modality is the same, only that it is used to exclude from tracking a URL, specific syntax, folders, files, etc.

Configuring the include and exclude section of Screaming Frog

Important! Whenever you want to include or exclude a specific route, you must add a dot and asterisk. Example: http://www.example.com/brand .*.

Both features allow you to make a list of the elements you want to incorporate or not, in order to customize your tracking.

Analyze the quality of the content on your website

This is the newest functionality which has incorporated Screaming Frog SEO Spider into its tracking, and you can find it starting with version 13.0.

The quality of the content is increasingly important, especially since it was proven that Google's algorithm is able to evaluate the level of the text, influencing the level of positioning of the domain.

The SEO Spider Content feature performs an algorithmic analysis of the text and identifies grammatical and typographical errors. All this taking into account the recommendations of the RAE.

Website content analysis in Screaming Frog

Important! Depending on the language your page is in, this is how you should configure grammatical tracking. This way you will avoid errors in the analysis.

In addition, you will be able to know the percentage of similar content between pages, duplicate content, pages with a low content level. This last point is very important, because Google is not yet able to identify and “read” images, so if you have a website that mostly has photographs, you could qualify it as low content and it would affect your positioning.

This functionality allows you to identify exactly which URLs are considered to have a low level of content development.

Bonus track: list mode and SERP mode

The SEO Spider mode comes by default every time you open Screaming Frog, however, it can be very useful to know the other two modalities that the tool incorporates: list and SERP.

Screaming Frog List Mode and SERP

Screaming Frog SEO Spider list mode:

• If you have a specific list of URLs that you would like to review to optimize, this mode is perfect since it will only focus on analyzing what you indicate, without having to waste resources or time crawling the entire website.
• It doesn't necessarily have to be an HTML file, it can be .txt, .xls, .xlsx, .csv, .xml, and many others. Just copy and paste the URL, and you're done!
• Important! To be able to trace, all files must begin with http://or https://.

Screaming Frog SEO Spider SERP mode:

• If you want to see what the metadata would look like and calculate the space you're using, in SERP mode you can upload an Excel file that contains three headers (with these names): “URL”, “Title” and “Description”.
• You can enter all the URLs you want to analyze in SERP mode and thus see what they would look like in the Google search engine.
• Important! In this mode, only the file you upload will be analyzed and it doesn't update anything on your website, it's more like a space to preview your work.

Both the List mode such as the SERP, they allow you to export all the analyzed data including the details you see in the software.

Things to consider with SEO Spider:

• Both to use SEO Spider and its other Screaming Frog Log File Analyzer tool, you need to go to their website and download the program, since it's not available in the cloud.
• It is important that you look at the system and memory requirements that the software needs to work without problems, since as we mentioned before, if your computer does not have enough capacity, the program may have problems.
• You should keep in mind that Screaming Frog, and in particular SEO Spider, has a lot of difference between its free and paid versions, for example, with the paid version you have no limit on the number of pages to analyze, compared to the 500 that the free version has at most.
• Screaming Frog can be integrated with the platforms of Google Search Console, Google Analytics, PageSpeed Insights, Ahrefs, among others, with its API integration.
• The paid version allows you to save the traces you have made and reopen them whenever you want, so you have a backup of what you saw at that time and you can resume the analysis whenever you want, or rather compare the old traces with the current situation.

Now that you know what the main features of Screaming Frog SEO Spider we invite you to venture out with it and start using it, it will be quite an experience for you!

If you need more information about this tool, our SEO Manager will be happy to support you in whatever you need.

How can we help you?

If you need more information, do not hesitate to contact us.