At Salience, we love the Screaming Frog SEO spider tool. That is why we’ve written an entire article on what the Screaming Frog SEO tool is, why we use it, its awesome features and how it can help you.
Why I use Screaming Frog
In my eight years of industry experience, the Screaming Frog SEO tool has always been my number one SEO crawler when I need to gather data on a client’s website.
The Screaming Frog crawler should be an indispensable tool in any SEO professional’s toolkit, and today I’m going to show you why.
Below I’ll give a brief intro on what the tool is, how it can be used for SEO and show you how to crawl a website with the Screaming Frog spider tool.
What is Screaming Frog SEO spider?
The Screaming Frog SEO Spider is a website crawler that collects linked URLs from a website along with corresponding data to help webmasters technically analyse and audit their website.
What does Screaming Frog do?
The Screaming Frog tool fetches SEO data for any website or list of URLs. This can include basic data such as title tags to more advanced information such as international hreflang targeting.
How to crawl a website with Screaming Frog
- Launch the Screaming Frog desktop application
- Enter your domain into the search bar
- Hit ‘Start’ to begin crawling
You can also crawl a list of individual URLs instead of a whole website. To do this:
- Go to the menu navigation
- Select ‘Mode’
- Change the setting from ‘Spider’ to ‘List’:
- Next, click the ‘Upload’ button
- Choose either ‘From a File’, ‘Enter Manually’, ‘Paste’ URL’s, ‘Download Sitemap’ or ‘Download Sitemap Index’:
- Choose your desired configuration
- Hit the ‘Start’ button to begin crawling.
This is only the basic guide to starting a crawl on the tool. Screaming Frog has several advanced crawling features and configurations. To see more on those, we recommend reading their super helpful user guide.
The Screaming Frog Tool for SEO
I use the Screaming Frog SEO tool daily at Salience. While it can be used for several advanced crawling activities, as mentioned in the Screaming Frog user guide, I’m going to show you 5 Screaming Frog features that can help you in your SEO efforts.
How do I find meta tags on a website?
The Screaming Frog SEO tool, that’s how! This crawler fetches data on title tags, header tags and meta description tags.
Title tags are one of the most important ranking factors – and a title tag on a search engine results page could be the first impression a potential customer gets of a business.
They appear in three key places; search engine results pages, browser title bars and on social sites when a page is shared:
Finding and exporting the title tags from every page on a website can be a laborious and time-consuming task if attempted manually.
Thankfully, Screaming Frog not only identifies your title tags via the crawl, but it also shows you several aspects of your title tags such as if they are too short, too long, duplicate, missing and more:
Having this data at your fingertips is great when you want to spot long title tags, keyword cannibalisation and more.
Meta descriptions are a less significant ranking signal than title tags but are important to encourage clicks from the search results page.
Meta descriptions appear underneath the title of a webpage on search engine results pages:
Screaming Frog SEO Tool allows you to fetch missing meta descriptions, duplicates, descriptions that are too long or too short and more:
Header tags, <h1> to <h6>, are HTML tags used to define headings on webpages. They are ordered by importance with the H1 tag the most important and the H6 tag the least important.
Screaming Frog fetches data on the H1 and H2 tags, delivering information on duplicates, multiples, long headers and more:
International SEO targeting
A multilingual website offers content in more than one language, whereas a multi-regional website explicitly targets different countries. Google distinguishes between the two, and there is HTML markup and different setups to reflect this.
A website can also be multilingual and multi-regional, e.g. a website with different versions for the US and Canada could have French and English versions of the Canadian content and Spanish and English versions of the US content.
It’s very important for international setups to be implemented correctly, as otherwise content can be considered as duplicated, and even rank for the wrong country.
Thankfully, the Screaming Frog SEO tool allows webmasters to identify several international targeting aspects of a website, such as missing confirmation links, inconsistent & incorrect languages codes, non-canonical hreflang and more:
Sitemaps – how to create an XML sitemap
Sitemaps are incredibly helpful, as they help search engines discover and index a website’s most important pages.
It’s important to keep sitemaps updated when old pages are removed, and new pages are added.
URLs in sitemaps should not redirect or return errors – every page should return a 200 OK status code. Search engines may ignore sitemaps completely if they contain too many errors.
XML sitemaps should be linked to in robots.txt file, and they should also be submitted to Google Search Console and Bing Webmaster Tools, as well as other region-appropriate search engines (such as Yandex in Russia).
The ‘XML Sitemap’ feature allows webmasters to create an XML Sitemap for their website with all HTML 200 response pages discovered in a Screaming Frog crawl.
You can also include more advanced configuration over URLs to include last modified, priority and change frequency:
However, John Mueller at Google doesn’t view priority or frequency as important:
You can read the Screaming Frog detailed tutorial here on how to use the SEO Spider as an XML Sitemap Generator.
Redirected links – how to find redirect URL’s
A 301 redirect specifies a page has moved permanently. A 302 redirect specifies a page has moved temporarily.
Where possible, redirects should be updated to the destination link. This helps reduce load on the server. This applies to 301 and 302 redirects if they have moved permanently.
The Screaming Frog SEO tool helps you find temporary and permanent redirects, identify redirect chains and loops or upload a list of URLs to audit in a site migration:
There you have it, five quick Screaming Frog features and crawling tips to help you fetch relevant SEO data for your digital marketing efforts.
If the Screaming Frog SEO tool still feels a little overwhelming and you’d prefer the Salience team of experts to take this and other website responsibilities off your hands, get in touch today!