Yoast SEO can help with optimizing website pages. Screaming Frog SEO Spider can crawl your website and report where you may have issues with your technical SEO – such as missing meta-descriptions or canonical tags. You can also use it to check if your analytics tags are present on all pages. Let’s get started!
Getting Started
To get started, you can download Screaming Frog SEO Spider from the website – https://www.screamingfrog.co.uk. There is a free version and a paid subscription, for £149.00 Per Year. The main difference is that with the paid version, you can integrate Google Analytics & Search Console as well as schedule & save crawls.
Once you have installed Screaming Frog SEO Spider, to crawl a site just enter the domain in the textbox at the top and click the Start button. The Start button will change to a Stop button (if you stop the crawl, you can resume it by clicking the button again, which has changed to a Resume button). You can also clear the crawl by clicking the Clear button. You can see how how far along the crawl is by looking at the bar in the top right corner. You can see how many pages have been crawled in the right bottom corner.
Panels
There’s a lot of information that is captured when you run a crawl. There are four main panels:
- Top Left – List of URLs with the content for each by whichever tab you are currently viewing.
- Top Right – Summary Information for each of the tabs in the Top Left panel.
- Bottom Left – Detailed information for any URL selected in the Top Left panel.
- Bottom Right – Graphs for any section selected in the Top Right.
After the Crawl
Once the crawl has completed, check out the Overview tab in the top right panel. That will give you the summary information for each of the tabs in the top left panel. For a quick check of SEO elements, scroll down to the Response Codes section. Clicking any where in the section will change the top left panel to the Response Codes tab and the bottom right panel will display a graph of the response codes from the crawl.
Take a look at the Client Error (4xx) and Server Error (5xx) – how many pages and what percentage for each is reported from the crawl. When you click on either of them, the pages will show up in the top left panel. There should not be any pages for either of these response codes. If there is, they should be addressed ASAP.
Next, scroll down to Page Titles. Page titles should be shorter than 65 characters for the simple reason that Google only displays the first 65 characters of the title of the page. Also, you want to use at least half of that (~30 characters) to make for a good page title and you want to make sure the page title is not duplicated across your site.
Similarly, when you scroll down to Meta Description, it should not be longer than 155 characters and you should use at least 70 characters to make it useful. Meta descriptions should also not be duplicated across pages.
And finally, scroll down to Canonicals. The vast majority of your pages should have canonical tags. These are tags that indicate which URL should be used by Google if the page has more than one URL. This can happen if the page is listed in two different sections of the website and that is reflected in the URL.
But Wait…There’s More
You can review all of the HTML, JavaScript, Image, CSS, PDF and other files that are both internal as well as external to your site. These are available in the first two tabs of the upper left panel and can be filtered by file type. You can check the status code, size, word count, inlinks & outlinks and a number of other page attributes.
One important thing to check is status codes of external pages. You want to make sure that, just like with your internal pages, the external pages are not returning a 4xx or 5xx status code. If they are, you will need to update the link or remove it.
There is a lot more to Screaming Frog SEO Spider and I would encourage you to explore it. I hope that this introduction helps to get you started.