Ranking in search engines requires a website with flawless technical SEO and great, relevant content. Luckily, the Yoast SEO plugin takes care of (almost) everything on your WordPress site. Still, if you really want to get the most out of your website and keep outranking the competition, some basic knowledge of technical SEO is a must. In this post, we’ll explain one of the most important concepts of technical SEO: crawlability.
What is the crawler again?
A search engine like Google consists of a crawler, an index, and an algorithm. The crawler follows the links. When Google’s crawler — also known as Googlebot — finds your website, it’ll render it, read it and save the content in the index.
A crawler follows the links on the web. A crawler is also called a robot, a bot, or a spider. It goes around the internet 24/7. Once it comes to a website, it saves the HTML version in a gigantic database called the index. This index is updated every time the crawler comes around your website and finds a new or revised version of it. Depending on how important Google deems your site and the number of changes you make on your website, the crawler comes around more or less often.
Read more: SEO basics: what does Google do »
And what is crawlability?
Crawlability has to do with the possibilities Google has to crawl your website. You can block crawlers on your site. There are a few ways to block a crawler from your website. If your website or a page on your website is blocked, you’re saying to Google’s crawler: “do not come here.” Your site or the respective page won’t turn up in the search results in most of these cases.
There are a few things that could prevent Google from crawling (or indexing) your website:
- If your robots.txt file blocks the crawler, Google will not come to your website or specific web page.
- Before crawling your website, the crawler will take a look at the HTTP header of your page. This HTTP header contains a status code. If this status code says that a page doesn’t exist, Google won’t crawl your website. In the module about HTTP headers of our Technical SEO training we’ll tell you all about that.
- If the robots meta tag on a specific page blocks the search engine from indexing that page, Google will crawl that page, but won’t add it to its index.
This flow chart might help you understand the process bots follow when attempting to index a page:
Improve your site’s crawlability with Yoast SEO Premium
We’ve said it many times, crawlability is an important aspect of SEO. And so optimizing the crawlability of your site is something you should do. Using tools such as Yoast SEO Premium can help make this easier for you.
Crawlers may come to your site multiple times a day. Sometimes they visit your site to discover new content, sometimes they come by to check if there are any new updates. But every time they arrive on your site, they will surely crawl everything that looks like a URL to them. Because of this, it’s very likely for a URL to be crawled multiple times per day.
This is not the way things should be. For the most part, you’re unlikely to make multiple changes on a URL on a given day. Not to mention, almost every CMS output URLs that don’t make sense that crawlers can safely skip. But instead of skipping these URLs, crawlers will crawl them, again and again, every time they come across one. All this unnecessary crawling takes up energy resources which makes this a harmful practice for our planet.
That’s why in Yoast SEO Premium 18.6, we introduced the first addition to our crawl settings, allowing you to better manage what search engines can skip on your site. Within this section, which is still in beta, you can find many toggles that let you turn off various things that WordPress automatically adds to your site that most sites won’t miss. In Yoast SEO Premium 18.7, we’ve expanded on the list of settings you can toggle and we will continue to do so in future releases of Yoast SEO Premium.
Want to learn all about crawlability?
Although crawlability is just the basics of technical SEO (it has to do with all the things that enable Google to index your site), it’s already pretty advanced stuff for most people. Nevertheless, if you’re blocking – perhaps even without knowing! – crawlers from your site, you’ll never rank high in Google. So, if you’re serious about SEO, this should matter to you.
If you really want to understand all the technical aspects of crawlability, you should check out our Technical SEO training. This SEO course will teach you how to detect technical SEO issues and solve them (with our Yoast SEO plugin). Yoast SEO Academy is included at no extra cost in your Premium subscription.
Keep reading: What is technical SEO: 8 aspects everyone should know »
Marieke is the head of strategy at Yoast and founder of Yoast SEO academy. She loves coming up with new ideas and products to make SEO attainable for everyone, and ensure a healthy growth for Yoast!
