One of the most important aspects of SEO is optimizing the crawlability of your site. Search engines have near-endless resources, so they have the power to crawl everything they find — and they will. But, that is not the way it should be. Almost every CMS outputs URLs that don’t make sense and that crawlers could safely skip. With Yoast SEO Premium 18.6, we’re starting a series of additions to clean up those unnecessary URLs, feeds, and assets so that the more critical stuff stands a better chance of being crawled.
Making your site easier to crawl
Google and other search engines crawl almost everything they can find — as Yoast founder Joost de Valk proves in a post on his site. But it can be hard to get them to crawl what you want them to crawl. Moreover, crawlers can come by many times each day and still not pick up the important stuff. There’s a lot to gain for every party involved — from the crawlers, site owners, and environment — to make this process more sensible. Yoast SEO Premium will help search engines crawl your site more efficiently.
In Yoast SEO Premium 18.6, we’re introducing the first addition to our crawl settings, allowing you to manage better what search engines can skip on your site. In this release, we’re starting with those RSS feeds of post comments in WordPress, but we have a long list of stuff that we want to help you manage.
Head over to our new Crawl settings section in the General settings of Yoast SEO Premium and activate the first addition to preventing search engines from crawling the post comment feeds.
This feature is available to all Yoast SEO Premium subscribers in beta form, and we’ve selected not to activate this for every site. In some cases, there still might be sites that use this in a way we can’t anticipate. We’re rolling out more crawling options — big and small — in the coming releases.
Let’s all start cleaning up the crawling on our sites — it’s better for you, your visitors, search engines, and the environment. All with a little help from Yoast SEO Premium. Let’s go!
Go Premium and get access to all our features!
Premium comes with lots of features and free access to our SEO courses!
Only €99 EUR / per year (ex VAT) for 1 site
Keeping Bing updated on your site
Yoast SEO 19.0 and Premium 18.6 also help Bing find your XML sitemaps. Last week, Bing changed the way they previously handled XML sitemaps. Before, we could submit sitemaps URLs anonymously using an HTTP request, but Bing found that spammers were misusing it thanks to this anonymity. You have two options to submit your sitemaps to Bing: a link in the robots.txt file or Bing Webmaster Tools.
To make your sitemaps available to Bing, we’ve updated Yoast SEO to add a link to your XML sitemap to your robots.txt file — if you want. This ensures that Bing can easily find your sitemap and keep updated on whatever you publish or change on your site. If you haven’t made a robots.txt file yourself, we’ll now add one with a link to your sitemap.xml file. You can add the link yourself via the file editor in Yoast SEO if you already have one.
We’ve also updated the robots.txt itself. We have changed the Disallow: /wp-admin/ line to Disallow:. More on that in our updated best-practice guide for robots.txt on WordPress sites.
Also, this might be an excellent opportunity to check out Bing Webmaster Tools — there are some great insights to be gained into your site’s performance on Bing.
Other enhancements and fixes
Of course, we did another round of bug fixes and enhancements. There are two that we’d like to highlight here. We’ve enhanced the compatibility with Elementor, ensuring that our SEO analysis functions appropriately.
In addition, we enhanced our consecutive sentence assessment in the readability analysis. This threw warnings when you had multiple sentences starting with the same word in a list. We handle content in lists differently now, and having various instances with the same word should not throw a warning anymore.
Update now to Yoast SEO 19.0 & Premium 18.6
In this release, we’re introducing more ways to control crawling on your site. For Yoast SEO Premium, we’re starting with a small addition to manage post comment feeds, but we’re expanding that in the coming releases. The feature is in beta, so we welcome your feedback!
In addition, we’ve also made sure that Bing can still find your XML sitemap, and we’ve fixed a couple of bugs with Elementor and our readability analyses.
Edwin is a strategic content specialist. Before joining Yoast, he spent years honing his skill at The Netherlands’ leading web design magazine.
