More crawl settings • Yoast

As we’ve said many times, crawlability is an important aspect of SEO. Your site has to do away with technical hurdles that can hinder crawling while keeping the number of crawlable URLs in check. Crawlers are eager to crawl everything they can find on your site — even if you have a small site with content that hardly changes. The overhead is enormous because there are loads of URLs that make no sense for them to crawl. We’re doing something about that. In Yoast SEO Premium 18.7, we’re expanding our brand-new crawl settings with a long list of additional options.

More crawl settings in Yoast SEO Premium

In Yoast SEO Premium 18.6, we introduced the first addition to a new tab called Crawl settings. Within this section, you can find many toggles that let you turn off various things that WordPress automatically adds to your site and that most sites won’t miss. These WordPress additions generate URLs that search engines can and will crawl. At Yoast, we’re on a mission to drastically reduce the number of URLs that any given site outputs, as crawling all of these are a tremendous waste.

Only available in Yoast SEO Premium, these crawling features give you more control over what you want Google to crawl. The crawl settings are in beta, and we welcome your feedback. Together, we can help take your site to another level!

What can you expect from the crawl settings?

Here’s the basic crawling list with stuff added by WordPress that you can turn off to prevent search engines from crawling it. In our help documentation on the crawl settings, you’ll find all these options explained.

• Short links
• REST API links
• RSD / WLW links
• oEmbed links
• Generator tags
• Emoji scripts
• Pingback HTTP header
• Powered by HTTP header

You can also disable a huge amount of feeds generated by WordPress:

• Global feed
• Global comment feed
• Post comment feed
• Post authors feed
• Post type feed
• Category feeds
• Tag feeds
• Custom taxonomy feeds
• Search results feeds
• Atom/RDF feeds

These settings give you much more flexibility in deciding what you want Google to crawl. We don’t activate these settings for you automatically because it might be that your site benefits from or uses one of these things. We don’t want to break stuff!

Yoast SEO 19.1 enhancements

Yoast SEO 19.1 comes with several fixes and enhancements. For example, our content analyses improved, with support for em dashes as punctuation marks in the focus keyphrase field and correctly detecting caps in focus keyphrases with a period.

Among many other things, Yoast SEO recognizes keyphrases with hyphens in the slug. Our word forms feature and the filtering out of function words now works for hyphenated keyphrases. For example, the keyphrase ex-husband will be recognized in the slug ex-husband, but also the keyphrase ex-husbands.

With the word forms feature, we recognize different grammatical forms of your focus keyphrase in various places in your content. You don’t have to write the same form of your keyphrase each time but can diversify your language by changing forms. This makes you write much more naturally!

WooCommerce SEO 14.9: Global identifiers for product variations + Schema

Working with product variations has always been a hassle. WooCommerce, for instance, doesn’t allow you to add global identifiers for product variations — making it hard to add valuable data for Google to discover. Moreover, there’s no simple way to add structured data to this correctly! In WooCommerce SEO 14.9, we’re fixing this.

WooCommerce SEO now outputs the necessary Schema markup for product variations on individual products. Of course, the plugin already outputs all the product variations Schema markup through the offers property. Now, it will also output the global identifiers for product variations under the offers property. In addition, we’ve also added the variations’ SKU and URL to a product’s Schema.

Open the product variations tab in WooCommerce and fill in the global identifier details for each variation in the new Yoast SEO section. The details will be automatically added to the product structured data generated by WooCommerce SEO.

Learn how to add global identifiers on variable products with WooCommerce SEO.

Removing dates from product Schema

Google sometimes shows dates in your search results, which can help users find the most relevant content. But on your WooCommerce product pages, this doesn’t make much sense — so we remove some metadata from your page to discourage them from showing dates here. So, we now remove the datePublished and dateModified attributes on the ItemPage Schema markup for a product.

Update now!

Yoast SEO 19.1, Yoast SEO Premium 18.7, and WooCommerce SEO 14.9 are out today. In Premium 18.7, we’ve vastly expanded the possibilities of the new crawl settings we introduced in 18.6. You now have many options to manage crawling on your WordPress site.