Over the last couple of months, there’s been a lot of talk about a new protocol called IndexNow. This new protocol automatically notifies search engines that support it about all the updates on your site. While initially skeptical about this development, we’ve decided to add an IndexNow integration to Yoast SEO Premium 18.8.
What is IndexNow?
The IndexNow protocol was launched in October 2021 by Microsoft Bing, Seznam, and Yandex. The goal of the protocol is to make it easier for site owners to ping search engines when changes in content happen on their sites. In theory, this should make the search engines less dependent on their crawling processes.
By offering a single point of access for the pings, many technical processes of crawling the web might become less important. Crawling has a tremendous impact on every website and the search engines themselves. Every technology that can make them rely less on crawling is good for the environment, site owners, and search engines.
IndexNow is a good idea and has seen much support from search engines. Unfortunately, for now, Google is holding out, but hopefully, that will change when more search engines start using the protocol. Of course, if millions of sites start using the IndexNow protocol, that chance may increase.
First no, now yes — what happened?
Our initial thought was this, while it is a good idea in theory, in practice, the issue of discovering content in a timely fashion was solved years ago with the introduction of XML sitemaps. With sitemaps, you also ping the search engines and notify them of whatever you are doing on your site. With IndexNow, it would depend on the efficiency of the process in practice. Plus, with Google not supporting IndexNow, we’re not reaching the majority of the market.
But over the last couple of months, a few things influenced our thinking. The first and most important thing is that we’ve been deep-diving into the mess called crawling. Our founder, Joost de Valk, again showed that there are a ton of robots that crawl the web at an incredible rate. Small sites with static content get crawled all the time by tens of different crawlers.
The crawl issue became a focus of the focus for Yoast SEO. With the release of the crawl settings in Yoast SEO Premium, we’ve been trying to eliminate overhead in WordPress. We’re trying to clean up sites to make it easier for crawlers to do their work by focusing on what’s important.
In this regard, IndexNow offers a way to focus less on crawling everything all the time and more on doing that in a managed fashion.
The latest change to the protocol is the second significant development that made us change our minds about IndexNow. The IndexNow team changed the endpoint of the API from pinging each individual-supported search engine to a single API managed by the protocol. This means that sites only have to ping a single site, making the process much more efficient.
All of this fits into our view of a sustainable crawling and indexing solution, so now is the time to implement IndexNow with a new integration in Yoast SEO.
What does IndexNow offer you?
The IndexNow integration has only upsides: turn it on, and you are good to go. We’ll ping IndexNow whenever you publish or update your content. You don’t have to do anything — it all works silently in the background. Using it makes you feel safe about getting your content noticed by search engines.
Supporting IndexNow also means helping everyone move to a more efficient and environmental-friendly crawling process. Good for the planet!
Here’s hoping Google will support IndexNow soon
IndexNow is off to a great start, but there is much to win. With our integration, we’re adding millions of sites to the list of IndexNow users. Hopefully, Google will start adopting it too. If nothing else, we’re doing our part to improve discoverability for websites — and reduce our carbon footprint while we’re at it.
Edwin is a strategic content specialist. Before joining Yoast, he spent years honing his skill at The Netherlands’ leading web design magazine.
