You probably want Google and other search engines to show most of your content in the search results. But most websites also have a few pages or posts that don’t need to be featured in those search results. So how do you tell Google what pages to show and which ones not to? This is where setting pages to noindex comes into play. In this post, we’ll explain more about indexing and how Yoast SEO can help you be in control of what pages will be shown in the search results.
Let’s start with a short explanation of indexing. Before a search engine like Google can rank a page or a post and show it in the search results, it needs to index it. Even before that, a crawler has to discover a piece of content before it can evaluate whether it’s a valuable addition to its index. One of the ways crawlers discover pages, is by crawling the XML sitemap of a website. After a page has been indexed, a search engine will understand its content and be able to rank that piece of content if it fits the user’s search query best.
Which pages to show in the search results?
Determining what has to be indexed by crawlers and what not tends to be hard to understand and it’s easy to make a mistake. You wouldn’t be the first to unknowingly set a whole post type to noindex, making it unavailable to search engines. We’ve thought long and hard about this and drastically simplified this process for you in Yoast SEO. Now it all boils down to asking you a straightforward question: Do you want page x to appear in search engines?
You can find the individual settings for making your content available for indexing in the corresponding parts of Yoast SEO. In WordPress, you’ll find the settings for posts and pages in the Content Types part of the Search Appearance tab. Taxonomies like your category and tag pages can be found in the Taxonomies tab of Yoast SEO. On Shopify, you can find these settings per page type in the Settings tab of Yoast SEO for Shopify.
By saying Yes to the ‘Show Posts in search results’ question in the post settings, you make sure that your posts will appear in the XML sitemap and, therefore, in the search results. If you want to exclude something, let’s say you want to exclude your category pages, you can switch the toggle to No. That way the taxonomy or post type will not appear in the XML sitemap. Because of that, it will not appear in the search results. Whenever you set something here to not appear in search engines, it will be noindexed and kept from the XML sitemap.
Read more: Why doesn’t Google index my content properly? »
Exclude individual posts
If you really don’t want a specific URL to show up in the search results you’ll need to add a noindex, follow tag. Leaving it out of your XML sitemap doesn’t automatically mean Google won’t index the URL. If Google can find it by following links, Google can index the URL.
You can also exclude individual posts from the XML sitemaps with the Yoast SEO sidebar in the post editor. Or if you’re on Shopify, you can exclude individual products with the Yoast SEO sidebar. Click on the Advanced tab and select No to the ‘Allow search engines to show this Post in search results?’ question.
View your XML sitemap
When you use Yoast SEO on WordPress, our plugin automatically generates a thorough XML sitemap for your website. Making it easy for search engines to discover all the pages you want them to discover. You can check your sitemap to see if the content you want to include appears in the XML sitemap. While you’re there, you should also check if the content you want to exclude from the sitemap doesn’t appear in it.
You can find your XML sitemap in Yoast SEO by going to General > Features > XML Sitemaps > ? (click on the question mark). Here you can click on ‘See the XML sitemap’ and this will take you to your website’s XML sitemap.
We’ve taken away a lot of the confusion around indexing content and XML sitemaps by simplifying things in Yoast SEO. But, most importantly, it is now so much easier to tell search engines what you would and would not like them to show in the search results.
More on XML sitemaps
XML sitemaps are a kind of treasure map for search engine robots. They crawl them to discover new or updated content on your site. Every site benefits from a sitemap, that’s why Yoast SEO automatically generates a robust XML sitemap for you. Your rankings won’t soar if you add one, but it does help the crawlers to discover your content that much easier. Need more information about the use of XML sitemaps on your site? Then we have some further reading for you:
Edwin is a strategic content specialist. Before joining Yoast, he spent years honing his skill at The Netherlands’ leading web design magazine.
