Crawl Budget Optimization: How To Speed Up Indexing, Crawlability and Avoid Keyword Cannibalization.

Transcript

Patrick Babakhanian here from serppro.io. I would like to wish you a happy new year. 2020 is going to be a very exciting year for you and us because we're going to drop a lot of new content, introduce a lot of features to SERP Pro.

So in this video, I want to start with crawl budget optimization. I want to teach you how to optimize your crawl budget, so you have better indexing, better crawlability, and less keyword cannibalization, which is two pages competing for the same keyword.

Now what we want to do is we want to make sure that all the pages that we have on our site are checked. So every page, does this page provide value to my user, yes or no?

If it's a yes, you leave them to index, you leave Google to crawl them. But if the page doesn't provide any value to the user, then you have to noindex them.

So they do not get crawled by the Google spider, and they don't spend a lot of time on those pages because you want the Google spider to crawl the pages that you want to rank.

So that's the crawl optimization budget. Every website has its budget. They don't have an infinite amount of resources where Google can crawl your whole site every single day. So you want to make sure to noindex non-valuable pages.

So what are non-valuable pages? Think categories, archives, tags. You have squeeze pages; you have doorway pages that do not provide any value. You want to consider noindexing them.

Now before you could noindex pages like this through the robots.txt but Google stopped supporting that feature.

So there's a whole nother way to do it now, which is via your site. Now we're going to use an SEO plugin called Project Supremacy, but this works on any SEO plugin, Rank Math, All In One, Platinum, Yoast, anything. They all support the SEO settings feature.

So I'm going to show you real quick example of that. So you want to go to the SEO section, and you can see that you're able to noindex categories, tags, navigation, menus, link categories, type, formats.

This plugin does support a lot more features, but you want to noindex Most of these. You want to noindex the categories, the archives, the tag archives, the date archives, the squeeze pages, any page that does not provide any value to your users.

So as you can see here, I have a lot of things, noindexed because they do not provide any value.

However, say you're running an eCommerce store, and you have a ton of pages, then product categories are going to be very important to you because you do want to rank them, you want to put content on them. So make sure to leave this indexed for eCommerce stores.

But anything else that doesn't provide any value to your site, you have to noindex. So most of it you can do through the plugin.

However, a lot of websites have pages in Google that are not relevant anymore, and you want to also noindex stats. So what do you do?

I'm going to use SERPchampion as an example. What you want to do is type in the site and then serpchampion.com, or your website obviously, and then you're going to see how many results are in here.

So what you need to do is you've got to check out all of these pages and see if they provide any value to your users. So most of our pages are blog articles, so if you need more training, go check them out.

But we have a lot of pages. I've seen a few pages that I do want to noindex. For example, this homepage, which is not a homepage, we want to noindex and no crawl, right?

So what do we do? What we're going to do is we're going to go to that specific page first, right? So I'm going to show this page real quick. It was the homepage. Let's see, take my face away here.

Okay, so basically, this is the homepage. If you, the URL is in there. So what we're going to do is we're going to click on that and then what you need to do is if you're plugin, your SEO plugin, supports that you got to go to the script section or the HTML section and you need to put in a type of code. This SEO plugin supports this feature already, but I will show you the manual way.

So what I'm going to do is I'm going to turn this on and make sure Google does not index or follow this page. I need it removed from Google. I don't want it to be crawled because obviously, I want spiders to crawl the pages that do matter. So I'm going to click on the update.

So what you're going to do, if you want to add it manually, is you need to install a plugin called Header Footer Code Manager. So it's a free plugin, just install it.

And what you have to do is you have to go into settings, add new snippet, we'll type in "noindex, no follow" here, we'll go site display specific pages and then you can choose the pages that you want to noindex.

So now, with one setting, you can noindex all of the pages that you don't need. So for example, the home and yeah, this one as well. So, for example, we have this right? Location header, that's good.

And then what we're going to do is we're going to copy-paste this code. So this code will make sure that they will noindex and no follow.

I'm going to include this code underneath this video so you know how to do it, and you can copy-paste it and then all you do is save it. So now these pages, so if we go to no home page, obviously I'm going to delete this too, but page source. Let's see.

So as you can see, this page has the noindex and no follow. And let's check this real quick to make sure it doesn't appear on any other page. So as you can see here, this one is indexed and follow, which is Project Supremacy.

But there are no noindexations, so that's good. So that's how you include the noindex and no follow per page with one plugin called the Header Code Manager. So definitely use that and now you can make sure that your site map is much cleaner.

You can make sure that Google crawls the pages that matter. So do this and just get better indexing, better crawlability, less keyword cannibalization. And all in all, if you set up all these things right, your rankings will go up higher, and backlinks will have a faster impact.