Webmaster Hangout Provides Insight into Frequency
During a recent webmaster hangout, Google’s John Mueller provided answers about how often the search engine crawls URLs.
Answering a question about why a page set to noindex was still showing up in Google’s index, Mueller offered some insight into Google’s approach to crawling updated pages and what webmasters and SEOs could do on their end.
“[We] don’t crawl URLs with the same frequency all the time,” said Mueller. “So some URLs we will crawl daily. Some URLS maybe weekly. Other URLs every couple of months, maybe even once a half year or so.”
Mueller stressed that Google continually tries to find the right balance for this, “so that we don’t overload your server.”
What Triggers a Crawl?
Mueller offered some key insights into the crawl process and what webmasters can do to help trigger it. Without providing too much detail about Google’s processes, he did explain that big updates to a site were often a clear trigger for a crawl.
“If you made significant changes on your website across the board, then probably a lot of those changes are picked up fairly quickly,” he said, “but there will be some leftover ones.”
Circling back to the question that sparked the answer, he added, “If you do things like site queries then there’s a chance that you’ll see those URLs that get crawled like once every half year. They’ll still be here after a couple of months.”
He did suggest updating sitemaps and allowing Googlebot to find the last modified date could lead to a crawl of the pages in question.
Of course, there are other ways to get Google to crawl a website.
Using Google’s URL Inspection Tool, users can actually manually request indexing or reindexing. It’s a simple process that can be started from the Tool’s reports, but it will take up to two weeks for your request to make it through the queue and the indexing process.
This is handy if you just need to reindex a page or two.
Want to learn more about how Google handles re-crawl and reindexing requests? Check out the official process here: https://support.google.com/webmasters/answer/6065812
Watch the full Webmaster Hangout here: https://www.youtube.com/watch?v=gC7aVygbMIk&feature=