Crawler

A crawler (or web crawler) is a program that browses the web automatically, hopping from link to link to discover pages and read their content. It is the first stage in how a search engine works: before Google can show a page in results, it has to find and crawl it with its crawler, called Googlebot. What the crawler collects is then passed to the indexing ✕ Indexing is the process by which a search engine stores a page in its index after crawling it, so that the page can appear in search results. More in the glossary → process. You can guide its behaviour with the robots.txt file, which says which areas it may or may not visit, and with internal links, which help it reach all your pages. Because every site has a limited crawl budget, it pays to make its job easy with a clear structure, a sitemap ✕ A sitemap is a file that lists the important pages of your site to help search engines discover and crawl them more easily. More in the glossary → and by avoiding duplicate or low-value pages.