If you use
meta robots tags on your pages, you can give search engines instructions on how you’d like them to crawl or index parts of your website. This page lists an overview of all the different values you can have in the meta robots tag, what they do, and which search engines support each value.
The different robots meta tag values
The following values (‘parameters’) can be placed on their own, or together in the
content attribute of tag (separated by a comma), to control how search engines interact with your page.
Scroll down for an overview of which search engines support …read more