1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5 out of 5)
Loading ... Loading ...

If you're new here, you may want to subscribe to my RSS feed or follow me on Twitter. Thanks for visiting!

As you know Google and other search engines try to index everything they manage to find through spidering the web. But sometimes you may not want to show some part of your website to others and to search engines. There are some techniques that let you hide what you want from Google. Here they are:

1. Block entire website from Google using Robots.txt file.

  • To prevent only Google from indexing your entire website use:

User-agent: Googlebot
Disallow: /

  • To prevent all search engines from crawling your website use:

User-agent: *
Disallow: /

  • If you have http and https versions of your website and would like Google to stop indexing https version and still index http version you need this in your Robots.txt file:

For http version (http://www.domainname.com/robots.txt):
User-agent: *
Allow: /

For https version (https://www.domainname.com/robots.txt):
User-agent: *
Disallow: /

  • If you want to exclude a specific folder from Google place this code into Robots.txt file:

User-agent: Googlebot
Disallow: /foldername/

  • If you want to exclude certain file types (pdf for example) from being crawled by Google:

User-agent: Googlebot
Disallow: /*.pdf$

  • Remove an image from Google by adding:

User-agent: Googlebot-Image
Disallow: /image.jpg

  • Disallow Google from indexing all pictures on your site:

User-agent: Googlebot-Image
Disallow: /

2. Remove or block pages from being indexed by Google using meta tags.

  • To prevent Google from indexing a site, place this meta tag in your <HEAD> section of that page:

<meta name="robots" content="noindex, nofollow">

  • To exclude all images on a page from being indexed, but allow the site to be indexed use:

<meta name="robots" content="noimageindex">

  • Remove cached version of your website from Google:

<meta name="Googlebot" content="noarchive">

NOTICE: This will only prevent from indexing a cache version of the site, Google will continue to index the website and show it in result pages.

  • Prevent or remove snippet:

<meta name=”googlebot” content=”nosnippet”>

NOTICE: Removing snippets will also remove cached version of your site.

Sphere: Related Content