The Search Pages of Blogger, which is basically Post Labels and Navigation URL, this pages considered as bad because the produce Duplicate content error and Duplicate Title error. Search pages also have Dynamic URL which is not supported by all search engines. So when search engine bot Index your blog Labels, they index your all post which is all ready exist in there index, so it produces duplicate content.
So it necessary to have Robot.txt for your Blogger blog that tells all bot to not to Index your blog Archeive pages,Labels and Navigation pages. I have Robot.txt for Techofy which have vary positive results for my blog and every line is tested one by one, and it removes some error that are reported in Webmaster Tools.
- Remove duplicate Meta descriptions error from your Blogger Blog.
- Remove duplicate Title error from your Blogger Blog.
- Stop All search Bot to Index Search, Labels, and achieves of blog.
- Go to Blogger Dashboard > Setting and then Click on Search Preferences.
- In Crawler and Indexing section click Edit in front of Custom robots.txt.
- Click Yes and add your custom written Robots.txt and then click Save changes to save.
User-agent: * Disallow: /search Disallow: /*_archive.html Disallow: /cgi-bin/ Allow: / Sitemap: http://yourblogURL/atom.xml?redirect=false&start-index=1&max-results=500
Copy above code and paste into your blog custom Robots.txt but replace Sitemap section with you Blog URL and Save.
This lines applies to all bots and all search bot that support robots.txt exclusion.
This line tells bot to not Index you Blog labels like my blog label Software which stop duplicate content error.
This line tells bots to not to Index your blog monthly archive pages which contain list of all post of particular month.
This is optional line i have included it because some time webmaster report 404 pages from your blog containing /cgi-bin/ part in URL, this 404 pages are auto generates due to some blogger internal error so stop search engines to stop indexing this type of pages.
Allow: / Sitemap: http://yourblogURL/atom.xml?redirect=false&start-index=1&max-results=500
This is your Blog sitemap which make bots to search your blog link and Index it. Including sitemap into robots.txt which makes it easier to submit sitemap to all search engine. Otherwise you have to manually submit sitemap to all search engine.
See Robots.txt of Techofy and ask question if you have any .