Setting Up Robots.txt on Blogger

Blogger now allows custom robots.txt, this is very useful because we can set the visibility of our articles on search engines, if you Want to Study about New Features of blogger Search Engine Optimization For blogger blog then you can Read My Previous Post on Blogger's SEO Advanced Search Setting



By default, every blog that uses the Blogger platform will have a robots.txt as follows:

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://www.example.com/feeds/posts/default?orderby=updated

And has the following explanations:

Mediapartners-Google is a robot from Google, leave it as is because if you mistakenly change that than the ads served will not fit with your content.

The next line is for all the robots and marked with an asterisk (*). On the default configuration, it is clear that the label of our blog is not indexed Disallow: /search.

Keep in mind that This slash (/) is as your homepage, so for example if you want the label to get indexed, do not just fill up with a slash like this Disallow: / because that would be you do not allow the robot tracing your blog, but it should like the example below:

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow:
Allow: /

Sitemap: http://www.example.com/feeds/posts/default?orderby=updated

With the configuration as above then all of the articles and the label will be indexed. And to block a robot for particular page (I take the example of my Forum page) you can simply write as follows:

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /p/forum.html
Allow: /

Sitemap: http://www.example.com/feeds/posts/default?orderby=updated

After the changes, Check Your Robots.txt by visiting www.example.com/robots.txt or www.example.blogspot.com/robots.txt Replace the Example.com with your domain name.

Warning!

Incorrect use of these features can result in your blog being ignored by All search engines and your Blog will De indexed Be Careful...!