How to Enable and Customize Blogger Robots.txt File?

Hi ! In this tutorial I am going to show you how to edit your Blogger blog robots.txt file. Webmasters use custom robots.txt file to control search engine web robots (also known as Web Wanderers, Crawlers, or Spiders) to crawl some directories and web pages or links of website or blog. when we take settings robots.txt file in blogger then search engine automatically index or remove pages from search engine according to the settings. 

By default each website allows the Search engines robots however if you would like to restricts the robots either to not crawl any bound directory, file or the complete website then you may want the robots.txt file in which you've got to write instructions for search engine bots.

Steps to edit robots.txt on blogger

Your Site Settings › Search preferences › Crawlers and indexing  

Here you will able to see two options Custom robots.txt and Custom robots header tags.These two
 options would offer you the flexibility to customise your robot.txt file.In the last post i told you about
Custom robots header tags.

Now press edit button which is present after  'Custom robots.txt' option. After pressing on edit button
you can see a message " Enable custom robots.txt content? " so press "Yes" and proceed to next step.

Now you can see  text area, type the content which you want to exclude a content from crawling.
Click on Save Changes button.

You are done!
How to block Link from search engines?
No, you have to write the URL. For example, if your want to stop robots from crawling this URL
(www.ichibikarin.blogspot.com/p/about.html) then, in the robot.txt file you will enter this command.

User-agent: *Disallow: p/about.html

Your robots.txt file located under your main blogspot directory as for Blogger Known its located on the following url:

http://www.ichibikarin.blogspot.com/robots.txt

By default blogger robots.txt contains the following content:

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://ichibikarin.blogspot.com/feeds/posts/default?orderby=UPDATED

I know this post is not a new thing but I hope it might help many newbies to understand the importance of robots.txt.