I recently here about this robots.txt file. What is it? How to make this file? Where can I attach it in my blog website? What should I write inside this file?
What is robots.txt file?
- Posted:
- 3+ months ago by SteveFort
- Topics:
- blog, robot, inside, write, file, website, web development, website development
Details:
Answers (1)
The robots.txt is a file that is read by search engines and the like (such as web-crawlers) to indicate which files and pages shouldn't and mustn't be accessed. For example, if you include the page "Disallow: /photogallery/" in a robots.txt file, it simply won't appear in search results.
If you are using any of the blogging website such as blogger.com or wordpress.com, they basically have it done for you; however, there is also an option to go to the settings and change the robots.txt - depending on the service provider you are using.
If you don't know what you are doing you could lose the chance for your blog to appear in the search results. However, making a robots.txt file is as easy as making batch CMD files. Here is an easy guide:
http://www.bing.com/webmaster/help/how-to-create-a-robots-txt-file-cb7c31ec
In addition, you should check out the robots.txt generator from Google Webmasters http://www.google.com/webmasters/