PDA

View Full Version : Robots.txt


potski
Apr 10, 2006, 02:32 AM
Hello!

I need help in writing a correct robots.txt file.
I don't know anything about this and I just want to
Make sure I end up getting the right codes based on
How I want things to be.


These are exactly the files and folders inside my server:

FILES:
- index.html
- welcome.html
- logo.jpg

FOLDERS:
- stats
- family
- sports

I have a website inside the "family" and "sports" folder.
I also have another one which is the "index.html" file in
The root directory.


I want robots, agents, spiders, or whatever to be able to
Crawl everything BUT the website and files I have iside the
"family" folder. Those are for my family only and I'd like
To keep it private. Everything else, I want it to be seen and
To be able to be indexed in search engines.


I would really appreciate any help I can get. Thank you!

PS: Where do I put the robots.txt file? Do I put it in the
Root directory?

LTheobald
Apr 10, 2006, 02:52 AM
As you guessed, you have to place the robots.txt file in the root directory of your site. As for the contents the file should hold, this should do the job:



User-agent: *
Disallow: /family/


For more information, check out these links:
The Web Robots Pages (http://www.robotstxt.org/wc/robots.html)
Robots.txt Syntax Check (http://tool.motoricerca.info/robots-checker.phtml)

potski
Apr 10, 2006, 02:57 AM
Big thanks! Appreciate that!