Asked Apr 10, 2006, 02:32 AM
I need help in writing a correct robots.txt file.
I don't know anything about this and I just want to
Make sure I end up getting the right codes based on
How I want things to be.
These are exactly the files and folders inside my server:
I have a website inside the "family" and "sports" folder.
I also have another one which is the "index.html" file in
The root directory.
I want robots, agents, spiders, or whatever to be able to
Crawl everything BUT the website and files I have iside the
"family" folder. Those are for my family only and I'd like
To keep it private. Everything else, I want it to be seen and
To be able to be indexed in search engines.
I would really appreciate any help I can get. Thank you!
PS: Where do I put the robots.txt file? Do I put it in the