Hello!

I need help in writing a correct robots.txt file.
I don't know anything about this and I just want to
make sure I end up getting the right codes based on
how I want things to be.


These are exactly the files and folders inside my server:

FILES:
- index.html
- welcome.html
- logo.jpg

FOLDERS:
- stats
- family
- sports

I have a website inside the "family" and "sports" folder.
I also have another one which is the "index.html" file in
the root directory.


I want robots, agents, spiders, or whatever to be able to
crawl everything BUT the website and files I have iside the
"family" folder. Those are for my family only and I'd like
to keep it private. Everything else, I want it to be seen and
to be able to be indexed in search engines.


I would really appreciate any help I can get. Thank you!

PS: Where do I put the robots.txt file? Do I put it in the
root directory?

LTheobald

As you guessed, you have to place the robots.txt file in the root directory of your site. As for the contents the file should hold, this should do the job:

For more information, check out these links:
The Web Robots Pages
Robots.txt Syntax Check

potski

Big thanks! Appreciate that!