Ask Experts Questions for FREE Help !
Ask

Robots.txt

Asked Apr 10, 2006, 02:32 AM — 2 Answers
Hello!

I need help in writing a correct robots.txt file.
I don't know anything about this and I just want to
Make sure I end up getting the right codes based on
How I want things to be.


These are exactly the files and folders inside my server:

FILES:
- index.html
- welcome.html
- logo.jpg

FOLDERS:
- stats
- family
- sports

I have a website inside the "family" and "sports" folder.
I also have another one which is the "index.html" file in
The root directory.


I want robots, agents, spiders, or whatever to be able to
Crawl everything BUT the website and files I have iside the
"family" folder. Those are for my family only and I'd like
To keep it private. Everything else, I want it to be seen and
To be able to be indexed in search engines.


I would really appreciate any help I can get. Thank you!

PS: Where do I put the robots.txt file? Do I put it in the
Root directory?

Search this Thread
Share |
2 Answers
LTheobald's Avatar
LTheobald Posts: 1,069, Reputation: 650
Ultra Member
  
#2

Apr 10, 2006, 02:52 AM
As you guessed, you have to place the robots.txt file in the root directory of your site. As for the contents the file should hold, this should do the job:

Code:
User-agent: *
Disallow: /family/
For more information, check out these links:
The Web Robots Pages
Robots.txt Syntax Check
Helpful
potski's Avatar
potski Posts: 9, Reputation: 1
New Member
  
#3

Apr 10, 2006, 02:57 AM
Big thanks! Appreciate that!
Helpful
Not your question? Ask your question View similar questions
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search

Add your answer here.

Remove Text Formatting
 

Undo
Redo
  
Decrease Size
Increase Size
Bold
Italic
Underline
Align Left
Align Center
Align Right
Ordered List
Unordered List
Decrease Indent
Increase Indent
Insert Email Link
Wrap [QUOTE] tags around selected text
Wrap [CODE] tags around selected text
Wrap [HTML] tags around selected text
Wrap [PHP] tags around selected text
Wrap [YOUTUBE] tags around selected text
Notification Type:



Check out some similar questions!

1980 sci fi movie black masked robots [ 1 Answers ]

Hello, I remember this movie or TV show when I was a kid. Its about a young boy and a short black masked robot that had a hole drilled in his face plate by a big black robot so they know his face "because all the robot look alike". The robot all have black faces and helmet on and black jump...

View more Internet & the Web questions Search