PDA

View Full Version : stop a spider



ncoastmd
08-02-2006, 06:07 PM
I have a couple pages on my site that are password protected. how do i stop the search engines from spidering these pages as well as my own search engine on my site?

ddadmin
08-02-2006, 06:26 PM
for good robots, you can use the robots.txt file to politely ask them not to crawl these pages. Bad bots ignore robots.txt entirely, in which you'll need to ban them outright, using .htacces for example.

Some resources:

robots.txt (http://www.javascriptkit.com/howto/robots.shtml)
Banning bots using htaccess (http://www.javascriptkit.com/howto/htaccess13.shtml)