View Full Version : stop a spider

08-02-2006, 07:07 PM
I have a couple pages on my site that are password protected. how do i stop the search engines from spidering these pages as well as my own search engine on my site?

08-02-2006, 07:26 PM
for good robots, you can use the robots.txt file to politely ask them not to crawl these pages. Bad bots ignore robots.txt entirely, in which you'll need to ban them outright, using .htacces for example.

Some resources:

robots.txt (http://www.javascriptkit.com/howto/robots.shtml)
Banning bots using htaccess (http://www.javascriptkit.com/howto/htaccess13.shtml)