View Full Version : stop a spider
08-02-2006, 07:07 PM
I have a couple pages on my site that are password protected. how do i stop the search engines from spidering these pages as well as my own search engine on my site?
08-02-2006, 07:26 PM
for good robots, you can use the robots.txt file to politely ask them not to crawl these pages. Bad bots ignore robots.txt entirely, in which you'll need to ban them outright, using .htacces for example.
Powered by vBulletin® Version 4.2.1 Copyright © 2013 vBulletin Solutions, Inc. All rights reserved.