Dynamic Drive Forums
>
General Coding
>
PHP
> Twitter's robots.txt question:
Log in
View Full Version :
Twitter's robots.txt question:
Hemat
12-24-2010, 11:02 AM
Twitter's robots.txt, It shows everything is disallowed, but surprisingly search engines are crawling and indexing everybody's profiles pages, Why?
Powered by vBulletin® Version 4.2.2 Copyright © 2021 vBulletin Solutions, Inc. All rights reserved.