Results 1 to 2 of 2

Thread: robots.txt

  1. #1
    Join Date
    Apr 2012
    Location
    Central New Jersey
    Posts
    143
    Thanks
    52
    Thanked 0 Times in 0 Posts

    Default robots.txt

    Here is my robots.txt file:

    User-agent: *
    #
    # See Wikipedia article for specs
    #
    Disallow: /test.*
    Disallow: /pgp_key.php
    Disallow: /googlef110bd7f27a60db7.html

    My web guru set up my site structure such that efffective URL for individual pages are in the form of marainlaw.com/page.php?here=[pagename]. I wanted spiders to not visit marainlaw.com/page.php?here=test, but I see that googlebot went there anyway. Should my robots.txt file be specified differently?
    Last edited by jscheuer1; 10-20-2012 at 04:32 PM. Reason: Format

  2. #2
    Join Date
    May 2012
    Location
    Hitchhiking the Galaxy
    Posts
    1,013
    Thanks
    46
    Thanked 139 Times in 139 Posts
    Blog Entries
    1

    Default

    Well /*? Is anything with a session id, so you could do something like
    Code:
    User-agent: *
    #
    # See Wikipedia article for specs
    #
    Disallow: /test.*
    Disallow: /pgp_key.php
    Disallow: /googlef110bd7f27a60db7.html
    Disallow: /*?here=test
    "Most good programmers do programming not because they expect to get paid or get adulation by the public, but because it is fun to program." - Linus Torvalds
    Anime Views Forums
    Bernie

  3. The Following User Says Thank You to bernie1227 For This Useful Post:

    marain (10-21-2012)

Similar Threads

  1. Twitter's robots.txt question:
    By Hemat in forum PHP
    Replies: 0
    Last Post: 12-24-2010, 11:02 AM
  2. robots.txt only allow index
    By onestopplay in forum HTML
    Replies: 0
    Last Post: 12-18-2010, 06:19 PM
  3. robots.txt and .htaccess
    By chechu in forum Looking for such a script or service
    Replies: 1
    Last Post: 08-10-2008, 10:11 AM
  4. Robots Text Script
    By snail in forum Looking for such a script or service
    Replies: 1
    Last Post: 06-15-2007, 10:11 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •