SEO question

[mikster]

Hi,

I was wondering if SEO bots picks up/index links for pages that don't exist. Hmm.. let me explain what i mean. I told my programmer to script the cms path to look like this "/members/album/picture-of-me" I just discovered that how he did it was that he used a script to somehow simulate those url because when i check for those folders "/members" they don't exist! The cms and pages are viewable its just the folders don't exist. Let me know what you think. My guess is the bot wouldn't be able to find the pages cause it doesn't exist. Will a sitemap solve he problem maybe?

[djr33]

Bots won't guess about URLs. Even if you visit 1/2/3/, they won't check 1/ or 1/2/. But if you have links anywhere pointing to 1/ or 1/2/, then they may try to index these and find them broken. I'm not sure what they do with broken pages, but it won't matter as long as you don't like to them. Does that make sense?

[mikster]

Hmm...But the links aren't broken. /1/2/3/ links actually works its just the actually folder and files are not there. I asked him how it did it and he apparently did it with .htaccess mod rewrite. Please have look at this article http://zenverse.net/seo-friendly-urls-with-htaccess/ it has codes to change /id?=34 to /seo/item35. Let me know if you know anyone else using htaccess mode rewrite and have their pages index in google or yahoo.

[mikster]

Actually after reading the ariticle carefully I just checked my wordpress blog and I think wordpress is using the mod_rewrite..not sure can someone confirm cause I'm looking at my blog and the url is like /category/boxing-article and yet there's not "category" folder anywhere on the domain. Well if wp is using mod_rewrite then its obviously working and SEO friendly cause my wp blog are indexed in google. I'm guessing maybe a sitemap will fix the job because it tells bot about all that links that exist on my site no?

[djr33]

It doesn't matter what files your server actually has. To a visitor (including any bots) the URL is correct, even if, technically, those files don't exist.

My point is something else:
If you have a link to 1/2/3/ and that works, then the bot will index it.
1/2/ may not exist. But that is NOT a problem unless the bot actually tries to go there.

As I said, bots don't guess about URLs. So the bot will NEVER try to go to 1/2/ UNLESS you have a link to it.

You won't need a site map for this. Search engines now are smarter than that, although adding a site map also won't hurt anything (and might help visitors).


Here's the basic summary:
Bots search by following all links. Any page that is linked to on your site will eventually be visited by bots.
Bots will never visit pages that are never linked to. This includes pages that don't exist.
If there is a page that does not exist, but it IS linked to, then you might have a problem.


All of that is unrelated to .htaccess, which can do fancy things like that with the URL, and it can be helpful, if confusing to manage. But as far as SEO is relevant, those pages DO exist (just as much as they would if they were real folders and files).



You can think of a much simpler situation:
1. Create a new directory called "/test/" on your web host.
2. Create a file called "/test/test.htm"

Now, if a search engine visits that page, it will index it. Remember that it would need to be linked to from somewhere (for example, your site's homepage, or a different website).
But the search engine will not try to access "/test/" which does "not exist" in the sense that there is nothing to load there and you will get some sort of error (either 403 "forbidden" or 404 "not found").

There's nothing different from the SEO point of view about the current situation, it's just more complicated.

[mikster]

Very well explained...Thank You very much = )


It doesn't matter what files your server actually has. To a visitor (including any bots) the URL is correct, even if, technically, those files don't exist.

My point is something else:
If you have a link to 1/2/3/ and that works, then the bot will index it.
1/2/ may not exist. But that is NOT a problem unless the bot actually tries to go there.

As I said, bots don't guess about URLs. So the bot will NEVER try to go to 1/2/ UNLESS you have a link to it.

You won't need a site map for this. Search engines now are smarter than that, although adding a site map also won't hurt anything (and might help visitors).


Here's the basic summary:
Bots search by following all links. Any page that is linked to on your site will eventually be visited by bots.
Bots will never visit pages that are never linked to. This includes pages that don't exist.
If there is a page that does not exist, but it IS linked to, then you might have a problem.


All of that is unrelated to .htaccess, which can do fancy things like that with the URL, and it can be helpful, if confusing to manage. But as far as SEO is relevant, those pages DO exist (just as much as they would if they were real folders and files).



You can think of a much simpler situation:
1. Create a new directory called "/test/" on your web host.
2. Create a file called "/test/test.htm"

Now, if a search engine visits that page, it will index it. Remember that it would need to be linked to from somewhere (for example, your site's homepage, or a different website).
But the search engine will not try to access "/test/" which does "not exist" in the sense that there is nothing to load there and you will get some sort of error (either 403 "forbidden" or 404 "not found").

There's nothing different from the SEO point of view about the current situation, it's just more complicated.