googlebot

[marain]

I have five active domains that get decent traffic. Until recently, a frequent visitor was the googlebot spider, frequent meaning maybe 4 to 5 percent of traffic. Of late googlebot does not seem to have stopped by at all. Traffic as a whole does not seem to have fallen off--at least not yet. I am wondering whether I should be concerned over googlebot's absence.

A.

[jscheuer1]

Probably not. Although I don't know for sure, I would think if you're updating your content a lot the bot has to visit more frequently. Once you've settled on things and the site is indexed, the bot doesn't have to visit as often. This would only have an impact on your human visitors if one of the main reasons they're coming is for frequent updates.

[marain]

Well the bot doesn't know whether it's being updated unless it stops by! I made significant changes to some pages on Saturday that google hasn't picked up on yet. On previous occasions, google reflected these changes within a day. All visits have stopped entirely, on all my domains.

A.

[djr33]

It might have noticed a change in the frequency of updates, so that it now visits less regularly (because it predicts fewer updates). But that wouldn't explain why it never visits.

It's hard to say what's going on. I'd guess either:
1. There's a new IP or other reason you're not recognizing it (but it's still there).
or
2. For some reason Google has flagged and rejected your website. I have no idea why that would be, but it's the only thing I can imagine if you get no visits.

Have you tried logging into your account with google to see what's going on? Webmaster tools, and so forth.

[marain]

The new IP suggestion appears to be not it: I do searches using keywords on the changed page that were not there previously, and they're not reflected in search results. At the same time (saints be praised), searches using preexisting keywords do yield the same good placement that I've long enjoyed, so at least it seems I'm not being blacklisted.

Concerning google's webmaster tools, no, I've not considered that. I was hoping this problem would resolve without need for deep analysis. My original question was, should I be concerned? So long as my previous placement continues, I can continue for a rather long time. But it's not something that I can let go forever.

A.

[jscheuer1]

When was the last ime you saw the bot?

[marain]

I last saw it maybe six weeks ago. That does not mean it's been MIA all the while--I hadn't been checking it.

My monitoring service allows me details of only the most recent 100 visitors. One of the sites averages about 100 visits daily. Another, about 25. Since I resumed looking, last Saturday, I've not seen the bot.

I resumed looking last Saturday because I'd made considerable changes to one of the sites, and wanted to start monitoring their effect. Not surprisingly (in view of the absent bot), the Google search engine is still unaware of those changes. A different search engine, DuckDuckGo.com, has picked up my changes. I do not know what DuckDuckGo bot's IP is, so can't do any frequency comparisons.

A.

[Beverleyh]

If you sign up for Google Webmaster Tools (its free) and submit an XML/txt sitemap, you can check when your site was last spidered via their dashboard.



Mine are usually crawled automatically every 7-10 days or so, but you can push URLs through manually to speed-up the process, via their Fetch as Google feature: http://support.google.com/webmasters...answer=1352276

Google Webmaster Tools and Google Analytics are 2 excellent, feature rich tools that every website owner can learn a lot from. Definitely good things to have in your SEO/Marketing arsenal.

EDIT: conveniently, this just popped up on my Twitter feed (tips to master Google Analytics): http://www.hongkiat.com/blog/google-...ics-data-tips/

[Beverleyh]

I dont know if you already do, but if you just want to check which pages of your site Google has indexed, you could always do a specific "site:" prefix search. To do that, just go to Google and type "site:http://www.mywebsite.com" (with the url of the website to check) into the search bar. The results should just list all of the pages from your website that Google has on record - if a page isn't there, Google hasn't indexed it.

[marain]

OK. You've given me new toys to play with. I took the time to dabble with them and am highly impressed. But I must now put down my toys and return to (what for me is) the real world. I'll be back.

Thanks! To the lot of you!

A.