Google Bot

[Gnome_101]

Google keeps hitting my site, but goes no further than the first two pages. Do you have to have the robots.txt file it keeps requesting?

I don't understand why google isn't going deeper into my site.

Thanks for any comments...

[tfbpa]

it mostly has to do with the PR you have got and the quality of links coming in. Try to get a few high PR links and your site will be spidered more often and deeper.

Also it doesn't help that your site uses frames, but this should not be that big a deal.....

[jocelyn]

Until you get a PR5 on your home, Googlebot will just grab a few pages when it happens to pass by. When you'll have a PR5 then it will chew your homepage and pages linked on it almost everyday.

[Gnome_101]

Thanks guys.. I've been working a lot to get links...so hopefully they'll help when the time comes. I didn't know you had to have PR5 to get 'deep' crawled.

I'm working to get rid of the frames as well so hopefully by Christmas everything 'll be in order!

:cool:

[jocelyn]

[QUOTE=Gnome_101]I didn't know you had to have PR5 to get 'deep' crawled.
[/QUOTE]
I did not say to get deepcrawled, I mean to have more visit from freshbot. All sites get deepcrawled if they have a few links. But you can't modify your pages as often as if you were refreshed almost everyday with a PR5.

[Gnome_101]

So I need to leave the site pretty much alone after the PR5? I just think it's kind wierd that I have about 2000+ pages, and google only has 167 of them listed when I do a site: command.

[jocelyn]

You have to get links to home and make it go to PR5. Once it's PR5, the pages on home will be refreshed every day. You have to use a sitemap to help googlebot find the pages, but just like your home it has to be PR5 to get it's links refreshed daily. So usually at 1st you use a pyramidal internal linking structure to focus PR on home and some main pages. Once you get a PR5 home you can slowly start to flatten this structure to send PR more evenly to more pages.