View Full Version : Google Bot
07-29-2004, 06:46 AM
Google keeps hitting my site, but goes no further than the first two pages. Do you have to have the robots.txt file it keeps requesting?
I don't understand why google isn't going deeper into my site.
Thanks for any comments...
07-29-2004, 08:31 AM
it mostly has to do with the PR you have got and the quality of links coming in. Try to get a few high PR links and your site will be spidered more often and deeper.
Also it doesn't help that your site uses frames, but this should not be that big a deal.....
07-29-2004, 08:35 AM
Until you get a PR5 on your home, Googlebot will just grab a few pages when it happens to pass by. When you'll have a PR5 then it will chew your homepage and pages linked on it almost everyday.
07-29-2004, 08:52 AM
Thanks guys.. I've been working a lot to get links...so hopefully they'll help when the time comes. I didn't know you had to have PR5 to get 'deep' crawled.
I'm working to get rid of the frames as well so hopefully by Christmas everything 'll be in order!
07-29-2004, 09:25 AM
[QUOTE=Gnome_101]I didn't know you had to have PR5 to get 'deep' crawled.
I did not say to get deepcrawled, I mean to have more visit from freshbot. All sites get deepcrawled if they have a few links. But you can't modify your pages as often as if you were refreshed almost everyday with a PR5.
07-29-2004, 09:30 AM
So I need to leave the site pretty much alone after the PR5? I just think it's kind wierd that I have about 2000+ pages, and google only has 167 of them listed when I do a site: command.
07-29-2004, 09:49 AM
You have to get links to home and make it go to PR5. Once it's PR5, the pages on home will be refreshed every day. You have to use a sitemap to help googlebot find the pages, but just like your home it has to be PR5 to get it's links refreshed daily. So usually at 1st you use a pyramidal internal linking structure to focus PR on home and some main pages. Once you get a PR5 home you can slowly start to flatten this structure to send PR more evenly to more pages.
vBulletin v3.0.3, Copyright ©2000-2013, Jelsoft Enterprises Ltd.