Issues Getting Indexed By G

[SanDiegoSEO]

I've never had this issue before, and cannot seem to find similar issues.

I have a site, brand new domain (3 month old domain,) all unique content, solid related links pointing to it (ranging from PR3 to PR6,) and indexed in both Y! and MSN. I just cannot get indexed by G. I could understand being indexed without rankings, but I'm not even indexed. The site has been up for 3 months, and has been using a Google Sitemap for about a month (and it gets picked up daily,) but still nothing in the index to the site. A search for the domain name results in over 50 URL's that contain a link to the site, but none from the site. Also G is reporting zero links to the site, when most of the 50 that contain our domain name have the links on those pages.

Anyone else have it take this long to get indexded?? It's been since May.

[Paz]

Hi,

Yes, 3 months is too long. Are you being spidered?

What's the url?

Cheers,
Paz.

[SanDiegoSEO]

The sites URL is www.haborcredit.com I'm not noticing any traffic from the bot at all. Its very odd. The sitemaps tool is showing the sitemap file is being picked up everyday, but no bot traffic at all.

[dazzlindonna]

I've had this exact problem for about the same amount of time. I have to assume that the domain name was banned by a previous owner, although I cannot determine that based on archive.org (which only shows a domain for sale page). I submitted a reinclusion request about a month ago. Got the standard auto reply. A few days ago I noticed a referrer in my logs from mail.google.com, so perhaps someone actually looked at the site. Still not indexed though. Totally clean site. Good backlinks. Doing well in yahoo and msn. googlebot retrieves the sitemap every day (several times each day), and recently started retrieving the home page every two days. Still not indexed. Like you, I don't expect rankings. I just expect to be indexed. Don't know what else to do.

[tasmaniac]

I've had this exact problem for about the same amount of time. I have to assume that the domain name was banned by a previous owner, although I cannot determine that based on archive.org (which only shows a domain for sale page).

Donna, did the domain name registration expire? If so, it's possible that the domain is not being indexed due to the expiration and not a previous ban. Not sure how Google is treating reinclusion requests for expired names now, but they usually added them back if there was actual content (in about 3 months). It's been awhile since I know of anyone using an expired domain name.

Searching for the URL in Google will return the same results as a banned domain, ie. "Sorry, no information is available for the URL www.expireddomain.com" without the option "If the URL is valid, try visiting that web page by clicking on the following link: www.expireddomain.com"

[Paz]

Sandiego,

I can't reach your site! I tried last night and again this morning....

(harborcredit could not be found, check the name and try again)

Is the url correct?


Cheers,
Paz.

[finders]

I had same problems with new site as well.
specifically with Google.

this time its nothign to do with previous ban or anything like that.
its google.

they take a much longer time to index - i donno why.
googlebot comes in and out every few days, but the site's not indexed.
I am still checking it out.

its been 4 weeks already....
everythings fine with my site.. hope it relates to yr problem..

[finders]

this is what I posted in another forum a few weeks back?

[COLOR=Green]
google is acting weird to index my new sites too.
it takes too long yahoo and msn already indexed my new sites.

rtchar sent me an article which I find interseting.
the articles mentioned most of sypmtons that my new site is experiencing....

but I disagree that its a san box though...
Disagreed with some other statements made in article too....
Take a pinch

this article ..............................................................................
Quote:
Big Sandbox for Google, AskJeeves & Yahoo. MSN Indexes Quickest
By Mike Banks Valentine (c) 2005

Listing delays that have come to be called the Google Sandbox
effect are actually true in practice at each of four top tier
search engines in one form or another. MSN, it seems has the
shortest indexing delay at 30 days. This article is the
second in a series following the spiders through a brand new
web site beginning on May 11, 2005 when the site was first
made live on that day under a newly purchased domain name.
http://publish101.com/Sandbox2

Previously we looked at the first 35 days and detailed the
crawling behavior of Googlebot, Teoma, MSNbot and Slurp as
they traversed the pages of this new site. We discovered that
each robot spider displays distinctly different behavior in
crawling frequency and similarly differing indexing patterns.

For reference, there are about 15 to 20 new pages added to
the site daily, which are each linked from the home page for
a day. Site structure is non-traditional with no categories
and a linking structure tied to author pages listing their
articles as well as a "related articles" index varied by
linking to relevant pages containing similar content.

So let's review where we are with each spider crawling and
look at pages crawled and compare pages indexed by engine.

The AskJeeves spider, Teoma has crawled most of the pages on
the site, yet indexes no pages 60 days later at this writing.
This is clearly a site aging delay that's modeled on Google's
Sandbox behavior. Although the Teoma spider from Ask.com has
crawled more pages on this site than any other engine over a
60 day period and appears to be tired of crawling as they've
not returned since July 13 - their first break in 60 days.

In the first two days, Googlebot gobbled up 250 pages and
didn't return until 60 days later, but has not indexed even
a single page in 60 days since they made that initial crawl.
But Googlebot is showing a renewed interest in crawling the
site since this crawling case study article was published
on several high traffic sites. Now Googlebot is looking at a
few pages each day. So far no more than about 20 pages at a
decidedly lackluster pace, a true "Crawl" that will keep it
occupied for years if continued that slowly.

MSNbot crawled timidly for the first 45 days, looking over
30 to 50 pages daily, but not until they found a robots.txt
file, which we'd neglected to post to the site for a week and
then bobbled the ball as we changed site structure, then
failed to implement robots.txt in new subdomains until day
25 - and THEN MSNbot didn't return until day 30. If little
else were discovered about initial crawls and indexing, we
have seen that MSNbot relies heavily on that robots.txt file
and proper implementation of that file will speed crawling.

MSNbot is now crawling with enthusiasm at anywhere between
200 to 800 pages daily. As a matter of fact, we had to use
a "crawl-delay" command in the robots.txt file after MSNbot
began hitting 6 pages per second last week. The MSN index now
shows 4905 pages 60 days into this experiment. Cached pages
change weekly. MSNbot has apparently found that it likes how
we changed the page structure to include a new feature which
links to questions from several other article pages.

Slurp gets strangely inactive then alternately hyperactive
for periods of time. The Yahoo crawler will look at 40 pages
one day and then 4000 the next, then simply look at the home
page for a few days and then jump back in for 3000 pages the
next day and back to only reviewing robots.txt for two days.
Consistency is not a curse suffered by Slurp. Yahoo now shows
6 pages in their index, one an errors page and another is a
"index/of" page as we have not posted a home page to several
subdomains. But Slurp has crawled easily 15,000 pages to date.

Lessons learned in the first 60 days on a new site follow:

1) Google crawls 250 pages on first discovery of links to site.
Then they don't return until they find more links and crawl
slowly. Google has failed to index new domain for 60 days.

2) Yahoo looks for errors pages and once they find bad links
will crawl them ceaselessly until you tell them to stop it.
Then won't crawl at all for weeks until crawling heavily
one day and lightly the next in random fashion.

3) MSNbot requires robots.txt files and once they decide they
like your site, may crawl too fast, requiring "crawl-delay"
instructions in that robots.txt file. Implement immediately.

4) Bad bots can strain resources and hit too many pages too
quickly until you tell them to stay out. We banned 3 bots
outright after they slammed our servers for a day or two.
Noted "aipbot" crawled first then "BecomeBot" came along
and then "Pbot" from Picsearch.com crawled heavily looking
for image files we don't have. Bad bots, stay out. Best to
implement robots.txt exclusions for all but top engines if
their crawlers strain your server resources. We considered
excluding the Chinese search engine named Baidu.com when
they began crawling heavily early on. We don't expect much
traffic from China, but why exclude one billion people?
Especially since Google is rumored to be considering a
possible purchase of Baidu.com as entry to Chinese market.

The bottom line is that we've discovered all engines seem to
delay indexing of new domain names for at least thirty days.
Google so far has delayed indexing THIS new domain for 60
days since first crawling it. AskJeeves has crawled thousands
of pages, while indexing none of them. MSN indexes faster than
all engines but requires robots.txt file. Yahoo's Slurp crawls
on again off again for 60 days, but indexes only six of total
15,000 or more pages crawled to date.

We seem to have settled that there is a clear indexing delay,
but whether this site is officially "Sandboxed" and whether
that delay is universal is less clear. Many webmasters claim
that they have been indexed fully within 30 days of first
posting a new domain. We'd love to see others track spiders
through new sites following launch to document their results
publicly so that indexing and crawling behavior are proven.






i donno who this author is and i disgreeed with these statements made in his "lessons learned in the first 60 days"


his site symptons are abt the same as my new sites though
[/COLOR]

[nitin_gupta2003]

what is ur correct domain ,u wrote in forum this domain
"www.haborcredit.com" but i am unable to find this domain on yahoo,and google both ,i think ur domain is www.harborcredit.com.

[SanDiegoSEO]

Sorry for the typo, yes the domain name is supposed to be www.haRborcredit.com

[dazzlindonna]

yes, i assume the domain i purchased was an expired domain. i completely understand google wanting to dump all previous PR and backlink credit for a newly owned / previously expired site, but i don't understand why they wouldn't even index it. that makes no sense to me at all. luckily, i'm getting lots of traffic from msn and yahoo, so i'm not terribly concerned, but it puzzles me. the one other thing i might try is changing hosts. as far as i can tell, the host and/or ip is not on any blacklist, but i might try changing anyway just for kicks.

[nitin_gupta2003]

Have a look on these urls in which ur site is present.these all are banned sites.

iranagrifair.com/search-UncleWebster-com-Anchorage-Alaska-s-Complet...
www.americanadvance.com/AA/credit-scores.aspx
www.inanyevent.gr/credit_card_debt_negotiations.html
www.platinumcashadvance.com/Platinum/credit-scores.aspx
www.filipinobarkada.com/27860/harbor-one-credit-union-.php
www.oncourseliving.com/classifieds.htm
www.paydayloanresource.com/PaydayLoanResource/credit-scores.aspx
zarzadzanie.info/national-education-student-loan.html
mortgage.pilotgroup.net/directory/index.php?c=334
beatkings.de/credit_card_debt_negotiations.html

So just check for ur database.may be the reason for not indexing. all this sites that Google banned i think.

[Paz]

Hi Nitin,

excellent work!!!!!

I've removed the live links though.... I'm sure seo guy doesn't want too many links to banned sites.... ;)


Cheers,
Paz.

[nitin_gupta2003]

So many sites are being banned by google in the crawl of 27 july.

and i also seen in a forum in which u have mentioned that "there is a deep crawl of Google on 27 july".

so is this possible that the sites that are banned may be for some time period or they banned permanently by google.is there is any chances for banned sites comeback with their Page Rank.

[Paz]

Hi Nitin,

it depends on the ban.... sometimes a site can be back from a penalty in 2 weeks, but a complete ban is taking longer these days. I don't know for sure, but Google's seems to be more aggressive in the last few weeks, and to be honest I haven't seen one come back yet - but it's still early.

Cheers,
Paz.

[SanDiegoSEO]

Well...I've checked the sites on that list, and the only ones that are in my database, are deffinatly not banned from Google, so I think the issue was from it being an expired domain name.

I did submit a reinclusion email, based on the fact my domain had been previously owned (2001) and may have been an expired domain, and one day later, its in the G index.

[Paz]

Sandiego,

Congratulations on getting back in the index!

The sites that I checked in that list are banned (grey pr and then a check for pages indexed in Google).

They're are all credit links and I'm sure loans site had links to them at one stage... did you remove them yourself - if you did can you tell us when you did remove them?

Cheers,
Paz.

[Paz]

Hi DD, Your sites been down all morning and afternoon.... in case you didn't know already. Still waiting for my seo scoop news of the day!

Cheers,
Paz.

[dazzlindonna]

Yeah, Paz, I know. Got a request into the host to fix it. I'm sure it'll be back soon. Thanks for letting me know though!