Yahoo! seems to be stuck crawling my site. The site is www.IntesiResources.com an I'm getting 1,500 to 3,000 pages/day from Yahoo, 95% of the listed as reloads. My site is dynamic built on ASP .NET framework and SQL. I has very deep links through SQL. Yahoo seems to be getting stuck in some kind of a spider trap. They are reloading my product pages so I don't want to keep them out with a statement in my robots.txt.

I have good results in Yahoo's Site Explorer but few Yahoo hits. Google send +200 visitors/day. I get very few Yahoo hits.

The description of the Yahoo Visitor by HitsLink is:

From Domain: yahoo.com

Domain I.P. Address: 68.142.211.98

Referred By: Bookmark or direct

Visit No. 3

Last Visit: 11/14/2005 9:15:00 PM

Browser: Konqueror CafeKelsa

Platform: FreeBSD i386 X11

Language: English (United States)

Script Version: Javascript 1.3

Screen Resolution: 1280 x 1024

Color Palette: 16 colors

Java Support: Yes

Cookie Support: Yes

My dynamic pages are remapped to look static to spiders and in the URL return, but the page Yahoo is getting stuck or reloading from internal links is: /_coremodules/eshopping/productdetail.aspx

I don't get the full address because HitsLink truncates dynamic pages.

Does anone have an explaination??

Regards,

Intesi Steve

Hi steve,

welcome to the forum!

I didn't know the answer to the problem, so I thought someone else would have a go... but I guess there's not enough information to go on.

Do you have any raw server logs we can look at, apart from the hitlink entry?

Cheers,
Paz.

__________________
10.3 million entries for Hotels in Turkey but I'm still chipping away.

Paz,

Here is a 5 minute snap shot:

2005-11-13 04:20:22 64.58.168.163 GET /uploads/images/regular/200.jpg - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...uctMasterID=125 200 0 0 8087 692
2005-11-13 04:20:22 64.58.168.163 GET /siteSpecific/images/inscapelogo.gif - 80 - 198.36.32.45 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+Q312461 ;+SV1;+.NET+CLR+1.0.3705;+.NET+CLR+1.1.4322) ASP.NET_SessionId=gqqx0e551acka3550zpp4f45;+__supp ort_check=1 http://www.intesiresources.com/pd_189.aspx 200 0 0 5026 535
2005-11-13 04:20:25 64.58.168.163 GET /siteSpecific/images/buttonBG_Small.gif - 80 - 198.36.32.45 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+Q312461 ;+SV1;+.NET+CLR+1.0.3705;+.NET+CLR+1.1.4322) ASP.NET_SessionId=gqqx0e551acka3550zpp4f45;+__supp ort_check=1 http://www.intesiresources.com/pd_189.aspx 200 0 0 573 538
2005-11-13 04:20:27 64.58.168.163 GET /default.aspx noTrack=true 80 - 64.58.168.163 - - - 200 0 64 0 74
2005-11-13 04:21:02 64.58.168.163 GET /siteSpecific/images/boxHeader.gif - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...uctMasterID=125 404 0 0 1795 695
2005-11-13 04:21:02 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=125 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 90900 12739
2005-11-13 04:21:17 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=74 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 64905 8813
2005-11-13 04:21:26 64.58.168.163 GET /default.aspx noTrack=true 80 - 64.58.168.163 - - - 200 0 64 0 74
2005-11-13 04:22:02 64.58.168.163 GET /siteSpecific/images/boxHeader.gif - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=74 404 0 0 1795 694
2005-11-13 04:22:02 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=74 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 63139 9048
2005-11-13 04:22:17 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=31 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 74767 10367
2005-11-13 04:22:26 64.58.168.163 GET /default.aspx noTrack=true 80 - 64.58.168.163 - - - 200 0 64 0 74
2005-11-13 04:22:43 64.58.168.163 GET /siteSpecific/images/boxHeader.gif - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=31 404 0 0 1795 694
2005-11-13 04:22:43 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=31 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 73430 10602
2005-11-13 04:22:43 64.58.168.163 GET /uploads/images/regular/27.jpg - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=31 200 0 0 8598 690
2005-11-13 04:22:52 64.58.168.163 GET / - 80 - 64.58.169.117 PageSentry+(Macintosh) - - 200 0 0 85785 103
2005-11-13 04:22:52 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=31 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 74773 10602
2005-11-13 04:23:17 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=99 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 71102 9005
2005-11-13 04:23:27 64.58.168.163 GET /default.aspx noTrack=true 80 - 64.58.168.163 - - - 200 0 64 0 74
2005-11-13 04:23:57 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=99 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 69429 9240
2005-11-13 04:23:57 64.58.168.163 GET /siteSpecific/images/boxHeader.gif - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=99 404 0 0 1795 694
2005-11-13 04:23:59 64.58.168.163 GET /uploads/images/regular/41.jpg - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=99 200 0 0 11560 690
2005-11-13 04:24:17 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=91 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 65383 9555
2005-11-13 04:24:27 64.58.168.163 GET /default.aspx noTrack=true 80 - 64.58.168.163 - - - 200 0 64 0 74
2005-11-13 04:24:44 64.58.168.163 GET /siteSpecific/images/boxHeader.gif - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=91 404 0 0 1795 694
2005-11-13 04:24:44 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=91 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 63569 9790
2005-11-13 04:24:44 64.58.168.163 GET /uploads/images/regular/102.jpg - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=91 200 0 0 10153 691
2005-11-13 04:24:54 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=91 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 63569 9790
2005-11-13 04:25:17 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=104 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 65976 9569
2005-11-13 04:25:27 64.58.168.163 GET /default.aspx noTrack=true 80 - 64.58.168.163 - - - 200 0 64 0 74
2005-11-13 04:25:52 64.58.168.163 GET / - 80 - 64.58.169.117 PageSentry+(Macintosh) - - 200 0 0 85785 103
2005-11-13 04:25:53 64.58.168.163 GET /siteSpecific/images/boxHeader.gif - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...uctMasterID=104 404 0 0 1795 695
2005-11-13 04:25:53 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=104 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 65982 9804
2005-11-13 04:26:16 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=75 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 65007 8813
2005-11-13 04:26:26 64.58.168.163 GET /default.aspx noTrack=true 80 - 64.58.168.163 - - - 200 0 64 0 74
2005-11-13 04:26:49 64.58.168.163 GET /siteSpecific/images/boxHeader.gif - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=75 404 0 0 1795 694
2005-11-13 04:26:49 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=75 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 65013 9048
2005-11-13 04:26:59 64.58.168.163 GET /siteSpecific/images/boxHeader.gif - 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) - http://www.intesiresources.com/_cor...ductMasterID=75 404 0 0 1795 694
2005-11-13 04:26:59 64.58.168.163 POST /_coreModules/eShopping/productDetail.aspx productMasterID=75 80 - 68.142.211.98 YahooSeeker/CafeKelsa+(compatible;+Konqueror/3.2;+FreeBSD+;+http://help.yahoo.com/+help/hotjobs/webmaster)+(KHTML,+like+Gecko) ASP.NET_SessionId=kga2fc45lszswueeimhwqzer - 200 0 0 63241 9048

Regards,

Intesi Steve

Hi,

I'm not sure it it's stuck, but I can tell you a bit more.

It's not the standard yahoo slurp bot. It's a fairly new yahoo bot that looks for job listings:
http://www.ihategoogle.org/?postid=208

So you can ban it with your robots.txt if you like. The webmaster url isn't working which is a bit naughty of yahoo, but you ban ask them directly about the problem here:
http://www.ysearchblog.com/archives/000078.html

I would delete the IP addresses from your post, once you've solved the problem. Maybe you could ask jocelyn to look at it - he's a bit of a wizard with bots....

Cheers,
Paz.

__________________
10.3 million entries for Hotels in Turkey but I'm still chipping away.

Paz,

I have worked it out with Yahoo! It was a Yahoo HotJobs crawler looking for job postings.

I was sent a request form from Yahoo HotJobs and they will stop crawling my site in 1-2 business days,

Thanks for your help.

Steve
Intesi! Resources

Hi,

I'm a little suprised that they didn't offer you a "disallow" entry in your robots.txt as a solution, but if it's sorted then great!

Cheers,
Paz.

__________________
10.3 million entries for Hotels in Turkey but I'm still chipping away.