Hi all,

I want to disallow certain directories from being crawled and indexed by any SE bots for one of my client's site..
so i have placed this tag in my robots.txt (to disallow all bots)
User-Agent: *
Disallow: /folder1/
Disallow: /folder2/
Disallow: /floder3/
Disallow: /floder5/

however i still see at msn fresh pages from these folder being cached. Those pages are not related to the site i m promoting n that's why i don't them to appear on any SE index.

Do i need to put this for specially for msnbot
User-agent: msnbot
Disallow: /folder1/
Disallow: /folder2/
Disallow: /floder3/
Disallow: /floder5/

All these folders are reside on root. and when they are crawled the path looks like this www.example.com/foldername/filename which is not related to example site..

I would like to know is there any other way to prevent those bots from crawling.. If not is there any solution or way out to get rid of this..

Kindly advice me..

thank you

__________________
IPad External Battery
IPhone 4 Battery Reviews

Hi Kate,

yes I've noticed that if you disallow pages with the robots.txt in MSN they index the pages as "link only", inother words in stead of showing the normal:

Page Title:
Here is a snippet usually taken from your meta description tag

they are showing the url with no title or description with

http://www.example.com/folder1/index.htm]

Cheers,
Paz.

__________________
10.3 million entries for Hotels in Turkey but I'm still chipping away.

Hi Paz,

Thank you for kind reply.. still i don't get this one. msn index the pages link only won't SE spiders get confused due to these links? Will they crawl them even if there is no title n description?
N what about other imp pages which i should show up at first why they are buried deep inside.. coz the pages that are listed are usually such links from disallowed folders and due to this other pages have been pushed down in the index..
How can i get those pages to show at first place? Can you advice Paz?

Thank you once again

__________________
IPad External Battery
IPhone 4 Battery Reviews

Hi,

other search engines don't normally index each other's indexes, so the url only links won't matter. Anyway, don't confuse the site: command results and rankings. I know a site that ranks very well in MSN depsite the fact that the site command returns a few url only links:
http://search.msn.com/results.aspx?...hitecompany.com

I think it's just a little warning from MSN.

Cheers,
Paz.

__________________
10.3 million entries for Hotels in Turkey but I'm still chipping away.

I have been getting the same thing, thought i was going mad have changed my tags 3 times to try and overcome this problem.