Other dirs causing bad results?

[lemmybrown]

Hi there. The site I wish to promote resides in a domain next to lots of other directorys that are used by my customers to feed pages through the web to their intranets.

I'm worried that the marketing pages that mostly sit in their own dir next to all the customer dirs will be getting marked down by SEs because the content may be getting watered down by all the repetition and other data on the customer only pages in the other directorys. I thought about using robots.txt to exclude the other stuff, but if I do that I'll be revealling things I want to keep quiet.

Would the best solution be to move the marketing page to thier own domain.?

[jocelyn]

robots.txt will stop honorable bots, but some may not be as honorable. So there is still a hole here.

You have to put an index.html or whatever default page so people have to know the name of the file they wish to see. If they go in the dir, it loads the index page that say whatever you feel like. Obviously, they may know standardly used names and get to some files anyway. I get hit for mailform.ml or something like that by spamming softwares all the time....lol


I can't wait to see what other dudes will suggest.

[lemmybrown]

I had no idea that some would get around robots.txt.. thanks for that.

[jocelyn]

[QUOTE=lemmybrown]I had no idea that some would get around robots.txt.. thanks for that.[/QUOTE]
If you restrict using * for all robots in your robots.txt, most will stop. But I'm not 100% sure it stops everyone... I mean spam email softwares are not asking permission, and do they really respect robots.txt... not too sure. Most use that to exclude their external javascript, css... from being crawling. This is why I'm curious about the sensitive files and this...

[seobook]

each page is ranked upon its own merit. while search engines such at teoma heavily use clustering the other major search engines are not considered theme based and still count all sorts of links.

though my site has nothing to do with currency conversion I link to a calculator so people in other countries can see the price of my product...those types of links still help organize the web and thus each page stands on its own and links from off topic pages still usually have some weighting.

people use search engines versus directories because there is more than one way to classify a site and often sites have many various underlying themes

[lemmybrown]

Thanks all, that totally clears up that question.