I am in the midst of completely re-arranging my site and because of this there will be a lot of pages which will no longer exist. Almost all pages are already indexed by GG and I don't think they, or any other searchengine for that matter, appreciates dead links.
Is there any way around this, like a redirect so my site won't be penalized? A custom 404 which redirects to the mainpage is acceptable for me since I don't have any real visitors yet, but that's not so good for searchengines I think.
I have read something about permanent redirects, a 301, but to my knowlegde this has to be done for every single page and as my site has about 10000 pages this is not an option, I still sorta have a life!
So if anybody has any suggestions, I am all ears!
Thanks.

Ok, I have been looking into it a bit more and there are quit a few stories that tell me that you shouldn't remove dead links, but cherish them. This mostly counts for websites that have actually good ranking for their keywords, mine doesn't so for this reason it is not neccesary.
But on the other hand there is enough space on my server for both versions. Only this means that all pages have identical brothers/sisters and identical content is something to avoid, I think.
Just thinking out loud here, if you want to do the same......

I don't see how broken links could ever be a good thing, they must always be bad. I check my site as soon as I've added something to ensure there are no broken links. Put it this way, if googlebot is crawling you and it reaches a deadlink, it hits the 404 page and gives up.

I don't think that when GG hits a 404 it gives up. I already have some non existing pages for a few months and they are still in the index. Even more strange, I just noticed that those pages have gone up in PR, from 2 to 3! I know it's really strange, but it is a fact.
So if 404's don't work, what will??

I didn't mean it gives up, it just stops indexing your site for that particular crawl, it may try again the next day.

Ok, thanks for explaining.
But if there are 10000 404's, which will be the case when I update the site, it will take forever for GG to spider my site completely.
So do you have any suggestions what to do with those pages?

Try and work as quickly as possible, it may even be better to use robots.txt to disallow them indexing your pages until you're done.

If possible set up 301 redirects from the old url's to the new url's. This will help to make sure you don't lose too much traffic from the search engines (especially the ones that don't index as fast as Google.) Besides that, when that page is crawled again, it will find the 301 (permanent redirect) and that can help to speed up the update of the index.

Regards,

Peter

__________________
Peter Faber
Knowledge-finder.com (Article) Directory
Text Link Brokers - Buy and Sell Text Links

301 is not easy for him... if the pages are generated, might be possible to make a generic THIS PAGE CAN NOW BE FOUND... BLA BLA BLA... with the link to the new site. Do not use a HTML refresh tag... not recommended.
Then slowly you 301 the pages keeping the old site live until you are done.
Visitors will find the new site and the links will show the bots where to go too.
This is how I see it, as less painfull, and most effective for you.

The reason I advise the 301's is that once the pages are not online anymore,. they will still be ranking in the search engines,. the 301's make sure that the searcher still ends up on a real page in stead of a lousy 404 page.

Perhaps in Google it can go relatively fast (but can still take months with 1000+ pages), but other search engines are way slower. If you add it all up,. it is quite a lot of visitors that you would be missing.

Peter

__________________
Peter Faber
Knowledge-finder.com (Article) Directory
Text Link Brokers - Buy and Sell Text Links