The Web is growing faster than ever before, or that is at least the general assumption. The Web is also an ever changing environment. Web sites are born and Web sites disappear from the Web's horizon. Web pages are created and added to existing Web sites and Web pages are deleted from servers and some are moved to new locations. The process of setting up a new domain and server is easy and it is even a lot easier to delete pages from a server. The whole process of adding and deleting files on Web servers is simply a piece of cake, but for some search engines adding and deleting files from their indexes isn't that easy and, in some cases, can actually take years.

The general consensus on how to get Web pages removed from search engine indexes isn’t clear and the best advice out there is from webmasters that have learnt the process by trial and error. The strange thing is how little information is made available on the process and the guidelines provided by the search engines don’t always work as expected.

Webmasters are often told by the search engines and search engine specialists that they should focus on creating the best Web site for the Web user and if it is good for the user it is good for their search engine performance. Making a Web site better often involves the deletion of Web pages from a server and often a lot of them. Some search engines hold on to deleted Web pages for a long time and it seems to take years for some of them to delete Web pages from their indexes even though they get the correct error code and the webmaster has told them via mod-rewrite and robots.txt files the Web pages are gone forever and new ones have taken over the job of serving the visitors.

Honest and hard working webmasters don’t want visitors to arrive on their Web sites via Web page that isn’t found there anymore, but that is a problem mainly originated from outdated search engine listings. To help the users get the best experience from the first visit many webmasters create customized 404 pages or error pages. Instead of arriving at a typical browser error page the visitor is welcomed and guided to the parts of the Web site that are functioning and ready to serve him. Sounds good?

Well, it is, but the truth is that customized error pages can have the webmaster perform search engine hara-kiri on their Web site if not configured correctly. What is good for the Web user can have devastating effects on the Web site’s performance with the search engines. We are not going into details on how you can configure your server, start the mod-rewrite engine and make all the arrangements to hopefully have the search engine correctly interpret your intentions with the error pages. If you have doubts about your Web site’s error page handling and management you should really look into it. There is however one problem. The major search engines don’t tell you exactly how you should do it and how they treat your Web site if you don’t do it correctly.

The major search engines can do better in educating webmasters on how they should configure their Web servers to serve the search engine crawlers with the correct Web pages. Educating webmasters might actually help the major search engines in maintaining their data centers and indexes with fewer crawling errors, fewer Web pages to crawl and preventing all kinds of misleading server configurations tests from taking place in volumes.