The freshness factor of any Web search engine is important. There are millions of new Web pages created daily that each deserves to be crawled, indexed and made available in search. There are probably more Web pages created daily that do not deserve to be crawled, indexed and made available in search. The criteria for which Web pages should be crawled, indexed and made available in search is nothing less than a puzzle for the large search engines to figure out, but why is it a puzzle?

The search technology the search engines have at their disposal as of today has advanced in the past decade or so. We have no idea on the technology behind the major search engines, but we can make certain assumptions based on common knowledge on advances in computer and Internet technology.

We are certain that the major search engines could easily discover and index the majority of new Web pages that are created today and make them available to us by tomorrow via their search interface. We can tell you with considerable confidence that the major search engines aren’t doing this, they could do it, but they aren’t doing it. The question arises why the search engines aren’t giving us, the Web users, the opportunity to get up to date search results?

The reasons could be many and here are few possibilities:

Search engines probably don’t like the idea that someone could create a Web page today and have it appearing in search results tomorrow without knowing more about the publisher and what he might publish.

Search engines probably don’t like the idea that someone could use the freshness factor to create optimized Web pages and test daily their performance in search knowing the search index updates daily.

Option to sort results by freshness or date takes certain control from the search engines over what search results are displayed that could help webmasters manipulate with the search results.

Data integrity is one possibility. The Web environment as it is today has gone far from the innocent environment it was a decade ago. Search engines need to protect themselves from all kinds of scenarios that could harm their business operations and their users. Understandably, Web pages and search results have to be checked, calculated and taken through different kind of filters before published to the public.

We mentioned only four possible reasons for why we, the Web users, don’t have the opportunity to search up to date Web site information and sort information by date, but there are likely more reasons for it and reasons that we even can’t imagine.

We look forward to the day when search engines can, in a day, discover, crawl, check, and filter and make available to us, the Web users, in search what was created, designed, invented and written yesterday.