The major search engines have the technology to identify new content, we all know that, but are they using it? Are we presented with the most recent content available online or do we get search results that are based on different things than freshness? The freshness factor may not be important if we are looking for an essay on Shakespeare. In that case we would be quite as well interested in a document created 20 years ago, but of course we would still be interested in newer documents and documents that are updated. However, the freshness should be a large factor when searching for information on technology.

Recently, we were searching for Apache related information and we started out by using our favorite search tool, Google, to lookup certain technical information. A lot of the first page results we got were documents created 5 years ago. The old documents may have more and better links pointing to them than the new ones, but you don’t use 5 year old Apache information if you are running a version earlier than that. We should mention that we tried to limit results down to 3 months, but still got old documents. The algorithm should have advanced to the stage to differentiate between technology related words and words related to history. We should also mention that by using the advanced search form in Google and by building more complicated conditions and queries it is possible to get highly targeted results, but we must realize the majority of search engine users may not use the advanced search options and if they did we are certain the advanced link would get a lot more visibility than it already has.

We have been using Web search for years. After we figured out the “old" factor in the search results we were able to identify the best source for the information we were looking for and spent our time there until we found the desired solution. We assumed our search experience was just a glitch related to our queries, not a trend, but since then we have discovered otherwise. Google’s search results are, at least to us, actually getting older. Here’s another example: We recently did a search for “analyzing search technology” (maybe not the best query) and what did we get: in position #2 in our favorite search tool: a ten year old document.  Is this the result we expect from today’s advanced search technology?

Google’s search technology is built around link popularity. A Web site that has more links pointing to it and better links than others is more likely to be in the top positions in search results. Links to a Web page are created over a long period of time. One link here and one link there. This means the older the page is the better chance it has of being in top position in search results. Newer pages with fresh information and perhaps more useable information, but with few links pointing to them don’t have a chance competing against documents lying around for years. This link popularity/qualitylinks ranking technology is making the Web older, less useful and less exciting. The freshness factor is missing.

Google has been improving its algorithm along the introduction of the “new data center and has said so recently. We have heard a new factor is having an impact on how Google calculates its search results and that is the trust factor. The more Google trust the source the higher the source is likely to be positioned within the search results. Sounds nice, but is it? This may actually be the reason why we got a 10 year old document, published by the best known software company in the world, in position #2 when we searched for “analyzing search technology”. Old documents, with many old links pointing to it, published by the most trusted Web sources are likely to get the best positions in the search results.

How about a “sort results by date” link? That’s all we ask for. That shouldn’t be a complicated task in a world of SQL, Ajax and Web 2.0 technology. Google already has a “sort results by date” link on the news search engine’s results and also on the blog search engine, very useful indeed. The question pops up: why isn’t Google allowing users of their main search engine to sort results by date? We say “allowing” because we don’t think Google has to create a brand new technology to offer search results sorted by date. They want to improve the search experience for the user, we know that. The “sort results by date” option would at least improve our search experience a lot. Is it possible that by offering users the chance to sort results by date it would make controllable search results uncontrollable? We say no more. We are going to check the freshness factor on other search engines and maybe share our experience.