We have often written about the major search engines and how they need to improve. We have also discussed about how important it is for the webmasters to know which rules and guidelines they should follow. The majority of webmasters aren't trying to fool the search engines; they just want to have their Web sites indexed and findable. How they can do that without being mod rewrite experts or preventing their Web sites from falling into duplicate penalty traps or sandboxes is perhaps their biggest challenge and even bigger than the designing and making of their Web sites.

The Web is growing faster than ever before, or that is at least the general assumption. The Web is also an ever changing environment. Web sites are born and Web sites disappear from the Web's horizon. Web pages are created and added to existing Web sites and Web pages are deleted from servers and some are moved to new locations. The process of setting up a new domain and server is easy and it is even a lot easier to delete pages from a server. The whole process of adding and deleting files on Web servers is simply a piece of cake, but for some search engines adding and deleting files from their indexes isn't that easy and, in some cases, can actually take years.

Search engine is a simple concept in itself. Search engine allows a Web user to search its index of crawled Web pages and displays results accordingly. The Web started with one document, then two etc., Ten years later the Web was billions of documents and 15 years later it was tens of billions of documents. How can a Web user get sensible results when searching billions of documents for a word like "business"?

The first PCs ever made had only few colors available, 16 if we remember it correctly, based on red, green and blue or RGB. Computer users needed more and today we take millions of colors on our screens for granted. Going back to 16 colors is not an option for us. Ten years ago (when color resolution was still an issue) the search engines weren't as powerful as the large search engine are today and there were fewer technical issues to solve, but Web users had a variety of search engines to search for Web information. The search engine landscape was, perhaps, more colorful than it is today.

The freshness factor of any Web search engine is important. There are millions of new Web pages created daily that each deserves to be crawled, indexed and made available in search. There are probably more Web pages created daily that do not deserve to be crawled, indexed and made available in search. The criteria for which Web pages should be crawled, indexed and made available in search is nothing less than a puzzle for the large search engines to figure out, but why is it a puzzle?

There is an ongoing talk online about the Web's biggest and most popular search engines, but the talk is often about anything else than search. The search companies are constantly adding new Web services and coming up with new "innovations". The online media, including blogs, waits for the next product release or the next Beta service and discusses in what direction the search engines are going. There is one thing, in our opinion, that is missing: how to improve and advance search technology to help us, the users, get the best search experience and actually find the Web pages and information that make up the Web today.

The major search engines have the technology to identify new content, we all know that, but are they using it? Are we presented with the most recent content available online or do we get search results that are based on different things than freshness? The freshness factor may not be important if we are looking for an essay on Shakespeare. In that case we would be quite as well interested in a document created 20 years ago, but of course we would still be interested in newer documents and documents that are updated. However, the freshness should be a large factor when searching for information on technology.

We have written many pieces about the major search engines and we are particularly interested in Google. Google has the biggest market share in search and is a dominating force in distributing or re-routing Web traffic and that is why 80 percent of our search pieces are about Google. The other major search engines, Yahoo and MSN, are of course very important and either one of them could become the next search destination for majority of searchers, but the present dominator in search is Google.

The new landscape Google has created with its new data center, nicknamed "Big Daddy", just became a lot clearer. Matt Cutts recently shared some very valuable comments on the new data center and Google's new algorithm. Matt Cutts says Google crawlers can now do a better job in crawling pages that deserve to be crawled and determining which Web sites are not worth the crawl. The cold and hard reality of Google's new data center and algorithm change is sending chills down many webmaster's spines and for many the chill can easily turn into a deadly flu for their Web sites.

The major search engines rely a lot on the quality and number of links pointing to a Web site when calculating its rank and position in search results. This has in many ways skewed search engine results in a way they do not always represent the most relevant results.

The major search engines of the Web may seem what they were originally intended to be; search tools to find information on the Web and in databases, but perhaps that is not exactly what they want to be and how they want you to use them. If every user would only search for what they are looking for and then click on the results presented in the organic results then search engines would face a serious problem; they wouldn't make a single dime. The fact is they want you to do something else than click on the organic links.

From the beginning of the World Wide Web people are always trying to control the Web, the search engines for their own benefits. Since they arrived on the Internet horizon the major search engines have been combating all kinds of manipulation of their search results and fortunately, at least for now, they have always managed to stay on top.

Looking back a few years there was always the excitement in the air that a new great search engine might appear on the Web landscape and they did. Almost daily there was a new search engine introduced, but then things started to change. Most of the new search engines were using Google's index or providing results from multiple search engines presented in a different way. Mindful of the glorious days of search we decided to try to find a brand new search engine launched in the past few weeks.

The Web has been the enlightenment for the creation of many new words. Those new words are sometimes called buzzwords. Buzzwords are terms or words that are heavily discussed in the Web community and describe something new. The latest buzzwords/terms in search engine marketing are among others "natural links" and "organic links". When the major search engines managed to effectively combat link spamming a new term emerged - natural link building.

Vulnerabilities in PC operating systems and software have ignited an industry of malicious software protection. Vulnerabilities or imperfections in the world's most popular search engines have ignited an industry called Search Engine Optimization.

Since Google became the search engine number one it has given and it has taken. The early days of Google's reign were colored with thankful mom-and-pop Web businesses that got enough visitors from Google to make their businesses work. The positive effects Google had on small and micro Web businesses was a force that helped build the most valuable Internet company to date.

There is no question about the fact Google is the world's leading search engine in popularity measures and perhaps also in all other important measures. Yahoo, currently the search engine number two is no longer trying to be the number one search engine.

Microsoft�s plan to go full speed in the summer with their advertising center will be waited with anticipation by many. If they succeed in creating an advertising network that works for the advertisers as well as the publishers then we could see some really interesting development taking place in the search engine marketing industry.

The Web is aging. Some Web sites are frequently updated, renovated and redesigned while others have stayed the same for years. Frequently old Web sites, with the exact same design and content for years, occupy top search engine positions in both Google and Yahoo. The reason is likely the algorithms and filters the search engines use to rank and value Web sites.

Since Google introduced PageRank the art of ranking Web pages has become an important subject for all search engine marketers. The idea behind ranking Web pages is simple. It is to find a way to sort Web pages based on other elements than pure keyword searching. Pure keyword searching doesn't work when there are billions of Web pages to pile through. Keyword searching would give us long lists of identical results.

The latest trend word on the Web is "Web 2.0". Web 2.0 is a concept that covers new type of Web services that use new methods in communicating between Web servers, databases, software and people. The development environment for Web 2.0 is referred to as AJAX, which is a combination of Java, RSS, XML and HTML. Too complicated to go into details, we care about the products and Web services Web 2.0 software developers stir together and roll out to us, the users.

Google's success has brought the entire search engine industry into focus. All of a sudden everyone seems to care about Web search and search technology. We remember a time not so long ago when search wasn't seen as a profitable business. All that changed with Google's success. Google found a way to make money from search and thanks to Google's success, search is sexy today. Things were easier for us, the webmasters, in the good old days, but who says things should be easy?

One of the most exciting things to think about on the future of the Web is Web searching. Search technology is almost in its infancy and it's a field where most advances could be made in the future. There is urgency to advance Web search technology. The Web is growing faster than any computer system can track and there is an ongoing need for an advanced technology to retrieve, organize and make all this information easily accessible for all Web users at lightning speed.

The major search engine players occupy the news on a daily basis, but there are a lot of search engines that are quite popular even though they aren't near the big players in terms of popularity and earned advertising dollars. The smaller search engines are not only less popular than the big ones, but also maintain databases that are usually smaller. It takes humongous amount of storage and computer power to index the entire Web. For smaller search engines that task may be too large to handle for now, but that could change in the near future.

The search engine industry has winners and losers. The winners are the ones we know most about. Searching has become a newsworthy attraction and many want to get on the Web search bandwagon. That is not easy. The Web search industry is occupied by three major players, Google, Yahoo and MSN Search, and of those three one is the most popular and frequently in the news, Google. All of the three major players are rich companies with thousands of scientists and engineers working for them.

A theory appeared well over a year ago on whether the world's most popular search engine had created a mechanism to delay new Web sites from being included in search results. To cut a long story short we can say with reasonable certainty there is no official "Sandbox", but that doesn't mean there aren't any filters that behave in a "sandboxed" way.

One of the interesting side effects of the recent Google AOL deal is the fact Microsoft lost the deal to Google and many see that as a blow to the software giant's effort to promote their search engine. Fighting Google in the search engine war would have been easier for Microsoft if they had won the deal. Perhaps Google knew that and were willing to compromise and go arms length to impress AOL. Nothing is actually changed. Google was with AOL before. What will be really interesting to watch and could change things is what the King of software will do to fight the Emperor of search.

The Search Engine Optimization (SEO) industry was born mainly from influence of optimizing Web pages for higher positions in Google. Search engine optimization was of course done for other search engines, but Google was the most popular search platform and still is and was and still is the big influence on the SEO industry.

There are rumors/news floating around that Microsoft may have shown interests to buy Yahoo. It wouldn't surprise us at all if Microsoft bought, acquired or merged with Yahoo. It could be the cleverest way to compete with Google head on.

Some have said searching is the least developed computer technology. It may sound surreal since searching is one of the most common task any Internet user does, but it can be said with more certainty than it looks like at first glance. Why should it be so difficult to provide a technology that searches plain words and delivers common sense results?