Search and ye shall find - the future of search

In the year 2008, there is one 80-pound-gorilla in the corner in the world of search, and its name is Google. FY2007 revenues were $16.6BN with net income of $4BN. In any industry, and especially one as R&D- (and hence expense-) intensive as technology and search, net margins of 24.1% is nothing to laugh it. Microsoft had slightly higher consolidated net margins (29%), while Cisco had only 21%, and both are much more mature businesses, having been around a lot longer than baby Google.

Several factors combine to create great opportunities in search:

1. Blinded by the size: It is hard to remember that Google didn't really exist a decade ago, and most people searched, if at all, using Yahoo or similar. Given that history, Google's invincibility should be taken carefully. No company is perfect or immune to a driven upstart, and larger companies tend to be more bureaucratic and slower-moving. More and more reports have been coming out of Silicon Valley disparaging some of Google's internal issues, including poor hiring practices, bad politics and even internecine warfare. Personally, I take no pleasure in these - Google is a good company that did much good in and to the Internet, despite its warts - except insofar as it indicates that Google is going through some of the maturing growth that all companies go through. What it further indicates is that Google is becoming a company like any other - probably a lot of fun to work at, good perks, but quickly losing some of the hypercompetitive edge that and nimbleness that allows it to lead the pack.  Google is big, but there are serious chinks in the armour.
2. Speed of light: Technology evolves very rapidly, and what was great yesterday is basic expectation today. As a colleague I was working with today described, 5 years ago chat/messaging on a dating site was an extra; today no dating site can survive without it. It is simply the base level required. Put in other terms, the bar keeps getting higher. Google has done an excellent job on making its infrastructure faster, its index larger, its crawling better, and its PageRank algorithm truer. It has also been aggressive in adoption of new technologies or paradigms, especially some of the Ajax work, to make its interfaces better. Nonetheless, at its heart, Google relies on its index of keywords to pages. In 1995, when Sergey Brin and Larry Page were working on BackRub, this was the wave of the future, since the alternative was either no search, one driven by paid placing, manual search or manual placement. Clearly this system is much better. However, as users have gotten used to the abilities of index-based searching, they have come to demand more, that it more closely reflect human interaction. You do not ask your HR department, "2007 holiday schedule officer rank seattle." Rather, you say, "what is the 2007 holiday schedule for officers who are based on seattle but rotate around the country supporting sales?" Anyone who grew up on Star Trek in the 1960s or 1970s cannot forget the crew talking to the female-sounding computer in normal, human language. Even back then, they got it.
3.  The Content Monster: The amount of content on the Internet, let alone corporate Intranets or the so-called "Deep Web," has grown exponentially, or probably more. What provides reasonable answers for 1MM pages of fairly narrow reliability does not necessarily do so for 1BN or 1TN. This is completely separate of the operational question of having a good enough engine to both index these pages and provide responses to queries in user-acceptable time. Even given those (which are not insignificant challenges), the results can become poor as too much irrelevant or unreliable content overwhelms the desired results. In engineering terms, this is known as having a low and dropping signal-to-noise ratio.

Given the above - inherent weakness of the dominant player, rising expectations, improved technology and overloaded content - there is plenty of room for several small, nimble competitors today to enter the search space and make meaningful advances. Their success will largely depend upon:

1. How well they really understand user requirements. They must meet and exceed user expectations, but not to a point that users cannot appreciate it. This is the classical engineer's problem: engineering a solution too far ahead of what people want.
2. How well they manage their operations. Too many companies, especially small start-ups, underestimate what it takes to grow, especially in the technology space, where operations costs can be huge, and the cost of failure of either results or availability on the long-term can be extraordinarily painful, sometimes fatal.  
3. How well they market. Anyone in marketing (including the author) will tell you that acquiring consumers is a very expensive proposition. Most wireless carriers measure the cost of acquiring a customer and the cost of losing one to the penny. The "churn rate", akin to employee turnover rate but as applied to wireless customers, is measured and followed rigidly (at least the better run ones, if such a thing exists in the United States). The newer players must account for customer acquisition costs. 
4. Business-model validity. Google built itself up on advertising. The advertising market is shrinking due to the current perceived economy and diffusing across the proliferation of publishers, making building a large business on advertising harder.
5. Investor patience. It takes a long time to build up a solid profitable consumer-oriented search business. Most investors, especially in Silicon Valley and the tech sector in general, lack the patience to wait for these models to play out.
6. How well Google responds. Notice this is last. Google will likely be around for a long time. Whether or not they can be nimble enough to respond to this depends on the company, its culture, its structure, and its ability to step outside itself. This is akin to the music labels in Part II of my analysis of the music industry.

Most new search entrants are focused on the semantic search space. Semantic search depends not on keywords, at least not directly, but on really understanding what a query means. I recently interviewed the founder & CEO of one of the leading firms in this space, hakia. The interview was on behalf of ArnoldIT, one of the top experts in search. The owner's blog is listed in the blogroll, at right, and the interview itself is here. I will review hakia and its interview in a follow-up article.