Jeroen Mom

Talking about search in the professional environment, I see that many people do not realize what exactly it is they are after. Terms like enterprise search and federated search often seem to be either new, or at least suffer from unclarity and many prejudices. There seems to a need for clarification about what it is and would fit best in the situation at hand.

One obvious desire when it comes to finding information is to be able to search through all available resources, in parallel and with all results in one combined overview. So with search goes integration. And with integration goes ranking (sorting the results according to relevance in relation to the query at hand).

Now as long as we are talking about company-internal information, one can think of solving the case using enterprise search engines. The basic principle for this kind of solution being the availability of the data to the search engine, so that it can build up its own indexes. This will enable the strongest possible searching as the engine can make use of the maximum amount of content both in indexing and in searching.

When it comes to integrating external information into the search, it gets more exciting. Any information available on the web could be pulled into the engines own reach (also called ‘harvesting’) to be indexed. There are however some important issues to consider. First of all there’s the boundless nature of the web, which may lead to a vast amount of data to be stored and managed in your enterprise environment. Secondly there’s the need for up to date information which requires special attention when pulling external  information in. Thirdly, as most information is not publicly available, one will need to find a solution for accessing what is referred to as the ‘deep web‘.

The issues mentioned above are typically handled by so called ‘Federated Search’ engines. These engines generally connect to many different sources, query them, and gather the search results into one central interface. Major difference with the before mentioned Enterprise Search engines is that in fact these engines do not really search themselves; they query the search engines of the required sources which then do the actual searching. Therefore they do not need to pull the information into an own index, thus preventing duplication of the data. And when it comes to currency of the information, users will always avail of the latest information released on the native publishers site.

An important benefit of Federated Search is the ease of searching multiple sources. This comes via a combination of factors: having one common interface, entering one single query in one common query language, eliminating the need to logon to the various sources, etc. To those already enjoying the fruits of Enterprise Search, there’s nothing spectacular in this… but then a new desire pops up: can we integrate the two?

As I can search internally, and I can also search externally, please integrate the resultlists into one! There is just a need for relevant information. Right here, right now; with one query, in one place. From a professionals point of view there’s no interest in the technical challenges. And rightly so. This is something we should solve as solution developers.

In fact we have come far by just harvesting the necessary information to be integrated into our Enterprise Search environment. The other way also results in a workable solution: to offer access to company internal information in such a way that it can be queried by the Federated Search engine. In the end, however, it shouldn’t be the technical (im)possibilities of the one or the other that set the direction. The optimal solution should be determined by the nature of the resources and the need of the professional.

Therefore, professional search in the near future can no longer be a question of either / or. Searching solutions will combine both internal and external, public and proprietary, structured and unstructured information. They will include both direct searching and federated querying.

If we really want to support our customers, this is one requirement to meet.