Optimising Performance of Competing Search Engines in Heterogeneous Web Environments

Abstract

Distributed heterogeneous search environments are an emerging phenomenon in Web search, in which topic-specific search engines provide search services, and metasearchers distribute user’s queries to only the most suitable search engines. Previous research has explored the performance of such environments from the user’s perspective (e.g., improved quality of search results). We focus instead on performance from the search service provider’s point of view (e.g, income from queries processed vs. resources used to answer them). We analyse a scenario in which individual search engines compete for queries by choosing which documents to index. We propose the COUGAR algorithm that specialised search engines can use to decide which documents to index on each particular topic. COUGAR is based on a game-theoretic analysis of heterogeneous search environments, and uses reinforcement learning techniques to exploit the sub-optimal behaviour of its competitors.