A Method to Assess Search Engine Results

an article by Judit Bar-Ilan (Bar-Ilan University) and Mark Levene (Birkbeck University of London) published in Online Information Review Volume 35 Issue 6

Abstract

Purpose
To develop a methodology for assessing search results retrieved from different sources.
Design/methodology/approach
This is a two-phase method, where in the first stage users select and rank the ten best search results from a randomly ordered set. In the second stage they are asked to choose the best pre-ranked result from a set of possibilities. This two-stage method allows the users to consider each search results separately (in the first stage) and to express their views on the rankings as a whole, as they were retrieved by the search provider. The method was tested in a user study that compared different country-specific search results of Google and Live search (now Bing). The users were Israelis and the search results came from six sources: Google Israel, Google.com, Google UK, Live Search Israel, Live Search US and Live Search UK. The users evaluated the results of nine pre-selected queries, created their own preferred ranking and picked the best ranking from the six sources.
Findings
The results indicate that the group of users in this study liked most the local Google interface, i.e. Google succeeded in it its country-specific customisation of search results. Live.com was much less successful in this aspect.
Research limitations/implications
Search engines are highly dynamic, thus the findings of the case study have to be viewed cautiously.
Originality/value
The main contribution of the paper is a two-phase methodology for comparing and evaluating search results from different sources.

---