Archive for October 20th, 2004

Yesterday, I went to a presentation by David Harper of the Robert Gordon University in Aberdeen on ProfileSkim with its “Relevance Profile Meter that displays the relevance profile across the document and a document browser that supports highlighting of the query words.”
Best to see/read the developers’ description of the project which really does look like an innovative way to present search results once a document has been retrieved.
Here’s how David described his talk:

Query-Based Document Skimming: A User-Centred Evaluation of Relevance

We present a novel within-document retrieval tool, ProfileSkim. The tool computes a relevance profile for a single document and query, and presents the profile in the form of an interactive relevance profile meter. Using this meter, a user can identify and visit relevant sections of a (long) document. ProfileSkim has been integrated as a plug-in within Adobe Acrobat Reader and Professional, and we will demonstrate this plug-in.

Much of the seminar will focus on a user-centred evaluation of ProfileSkim. ProfileSkim is compared against a document skimming tool based on a sequential “find” function, FindSkim. A novel simulated work task was devised, where experiment participants are asked to identify (index) relevant pages of an electronic book, given subjects from the existing book index. This subject index provides the ground truth, against which the indexing results can be compared. Our major hypothesis was confirmed, namely ProfileSkim proved significantly more efficient than FindSkim, as measured by time for task. Moreover, indexing task effectiveness, measured by typical IR measures, demonstrated that ProfileSkim was better than FindSkim in identifying relevant pages, and in the case of precision, almost significantly so. The experiments confirm the potential of relevance profiling for improving query-based document skimming, which should prove highly beneficial for users trying to identify relevant information within long documents.

Related Paper

Harper, D. J., Koychev, I., Sun, Y., and Pirie, I. (2004). Within-document retrieval: A user-centred evaluation of relevance profiling. Information Retrieval, 7, 265-290.

My question though is how to test the efficiency of a method or tool which in competely unknown to the subject against a method or tool that is very familiar — in this case highlighted search terms within the document? When presented with something novel, like ProfileSkim, subjects (users/people) tend to explore the new tool or method for quite a while before becoming comfortable in using it. Yes you can look for changes (deltas) but for how long? Is there a good or accepted way to measure technology adaptation by a subject? (I’ll take either ;->)

Comments No Comments »