Show simple item record

dc.contributor.authorRahbariasl, Shahin 13:59:00 (GMT) 13:59:00 (GMT)
dc.description.abstractAssessors who judge the relevance of documents to the search topics and perform the relevance assessment process are one of the main parts of Information Retrieval (IR) sys- tems evaluations. They play a significant role in making test collections which can be used in evaluations and system designs. Relevance assessment is also highly important for e-discovery where relevant documents and materials should be found with acceptable cost and in an efficient way. In order to study the relevance judging behavior of assessors better, we conducted a user study to further examine the effects of time constraints and document excerpts on relevance behavior. Participants were shown either full documents or document excerpts that they had to judge within 15, 30, or 60 seconds time constraint per document. For producing document excerpts or paragraph-long summaries, we have used algorithms to extract what a model of relevance considers most relevant from a full document. We found that the quality of judging slightly differs within each time constraint but not significantly. While time constraints have little effect on the quality of judging, they can increase the judging speed rate of the assessors. We also found that assessors perform as good and in most cases better if shown a paragraph-long document excerpt instead of a full document, therefore, they have the potential to replace full documents in relevance assessment. Since document excerpts are significantly faster to judge, we con- clude that showing document excerpts or summaries to the assessors can lead to better quality of judging with less cost and effort.en
dc.publisherUniversity of Waterlooen
dc.subjectAssessing Behavioren
dc.subjectInformation Retrievalen
dc.subjectData systemsen
dc.subjectTime Constraintsen
dc.subjectDocument Excerptsen
dc.subjectRelevane Assessmenten
dc.titleThe Effects of Time Constraints and Document Excerpts on Relevance Assessing Behavioren
dc.typeMaster Thesisen
dc.pendingfalse R. Cheriton School of Computer Scienceen Scienceen of Waterlooen
uws-etd.degreeMaster of Mathematicsen
uws.contributor.advisorD. Smucker, Mark
uws.contributor.affiliation1Faculty of Mathematicsen

Files in this item


This item appears in the following Collection(s)

Show simple item record


University of Waterloo Library
200 University Avenue West
Waterloo, Ontario, Canada N2L 3G1
519 888 4883

All items in UWSpace are protected by copyright, with all rights reserved.

DSpace software

Service outages