Quotation Zlabinger, Markus, Sabou, Reka Marta, Hofstätter, Sebastian, Hanbury, Allan. 2020. Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports. Empirical Methods in Natural Language Processing, In Findings of ACL: EMNLP 2020, Hrsg. EMNLP 2020, 3064-3074. online: None.


RIS


BibTeX

Abstract

The search for Participants, Interventions, and Outcomes (PIO) in clinical trial reports is a critical task in Evidence Based Medicine. For an automatic PIO extraction, high-quality corpora are needed. Obtaining such a corpus from crowdworkers, however, has been shown to be ineffective since (i) workers usually lack domain-specific expertise to conduct the task with sufficient quality, and (ii) the standard approach of annotating entire abstracts of trial reports as one task-instance (i.e. HIT) leads to an uneven distribution in task effort. In this paper, we switch from entire abstract to sentence annotation, referred to as the SEN-BASE approach. We build upon SENBASE in SENSUPPORT, where we compensate the lack of domain-specific expertise of crowdworkers by showing for each task-instance similar sentences that are already annotated by experts. Such tailored task-instance examples are retrieved via unsupervised semantic short-text similarity (SSTS) method – and we evaluate nine methods to find an effective solution for SENSUPPORT. We compute the Cohen’s Kappa agreement between crowd-annotations and gold standard annotations and show that (i) both sentence-based approaches outperform a BASELINE approach where entire abstracts are annotated; (ii) supporting annotators with tailored task-instance examples is the best performing approach with Kappa agreements of 0.78/0.75/0.69 for P, I, and O respectively.

Tags

Press 'enter' for creating the tag

Publication's profile

Status of publication Published
Affiliation External
Type of publication Contribution to conference proceedings
Language English
Title Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports. Empirical Methods in Natural Language Processing,
Title of whole publication Findings of ACL: EMNLP 2020
Editor EMNLP 2020
Page from 3064
Page to 3074
Location online
Year 2020
Open Access Y
Open Access Link https://aclanthology.org/2020.findings-emnlp.274.pdf

Associations

People
Sabou, Reka Marta (Details)
External
Hanbury, Allan (Technical University of Vienna, Austria)
Hofstätter, Sebastian (Technical University of Vienna, Austria)
Zlabinger, Markus (Technical University of Vienna, Austria)
Organization
Institute for Data, Process and Knowledge Management (AE Sabou) (Details)
Research areas (ÖSTAT Classification 'Statistik Austria')
1109 Information and data processing (Details)
1122 Artificial intelligence (Details)
1127 Information science (Details)
1138 Information systems (Details)
1140 Software engineering (Details)
Google Scholar: Search