Retrieval augmented scientific claim verification
Document Type
Article
Publication Date
4-1-2024
Journal / Book Title
JAMIA Open
Abstract
Objective: To automate scientific claim verification using PubMed abstracts. Materials and Methods: We developed CliVER, an end-to-end scientific Claim VERification system that leverages retrieval-augmented techniques to automatically retrieve relevant clinical trial abstracts, extract pertinent sentences, and use the PICO framework to support or refute a scientific claim. We also created an ensemble of three state-of-the-art deep learning models to classify rationale of support, refute, and neutral. We then constructed CoVERt, a new COVID VERification dataset comprising 15 PICO-encoded drug claims accompanied by 96 manually selected and labeled clinical trial abstracts that either support or refute each claim. We used CoVERt and SciFact (a public scientific claim verification dataset) to assess CliVER's performance in predicting labels. Finally, we compared CliVER to clinicians in the verification of 19 claims from 6 disease domains, using 189 648 PubMed abstracts extracted from January 2010 to October 2021. Results: In the evaluation of label prediction accuracy on CoVERt, CliVER achieved a notable F1 score of 0.92, highlighting the efficacy of the retrieval-augmented models. The ensemble model outperforms each individual state-of-the-art model by an absolute increase from 3% to 11% in the F1 score. Moreover, when compared with four clinicians, CliVER achieved a precision of 79.0% for abstract retrieval, 67.4% for sentence selection, and 63.2% for label prediction, respectively. Conclusion: CliVER demonstrates its early potential to automate scientific claim verification using retrieval-augmented strategies to harness the wealth of clinical trial abstracts in PubMed. Future studies are warranted to further test its clinical utility.
DOI
10.1093/jamiaopen/ooae021
Journal ISSN / Book ISBN
85187374086 (Scopus)
Montclair State University Digital Commons Citation
Liu, Hao; Soroush, Ali; Nestor, Jordan G.; Park, Elizabeth; Idnay, Betina; Fang, Yilu; Pan, Jane; Liao, Stan; Bernard, Marguerite; Peng, Yifan; and Weng, Chunhua, "Retrieval augmented scientific claim verification" (2024). School of Computing Faculty Scholarship and Creative Works. 47.
https://digitalcommons.montclair.edu/computing-facpubs/47
Published Citation
Hao Liu, Ali Soroush, Jordan G Nestor, Elizabeth Park, Betina Idnay, Yilu Fang, Jane Pan, Stan Liao, Marguerite Bernard, Yifan Peng, Chunhua Weng, Retrieval augmented scientific claim verification, JAMIA Open, Volume 7, Issue 1, April 2024, ooae021, https://doi.org/10.1093/jamiaopen/ooae021
Comments
This is an open access article distributed under the terms of the Creative Commons CC BY license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.