Format

Send to

Choose Destination
Environ Int. 2020 Mar 20;138:105623. doi: 10.1016/j.envint.2020.105623. [Epub ahead of print]

SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimation.

Author information

1
Sciome LLC, 2 Davis Drive Durham, NC 27709, USA. Electronic address: brian.howard@sciome.com.
2
Sciome LLC, 2 Davis Drive Durham, NC 27709, USA.
3
Integrated Risk Information System (IRIS) Division, Environmental Protection Agency, 109 T.W. Alexander Drive RTP, NC 27709, USA.
4
National Toxicology Program (NTP)/National Institute of Environmental Health Sciences (NIEHS), 111 T.W. Alexander Drive RTP, NC 27709, USA.

Abstract

BACKGROUND:

In the screening phase of systematic review, researchers use detailed inclusion/exclusion criteria to decide whether each article in a set of candidate articles is relevant to the research question under consideration. A typical review may require screening thousands or tens of thousands of articles in and can utilize hundreds of person-hours of labor.

METHODS:

Here we introduce SWIFT-Active Screener, a web-based, collaborative systematic review software application, designed to reduce the overall screening burden required during this resource-intensive phase of the review process. To prioritize articles for review, SWIFT-Active Screener uses active learning, a type of machine learning that incorporates user feedback during screening. Meanwhile, a negative binomial model is employed to estimate the number of relevant articles remaining in the unscreened document list. Using a simulation involving 26 diverse systematic review datasets that were previously screened by reviewers, we evaluated both the document prioritization and recall estimation methods.

RESULTS:

On average, 95% of the relevant articles were identified after screening only 40% of the total reference list. In the 5 document sets with 5,000 or more references, 95% recall was achieved after screening only 34% of the available references, on average. Furthermore, the recall estimator we have proposed provides a useful, conservative estimate of the percentage of relevant documents identified during the screening process.

CONCLUSION:

SWIFT-Active Screener can result in significant time savings compared to traditional screening and the savings are increased for larger project sizes. Moreover, the integration of explicit recall estimation during screening solves an important challenge faced by all machine learning systems for document screening: when to stop screening a prioritized reference list. The software is currently available in the form of a multi-user, collaborative, online web application.

KEYWORDS:

Active learning; Document screening; Evidence mapping; Machine learning; Recall estimation; Systematic review

PMID:
32203803
DOI:
10.1016/j.envint.2020.105623
Free full text

Conflict of interest statement

Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center