Send to

Choose Destination
NPJ Digit Med. 2019 Nov 18;2:111. doi: 10.1038/s41746-019-0189-7. eCollection 2019.

Human-machine partnership with artificial intelligence for chest radiograph diagnosis.

Author information

1Department of Radiology, Stanford University School of Medicine, 300 Pasteur Dr., H1307, Stanford, CA 94305 USA.
Unanimous AI, 2443 Fillmore Street #116, San Francisco, CA 94115-1814 USA.
3Department of Computer Science, Stanford University School of Medicine, 353 Serra Mall (Gates Building), Stanford, CA 94305 USA.
4Department of Radiology, Duke University Medical Center, Box 3808 Erwin Rd, Durham, NC 27710 USA.


Human-in-the-loop (HITL) AI may enable an ideal symbiosis of human experts and AI models, harnessing the advantages of both while at the same time overcoming their respective limitations. The purpose of this study was to investigate a novel collective intelligence technology designed to amplify the diagnostic accuracy of networked human groups by forming real-time systems modeled on biological swarms. Using small groups of radiologists, the swarm-based technology was applied to the diagnosis of pneumonia on chest radiographs and compared against human experts alone, as well as two state-of-the-art deep learning AI models. Our work demonstrates that both the swarm-based technology and deep-learning technology achieved superior diagnostic accuracy than the human experts alone. Our work further demonstrates that when used in combination, the swarm-based technology and deep-learning technology outperformed either method alone. The superior diagnostic accuracy of the combined HITL AI solution compared to radiologists and AI alone has broad implications for the surging clinical AI deployment and implementation strategies in future practice.


Computer science; Radiography

Conflict of interest statement

Competing interestsThe authors had control of the data and the information submitted for publication. Four authors (L.R., D.B., G.W. and M.L.) are employees of Unanimous AI, who developed the swarm platform used in this study. All other authors are not employees of or consultants for Unanimous AI and had control of the study methodology, data analysis, and results. There was no industry support specifically for this study. This study was supported in part by NSF through Award ID 1840937.

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center