Format

Send to

Choose Destination
Acad Radiol. 2019 Nov 6. pii: S1076-6332(19)30488-X. doi: 10.1016/j.acra.2019.10.006. [Epub ahead of print]

Exploring Large-scale Public Medical Image Datasets.

Author information

1
Australian Institute for Machine Learning, North Terrace, Adelaide, Australia; School of Public Health, University of Adelaide, North Terrace, Adelaide 5000, Australia; Royal Adelaide Hospital, North Terrace, Adelaide, Australia. Electronic address: luke.oakden-rayner@adelaide.edu.au.

Abstract

RATIONALE AND OBJECTIVES:

Medical artificial intelligence systems are dependent on well characterized large-scale datasets. Recently released public datasets have been of great interest to the field, but pose specific challenges due to the disconnect they cause between data generation and data usage, potentially limiting the utility of these datasets.

MATERIALS AND METHODS:

We visually explore two large public datasets, to determine how accurate the provided labels are and whether other subtle problems exist. The ChestXray14 dataset contains 112,120 frontal chest films, and the Musculoskeletal Radiology (MURA) dataset contains 40,561 upper limb radiographs. A subset of around 700 images from both datasets was reviewed by a board-certified radiologist, and the quality of the original labels was determined.

RESULTS:

The ChestXray14 labels did not accurately reflect the visual content of the images, with positive predictive values mostly between 10% and 30% lower than the values presented in the original documentation. There were other significant problems, with examples of hidden stratification and label disambiguation failure. The MURA labels were more accurate, but the original normal/abnormal labels were inaccurate for the subset of cases with degenerative joint disease, with a sensitivity of 60% and a specificity of 82%.

CONCLUSION:

Visual inspection of images is a necessary component of understanding large image datasets. We recommend that teams producing public datasets should perform this important quality control procedure and include a thorough description of their findings, along with an explanation of the data generating procedures and labeling rules, in the documentation for their datasets.

KEYWORDS:

Artificial intelligence; dataset; deep learning; exploratory analysis; quality control

PMID:
31706792
DOI:
10.1016/j.acra.2019.10.006

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center