Format

Send to

Choose Destination
J Am Acad Dermatol. 2018 Feb;78(2):270-277.e1. doi: 10.1016/j.jaad.2017.08.016. Epub 2017 Sep 29.

Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images.

Author information

1
Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, New York.
2
IBM Research Division, Thomas J. Watson Research Center, Yorktown Heights, New York.
3
Departments of Neurology, Psychiatry, and Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
4
Kitware Inc, Clifton Park, New York.
5
Stoecker & Associates, Rolla, Missouri.
6
Melanoma Unit, Department of Dermatology, Hospital Clinic, Institut d'Investigacions Biomèdiques August Pi i Sunyer, CIBER de Enfermedades Raras, Instituto de Salud Carlos III, University of Barcelona, Barcelona, Spain.
7
Department of Computer Science, University of Central Arkansas, Conway, Arkansas.
8
Dermatology Service, Aurora Centro Especializado en Cáncer de Piel, Medellín, Colombia; Department of Dermatology and Cutaneous Surgery, University of Miami Miller School of Medicine, Miami, Florida.
9
Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, New York; Department of Dermatology, Sheba Medical Center, Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel.
10
Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, New York. Electronic address: halperna@mskcc.org.

Abstract

BACKGROUND:

Computer vision may aid in melanoma detection.

OBJECTIVE:

We sought to compare melanoma diagnostic accuracy of computer algorithms to dermatologists using dermoscopic images.

METHODS:

We conducted a cross-sectional study using 100 randomly selected dermoscopic images (50 melanomas, 44 nevi, and 6 lentigines) from an international computer vision melanoma challenge dataset (n = 379), along with individual algorithm results from 25 teams. We used 5 methods (nonlearned and machine learning) to combine individual automated predictions into "fusion" algorithms. In a companion study, 8 dermatologists classified the lesions in the 100 images as either benign or malignant.

RESULTS:

The average sensitivity and specificity of dermatologists in classification was 82% and 59%. At 82% sensitivity, dermatologist specificity was similar to the top challenge algorithm (59% vs. 62%, P = .68) but lower than the best-performing fusion algorithm (59% vs. 76%, P = .02). Receiver operating characteristic area of the top fusion algorithm was greater than the mean receiver operating characteristic area of dermatologists (0.86 vs. 0.71, P = .001).

LIMITATIONS:

The dataset lacked the full spectrum of skin lesions encountered in clinical practice, particularly banal lesions. Readers and algorithms were not provided clinical data (eg, age or lesion history/symptoms). Results obtained using our study design cannot be extrapolated to clinical practice.

CONCLUSION:

Deep learning computer vision systems classified melanoma dermoscopy images with accuracy that exceeded some but not all dermatologists.

KEYWORDS:

International Skin Imaging Collaboration; International Symposium on Biomedical Imaging; computer algorithm; computer vision; dermatologist; machine learning; melanoma; reader study; skin cancer

PMID:
28969863
PMCID:
PMC5768444
DOI:
10.1016/j.jaad.2017.08.016
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Elsevier Science Icon for PubMed Central
Loading ...
Support Center