Format

Send to

Choose Destination
Ophthalmology. 2019 Apr;126(4):552-564. doi: 10.1016/j.ophtha.2018.11.016. Epub 2018 Dec 13.

Using a Deep Learning Algorithm and Integrated Gradients Explanation to Assist Grading for Diabetic Retinopathy.

Author information

1
Google Research, Google, LLC, Mountain View, California.
2
Department of Ophthalmology, Palo Alto Medical Foundation, Palo Alto, California.
3
Verily Life Sciences, South San Francisco, California.
4
Department of Ophthalmology, Emory University, Atlanta, Georgia.
5
Ophthalmic Consultants of Boston, Boston, Massachusetts.
6
Magruder Laser Vision, Orlando, Florida.
7
Denver Health Medical Center, Denver, Colorado; Department of Ophthalmology, University of Colorado School of Medicine, Aurora, Colorado.
8
Department of Ophthalmology, Massachusetts Eye and Ear Infirmary, Harvard Medical School, Boston, Massachusetts.
9
Google Research, Google, LLC, Mountain View, California. Electronic address: lhpeng@google.com.

Abstract

PURPOSE:

To understand the impact of deep learning diabetic retinopathy (DR) algorithms on physician readers in computer-assisted settings.

DESIGN:

Evaluation of diagnostic technology.

PARTICIPANTS:

One thousand seven hundred ninety-six retinal fundus images from 1612 diabetic patients.

METHODS:

Ten ophthalmologists (5 general ophthalmologists, 4 retina specialists, 1 retina fellow) read images for DR severity based on the International Clinical Diabetic Retinopathy disease severity scale in each of 3 conditions: unassisted, grades only, or grades plus heatmap. Grades-only assistance comprised a histogram of DR predictions (grades) from a trained deep-learning model. For grades plus heatmap, we additionally showed explanatory heatmaps.

MAIN OUTCOME MEASURES:

For each experiment arm, we computed sensitivity and specificity of each reader and the algorithm for different levels of DR severity against an adjudicated reference standard. We also measured accuracy (exact 5-class level agreement and Cohen's quadratically weighted κ), reader-reported confidence (5-point Likert scale), and grading time.

RESULTS:

Readers graded more accurately with model assistance than without for the grades-only condition (P < 0.001). Grades plus heatmaps improved accuracy for patients with DR (P < 0.001), but reduced accuracy for patients without DR (P = 0.006). Both forms of assistance increased readers' sensitivity moderate-or-worse DR: unassisted: mean, 79.4% [95% confidence interval (CI), 72.3%-86.5%]; grades only: mean, 87.5% [95% CI, 85.1%-89.9%]; grades plus heatmap: mean, 88.7% [95% CI, 84.9%-92.5%] without a corresponding drop in specificity (unassisted: mean, 96.6% [95% CI, 95.9%-97.4%]; grades only: mean, 96.1% [95% CI, 95.5%-96.7%]; grades plus heatmap: mean, 95.5% [95% CI, 94.8%-96.1%]). Algorithmic assistance increased the accuracy of retina specialists above that of the unassisted reader or model alone; and increased grading confidence and grading time across all readers. For most cases, grades plus heatmap was only as effective as grades only. Over the course of the experiment, grading time decreased across all conditions, although most sharply for grades plus heatmap.

CONCLUSIONS:

Deep learning algorithms can improve the accuracy of, and confidence in, DR diagnosis in an assisted read setting. They also may increase grading time, although these effects may be ameliorated with experience.

PMID:
30553900
DOI:
10.1016/j.ophtha.2018.11.016
Free full text

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center