Harnessing Event Report Data to Identify Diagnostic Error During the COVID-19 Pandemic

Introduction: COVID-19 exposed systemic gaps with increased potential for diagnostic error. This project implemented a new approach leveraging electronic safety reporting to identify and categorize diagnostic errors during the pandemic.

Methods: All safety event reports from March 1, 2020, to February 28, 2021, at an academic medical center were evaluated using two complementary pathways (Pathway 1: all reports with explicit mention of COVID-19; Pathway 2: all reports without explicit mention of COVID-19 where natural language processing [NLP] plus logic-based stratification was applied to identify potential cases). Cases were evaluated by manual review to identify diagnostic error/delay and categorize error type using a recently proposed classification framework of eight categories of pandemic-related diagnostic errors.

Results: A total of 14,230 reports were included, with 95 (0.7%) identified as cases of diagnostic error/delay. Pathway 1 (n = 1,780 eligible reports) yielded 45 reports with diagnostic error/delay (positive predictive value [PPV] = 2.5%), of which 35.6% (16/45) were attributed to pandemic-related strain. In Pathway 2, the NLP-based algorithm flagged 110 safety reports for manual review from 12,450 eligible reports. Of these, 50 reports had diagnostic error/delay (PPV = 45.5%); 94.0% (47/50) were related to strain. Errors from all eight categories of the taxonomy were found on analysis.

Conclusion: An event reporting-based strategy including use of simple-NLP-identified COVID-19-related diagnostic errors/delays uncovered several safety concerns related to COVID-19. An NLP-based approach can complement traditional reporting and be used as a just-in-time monitoring system to enable early detection of emerging risks from large volumes of safety reports.