Utility of a Clinical Support Tool for Outpatient Evaluation of Pediatric Chest Pain

Abstract

Introduction

Appropriate resource utilization is a key element with respect to successful reform of the US healthcare delivery system^1,2. While appropriate use criteria exists for many medical disciplines, this is not necessarily the case in pediatric cardiology. Clinical pathways have been developed to streamline care and limit variability in physician practices^3,4, with some focusing on common outpatient pediatric cardiology complaints such as chest pain⁵. It is important to note that the vast majority of children seen with chest pain have no underlying cardiac pathology, yet the possibility of serious and life-threatening pathology often leads to expensive testing (i.e., echocardiography). The purpose of this study was to use data mining techniques to evaluate the clinical pathway currently being employed at a large single-center pediatric cardiology practice.

Electronic health records (EHRs) are intended to improve the quality of patient care by providing accurate and complete patient data across healthcare organizations. However EHRs also act as a rich data source for healthcare researchers. Many studies utilize EHRs to identify predicting variables, such as predictors of asthma exacerbations⁶ or immunization delay⁷. This study analyzes EHRs to confirm or reject the clinical pathway’s assumed key predictors of cardiac disease. We supplement this analysis by developing and evaluating a diagnostic aid based on significant predictors.

Method

This study presents a three stage analysis. Stage-one performs univariate analysis to identify subgroups of patients which are more likely to have cardiac disease. Stage-two analysis uses logistic regression to predict whether a patient has cardiac disease while stage-three analysis develops a diagnostic aid to help physicians identify patients potentially at-risk for cardiac disease.

During stage-two analysis, developing a logistic regression model, variables with p-values less than 0.35 in stage-one analysis were included as initial variables in the model. The model was trained using 80% records, and tested with the remaining 20%. This dataset exhibits class imbalance issues which can cause logistic regression to perform poorly. However, oversampling the minority class can considerably improve performance and obtains comparable results to more complex solutions⁸. The minority class was oversampled until the number of majority and minority records were the same. The logistic regression model was trained on the oversampled data. The resulting parameters were used to identify significant predictors in the final model (i.e. factors with p-values greater than 0.05 were removed).

In order to avoid overfitting the data with too few outcome events per predictor variable, a general rule is there should be at least 10 outcome events per independent variable in the prediction rule⁹. As such, if the events-to-predictor ratio was less than 10, the m (where m = # outcome events/10) most significant predictors were kept while the remaining were removed.

Stage-three analysis develops a risk model by assigning a risk score to each significant predictor from the logistic regression model. This value was calculated by dividing the β-coefficient by one-half the sum of the two smallest (absolute sense) coefficients in the model and rounding to the nearest integer¹⁰. The overall risk score was calculated by summing the risk scores from each predictor.

The discrimination of our models was assessed by several metrics. We employed the metrics area under the receiver operating characteristic curve (AUC) and Matthews correlation coefficient (MCC). AUC assesses a classifier’s ability to balance the true positive and false positive rates. An AUC of 1.00 corresponds to perfect discriminatory power while an AUC of 0.50 indicates no ability to discriminate. New research indicates the AUC is a noisy classification measure¹¹. As a result we also included MCC as it considers both true and false positive and negative rates. MCC is considered a more balanced measure than AUC and can also be used when classes are imbalanced. A MCC of 1.00 indicates perfect discrimination while a MCC 0.00 indicates no better than random classification.

Dataset

The dataset contained anonymous records of 1,997 pediatric patients seen in one of 17 outpatient clinics (11 permanent and 6 outreach). The permanent clinics are typically accessible 5 days per week and mainly serve metropolitan areas. The outreach clinics accessibility varies from one day every other week to an average of 1.5 days per week, and typically serve more rural areas in a single state. The clinical protocol was developed within the pediatric cardiology practice utilizing available best practices, local clinical expertise and group consensus. The clinical evaluation consists of the following three components: a) patient questionnaire and family history completed prior to seeing physician and again reviewed with physician, b) comprehensive physical examination, and c) 12-lead electrocardiogram (EKG). If no ”red flags” are raised or noted with a, b, and c, then the evaluation is complete and the patient is diagnosed with non-cardiac chest pain. If red flags are raised the patient is referred for further testing at the discretion of the treating cardiologist. Any deviations from the pathway must be documented and explained by the treating physician.

As a result of this procedure each record is composed of five components: 1) clinic and attending physician identifiers, 2) results of the physical exam, 3) responses to the patient and family history questionnaires, 4) outcomes of diagnostic testing, and 5) ICD-9 codes for diagnoses.

The physical exam records the following values 1) height, 2) weight, 3) BMI, 4) respiratory rate, 5) heart rate, 6) blood pressure, and 7) physical exam outcome. The outcome of the physical exam is a free text field written by the physician. The patient and family questionnaires comprise the majority of the patient record. The patient questionnaire consists of over 10 questions regarding 1) where the site of pain occurs, 2) how long the pains persists, 3) how often the pain occurs, 4) when the patient first noticed the pain, 5) how intense is the pain is, 6) what symptoms best describes the pain, and 7) what actions increase the pain. Each question, except concerning the site of pain, provides a set of fixed responses to which the patient indicates which responses most accurately describe his/her condition. The family history questionnaire consists of 22 questions regarding family history of cardiac disease, seizure, sudden death, etc. All questions require a yes/no response, with the exception of a single field provided for additional relevant information.

The ”red flags” on the patient questionnaire are as follows: 1) chest pain described as ”pressure”, 2) duration of chest pain is less than 48 hours or 2–7 days, 3) chest pain occurring during exertion/exercise, 4) chest pain made worse with exercise or lying down, 5) associated palpitations, lightheadedness, syncope or nausea, and 6) past history of rheumatic fever or Kawasaki disease. The ”red flags” on the clinical and family history are as follows: 1) ”has your child passed out during or after exercise, emotion, or startle?”, 2) ”has your child ever had extreme shortness of breath during exercise?”, 3)”has your child had extreme fatigue associated with exercise (different from other children)?”, 4) ”has your child ever had discomfort, pain or pressure in his chest during exercise?”, 5) ”are there any family members who died suddenly of heart problems before age 50?”, 6) ”are there family members who have had unexplained fainting or seizures?”, and 7) ”are there any relatives with an enlarged heart (hypertrophic or dilated cardiomyopathy)?”. These ”red flags” do not necessarily mean the child has cardiac chest pain, rather signify that these patients might require a more intensive evaluation to rule-out cardiac disease. From the patient questionnaire most experts would agree that exertional chest pain and associated syncope (passing out) and/or palpitations (potential heart rhythm abnormality) could signal significant cardiac disease.

The questionnaires were provided to patients in paper format and were later transcribed as EHRs. Each question was coded in a systematic method with the exception of free response fields. We assume the errors induced during transcription are minimal as free response fields were limited. However, the accuracy of free response questions may be questionable; especially concerning the site of maximal pain which asks patients to indicate on a diagram where the pain occurs. Each diagnostic test performed included its CPT code and a text field describing the outcome. Diagnoses were encoded as both an ICD-9 code and a text value. The vast majority (86.7%) of patients were assigned at least one of over 80 unique diagnoses in the dataset.

In addition to the patient records, we were provided each physician’s age, each clinic’s status (either permanent or outreach), and the diagnosis severity (no, minor, or major cardiac disease). The diagnosis severity was determined by a panel composed of physicians from this practice. Each physician independently reviewed the diagnoses and assigned them one of the three possible outcomes. If the reviewers’ selections were unanimous, the action was considered valid. However if the reviewers were not unanimous in their selection, discussion followed and a selection was made as a panel. The panel identified 20 diagnoses as major cardiac disease, 14 as minor, and the remainder as no cardiac disease.

Before developing a support tool, the data set was cleaned. Patients who had not completed the questionnaires were removed from the data set, leaving 1,888 remaining records. The question ”Does chest pain occur during exercise, rest, or both?” had 43 missing values while the question ”what is the severity of the pain (1–10)?” had 88 missing values. The regression method presented above was used to estimate the missing values.

Results

Before stage one analysis was performed missing entries in the data were estimated. Our regression models included the variables from the patient and family history questionnaires and the outcomes of the physical exam and EKG. These variables compose the results of the standard clinical procedure. The regression model to predict if pain occurred during exercise, rest, or both achieved AUCs of 0.871, 0.864, and 0.771 respectively. These results indicated the classifiers had good discriminatory power and were used to predict the 43 missing values. However, the linear regression model to predict the severity of pain had poor performance (RMSE = 2.542). As such, the question was removed from the dataset. The question concerning the site of pain was also removed as there was no systematic method to interpret the response.

For this analysis, positive cardiac disease is identified as both major and minor cardiac disease. The univariate analysis of the questionnaires identified 16 significant (p-value ≤ 0.05) questions. The set of questions concerning what symptoms co-occur with chest pain was the only question set with all the questions identified as significant. The set of questions concerning family history was the only question set with no significant differences. The univariate analysis of the patient symptom questionnaire is found in Table 3 while the analysis of the family history questionnaire is found in Table 4. A positive physical exam had an odds ratio 0.84 (0.66–1.07 95% CI) and a p-value of 0.15. A positive EKG had an odds ratio of 1.82 (1.40–2.34 95% CI) and a p-value < 0.01.

During stage-two analysis, developing a logistic regression model for cardiac disease (major or minor), we included all variables in stage-one analysis with p-values less than 0.35 as initial explanatory variables. At the end of the stepwise logistic regression model, there were 15 variables with p-values less than 0.05. These variables, along with their corresponding coefficients and odds ratios, are included in Table 5. The most significant predictors within the model were syncope (odds ratio = 2.93) and palpitations (odds ratio= 3.52). The model achieved an AUC = 0.726 indicating moderate discriminatory power (Table 6).

As part of stage-three analysis, each significant predictor from stage two was assigned a risk score by dividing its β-coefficient by one-half the sum of the two smallest (absolute sense) coefficients in the model and rounding to the nearest integer¹⁰. These risk scores formulate a diagnostic aid by assigning a patient a cumulative risk score, the sum of the risk scores from each predictor. The diagnostic aid had a sensitivity of 0.50 and specificity of 0.65. The cut-off point was selected where the sum of the sensitivity and specificity was the largest. The likelihood of cardiac disease increased with the cumulative risk score as shown in Table 6.

When comparing the ”red flags” in the clinical protocol with the univariate analysis of cardiac disease (major or minor), nine of the 20 ”red flags” are considered significant. Seven of these significant variables agree with the ”red flag” interpretation. However two significant variables, 1) pain occurs with exercise and 2) exercise increases pain, contradict the ”red flag” interpretation. They indicate pain with exercising is correlated with no cardiac disease. The regression model agrees with the ”red flags”, onset of pain within the past 2–7 days, syncope, and palpitations.

Stage one through three analysis was repeated, however cardiac disease was defined as only major cardiac disease. The univariate analysis for major cardiac disease is not reported in this study. However stage-two results are found in Table 7. Ten variables were significant predictors of major cardiac disease with the strongest predictor being a positive EKG. However, this model had an AUC = 0.60 (Table 8) indicating very poor discriminatory power. Risk scores were also assigned to each predictor in a similar manner. The major cardiac disease diagnostic aid had a sensitivity of 0.87 but only 0.39 for specificity (Table 8). This model has sufficiently poor performance that extreme caution should be taken when interpreting its results.

Discussion

Evaluating physician and pathway performance is essential to quality assurance and has long been a central issue in medicine. While many studies have focused on analyzing physician performance^12,13,14 and the impact of continuing medical education¹⁵, our study focuses on evaluating the performance of a clinical pathway in a large single-center pediatric cardiology practice. In addition to evaluating the pathway, we developed a diagnostic aid for cardiac pathology. However, the regression model for major cardiac disease had sufficiently poor performance that no confidence should be taken in interpreting its results.

Given the low incidence of major cardiac disease in this population, it is not surprising that our analysis did not identify a single predictor of major cardiac pathology with the current clinical pathway. However, there were a number of clinical and historical data elements that favored a presence or absence of minor cardiac pathology. This data should be evaluated with caution as some of the results were contrary to what most clinical cardiologists would view as significant/pertinent positives. For example, this analysis revealed a negative correlation of exertional chest pain with minor cardiac disease, while most would agree that these symptoms are often the most concerning. One explanation for this anomaly would be that children are not often reliable historians, thus relying too much on their description of the events might lead one to an incorrect conclusion. It is also possible that exertional chest pain is negatively associated with minor cardiac disease but positively associated with major cardiac disease. These results may be affected by a low incidence of cardiac pathology in this subset which makes interpretation difficult at this time. Although the panel labelled diagnoses as major or minor cardiac disease, they did feel that the minor diagnoses were not necessarily related to contributing to the chest pain. Our hope is that the current model can be applied to an increasing number of patients and provide more robust results/analysis.

Despite these limitations, our results did confirm a number of ”red flags” utilized in the clinical pathway, such as syncope, palpitations, and having noticed pain for 2–7 days. This study also raises potential ”red flags”, such as pain occurring daily, which should be further studied. This study highlights the difficulty in drawing significant conclusions primarily from patient and family history. However, empirical analysis of patient symptoms can lead to improved clinical pathways and patient outcomes.

References