Background
Congenital heart disease (CHD) is a relatively common disorder in childhood, affecting approximately 8–9 per 1000 live-born infants annually in the UK. The crude prevalence of CHD was estimated in 2008 to be 3.05 per 1000 patients registered with UK general practitioners. CHD often involves serious abnormalities and is an important cause of childhood mortality, morbidity and disability. Around one-third of deaths due to CHD occur before 14 years of age. Infancy is the highest risk period, with 21% of CHD deaths occurring in the first year of life, but the proportion of deaths in adults > 45 years is increasing. In 2006, CHD accounted for 3% of child deaths.
It is generally recognised that it is important and valuable to monitor outcomes in cardiac surgery and that, to do so fairly and effectively, one needs to risk stratify the case load of each unit. Risk stratification of adult cardiac surgery patients is an essential part of the audit process, which reduces the prospect of unfair assessment of outcomes attributed to a surgeon or team whose mortality rate is relatively high simply because it reflects patients who were inherently higher risk. There is evidence that, since outcome monitoring in adult cardiac surgery became mandatory and routine, outcomes have improved, and there has been no consequent negative effect in terms of centres turning away high-risk cases, as was originally feared. There were consequent benefits to patients and their families in terms of the quality of care and the improved information they received. Analytical methods for outcome monitoring are well advanced for adult cardiac surgery and the use of graphical techniques such as the variable life-adjusted display (known by the acronym VLAD) to display risk-adjusted outcome charts is now common in the quality assurance process.
At present, no process for routinely monitoring risk-adjusted outcomes in paediatric cardiac surgery exists, and achieving this is clearly desirable. The challenge is due to the great diversity of the patient population in terms of the diagnoses, indications for surgery, operations performed, age at operation and other factors as well as the logistics of co-ordinating such an endeavour across many disparate cardiac centres in a geographical area.
Objectives
Our objectives for this project were
to test an existing risk model based on the Risk Adjustment for Congenital Heart Surgery-1 (RACHS-1) score and patient age, derived from outcomes at one centre (Great Ormond Street Hospital), in the Central Cardiac Audit Database (CCAD) data from all centres across the UK
to understand the contribution that diagnostic information can make to risk estimation and monitoring of outcomes, establish whether or not information concerning comorbidities can contribute to improved methods of risk estimation and, if indicated and possible, revise the existing risk model such that it is suitable for use at other centres and by CCAD
to examine the implications of reporting mortality outcomes by diagnosis as well as by procedure category
to disseminate our findings and any risk models and monitoring tools developed to UK centres and to CCAD, so that it can consider how best to share the information with stakeholders, including through its ‘public portal’ web pages.
Methods
Data
Since 2000, quality assurance in paediatric cardiac surgery in the UK has been underpinned by CCAD, which provides aggregated national and institution-specific data. Mandatory data submissions to CCAD are requested every 3 months from hospitals performing cardiac surgery in the UK, including details about patients and the operations performed. Patient identifiers, including NHS number, are provided to CCAD, which periodically requests information on patients' survival status from the Central Register of NHS patients. This process is approved by the National Information Governance Board for Health and Social Care with consent requested from parents for participation in national audit of outcomes. There is an extensive quality assurance process incorporating a rolling programme of professionally led site visits.
We received a data set of pseudonymised records from CCAD in December 2010. After removing all records in which the patient was > 16 years at the time of the procedure and all records for patients who underwent only catheter procedures, we split the data set into development (70% of patients) and test (30% of patients) data sets using random allocation stratified by year and institution of first procedure for a patient. All further analysis was performed on the development set, which contained 34,385 records, corresponding to 22,449 unique patients. The quarantined test set contained 14,316 records, corresponding to 9354 unique patients, and played no part in risk model development.
Data cleaning
Considerable effort was required to prepare the data set for analysis. This data cleaning process involved identifying and removing duplicate records, identifying and resolving instances in which the pseudonymisation process had resulted in different patients being given the same ID number, and identifying and resolving instances of inconsistent patient details within a sequence of records. The inclusion of the CCAD senior strategist, Dr David Cunningham, on the project team was essential to resolving issues of data quality and interpretation in a timely manner.
Outcome measure
The unit of analysis was a ’30-day episode’. For each patient an episode started with their first surgical procedure. Any further surgical procedures that the same patient underwent within 30 days of this first procedure were not included in the model development. The next surgical procedure recorded for the same patient > 30 days after the first surgical procedure was treated as the start of a new 30-day episode.
The outcome measure for each episode was death within 30 days of the start of that episode.
Candidate risk factors
For each episode there were data fields available providing the specific procedure performed (one of 36 groups defined by CCAD or ‘unassigned’), whether or not the operation was performed on bypass, up to six separate diagnostic codes, data on comorbidity, whether or not the patient had an antenatal diagnosis, patient sex, age and weight, ethnicity and a measure of deprivation. Comorbidities considered included prematurity (< 37 weeks' gestation), Down syndrome, genetic syndromes and clinical constellations of features that constitute a recognised syndrome, congenital structural defects of organs other than the heart and acquired conditions.
Preparation of diagnostic information
There were up to six raw diagnostic codes for each patient record and each raw diagnostic code can take on one of several hundred values. The research team enlisted the support of other clinical experts in developing a new hierarchy of 24 primary cardiac diagnoses based on the raw diagnostic codes available for each episode.
Additionally, we identified within the raw codes those combinations which indicated that the patient had a single functioning ventricle.
Analysis
Candidate risk factors were first assessed in terms of data quality, clinical face validity and univariate association with death within 30 days. Classification and regression tree analysis was used to identify groupings of primary cardiac diagnoses that were associated with death within 30 days.
Model development followed an iterative process involving multiple logistic regression, assessment of model performance, assessment of model stability, clinical discussion and variable simplification. Ultimately, model choice was not a purely statistical consideration. Consideration of uptake by CCAD and centres supported the use of as many specific procedure groups as possible and we also took account of the opportunity presented by the risk model to drive improvements in completeness and data quality concerning diagnosis and comorbidity.
Results
Mortality rates
In the development set there were 25,665 episodes that resulted in survival to 30 days, 693 episodes in which the vital status at 30 days was unknown and 854 episodes that resulted in death within 30 days (mortality 3.2% overall).
The risk model
The final risk model, decided on jointly by the clinical and analytical teams, was a logistic regression model with the following variables:
age (both as a continuous measure and as neonate/infant/child bands)
weight (as a continuous measure)
specific procedure (one of 27 CCAD groups, ‘no specific procedure’ or ‘low-volume specific procedure’)
procedure type (bypass or non-bypass)
broad diagnosis group (low-, medium- or high-risk group)
univentricular heart attribute (indicator variable)
presence/absence of a recorded non-Down syndrome comorbidity
indicator variable for whether an episode occurred pre 2007 or from 2007 onwards.
This last variable was introduced to account for decreasing mortality over time in the development set and to ensure that prospective use of the risk model would provide a means of benchmarking against recent national outcomes.
Evaluation of the risk model
In the test set it was possible to calculate a risk score in 95% of cases. Weight was missing in 4.9% of cases and age in 0.2% of cases. There were also 392 (3.4%) episodes without diagnostic information but these were included in the analysis within the diagnostic category ‘empty/missing diagnosis’. Additionally, the 30-day life status was ‘unknown’ for 226 (2.0%) episodes in the test set. There were differences in the mortality rates associated with individual risk factors between the test set and the development set. In particular, there was a higher mortality rate among neonates in the test set than among those in the development set.
The area under the receiver operating curve (AUC) for the model in the test set across all years is 0.77, similar to the value calculated for the development set (0.78). This indicates good discrimination between groups of patients with high and low mortality. The total number of observed deaths was 335 compared with 329 predicted. There were statistically significant discrepancies between observed and predicted mortality in deciles of predicted risk.
Given the intention for this model to be used prospectively, we also evaluated the model in test episodes since 2007. Here, there was greater discrimination (0.81 vs 0.78), but the model underestimates risk among very high-risk episodes and is, as a result, less accurate overall in these more recent data (128 observed deaths vs 113 predicted post 2007).
Case mix
The distribution of predicted risk in the two data sets is very similar and shows that just over 30% of episodes have a predicted risk of 30-day mortality of ≤ 1% and 80% have a predicted risk of ≤ 4%, but that around 5% of episodes have a predicted risk of > 10%.
Many of the institutions have quite similar profiles of predicted risk but there are some marked differences. For instance, at one institution 12% of episodes have a predicted risk of 30-day mortality of > 10%, whereas at another institution this proportion is < 1%.
Conclusions
A risk model has been developed for monitoring short-term surgical outcomes following paediatric cardiac surgery that, for the first time, makes use of diagnostic information as well as procedural data. The model shows good discrimination between groups of patients with high and low mortality and reasonable accuracy.
Given that the model underestimated risk at the very high-risk end of the recent (since 2007) test data, risk adjustment using the model as currently parameterised will potentially give an unfair assessment of outcomes at those centres with a high proportion of high-risk cases. This is an important caveat to the interpretation of risk-adjusted outcomes within and across institutions that will need to be discussed with CCAD and the institutions as this work is taken forward.
Recommendations for future research
The completion of this project has highlighted a number of research priorities and opened possibilities for future research:
It would be desirable for any implementation of routine monitoring of risk-adjusted outcomes in paediatric cardiac surgery to be piloted and, if appropriate, rolled out and evaluated, with evaluation including qualitative research on drivers and barriers to adoption.
The availability of a risk model makes possible research into differences between case mix across centres nationally and into trends in case mix over time.
The scheme developed as part of this project for the classification of primary cardiac diagnosis based on the codes available within the CCAD data set enables important research to be conducted on the long-term outcomes achieved for groups of surgical CHD patients defined by diagnosis. This would provide valuable information to patients, carers, commissioners and clinicians alike.
Our next steps
One of the co-investigators has, over the course of the project, joined the CCAD steering committee. This has further improved the link between CCAD and the project team, which was already robust given the presence of the senior strategist for the UK cardiac audits as a co-investigator.
This project and future use of the risk model was discussed at the CCAD steering committee meeting in October 2011. The CCAD steering committee has a strong interest in the work given their commitment to audit and quality assurance. CCAD and several centres have expressed an interest in piloting the risk model and the accompanying monitoring tool.
Funding
The National Institute for Health Research Health Services and Delivery Research programme.
