Predictive performance of critical illness scores and procalcitonin in sepsis caused by different gram-stain bacteria

OBJECTIVES: To compare the early and late predictive values of several critical illness scores (CISs) and biomarkers in sepsis-3 patients with bloodstream infections (BSIs) and to identify the prognostic value of procalcitonin (PCT) for different gram-stain bacteria infections. METHODS: Patients with at least one positive blood culture within 24h of emergency department admission and with a final diagnosis of sepsis/septic shock were enrolled. CISs were calculated based on the first parameters on the day of admission. The receiver operating characteristics curve was used to analyze the predictive value of CISs and biomarkers for early and late mortality. RESULTS: Of 834 enrolled patients with sepsis-3, death occurred in 214 patients within 28 days and in 273 patients within 60 days. Compared with biomarkers, CISs showed a significantly higher area under the curve (AUC) in the prediction of early and late mortality (p<0.01), especially for patients with GNB infection. The Sequential Organ Failure Assessment score showed a higher AUC for predicting early mortality than the Mortality in Emergency Department Sepsis score (p=0.036). Compared with GNB infections, the AUC values of the PCT for gram-positive bacteria (GPB) infections were higher for predicting early or late mortality; PCT showed higher AUC than high-sensitivity C-reactive protein and white blood cells for predicting early mortality (p<0.05). CONCLUSIONS: CISs were more advantageous in the assessment of early and late prognosis, especially for patients with GNB infections; however, for sepsis with GPB infection, PCT can be used for the prediction of early mortality.


' INTRODUCTION
Sepsis is one of the leading causes of critical illness and mortality worldwide (1)(2)(3) and has been increasing annually by 8%-13% over the past decade, primarily due to the increase in invasive procedures, immunosuppressive drugs, chemotherapy, and transplantation (4). The emergency department (ED), the main healthcare segment treating critically ill patients, requires effective strategies to identify patients at high risk for death, either through a readily available laboratory parameter or a clinical decision rule that can be assessed rapidly. However, it has been proven difficult to develop clinical and laboratory criteria that accurately predict the risk for mortality. Several potential scores for timely diagnosis, risk stratification, and evaluation of the prognosis of sepsis in the ED and intensive care unit (ICU) have come into focus in the last decade (5)(6)(7)(8). The Mortality in Emergency Department Sepsis (MEDS) score (Supplemental File 1) (9) incorporates available information in the ED to aid clinicians in accurately assessing a patient's disease severity and mortality risk for ED presentation (10,11). The Sequential Organ Failure Assessment (SOFA) score (Supplemental File 2) (12) uses clinical and laboratory variables to rank the severity of organ failure and is associated with mortality in ED patients hospitalized with sepsis (13,14). Acute Physiology and Chronic Health Evaluation II (APACHE-II) is now the most widely used severity-of-illness score worldwide; it is used to predict the mortality rate of critically ill patients admitted to the ICU. The logistic organ dysfunction system (LODS) score (Supplemental File 3) (15) has also been proven to be valid for predicting ICU mortality in critically ill patients (6,16).
Sepsis management guidelines recognize the value of biomarkers in providing supplemental data that can support clinical assessment and treatment decisions (17). A few nonspecific biomarkers are established markers for the prognostic evaluation of patients with sepsis. Procalcitonin (PCP), a peptide precursor of the hormone, calcitonin, has been shown to be an important reference marker for infection (18,19). C-reactive protein (CRP) is an acute-phase reactive protein that can interact with capsule C polysaccharides of Streptococcus pneumoniae. Previous studies have shown that PCT and CRP concentrations are related to the prognosis of patients with sepsis (20,21). The use of such parameters may improve the accuracy of infection prognosis.
Given that few studies have compared the various outcome-prediction scoring systems and biomarkers for sepsis and septic shock patients according to the sepsis-3 criteria established by the Society of Critical Care Medicine/ European Society of Intensive Care Medicine task force (22), the objective of this work was to identify the critical illness scores (CISs) and biomarkers that better predict sepsis-related mortality by comparing predictive validity for early (28-day) and late (29-to 60-day) mortality among patients with bloodstream infections (BSIs) in the ED. Further analysis was performed to identify the prognostic value of PCT for different gram-stain bacteria infections.

Data source
We reviewed the medical records of patients aged 418 years who had visited our ED with infection-related disease and who underwent at least one positive blood culture and laboratory tests, including PCT and high-sensitivity CRP (hs-CRP), within 24h of ED arrival. For patients with a mean arterial blood pressure (MAP) o65 mmHg or lactate concentration X2 mmol/L on ED admission, MAP and lactate concentration after fluid resuscitation were also collected to diagnose septic shock. The exclusion criteria were age o18 years, positive blood culture after 24h of ED arrival, repeated positive blood culture at one episode, missing data, contamination, or loss to follow-up.

Data collection
As previously described (23), patients' basic information, including age, sex, comorbidities, vital signs, laboratory results, imaging results, vasopressor use, and urine output, was obtained from the database. Only parameters that appeared for the first time after ED arrival were used. SOFA, LODS, MEDS, and APACHE II scores were calculated to identify patients at high risk for poor outcomes from the individual data elements collected within 24h after ED arrival. The primary outcomes were early (28-day) and late (29-to 60-day) all-cause mortality.

PCT levels and blood cultures
Plasma PCT assessment is usually performed in the ED in patients with clinically suspected sepsis. As previously described (23), serum PCT levels were measured using an automatic analyzer (Vidas B.R.A.H.M.S.; bioMérieux, Marcy l'Etoile, France), according to the manufacturer's instructions. The lower detection limit of the assay was 0.05 ng/mL, and the assay sensitivity was 0.09 ng/mL. According to well-established methods (32), for each sample, an aliquot of 5-10 mL whole blood was inoculated into Bactect aerobic and anaerobic bottles (Becton Dickinson, Sparks, MD, USA) that were then incubated in a Bactect FX automated blood culture system (Becton Dickinson). Aliquots were removed from positive cultures for Gram staining and streaked on a solid medium for subsequent analysis. Microorganisms were identified using conventional methods and matrix-assisted laser desorption/ionization-time-of-flight mass spectrometry (Bruker Daltonik, Bremen, Germany).

Pathogen identification
As previously described (23), microorganisms detected in blood cultures (BCs) are considered clinically relevant pathogens rather than contaminants if they meet the following conditions: (1) detection of two or more BCs and reported by the clinician as the cause of the infection episode; (2) detection in one set of BCs but consistent with the results of cultured samples from suspected infectious foci collected from the same patient during the same infectious episode; (3) detection in one set of BCs of a species included among the etiopathogenic agents of the patient's infectious disease; and (4) detection in one set of BCs reported by the clinician as the cause of the infection episode in the final diagnosis based on clinical, instrumental, and laboratory data. Coagulasenegative staphylococci, Corynebacterium spp., and other skin commensals were considered contaminants when isolated from one set of BCs alone (24) and in the absence of clinical and/or laboratory data suggesting a pathogenic role.

Definition of sepsis and septic shock
Sepsis was defined as life-threatening organ dysfunction due to a dysregulated host response to infection. Sepsis clinical criteria: organ dysfunction is defined as an increase of X2 points in the SOFA score. Septic shock was defined as an MAP o65 mmHg and lactate concentration X2 mmol/L despite fluid resuscitation with X30 mL/kg of crystalloid or 5% albumin in patients diagnosed with sepsis (1).

Statistical analyses
Statistical analyses were performed using MedCalc version 19.0.4 (MedCalc Software, Ostend, Belgium) and SPSS Version 23.0 software (IBM Corp., Armonk, NY, USA). Quantitative data were presented as means ± standard errors of the mean or as the median and interquartile range (i.e., 25 th -75 th percentiles), depending on the distribution of the data. Categorical data are summarized as ratios and percentages. Differences in means for continuous variables were compared using Student's t-test (two groups) or analysis of variance (multiple groups); differences in proportions were tested using an R Â C contingency table and Pearson's chisquared or Fisher's exact test as appropriate. Receiver operating characteristic (ROC) curve analysis and calculation of area under the curve (AUC) were conducted to determine the diagnostic utility of various PCT cut-offs and to assess the ability to predict mortality. Youden's indices (sensitivity+ specificity À 1) were calculated to determine the ideal discriminatory cut-off values. All statistical analyses were performed with SPSS 23.0, except for the comparison of AUCs, which was performed using MedCalc. All analyses were exploratory; a two-tailed p-valueo0.05 was considered as the cut-off for statistical significance.

Clinical characteristics
A total of 1092 positive BCs were obtained; 834 met the criteria for sepsis-3. A flow diagram of the included and excluded patients is shown in Figure 1. The patients' demographic and clinical characteristics are shown in Table 1: 522 (62.6%) were men and 312 (37.4%) were women; 666 (79.9%) fulfilled the criteria for sepsis, and 168 (20.1%) fulfilled the criteria for septic shock. Upon final diagnosis, most of the patients were diagnosed with respiratory infections (510 patients, 61.2%), followed by 186 (22.3%) with abdominal infections, 49 (5.9%) with urinary infections, 60 (7.2%) with catheter-related infections, and 10 (1.2%) with infective endocarditis. Regarding the pathogens detected, 393 patients (47.1%) were infected with gram-negative bacteria (GNB), slightly more than those infected by GPB (324 patients, 38.8%). Compared with the sepsis group, PCT and hs-CRP levels, SOFA, MEDS, LODS, and APACHE-II scores, and early and late mortality rates were found to be significantly higher in the septic shock group.  Table 2). Figure 2-A shows the ROC curves for the SOFA, LODS, MEDS, and APACHE-II scores and PCT, hs-CRP, and WBC for early mortality. Pairwise comparisons of the AUC of the scores were as follows: SOFA versus MEDS, p=0.036; SOFA versus LODS, p=0.213; SOFA versus APACHE-II, p=0.385; MEDS versus LODS, p=0.227; MEDS versus APACHE-II, p=0.134; and LODS versus APACHE-II, p=0.899. Pairwise comparisons of the AUC of the biomarkers were as follows: PCT versus hs-CRP, p=0.678; PCT versus WBC, p=0.239; and hs-CRP versus WBC, p=0.127. At a cutoff value of 5.0, the SOFA scores showed a sensitivity of 88.5%, a specificity of 56.5%, and a negative predictive value (NPV) of 92.7%; at a cut-off value of 0.84 ng/mL, PCT showed a sensitivity of 88.5%, a specificity of 56.5%, and an NPV of 92.7% (shown in Table 3).

' DISCUSSION
We evaluated four outcome prediction scoring systems and three biomarkers to predict early and late mortality among septis patients with BSIs. We found that CISs perform better than biomarkers for early and late mortality, especially for patients with GNB infection; the SOFA score offered better predictive performance than the MEDS score for early mortality. PCT achieved better predictive performance when used for GPB infections than hs-CRP and WBC for early mortality, probably performing better for GPB than for GNB.
The Third International Consensus Definitions for Sepsis and Septic Shock recommend that the SOFA score be used as a new clinical criterion for sepsis, and that the recommended criteria achieved moderate predictive performance for acute mortality [(In ICU patients with suspected infection: AUROC=0.74; 95% confidence interval [CI], 0.73-0.76); (for patients outside the ICU with suspected infection: AUROC=0.79; 95% CI, 0.78-0.80)] (1). This is comparable to the AUC of 80.5% in our study for early (28-day) mortality. Although the SOFA score uses physiological and laboratory variables alone and given that organ failure is an important predictor of the worst-case outcome, we found that the SOFA score performed better in predicting early mortality than the MEDS score. The MEDS score requires a subjective assessment of the likelihood of short-term mortality and may be less accurate in cases of higher illness severity. Jones et al. (25) found that the MEDS score had an AUC of 0.61 for patients admitted to the ED with septic shock. Similarly, Nguyen et al. (26) found MEDS to have an AUC of 0.63 in patients undergoing early goal-directed treatment in the ED. Our results for AUC were 0.761 for early mortality prediction and 0.744 for late mortality prediction, similar to the AUC of 0.78 found among patients with sepsis in the initial MEDS validation study (10). Patient selection may account for these different outcomes in that our study represents a more infirm group of patients with higher illness severity according to the sepsis-3 criteria.
The original APACHE score was developed in 1981 to classify groups of patients according to illness severity. It was divided into two sections: a physiology score to assess the degree of acute illness and a preadmission evaluation to determine the chronic health status of the patient (27). In 1985, the original model was revised and simplified to create APACHE-II (8), which is now the world's most widely used severity-of-illness score. In APACHE-II, there are 12 physiological variables, with the worst value recorded during the first 24 h of a patient's admission to the ICU used for each physiological variable. Furthermore, the effects of age and chronic health status are incorporated directly into the model and weighted according to their relative impact, likely rendering better predictive ability for late mortality. Our study found that the APACHE-II score had the largest AUC among the four prognostic scores and performed better than the MEDS score for late mortality prediction. This is in contrast to Pong JZ et al. (28), who found that the MEDS score outperformed the APACHE-II score in predicting 30-day in-hospital mortality among ED patients with suspected sepsis and fulfilled the systemic inflammatory response syndrome criteria. Chen YX et al. (29) found that the two scoring systems had similar AUC values for early mortality prediction among septic patients admitted to the ED. Patient selection may account for these different outcomes.
PCT, serving as a biomarker of bacterial infection, has been widely investigated for its prognostic value in patients with sepsis. A meta-analysis (30) including 23 studies with 3994 patients found that elevated PCT levels were associated with a higher risk for death. Nevertheless, a recently published study (31) also found that PCT was a promising biomarker for the prognosis of mortality in sepsis, with an AUROC of 0.81, which was higher than our results. This may be attributed to patient selection and PCT testing time. Given that PCT values were associated with bacterial types (23,32), we analyzed the predictive power of PCT for early and late mortality among sepsis patients with GNB and GPB infections, respectively; we found that AUCs were higher in GPB infections. This may be attributed to the fact that GPB lacks cell walls and stimulates PCT release through the production of distinct inflammatory cytokines alone (33). Consequently, to some extent, the PCT value may reflect the severity of the inflammatory response. Unlike GPB infections, GNB produces endotoxins that can also be released upon cell death, also affecting PCT levels (34).
We also found that PCT performed better than hs-CRP and WBC for the prediction of early mortality among sepsis patients with GPB infections; it did not show any advantage in mortality prediction before we segregated patients into two groups based on Gram stain testing. In contrast, the AUROC of CISs in early and late mortality prediction was larger for GNB infection than for GPB infection. To the best of our knowledge, this is the first study to demonstrate the predictive value of PCT in patients with GPB and GNB infections.
This study has some limitations. First, this is a subgroup analysis; although the subgroups have a large number of patients, it is still a post-hoc analysis of a single-center retrospective study. Therefore, the utility of these data is limited to emergency applications. Second, our data is just a snapshot of hospital admission, since sepsis severity is a continuum; a patient who is infected now may develop sepsis or septic shock tomorrow. Third, the predictive power ' CONCLUSIONS CISs were more advantageous in the assessment of early and late prognosis, especially for GNB infections. However, for sepsis with GPB infection, PCT can be used to predict early mortality.

Highlights of the Study
There were significant differences in critical illness scores (CISs) and biomarkers between the death and survival groups in predicting early and late mortality in patients with sepsis and septic shock. CISs showed a significantly higher area under the curve (AUC) in the prediction of early and late mortality based on receiver operating characteristic curve analysis than biomarkers, especially for patients with gram-negative bacteria (GNB) infections. Procalcitonin (PCT) might perform better for gram-positive bacteria (GPB) infections than for GNB infections in the prognosis of patients with sepsis-3. Meanwhile, PCT performed better than high-sensitivity C-reactive protein and white blood cells with GPB infections. The Sequential Organ Failure Assessment score showed a higher AUC for predicting early mortality than the Mortality in Emergency Department Sepsis score.

' ACKNOWLEDGMENTS
This study was supported by grants from the National Natural Science Foundation of China (grant number 81871600).

' AUTHOR CONTRIBUTIONS
Yan ST participated in the study design and coordination, data acquisition and analysis, performed statistical analyses and interpretation, and drafted the manuscript. Zhang GQ conceived the study, participated in its design and coordination, and participated in data analysis and interpretation. All authors read and approved the final version of the manuscript.