• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Arch Dis Child. Apr 1, 2011; 96(4): 361–367.
Published online Jan 24, 2011. doi:  10.1136/adc.2010.183111
PMCID: PMC3158666

Risk score to stratify children with suspected serious bacterial infection: observational cohort study

Abstract

Objectives

To derive and validate a clinical score to risk stratify children presenting with acute infection.

Study design and participants

Observational cohort study of children presenting with suspected infection to an emergency department in England. Detailed data were collected prospectively on presenting clinical features, laboratory investigations and outcome. Clinical predictors of serious bacterial infection (SBI) were explored in multivariate logistic regression models using part of the dataset, each model was then validated in an independent part of the dataset, and the best model was chosen for derivation of a clinical risk score for SBI. The ability of this score to risk stratify children with SBI was then assessed in the entire dataset.

Main outcome measure

Final diagnosis of SBI according to criteria defined by the Royal College of Paediatrics and Child Health working group on Recognising Acute Illness in Children.

Results

Data from 1951 children were analysed. 74 (3.8%) had SBI. The sensitivity of individual clinical signs was poor, although some were highly specific for SBI. A score was derived with reasonable ability to discriminate SBI (area under the receiver operator characteristics curve 0.77, 95% CI 0.71 to 0.83) and risk stratify children with suspected SBI.

Conclusions

This study demonstrates the potential utility of a clinical score in risk stratifying children with suspected SBI. Further work should aim to validate the score and its impact on clinical decision making in different settings, and ideally incorporate it into a broader management algorithm including additional investigations to further stratify a child's risk.

Introduction

Serious bacterial infections (SBIs) are a leading cause of morbidity and mortality among children worldwide. A recent review of child deaths in England, Wales and Northern Ireland confirmed that the largest single cause of death in children dying of an acute physical illness was infection, and found evidence that healthcare professionals in both hospital and primary care had difficulty identifying serious illness in children.1 In another study of over 400 children with meningococcal sepsis in the UK, only half were referred to hospital at their first primary care consultation, usually after a delay of several hours following the first symptoms and/or signs of sepsis.2 Early recognition of SBIs is associated with better treatment outcomes3 4 but is challenging as many of the early clinical features also occur in self-limiting illnesses. In developed countries, discriminating serious infection from self-limiting illness is further complicated by the low prevalence of serious infection in children presenting to hospital emergency departments as well as primary care settings.5 Better clinical tools are therefore required to improve the recognition of children with serious acute illness, particularly SBIs.1 6 7

What is already known on this topic

[triangle]

Serious bacterial infections (SBI) are a leading cause of childhood illness and death in the UK and worldwide.

[triangle]

In clinical practice, discriminating children with SBI from those with other self-limiting infections is challenging.

[triangle]

A simple, validated clinical tool to risk stratify and guide further management of children with suspected SBI would greatly improve their care.

What this study adds

[triangle]

The high specificity for SBI of some clinical signs supports their use in clinical guidelines to identify seriously ill children.

[triangle]

The utility of a simple clinical score to stratify children according to SBI risk is clearly demonstrated.

[triangle]

Combining this clinical tool with laboratory tests in a broader management algorithm may further improve SBI diagnosis pathways.

Clinical prediction rules utilising key clinical features to generate a diagnostic or prognostic score are increasingly used to risk stratify patients in a wide variety of clinical situations, particularly those in which diagnostic decisions are complicated, and in some areas have had a major impact on clinical service delivery.815 We aimed to develop a simple clinical score with the ability to risk stratify children with possible SBIs presenting to a paediatric emergency department.

Methods

Patients and setting

As part of a large prospective study of patient care pathways, detailed clinical data were collected prospectively on all children presenting to the Queen's Medical Centre Emergency Department in Nottingham between September 2000 and March 2001, and September 2001 and March 2002, with the exception of neonates and children requiring immediate emergency resuscitation at presentation. A triage nurse recorded vital signs prior to assessment by emergency department clinical staff. All clinical data including investigations, management and final diagnoses were directly entered onto a standard proforma. Completed proformas were then scanned using Formic electronic data capture software to minimise data entry errors. Study clinicians checked the data for completeness, resolved data gaps and inconsistencies by re-review of the clinical notes, and recorded additional clinical data on children who were admitted. Children who re-attended hospital within 1 week of discharge from either the emergency department or the ward were identified from the electronic patient register, their notes reviewed, and final diagnoses and SBI classification amended in the light of their second presentation. A consultant paediatrician re-reviewed the patient records of all those admitted to check the accuracy of the data, particularly in relation to the final diagnosis recorded.

Children were eligible for inclusion in this analysis if their differential diagnosis at presentation included acute infection. We excluded from the analysis children for whom data were insufficient to confidently assign outcome, or who had missing dates of birth.

Clinical definitions

Automated measurements were used for temperature (tympanic thermometer), pulse rate, blood pressure and pulse oximetry. Tachypnoea, tachycardia and hypotension were defined according to UK Advanced Paediatric Life Support guidelines16; children for whom no blood pressure recordings were available were assumed not to be hypotensive for the purpose of the analysis. We adopted pragmatic definitions of mild hypoxia (SaO2<95% and/or documented oxygen therapy) and severe hypoxia (SaO2<90% and/or SaO2<92% despite documented oxygen therapy), and defined a ‘risk factor for infection’ as a documented history of any condition known to increase a child's risk of invasive bacterial infection. Developmental delay was included separately as a potential risk factor, since several studies have documented an increased rate and severity of acute illness including infection among this group.17 18 Other definitions reflected those commonly used in clinical practice (table 1); data collected routinely on all children included level of consciousness, capillary refill time, hydration status, and presence and type of rash.

Table 1
Univariable associations between clinical variables and risk of serious bacterial infection (SBI) in children presenting to the emergency department

SBI was defined a priori as admission to hospital plus any of the following (in the absence of an alternative non-infective or non-bacterial diagnosis to explain the clinical and laboratory findings): positive bacterial cultures from blood or another normally sterile site in the appropriate clinical context, radiological signs of pneumonia, clinical meningitis plus a cerebrospinal fluid polymorphonuclear leukocytosis, acute febrile purpura, deep collection(s) requiring intravenous antibiotics±surgical drainage, a white blood cell count ≥20×109/l, a C reactive protein ≥120 mg/l, or a final diagnosis of septic arthritis, osteomyelitis, empyema or mastoiditis.

Statistical methods

Analyses were performed using Stata v 10 (StataCorp LP). The distribution of each variable was summarised with respect to SBI, and crude OR derived. The sensitivity, specificity, positive and negative predictive values (PPV, NPV) and positive and negative likelihood ratios (LR+, LR−) were reported for each variable.

We randomly assigned each child to either a ‘derivation’ or a ‘validation’ dataset, such that ~85% of children were in the derivation set. Multivariate logistic regression models were then derived using the derivation set to develop a clinical prediction score for SBI. We included in the model only variables with at least a weak association with SBI in the univariate analysis (likelihood ratio test (LRT) p value ≤0.1), first confining this to categorical variables (which are more easily applied in clinical practice), and then repeating the analysis using the available continuous data to explore to what extent this improved the model.

Backwards stepwise logistic regression was performed first using ‘standard’ selection criteria, such that variables not significantly associated with SBI (Wald p value <0.5; t test p value ≤0.1 for continuous variables) are sequentially dropped from the model, and then using ‘sign OK’ selection,19 whereby variables are eliminated from the model only if the sign (+ or −) of the regression coefficient (β) is discordant with known or expected clinical associations (since standard selection criteria may not be optimal in the analysis of smaller datasets15 1921).

Each model took the form: SBI=α+β1X12X23X3+…+βiXi, where α is a constant, βi are the regression coefficients for each variable (Xi) in the model, and Xi are coded as 0 or 1 (for binary variables), integers (for ordered categorical variables) or continuous numbers (for continuous variables). For each model we generated a score for each child, such that the score=β1X12X23X3+... +βiXi. The ability of each score to discriminate children with SBI was quantified in the derivation dataset, and validated in the validation dataset, by calculating the area under the receiver operator characteristics (ROC) curve (AUC) of score as a predictor of SBI.

Finally, the best model was selected to derive a simpler score for clinical use. Criteria for model selection included both discriminant ability (defined by the AUC) and model simplicity. We generated a simplified clinical prediction score using weightings defined by the β coefficients of the chosen model, multiplied if necessary by a constant, and rounded as far as possible to the nearest integer. The discriminant ability of this final ‘SBI score’ was tested by plotting an ROC curve of the score as a predictor of SBI in the entire dataset (derivation and validation sets combined). Risks of SBI were presented stratified by score.

Ethics approval

Ethics approval was granted by the Nottingham Research Ethics Committee and the ethics committee of the London School of Hygiene and Tropical Medicine.

Results

There were 2341 eligible children, of whom 17 were excluded because insufficient data were available to confidently determine outcome. A further 373 children were excluded because their date of birth and therefore their age could not be positively determined from the hospital records due to uncertainty in interpreting their handwritten unique patient identifier number on the proforma; the proportion of children with SBI did not differ significantly between those for whom age data were and were not available (p=0.47).

The final analysis therefore included 1951 children. Their median age was 19 months (range 1 month to 15 years) and 55.4% were boys. Five hundred and forty-six (28.0%) children were admitted at first presentation, of whom 72 had SBI. Thirty-six children re-attended hospital within a week of their initial presentation and within the same illness episode, 18 of whom were admitted and 2 of whom had SBI. The principal final diagnoses of the 74 (3.8%) children with SBI included pneumonia (52), sepsis without a clear focus (14), soft tissue infections (5), urinary sepsis (2) and tonsillitis plus sepsis (1).

Several clinical variables were strongly associated with SBI in the univariate analysis (table 1), including signs of a child's a neurologic, haemodynamic and respiratory state, and a known risk factor for infection. Although hypotension was not significantly associated with SBI, it is important to note that blood pressure data were only available for 103 (5.3%) children.

The predictive values of each variable are summarised in table 2. Apart from tachypnoea (sensitivity 71.6%), the sensitivity of most clinical signs was poor. However, some clinical signs, such as those of reduced level of consciousness/alertness and impaired perfusion were highly specific, and demonstrated more clinically useful positive likelihood ratios. The low prior probability of SBI in this population explains the apparently high negative predictive values of many variables, reflected in the poor utility of most negative likelihood ratios.

Table 2
Predictive value of clinical variables for serious bacterial infection

After random weighted allocation of children to the derivation and validation sets, the derivation set comprised 1600 (85.2%) children without SBI and 64 (86.5%) children with SBI. The models with the best discriminative ability were those derived using ‘sign OK‘ selection. For the ‘sign OK’ model confined to categorical variables, the AUCs in the derivation and validation sets were 0.76 (0.68 to 0.82) and 0.85 (0.71 to 0.99), respectively. Inclusion of continuous data in the model did not markedly improve performance (AUC 0.76 and 0.86 in the derivation and validation sets, respectively), so the simpler model confined to categorical variables was selected. Regression coefficients and adjusted ORs and 95% CIs for each variable included in this model are given in table 3. A modified SBI score was derived based on the β coefficients of this model, such that:

Table 3
Regression coefficients (β), and adjusted OR and 95% CI for variables included in the chosen model

The AUC for SBI score as a predictor of SBI in the derivation and validation sets combined was 0.77 (95% CI 0.71 to 0.83; figure 1). Table 4 shows the SBI score in tabular form more conveniently for clinical use, and risks of SBI stratified by SBI score are shown in table 5.

Figure 1
Receiver operator characteristics (ROC) curve of serious bacterial infection score as a predictor of serious bacterial infection in the entire dataset.
Table 4
Serious bacterial infection (SBI) score for prediction of serious bacterial infection
Table 5
Risks of significant bacterial infection (SBI) stratified by SBI score

Discussion

By exploring the feasibility of developing a clinical prediction score for risk stratification of suspected serious bacterial illness in children, this study addresses an important challenge in everyday clinical practice in the UK and other developed countries.1 2 5 7 Among children presenting to emergency care in whom infection was suspected (where the prior probability of SBI was 5%), our SBI score was able to rule out SBI with reasonable confidence and precision in three quarters of children in the dataset to whom the score was applied (SBI score ≤5 associated with <2% risk of SBI), to identify children at the highest risk of SBI (a third of children with an SBI score >8 had SBI) and to risk stratify the remaining children into groups with increasing risk.

We modelled continuous data as categorical and continuous variables in separate models. Although less suitable for the development of a clinical prediction score for clinicians, continuous variables maximise information15 22 and can be utilised in computer-assisted diagnostic pathways. However, their inclusion in the model did not improve discriminative ability sufficiently to justify the increased complexity.

Previous studies in developed countries have attempted to derive clinical scores as a tool to identify seriously ill children at presentation.2335 Most were carried out in populations with a higher incidence of SBI prior to the introduction of routine vaccination against Haemophilus influenza type b (Hib) and Streptococcus pneumoniae; others were designed to predict the need for admission to intensive care.9 Moreover, promising initial reports of diagnostic performance have not always been consistently reproducible.27 36 37 This probably reflects a combination of the inherent over-optimism of data-driven model predictions based on small datasets,15 19 21 3840 failure to validate models and/or scores in an independent dataset,15 19 and spectrum bias arising from the use of case–control designs.4143 In a recent study based mainly in primary care, a clinical algorithm was derived with the ability to rule out SBI with reasonable confidence.44 However, these results have not been validated in an independent dataset, and negative predictive values are likely to be lower in emergency department settings where the prior probability of SBI is higher.

The strengths of our study include the large number of children studied, detailed review of all clinical and laboratory data to assign outcome, validation of the model in an independent part of the dataset, and inclusion of all children presenting with suspected SBI in a cohort study design, which suggests the predictive value of our SBI score should be generalisable to other similar settings. Furthermore, focusing on children with suspected SBI rather than all children with serious illness allowed inclusion of clinical risk factors specific to infection, and may have helped improve model specificity. It is important to stress that while the SBI score should be generalisable to emergency departments in the developed world, it is unlikely to be applicable to developing world settings where the prevalence of SBI among paediatric presentations is much higher and there may be other important risk factors for infection such as malnutrition and HIV.5

A potential limitation of the study concerns the definition of SBI. No single, reliable gold standard exists for the diagnosis of invasive bacterial infections in children. Although cultures from blood and other sterile sites are highly specific, sensitivity is limited.4547 The definition of SBI used has been formulated to take account of all available clinical and laboratory information in defining each clinical syndrome, including a child's response to treatment, and is designed to capture all children with invasive or otherwise SBIs who require admission to hospital. A potential criticism is the risk of incorporation bias, since clinicians are not blind to the admission clinical variables studied, which are likely to influence admission decisions; inclusion of clinical and laboratory data from the entire admission in assigning outcome is likely to only partly mitigate this bias. Nevertheless, we believe the definition adopted represents a pragmatic working definition of SBI. Importantly, we were also able to identify those children discharged from hospital who re-attended within a week, and to amend their SBI classification as required. While we cannot exclude the possibility that other children may have re-presented elsewhere, the nearest emergency department, which does not have a dedicated paediatric section, is 18 miles away, so the chance of this is small. Similarly, since sudden childhood death is rare in this population and most cases are brought to hospital, the chance of a child in this study dying at home and not re-attending hospital is also small.

Missing data also limit the power of our analysis. Although the proportion of data missing for individual variables was generally small, this became more significant in the multivariate analysis. Blood pressure is not routinely measured in paediatric clinical practice except in the sickest children, so data were missing for most patients. Nevertheless, blood pressure may have an important role in predicting SBI that this analysis would not be able to identify. The number of children for whom age data were missing is unfortunate as it further limits study power; however, this is unlikely to have introduced any bias since the underlying problem (poor legibility of the proforma) is unlikely to be related to the clinical presentation and was not associated with SBI.

Finally, measurement error is also possible, since emergency department staff assessed each clinical sign using routine clinical tools. For some variables this is likely to be minimal since automated measurements were used. Arguably, even for other variables this may be seen as a strength of the study, since it increases its generalisability to the routine clinical setting.

The implications of this study for clinical practice are threefold. First, the specificity of some clinical signs for SBI (such as reduced activity or level of consciousness, prolonged capillary refill time and reduced skin turgor) support the National Institute for Health and Clinical Excellence clinical guidelines7 which use these clinical features as ‘red flags’ to identify seriously ill children. The presence of these features in a febrile child should prompt a careful clinical assessment. Second, it clearly demonstrates the potential utility of a clinical score to stratify children according to their risk of SBI. Such a score might be incorporated into electronic decision support and medical records. Finally, it underlines the need to use such a clinical score as part of a broader algorithm to guide management decisions (including additional investigations, plans for review or admission for observation) to further stratify a child's risk. Future research should explore the benefit of including in the model other risk factors such as immunisation status, routine blood pressure measurement, and other potential markers of sepsis such as limb pain or cool peripheries,2 investigate how laboratory tests such as C-reactive protein, procalcitonin and white blood cell count may best be incorporated into a clinical algorithm to refine a child's risk status, and further validate the performance and impact14 48 of the score in different acute paediatric settings, including primary care.

Acknowledgments

The authors would like to thank Dr Terence Stephenson for his contributions to the initial data collection.The study was carried out on behalf of the Royal College of Paediatrics and Child Health (RCPCH) working group on Recognising Acute Illness in Children (RAIC). MT is a member of University of Oxford Department of Primary Health Care which is part of the NIHR National School of Primary Care Research. ML is a member of the Leicester Collaboration for Leadership in Applied Health Research and Care (CLARCH) team and Director of the Health Education Research and Development Unit (HERADU).

Footnotes

Contributors AJB and ML are joint first authors.

Funding AJB is supported by a Wellcome Trust research training fellowship (081697). The authors wish to thank the Well Child Medical Charity for their funding of the studies in Nottingham and of this study.

Competing interests None.

Ethics approval This study was conducted with the approval of the Nottingham Research Ethics Committee and the ethics committee of the London School of Hygiene and Tropical Medicine.

Provenance and peer review Not commissioned; externally peer reviewed.

References

1. Pearson GAE. Why Children Die: A Pilot Study 2006; England (South West, North East and West Midlands), Wales and Northern Ireland. London: Confidential Enquiry into Maternal and Child Health, 2008.
2. Thompson MJ, Ninis N, Perera R, et al. Clinical recognition of meningococcal disease in children and adolescents. Lancet 2006;367:397–403. [PubMed]
3. Kumar A, Roberts D, Wood KE, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med 2006;34:1589–96. [PubMed]
4. Dellinger RP, Levy MM, Carlet JM, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock: 2008. Crit Care Med 2008;36:296–327. [PubMed]
5. Van den Bruel A, Haj-Hassan T, Thompson M, et al. Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review. Lancet 2010;375:834–45. [PubMed]
6. Tasche M, Oosterberg E, Kolnaar B, et al. Inventory of gaps in the evidence base of general practice. Huisarts Wet 2001;44:91–4.
7. National Institute for Health and Clinical Excellence Feverish Illness in Children: Assessment and Initial Management in Children Younger Than 5 Years. London: NICE, 2007.
8. Antman EM, Cohen M, Bernink PJ, et al. The TIMI risk score for unstable angina/non-ST elevation MI: A method for prognostication and therapeutic decision making. JAMA 2000;284:835–42. [PubMed]
9. Duncan H, Hutchison J, Parshuram CS. The Pediatric Early Warning System score: a severity of illness score to predict urgent medical need in hospitalized children. J Crit Care 2006;21:271–8. [PubMed]
10. Haines C, Perrott M, Weir P. Promoting care for acutely ill children-development and evaluation of a paediatric early warning tool. Intensive Crit Care Nurs 2006;22:73–81. [PubMed]
11. Mwachari C, Nduba V, Nguti R, et al. Validation of a new clinical scoring system for acute bronchitis. Int J Tuberc Lung Dis 2007;11:1253–9. [PubMed]
12. van Koningsveld R, Steyerberg EW, Hughes RA, et al. A clinical prognostic scoring system for Guillain-Barré syndrome. Lancet Neurol 2007;6:589–94. [PubMed]
13. Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to categorize patients probability of pulmonary embolism: increasing the models utility with the SimpliRED D-dimer. Thromb Haemost 2000;83:416–20. [PubMed]
14. Moons KG, Royston P, Vergouwe Y, et al. Prognosis and prognostic research: what, why, and how? BMJ 2009;338:b375. [PubMed]
15. Royston P, Moons KG, Altman DG, et al. Prognosis and prognostic research: Developing a prognostic model. BMJ 2009;338:b604. [PubMed]
16. Advanced Life Support Group. Advanced Paediatric Life Support: The Practical Approach. Fourth edition. Wiley-Blackwell, Oxford, UK: 2004.
17. Graham RJ. Specialty services for children with special health care needs: supplement not supplant the medical home. Arch Dis Child 2008;93:2–4. [PubMed]
18. Mahon M, Kibirige MS. Patterns of admissions for children with special needs to the paediatric assessment unit. Arch Dis Child 2004;89:165–9. [PMC free article] [PubMed]
19. Steyerberg EW, Eijkemans MJ, Harrell FE, Jr, et al. Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making 2001;21:45–56. [PubMed]
20. Sun GW, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 1996;49:907–16. [PubMed]
21. Harrell FE, Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361–87. [PubMed]
22. Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ 2006;332:1080. [PMC free article] [PubMed]
23. Egdell P, Finlay L, Pedley DK. The PAWS score: validation of an early warning scoring system for the initial assessment of children in the emergency department. Emerg Med J 2008;25:745–9. [PubMed]
24. McCarthy PL, Sharpe MR, Spiesel SZ, et al. Observation scales to identify serious illness in febrile children. Pediatrics 1982;70:802–9. [PubMed]
25. McCarthy PL, Jekel JF, Stashwick CA, et al. History and observation variables in assessing febrile children. Pediatrics 1980;65:1090–5. [PubMed]
26. McCarthy PL, Jekel JF, Stashwick CA, et al. Further definition of history and observation variables in assessing febrile children. Pediatrics 1981;67:687–93. [PubMed]
27. Kuppermann N, Fleisher GR, Jaffe DM. Predictors of occult pneumococcal bacteremia in young febrile children. Ann Emerg Med 1998;31:679–87. [PubMed]
28. Nelson KG. An index of severity for acute pediatric illness. Am J Public Health 1980;70:804–7. [PMC free article] [PubMed]
29. Bass JW, Steele RW, Wittler RR, et al. Antimicrobial treatment of occult bacteremia: a multicenter cooperative study. Pediatr Infect Dis J 1993;12:466–73. [PubMed]
30. Dershewitz RA, Wigder HN, Wigder CM, et al. A comparative study of the prevalence, outcome, and prediction of bacteremia in children. J Pediatr 1983;103:352–8. [PubMed]
31. McCarthy PL, Lembo RM, Baron MA, et al. Predictive value of abnormal physical examination findings in ill-appearing and well-appearing febrile children. Pediatrics 1985;76:167–71. [PubMed]
32. McCarthy PL, Lembo RM, Fink HD, et al. Observation, history, and physical examination in diagnosis of serious illnesses in febrile children less than or equal to 24 months. J Pediatr 1987;110:26–30. [PubMed]
33. Schwartz RH, Wientzen RL., Jr Occult bacteremia in toxic-appearing, febrile infants. A prospective clinical study in an office setting. Clin Pediatr (Phila) 1982;21:659–63. [PubMed]
34. Waskerwitz S, Berkelhamer JE. Outpatient bacteremia: clinical findings in children under two years with initial temperatures of 39.5 degrees C or higher. J Pediatr 1981;99:231–3. [PubMed]
35. Yilmaz HL, Yildizdas RD, Alparslan N, et al. Screening tools for bacteraemia in a selected population of febrile children. Ann Acad Med Singap 2008;37:192–9. [PubMed]
36. Baker MD, Avner JR, Bell LM. Failure of infant observation scales in detecting serious illness in febrile, 4- to 8-week-old infants. Pediatrics 1990;85:1040–3. [PubMed]
37. Teach SJ, Fleisher GR. Efficacy of an observation scale in detecting bacteremia in febrile children three to thirty-six months of age, treated as outpatients. Occult Bacteremia Study Group. J Pediatr 1995;126:877–81. [PubMed]
38. Efron B. Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 1986;78:316–31.
39. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 1997;277:488–94. [PubMed]
40. Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996;49:1373–9. [PubMed]
41. Banoo S, Bell D, Bossuyt P, et al. Evaluation of diagnostic tests for infectious diseases: general principles. Nat Rev Microbiol 2006;4:S20–32. [PubMed]
42. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999;282:1061–6. [PubMed]
43. Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 1978;299:926–30. [PubMed]
44. Van den Bruel A, Aertgeerts B, Bruyninckx R, et al. Signs and symptoms for diagnosis of serious infections in children: a prospective study in primary care. Br J Gen Pract 2007;57:538–46. [PMC free article] [PubMed]
45. Cutts FT, Zaman SM, Enwere G, et al. Efficacy of nine-valent pneumococcal conjugate vaccine against pneumonia and invasive pneumococcal disease in The Gambia: randomised, double-blind, placebo-controlled trial. Lancet 2005;365:1139–46. [PubMed]
46. Sáez-Llorens X, Vargas S, Guerra F, et al. Application of new sepsis definitions to evaluate outcome of pediatric patients with severe systemic infections. Pediatr Infect Dis J 1995;14:557–61. [PubMed]
47. Luck S, Torny M, d'Agapeyeff K, et al. Estimated early-onset group B streptococcal neonatal disease. Lancet 2003;361:1953–4. [PubMed]
48. Moons KG, Altman DG, Vergouwe Y, et al. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 2009;338:b606. [PubMed]

Articles from BMJ Open Access are provided here courtesy of BMJ Group

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...