PubMed Health. A service of the National Library of Medicine, National Institutes of Health.

Thompson M, Van den Bruel A, Verbakel J, et al. Systematic Review and Validation of Prediction Rules for Identifying Children with Serious Infections in Emergency Departments and Urgent-Access Primary Care. Southampton (UK): NIHR Journals Library; 2012 Mar. (Health Technology Assessment, No. 16.15.)

Appendix 7Research protocol

1. Project title: Systematic review and validation of clinical prediction rules for identifying children with serious infections in emergency departments and urgent-access primary care

2. How the project has changed since the outline proposal was submitted

The Board had several comments on the outline proposal which we have addressed.

a. We have increased the emergency care expertise on our team

Although our team already included Dr Henrietta Moll, Emergency Department Paediatrician at Sophia Children's Hospital in Rotterdam, we have also included Dr Shelly Segal who is the Lead Paediatrician in the Accident and Emergency Department at the John Radcliffe Hospital in Oxford, and Dr Monica Lakhanpaul who is a Consultant Paediatrician and who brings extensive experience as the project lead of the recent NICE guideline on the management of the febrile child and who is the clinical lead on the development of the nurse-led urgent care service in Leicester.

b. We have been more explicit about eligibility criteria

The Board wanted to see a strong justification of the patient eligibility criteria. We agree that for the proposed systematic review, more details of the inclusion and exclusion criteria are needed. We have included these in the full proposal.

c. We have reduced the scope of the study to focus on the systematic review element

The main concern of the Board was the feasibility of including the cross validation of prediction rules as well as the systematic review. Particular concern was expressed about our ability to secure access to the databases planned within the time frame. However we have already secured access to the five key datasets of which we are already aware and this will allow us to conduct an individual patient data based meta-analysis on at least this sub-set of studies.

The Board also had several comments noted on June 17, 2008 to which we responded on July 8, 2008 and which are incorporated into this final project description. Our responses are summarised as follows:

  1. The applicability of the clinical prediction tool needs to be clearly defined in terms of who will be able and likely to use it.
    We feel that the management of children presenting to emergency and urgent care settings with infections presents an ideal opportunity for application of a clinical prediction rule. In general clinical prediction rules are most likely to be helpful in situations where ‘decision making is complex, the clinical stakes are high, or there are opportunities to achieve cost savings without compromising patient care’ (McGinn, JAMA 2000). The clinical prediction tool that we will develop and cross-validate will incorporate components of the history, vital signs and basic examination findings. We feel that this prediction rule will therefore be most applicable to front line clinicians, such as GPs, paramedics, practice nurses, A&E triage nurses and nurse practitioners, and A&E junior medical staff. The advantage of the methods planned is that we will be able to validate this rule in multiple clinical settings with varying prevalence of serious infection, and thus the prediction rule will be applicable in many different acute care settings in the NHS. If the rule is found to be robust we will disseminate the findings widely via appropriate peer reviewed publications and by contact with the relevant professional bodies. In addition, we anticipate that it will complement the current NICE guideline on the assessment of feverish children (Feverish illness in children, NICE May 2007) by formally testing, simplifying and quantifying the accuracy of many of the clinical predictors used in that guideline. As with any clinical prediction rule, the impact of the rule will need to be evaluated once it has been implemented.
  2. The amount of time allocated to staff involved in the project appears too low in some cases, this should be reviewed along with the project costs.
    We agree that the time allocated particularly to senior staff on this project was far too low in our application and would like to revise this, with the permission of the Board. We propose the following changes to staff hours over the course of the 1 year project: 1) increase in Dr Thompson's (Lead applicant) hours from 132 hours to 330 hours, 2) Professors Mant and Glasziou from 11 to 44 hours, 3) Dr Lakhanpaul's hours from 36 to 82 hours. The costs allocated to the other staffing costs both at Oxford (Dr Perera & Research Assistant), Oxford Radcliff Trust (Dr Segal), Leuven (Belgium) (Professor Buntinx, Drs Aertgert and Van en Bruel) and Rotterdam (Dr Moll) and Maastricht (Dr Dinant) have not been altered. These changes are outlined in the accompanying spreadsheet and will increase the overall budget to £125,657 (see attached spreadsheet).
  3. The distribution of end points available for analysis should be described.
    In all datasets we have the main outcomes recorded of need for admission to hospital, and number of children with serious infection. The definition of serious infection will be standardised across all datasets, but will include clinical conditions such as meningitis, UTI, bacterial gastroenteritis, pneumonia, sepsis.
  4. The core items eligible for inclusion across the five datasets should be stated.
    We have provisionally examined the core items from each of the datasets, from which we will identify predictors of serious infection. This shows that all datasets include details of the general characteristics of the children such as age, gender, as well as the setting and whether referred or not referred. The completeness of the presenting clinical features, i.e. symptoms and signs varies between datasets. All the datasets include the core vital signs heart rate, respiratory rate, and temperature, and some also include oxygen saturations and capillary refill time. Five of the datasets include a large number of clinical features identified from parental history or initial triage/examination. Two of the datasets include fewer clinical features. The number of investigations performed on children varies with type of clinical setting, and we are likely only to consider results of white cell count or C-reactive protein (CRP) as predictors. We will also use the systematic review that we will be undertaking to assist us in deciding which predictors have been most useful in previous studies in this area.
  5. The project should consider alternatives to splitting the dataset randomly in half.
    We agree that splitting the data may not be the best method for validating the prediction rule. The two issues that we will aim to address are: a) over-optimistic estimates and b) transferability of the prediction rule across settings. To achieve this we will use k-fold cross-validation to obtain more realistic estimates and calibration using other datasets to test the transferability of the model. By validating the clinical prediction rules on patients in broader settings (and thus different disease prevalence and spectrum) from those used to derive the rule, we will be able to demonstrate the generalizability or external validity of the rule (McGinn et al., JAMA 2000). We anticipate that this process will require model revision and/or shrinkage methods (Steyerberg EW et al. Statist Med 2004).
  6. Service user involvement.
    We did not specify the level of service user involvement in the application as we did not feel that it was particularly relevant to this type of study. However, we agree that it would be useful to involve parents/carers input in assessing the likely impact of this rule in the real world setting, and to ensure that the predictors we identify (e.g. vital sign measurements, possibly blood tests) are acceptable to most parents/carers. We will therefore assemble a group of parents who have had personal experience with children in emergency care or urgent access primary care and obtain their input on the final prediction rules.

3. Planned investigation

Research objectives

The overall aim of this research is to systematically identify simple clinical decision rules which can allow children with self-limiting illness to be safely discharged from emergency and urgent primary care settings while not missing any cases of serious infection. We propose to undertake a systematic review of the literature on prediction rules for triaging children with acute illness in emergency and urgent care settings.

The specific objectives of the systematic review are:

  1. To identify the clinical features and decision rules which have already been shown to have predictive value for identifying (or excluding) children with severe infection.
  2. To identify and compare the best performing prediction rules from the literature.
  3. To explore the added value of including laboratory tests and vital signs to prediction rules based on clinical history and observation.

Clinical prediction rules are a simple pragmatic technology that can be used by clinical staff to assist them in assessment and clinical management. A widely implemented example which has been shown to reduce both resource use and missed diagnoses in A&E is the Ottawa Ankle Rule for ordering an X-ray.1 The marginal NHS cost of implementing a clinical prediction rule depends primarily on the cost of any additional staff time or investigations required. The prediction rules that we propose validating have very low marginal cost because the main components are an integral part of the standard clinical assessment of children that clinicians use in routine NHS practice (i.e. medical history, presenting complaints, vital signs and examination findings).

The main economic benefit to the NHS is the potential to reduce the need for urgent hospital admission by reliably identifying the vast majority of children who can safely be discharged home or to lower acuity care (e.g. GP follow up). However, more effective triage using a formal prediction rule will also improve the care of children with serious infections (e.g. by signalling the need for 999 transfer to A+E, for urgent diagnostic or therapeutic intervention, or for urgent paediatrician review), thus optimising use and effectiveness of emergency services. If shown to discriminate effectively, such a prediction rule would be used at several levels of the emergency medical system in the UK, including paramedics, walk-in or out of hours surgeries, paediatric assessment units, as well as A&E Departments.

Existing research

Acute illness is one of the most common problems encountered in children attending emergency departments as well as by urgent-access primary care services in the UK.

Between 27–47% of patients who present to A&E departments in the UK do so for medical illness, rather than trauma.2 For children, the most common medical reasons for attending A&E are breathing difficulty (31%), febrile illness (20%), diarrhoea/vomiting (16%), abdominal pain (6%), seizure (5%), or rash (5%).2

Children under 5 years of age also constitute a substantial part of the workload of urgent-access primary care services. Indeed, the patient group which presents most commonly to out-of-hours assessment clinics is children with acute infections.3,4 Similarly, acute illness in children is also a major component of the work of NHS Direct, where 22% of all telephone calls are related to children under 5 years of age.5

One of the key tasks in both hospital emergency departments and urgent-access primary care clinics is therefore to distinguish children who may have serious infections or complications of infections (e.g. meningitis, bacteraemia, hypoxia from bronchiolitis, dehydration from gastroenteritis) from the vast majority with self-limiting or minor infections who can safely be managed as outpatients or referred to primary care services. This task is challenging. With increasing A&E attendance rates in the UK, hospital admission of children is becoming more common despite a falling incidence of serious infection. At the same time, approximately half of children with meningococcal disease are still missed at first consultation with a doctor, which results in poorer health outcome.6 If the simple clinical decision rules we plan to assess are shown to be effective, they are likely to be welcomed and widely adopted.

There are several triage systems currently in use in emergency departments in the UK. The Manchester Triage System assigns the patient to one of five categories based on the maximum time that they can wait for full assessment.7,8 It provides only modest sensitivity (63%) to detect emergency or very urgent cases and is a generic instrument to deal with emergencies including trauma.9 Other triage systems used internationally include the Emergency Severity Index, the Paediatric Canadian Triage and Acuity Scale, Paediatric Risk of Admission Score, and the Paediatric Emergency Assessment Tool.1014 A number of more specific ‘scoring systems’ for children presenting to emergency departments with medical illness have been developed. None have shown sufficient ability to rule out serious infection in children to be widely adopted in an NHS context.1518

The National Institute for Health and Clinical Excellence (NICE) guideline for the management of feverish illness in children under 5 years of age was published in 2007.19 It is an important starting point for us because its recommendations are based on a literature review utilising stakeholders to identify key documents. However, we are aware of important recent studies which were not included and no attempt was made to explore the data at an individual patient level.

Conducting individual patient level meta-analysis is important to provide evidence to underpin several of the NICE recommendations – for example the recommendation that ‘Healthcare professionals should measure and record temperature, heart rate, respiratory rate and capillary refill time as part of the routine assessment of a child with fever’.19

Research methods

The proposed project will involve a systematic review of the literature on clinical predictors of serious infection in children, including systematic review and standard meta-analysis where appropriate of all studies and individual-patient data meta-analysis of identified studies on unselected populations (the studies most likely to provide reliable predictive values for triage in urgent access primary care and emergency care settings in the UK).

In our preparatory work developing this protocol (to assess the size of the task and the feasibility of individual patient data meta-analysis) we have identified four studies published in the past 20 years in unselected populations of children presenting to urgent-access primary care (from Belgium20) and children presenting to emergency department (from the Netherlands21,22) but have been made aware of three substantial but as yet unpublished datasets from the UK. The investigators of all seven studies (total sample 11,328) have agreed to supply us with their individual patient data and support this analysis (letters available).

Literature search

The literature will be searched in MEDLINE, EMBASE, DARE and CINAHL. The search strategy will consist of a combination of terms on serious infections, terms referring to ‘signs and symptoms’, laboratory tests, children, ambulatory care and infections, using both MeSH terms and free text words if appropriate. In addition, the reference lists of the articles thus retrieved will be checked. A search for any unpublished material will consist of contacting known researchers in the field.

The research team have already performed a provisional literature search using the proposed search strategy: 892 articles were identified from MEDLINE, 718 from EMBASE, 7 from DARE and 86 from CINAHL. After duplicates had been discarded, the total number of citations was 1578, as detailed below:

5InfectionsArthritis, Infectious”[Mesh] OR “Bone Diseases, Infectious”[Mesh] OR “Community-Acquired Infections”[Mesh] OR “Respiratory Tract Infections”[Mesh] OR “Sepsis”[Mesh] OR “Skin Diseases, Infectious”[Mesh] OR “Soft Tissue Infections”[Mesh] OR “Urinary Tract Infections”[Mesh] OR “Meningitis”[Mesh] OR meningitis OR serious infections OR “Gastroenteritis”[Mesh]586,606
7Signs and symptoms“Signs and Symptoms”[MeSH] OR signs and symptoms OR “Fever”[MeSH] OR fever OR fast breathing OR tachypnoea OR respiratory rate OR yale observation scale OR yale score OR yale scale OR Nelson score OR Nelson scale OR young infant observation scale OR “Tachycardia”[Mesh] OR fast heart rate OR capillary refill time1,324,204
9Laboratory tests“Laboratory Techniques and Procedures”[Mesh]1,158,471
12Child“infant”[MeSH Terms] OR “child”[MeSH Terms] OR “adolescent”[MeSH Terms] OR paediatric [All fields] OR pediatric [All fields] OR “pediatrics” [MeSH term] OR p*ediatric* OR child* OR infant* OR bab* OR neonat* OR newborn* OR toddler*2,584,184
14bisAmbulatory care“Ambulatory Care”[Mesh] OR “Family Practice”[Mesh] OR general practice OR GP OR “Physicians, Family”[Mesh] OR “Primary Health Care”[Mesh] OR “Emergency Service, Hospital”[Mesh] OR primary care208,882
24Combination5 AND (7 OR 9) AND 12 AND 14bis892
E1Infections‘infectious arthritis’/exp OR ‘hematogenous osteomyelitis’/exp OR ‘communicable disease’/exp OR ‘respiratory tract infection’/exp OR ‘sepsis’/exp OR ‘skin infection’/exp OR ‘soft tissue infection’/exp OR ‘urinary tract infection’/exp OR ‘meningitis’/exp OR ‘gastroenteritis’/exp OR serious AND infections73,777
E2Signs and symptoms‘physical disease by body function’/exp OR (signs AND symptoms) OR ‘fever’/exp OR fever OR (fast AND breathing) OR tachypnoea OR (respiratory AND rate) OR (yale AND observation AND scale) OR (yale AND score) OR (yale AND scale) OR (nelson AND score) OR (nelson AND scale) OR (young AND infant AND observation AND scale) OR ‘tachycardia’/exp OR (fast AND heart AND rate) OR (capillary AND refill AND time)4,255,612
E3Laboratory tests‘laboratory diagnosis’/exp91,178
Children‘infant’/exp OR ‘preschool child’/exp OR ‘school child’/exp OR ‘toddler’/exp OR ‘adolescent’/exp OR ‘pediatrics’/exp OR p*ediatric* OR child* OR infant* OR bab* OR neonat* OR newborn* OR toddler*2,578,259
E4Ambulatory care‘ambulatory care’/exp OR ‘general practice’/exp OR (general AND practice) OR gp OR ‘general practitioner’/exp OR (family AND physician) OR ‘primary medical care’/exp OR (primary AND care) OR ‘emergency ward’/exp362,300
E5CombinationE1 AND (E2 OR E3) AND E4718
D1Infections““Arthritis, Infectious”[Mesh] OR “Bone Diseases, Infectious”[Mesh] OR “Community-Acquired Infections”[Mesh] OR “Respiratory Tract Infections”[Mesh] OR “Sepsis”[Mesh] OR “Skin Diseases, Infectious”[Mesh] OR “Soft Tissue Infections”[Mesh] OR “Urinary Tract Infections”[Mesh] OR “Meningitis”[Mesh] OR meningitis OR serious infections OR “Gastroenteritis”[Mesh]254
D2Signs and symptoms OR laboratory tests(“Signs and Symptoms”[MeSH] OR signs and symptoms OR “Fever”[MeSH] OR fever OR fast breathing OR tachypnoea OR respiratory rate OR yale observation scale OR yale score OR yale scale OR Nelson score OR Nelson scale OR young infant observation scale OR “Tachycardia”[Mesh] OR fast heart rate OR capillary refill time)495
D3Laboratory tests(“Laboratory Techniques and Procedures”[Mesh])17
D4Child“infant”[MeSH Terms] OR “child”[MeSH Terms] OR “adolescent”[MeSH Terms] OR paediatric [All fields] OR pediatric [All fields] OR “pediatrics” [MeSH term] OR p*ediatric* OR child* OR infant* OR bab* OR neonat* OR newborn* OR toddler*0
D5Child“infant”[MeSH Terms] OR “child”[MeSH Terms] OR “adolescent”[MeSH Terms] OR paediatric [All fields] OR pediatric [All fields] OR “pediatrics” [MeSH term] OR child* OR infant* OR bab* OR neonat* OR newborn* OR toddler*0
D6Child“infant”[MeSH Terms] OR “child”[MeSH Terms] OR “adolescent”[MeSH Terms] OR “pediatrics” [MeSH term] OR child* OR infant* OR bab* OR neonat* OR newborn* OR toddler*0
D7Child““infant”[MeSH Terms] OR “child”[MeSH Terms] OR “adolescent”[MeSH Terms]973
D8Ambulatory care“Ambulatory Care”[Mesh] OR “Family Practice”[Mesh] OR general practice OR GP OR “Physicians, Family”[Mesh] OR “Primary Health Care”[Mesh] OR “Emergency Service, Hospital”[Mesh] OR primary care2346
D9CombinationD1 AND (D2 OR D3) AND D7 AND D87
C1Infectionsexp Arthritis, Infectious/OR exp Bone Diseases, Infectious/OR exp Community-Acquired Infections/OR exp Respiratory Tract Infections/OR exp SEPSIS/OR exp Skin Diseases, Infectious/OR exp Soft Tissue Infections/OR exp Urinary Tract Infections/OR exp Meningitis/OR exp GASTROENTERITIS/OR serious [mp=title, subject heading word, abstract, instrumentation]30,021
C2Signs and symptoms(signs and symptoms).mp. [mp=title, subject heading word, abstract, instrumentation] OR exp FEVER/OR exp Respiratory Rate/OR OR fast OR yale observation OR yale OR yale OR nelson OR nelson OR young infant observation OR exp TACHYCARDIA/OR fast heart OR capillary refill
C3Lab testsexp Diagnosis, Laboratory/38,138
C4Childexp INFANT/OR exp CHILD/OR exp Adolescence/OR OR exp Pediatrics/OR child$.mp. OR infant$.mp. OR newborn$.mp. OR bab$.mp. OR neonat$.mp. OR toddler$.mp. OR exp Child, Preschool/233,317
D4Ambulatory careexp Ambulatory Care/OR exp Family Practice/OR general OR OR exp Physicians, Family/OR exp Primary Health Care/OR exp Emergency Service/OR primary,377
D5CombinationC1 AND (C2 OR C3) AND C4 AND C586

Screening of titles and abstracts

Titles and abstracts will be screened by two independent reviewers, with discrepancies resolved by a third independent reviewer. The principal inclusion criterion will be a study on the predictive value of potential indicators for the diagnosis of serious infections in children; we will include systematic reviews and meta-analyses as well as primary studies. Serious infections will be defined as sepsis, pneumonia, meningitis, urinary tract infection, bacterial gastro-enteritis, cellulitis requiring hospital intervention, osteomyelitis and bronchiolitis requiring hospitalisation. Diagnostic indicators will be defined as any symptom, sign, test or other potential discriminator (e.g. doctor or parent opinion) used to predict or rule out the presence of illness.

We have drawn up the provisional reviewer guidelines below, which will be expanded and refined if studies come to light that are not easily included/excluded by the guideline:

CharacteristicInclude ifExclude If
  1. Cross-sectional study of immediate diagnostic accuracy
  2. Longitudinal study of predictive accuracy
  3. Systematic reviews of above studies
  1. Case series of <50 children
  2. Letters without research results
  3. Narrative study or comment only
  4. Therapy evaluation
  1. Includes children age 1 month-18 years (separately delineable)
  2. Otherwise healthy
  1. out of age range
  2. pre-existing illness
  1. General practice/family medicine
  2. Other Ambulatory care
  3. Paediatric assessment unit
  4. Pre-admission Emergency Care
  1. Post-admission secondary care
  2. Outside Europe, North-America, Australia/NZ

(i.e. serious infection)
  1. Hospitalisation with presumed or confirmed serious infection (e.g. LRTI or pneumonia, meningitis, sepsis; osteomyelitis; complications of gastrointestional or respiratory infection).
  2. Specific infections diagnosed in a community setting: Pneumonia (with x-ray confirmation); other LRTI with quantified hypoxia; UTI (with microbiological confirmation).
Diagnosis other than serious infection
Diagnostic procedures
  1. History and presenting symptoms – e.g. fever, cough, vomiting, pallor, crying pattern, lethargy, irritability
  2. Observation scales (e.g. McCarthy, Baby Check, Young infant observation scale) or triage scores (e.g. Manchester triage score)
  3. Physical examination – e.g. vital signs, meningeal signs, capillary refill time
  4. Near-patient tests – e.g. urine dispstick; influenza or RSV testing; CRP
  5. Rapid laboratory tests – e.g. WBC; inflammatory markers; urine microscopy
  1. Imaging
  2. Invasive testing
  3. In-lab microbiology other urine culture and microscopy.
NB These procedures may be used for confirmation of outcome

The selection process will be piloted on a sample of 20 articles, and interobserver agreement will be calculated for the entire sample.

Quality assessment

Selected articles will be assessed on quality by using the QUADAS instrument.23,24 The QUADAS instrument has 11 core items and 9 additional items. The use of QUADAS is currently endorsed by the Cochrane Collaboration in the new handbook of diagnostic systematic reviews. Some of the co-applicants were involved in the writing of this new handbook, which is due to be published shortly. Not all items will be applicable in our review.

1Was the spectrum of patients representative of the patients who will receive the test in practice?Yes
2Is the reference standard likely to correctly classify the target condition?Yes
3Is the time period between reference standard and index test short enough to be reasonably sure that the target condition did not change between the two tests?Yes
4Did the whole sample or a random selection of the sample receive verification using a reference standard or diagnosis?Yes
5Did patients receive the same reference standard regardless of the index test used?Yes
6Was the reference standard independent of the index test?Not always, e.g. sepsis
7Were the index test results interpreted without knowledge of the results of the reference standard?Yes
8Were the reference standard results interpreted without knowledge of the results of the index test?Not always
9Were the same clinical data available when test results were interpreted as would be available when the test is used in practice?Yes, although clinical data are index tests
10Were uninterpretable/intermediate test results reported?Yes
11Were withdrawals from the study explained?Yes
Additional items
12If a cut-off has been used, was it established before the study was started?Not always (e.g. ROC analysis)
13Is the technology of the index test likely to have changed since the study was carried out?No, unless for lab tests
14Did the study provide a clear definition of what was considered to be a 'positive' result?Yes
15Was treatment started after the index test was carried out but before the reference standard was performed?Yes
16Were data on observer variation reported?Less applicable
17Were data on instrument variation reported?Not applicable
18Were data presented for appropriate patient subgroups?Yes
19Was an appropriate sample size included?Yes
20Were objectives pre-specified?Yes

Data extraction

The following data will be extracted from the included articles:

  1. Design features, prospective or retrospective, consecutive patient inclusion.
  2. The setting: emergency department, ambulatory care, in hospital or other.
  3. The age and other patient characteristics.
  4. The outcome and how that was defined (reference standard).
  5. The index test, with details and cut-off used.
  6. The number of participants and the prevalence of the outcome.
  7. The results from the study, in sensitivity, specificity, positive or negative predictive value, odds ratios, area under curves (AUC) or p-values. Confidence intervals (CI) will be extracted where possible. When sufficient data are reported, 2 × 2 tables will be extracted.

The data will be extracted in duplicate by two independent researchers. If possible, authors will be contacted to supplement missing data.

Summarising the data

A. Study level meta-analysis

Depending on the nature of the available data, a meta-analysis will be performed. Diagnostic accuracy studies will be pooled using the bivariate method.2527 The bivariate approach preserves the two-dimensional nature of the original data. Pairs of sensitivity and specificity are jointly analyzed, incorporating any correlation that might exist between these two measures using a random effects approach. This method has been shown to be equivalent to the hierarchical summary receiver operating characteristic (ROC) model which is considered the gold standard for diagnostic meta-analyses, but the results of the bivariate method are easier to interpret in clinical terms.28,29

Before deciding to pool any studies, heterogeneity both in terms of clinical heterogeneity (by detailed study of the methods section of the paper and of protocol articles if available) and statistical heterogeneity (by calculating I2) will be assessed. If possible, pooling will take the natural order of tests into account – in clinical practice, history and clinical examination are done before requesting laboratory tests.

B. Individual patient data meta-analysis

As stated above, in our provisional work we have identified five studies (with an aggregate population of about 10,000 patients) on unselected populations and have formal agreement to use the crude data for IPD analysis. These datasets are detailed below:

  • Coventry, UK: 700 children presenting to hospital paediatric assessment unit/A&E with suspected acute infection (Dr Thompson).
  • Oxfordshire & Somerset, UK: 2000 children presenting to general practice and out of hours centres with acute infection (Dr. M. Thompson);
  • Nottingham, UK: 1700 children presenting to A&E with suspected acute infection (Dr Lakhanpaul)
  • Netherlands: 3 datasets – 595 children presenting to emergency department with fever without source; 400 children with meningeal signs; 1787 children presenting to emergency department with fever (Dr H. Moll)
  • Belgium: 4000 children presenting to primary care with acute infection (Dr Van Den Bruel)

These datasets will be complemented by the studies identified in the systematic review. Authors will be asked to contribute data in whatever format they prefer in order to facilitate contribution to the study. If further studies on unselected populations are identified we will attempt to include them, although we recognise that it is usually impossible to retrieve individual patient data for studies published more than 20 years ago, and sometimes difficult to get agreement for release of data for more recent studies. Moreover, we will need to address the applicability of studies performed prior to vaccination for Haemophilus influenzae, Streptococcus pneumoniae and Neisseria meningitidis C. We will explore bias and generalizability by comparing the test characteristics (e.g. sensitivity, specificity) generated by the IPD analysis with the results reported by any studies we identify on unselected populations for which full data is unavailable.

In conducting the IPD analysis we will take a two-stage approach, generating diagnostic algorithms using logistic regression (to generate odds ratios) in stage 1 and then assessing the predictive value and ROC characteristics of these algorithms in stage 2. In order to validate the prediction rule we will need to address two issues: a) over-optimistic estimates and b) transferability of the prediction rule across settings. To achieve this we will use k-fold cross-validation to obtain more realistic estimates and calibration using other datasets to test the transferability of the model. By validating the clinical prediction rules on patients in broader settings (and thus different disease prevalence and spectrum) from those used to derive the rule, we will be able to demonstrate the generalizability or external validity of the rule (McGinn et al., JAMA 2000). We anticipate that this process will require model revision and/or shrinkage methods (Steyerberg EW et al., Statist Med 2004).

As with the standard meta-analyses, to decide whether pooling of data for analysis is justified we will assessed heterogeneity between studies using I2, which describes the percentage of variation between studies due to heterogeneity rather than chance. The range for I2 lies between 0% (i.e., no observed heterogeneity) and 100%; we will pool if I2 is lower than 25% (p > 0·30).

In stage 1 a two-level multilevel regression model will be fitted for the diagnostic variables of interest, with patients corresponding to level one units and individual study as level two units. This will generate odds ratios for the likelihood of the main outcome (serious infection). Study effects will be represented by fixed effects, whilst patient effects will be represented by random effects. The diagnostic factors included in the analysis will be used as covariates. We will use a binary dummy variable to identify each study within the regression analysis.

To reduce bias and to increase statistical efficiency, we will impute missing data using the linear regression method (multivariate analyses) available in Spss (version 12.0). Regression will be based on the correlation between individual variables with missing values and all other variables, as estimated from the complete set of data. We will impute missing values only within individual studies.

In stage 2 we will construct a number of diagnostic algorithms, using the odds ratios for individual diagnostic markers derived in stage 1, and calculate their sensitivity, predictive value when applied to the second half of the dataset. To conduct sensitivity analyses, we will also report these results applied to each dataset separately. Confidence intervals around these test characteristics will be reported with 95% confidence intervals based ion the standard error of a proportion. Where appropriate we will develop two-level staged algorithms (e.g. undertaking a diagnostic test being dependent on presenting symptoms and signs) and present ROC curves where the algorithm includes a diagnostic test or marker generating a continuous variable (e.g. % oxygen saturation).

Subgroup analyses will be attempted. Subgroups will be based on patient age, i.e. children under the age of 1 year, children between 1 and 4 years, children between 5 and 12 years, and adolescents. Another subgroup is based on setting, reflecting increasing prevalence of serious infections: general practice – urgent access primary care – paediatric assessment unit – emergency department. A final category will be based on outcome. Generating additional separate algorithms for sepsis/meningitis and pneumonia would be desirable, as the first outcome requires immediate action, and the second outcome is the most prevalent serious infection in children in primary and secondary care.

Research Governance

The University of Oxford will be the nominated sponsor for this study.

4. Project timetable and milestones

The following will be the key milestones for the study:

MilestoneStart dateCompletion date
Perform literature searchMonth 1Month 1
Obtain data from 5 existing datasetsMonth 1Month 4
Screen titles and abstractsMonth 2Month 2
Quality grading of included studiesMonth 3Month 4
Data extraction from included studiesMonth 4Month 6
Summarising data, meta-analysisMonth 6Month 9
Individual patient data meta-analysisMonth 6Month 10
Writing final report and submitting for publicationMonth 10Month 12

5. Expertise

The research team that has been assembled for this project brings together methodological expertise in systematic reviewing, diagnostic test systematic reviewing, individual patient data meta analysis, as well as considerable clinical expertise in both emergency departments and primary care settings. Moreover it draws on this expertise not only from the UK, but also from Belgium and the Netherlands.

Dr Thompson is a Clinical Lecturer in Primary Care and half-time Principal in General Practice who also works regularly in an out of hours GP surgery. He has performed several research studies examining clinical predictors of serious infections in primary care and paediatric assessment units. These have included prospective studies of predictive value of vital signs, severity of illness scores and inflammatory markers in children a paediatric assessment unit. He has also published on the early signs of meningococcal disease in children. His systematic review experience includes the treatment of common upper respiratory tract infections with steroids, and he is also currently a member of two National Institute for Health & Clinical Excellence (NICE) guideline development groups (prescribing antibiotics for upper respiratory tract infections, and diagnosis and treatment of meningitis). In addition to his clinical experience and expertise in diagnostic studies he will be responsible for coordinating the proposed study and will supervise the staff funded by this grant.

Professor David Mant is the head of the Department of Primary Health Care at the University of Oxford and has an international reputation in primary care research. His research has included numerous seminal studies on childhood infections and cardiovascular disease. He was the PI of the MRC-funded Oxford Childhood infection study from 2001–6, and sits on national committees such as the Standing Advisory Committee on Antibiotic Resistance (paediatric sub-group) 2005–7 and National Expert Panel on New and Emerging Infections (2003–7). He will contribute extensive methodological input on the study design and meta-analysis, and will provide direct support to Dr Thompson.

Dr Glasziou has extensive expertise in conducting systematic reviews and individual patient data meta-analyses. He has published several textbooks on systematic reviewing, and authored numerous systematic reviews. He is a member of the Cochrane Collaborations diagnostics sub-group and currently Professor of Evidence Based Medicine at the University of Oxford. He is also a practising GP in Oxford. He will contribute extensively to the methodology of the systematic review and IPD meta-analysis.

Dr Van den Bruel has done research in the area of serious infections in children for the last 6 years. She has performed several studies, including one in which a clinical prediction rule for the exclusion of serious infections was developed. In addition, she has been working at the Belgian HTA agency for the last 4 years, where she is responsible for the evaluation of diagnostic tests and the methodology of systematic reviews. Previously she worked as a GP for seven years. In addition to sharing her dataset of children with acute infections, she will contribute expertise in systematic reviews of diagnostic studies.

Dr Moll is Head of the Paediatric Emergency Department of the Sophia's Children's Hospital – ErasmusMC in Rotterdam. Her research has focussed on emergency department triage, and the development and validation of prediction rules for acute paediatric infections. In particular she has performed studies on meningitis, fever/serious bacterial infections, RSV and pneumonia in the emergency department setting. She will contribute not only her dataset to this study, but also her clinical experience in emergency paediatrics, and research experience in diagnostic studies and validation of prediction rules.

Professor Buntinx has been working as a GP in Belgium for 32 years and as a researcher and Professor at the Departments of General Practice of the University of Maastricht (Netherlands) and Leuven (Belgium) since 1989. From the start of his research career, he has been focusing on the methodology and execution of diagnostic studies, including multivariate analyses and diagnostic meta-analysis. He has published some 175 papers in international peer reviewed journals and almost as many in Dutch language peer reviewed journals. He currently serves as the research director of the Department of General Practice in Leuven and as the founding president of the Belgian Centre of Evidence-based Medicine. With Prof. Knottnerus he also is the co-editor of the new edition of ‘The evidence base of clinical diagnosis’ (Blackwells, in press). In 2006, he wad elected member of the Belgian Royal Academy of Medicine. He will contribute to the methodological input on diagnostic studies and systematic review of diagnostic studies.

Professor Bert Aertgeerts is a GP in Belgium. He has done research on screening for alcohol abuse and dependence in different settings, and has conducted several systematic diagnostic reviews on various clinical topics. He was also responsible for the European First Aid Manual, led by Stijn Vandevelde from the Red Cross Flanders. He is the director (2001) of the Centre of Evidence-Based Medicine (Belgian Branch of the Dutch Cochrane Collaboration) and is currently head of the department of General Practice at the Katholieke Universiteit Leuven. In 2006, he was elected member of the Belgian Royal Academy of medicine. He will contribute to the systematic review methodology, particularly in relation to diagnostic studies.

Professor Geert-Jan Dinant is vice-chair of the Department of General Practice at the University of Maastricht in the Netherlands. He has extensive research experience in performing diagnostic studies in primary care on pneumonia and osteoporosis and has authored textbooks on evidence based clinical diagnosis. He will bring methodological expertise in diagnostic studies to the research team.

Dr Shelly Segal is a Consultant Paediatrician with special interest in Infectious disease and is the Clinical lead in the Paediatric Emergency department at the John Radcliffe Hospital in Oxford. In addition to her paediatric emergency experience, she has performed several studies on the genetic susceptibility to infectious diseases in children, particularly invasive pneumoccal disease. She will contribute her paediatric emergency clinical experience to the research team.

Dr Monica Lakhanpaul is a Consultant Paediatrician and Co-Director for National Collaborating Centre for Women's Health and Children's Health of the Royal College of Paediatrics and Child Health which has been responsible for undertaking several NICE reviews, in particular the recent guideline on management of the Feverish child. She is also clinical lead for developing nurse-led urgent care services in Leicester. She has recently completed a prospective study of children attending A&E in Nottingham. In addition to sharing dataset for the IPD meta-analysis, she will contribute clinical paediatric experience, and experience of literature review to the project team.

Dr Rafael Perera is a University Lecturer in Statistics and Director of Research Methodologies at the Centre for Evidence-Based Medicine. He has extensive experience conducting systematic reviews and IPD meta-analysis and has published numerous systematic reviews. He will contribute extensively to the data analysis for this study.

6. Service users

The research team has extensive current experience as front-line service clinicians in the provision of clinical care to children in emergency and urgent primary care settings. Dr Segal is in charge of paediatric emergency medicine at the John Radcliffe Hospital in Oxford, Dr Moll is Head of Paediatric Emergency Department at Sophia's Children Hospital in Rotterdam, and Dr Lakhanpaul is a Community Paediatrician in Leicester. Professors Mant, Glasziou, Dinant, Aertgeerts, Buntinx, and Drs Thompson and Van den Bruel are all general practitioners who have worked in general practice in England, the Netherlands, and Belgium. Dr Thompson also works in an out of hours GP centre in Oxford. We will also gather the input of parents/carers input in order to assess the likely impact of this rule in the real world setting, and to ensure that the predictors we identify (e.g. vital sign measurements, possibly blood tests) are acceptable to most parents/carers. We will therefore assemble a group of parents who have had personal experience with children in emergency care or urgent access primary care and obtain their input on the final prediction rules.

7. Justification of support required

The main support required for this project are salary support for a data manager, salary support for a statistician, and reimbursement for meetings of the research team. The data manager will be responsible for the retrieval of articles identified in the systematic review, obtaining and assembling the datasets required for the individual patient data meta analysis, as well as organising meetings of the research team. The data manager may be expected to undertake simple descriptive analysis of the datasets. Salary support has been requested for Dr Perera to undertake statistical work for this study. Dr Perera will be responsible for the summarising of the studies identified in the systematic review. He will also undertake the individual patient data meta-analysis. Dr Van den Bruel will be reimbursed for undertaking one part of the analysis, and will invoice the University of Oxford for a specified component of this work. In order to take advantage of the considerable expertise of the collaborators who have agreed to work on this project, reimbursement for attending four research team meetings during the study period will be provided to Professor Buntinx, Professor Aertgeerts, Professor Dinant, Dr Van den Bruel, Dr Moll, Dr Lakhanpaul and Dr Segal. Nominal salary support has been requested for Dr Lakhanpaul, Professor Mant, Professor Glasziou, Dr Segal and Dr Thompson. No salary support has been requested for Professor Buntinx, Professor Aertgeerts, Professor Dinant, or Dr Moll.

In order to minimise impact on the environment the Dutch and Belgian collaborators on this study will attend meetings in England by rail where possible rather than flying. This project also seeks to reduce the need and costs associated with further prospective studies of predictors of serious infection in children, by taking advantage of literature that has already been published on this topic, and by using individual patient data meta-analysis of studies that have already been carried out.

8. References

Stiell I, Wells G, Laupacis A, Brison R, Verbeek R, Vandemheen K, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. Multicentre Ankle Rule Study Group. BMJ. 1995;311(7005):594–7. [PMC free article: PMC2550661] [PubMed: 7663253]
Armon K, Stephenson T, Gabriel V, MacFaul R, Eccleston P, Werneke U, et al. Determining the common medical presenting problems to an accident and emergency department. Arch Dis Child. 2001;84(5):390–2. [PMC free article: PMC1718762] [PubMed: 11316679]
Avery AJ, Groom L, Boot D, Earwicker S, Carlisle R. What problems do patients present with outside normal general practice surgery hours? A prospective study of the use of general practice and accident and emergency services. J Public Health Med. 1999;21(1):88–94. [PubMed: 10321866]
Brogan C, Pickard D, Gray A, Fairman S, Hill A. The use of out of hours health services: a cross sectional survey. BMJ. 1998;316(7130):524–7. [PMC free article: PMC2665637] [PubMed: 9501716]
Munro J, Nicholl J, O'Cathain A, Knowles E. Impact of NHS direct on demand for immediate care: observational study. BMJ. 2000;321(7254):150–3. [PMC free article: PMC27434] [PubMed: 10894694]
Thompson MJ, Ninis N, Perera R, Mayon-White R, Phillips C, Bailey L, et al. Clinical recognition of meningococcal disease in children and adolescents. Lancet. 2006;367(9508):397–403. [PubMed: 16458763]
Mackway-Jones K. Emergency triage: Manchester Triage Group. BMJ Publishing Group; 1996.
Cooke MW, Jinks S. Does the Manchester triage system detect the critically ill? J Accid Emerg Med. 1999;16(3):179–81. [PMC free article: PMC1343329] [PubMed: 10353042]
Roukema J, Steyerberg EW, van Meurs A, Ruige M, van der Lei J, Moll HA. Validity of the Manchester Triage System in paediatric emergency care. Emerg Med J. 2006;23(12):906–10. [PMC free article: PMC2564249] [PubMed: 17130595]
Baumann MR, Strout TD. Evaluation of the Emergency Severity Index (version 3) triage algorithm in pediatric patients. Acad Emerg Med. 2005;12(3):219–24. [PubMed: 15741584]
Tanabe P, Gimbel R, Yarnold PR, Adams JG. The Emergency Severity Index (version 3) 5-level triage system scores predict ED resource consumption. J Emerg Nurs. 2004;30(1):22–9. [PubMed: 14765078]
Gouin S, Gravel J, Amre DK, Bergeron S. Evaluation of the Paediatric Canadian Triage and Acuity Scale in a pediatric ED. Am J Emerg Med. 2005;23(3):243–7. [PubMed: 15915392]
Chamberlain JM, Patel KM, Pollack MM. The Pediatric Risk of Hospital Admission score: a second-generation severity-of-illness score for pediatric emergency patients. Pediatrics. 2005;115(2):388–95. [PubMed: 15687449]
Gorelick MH, Lee C, Cronan K, Kost S, Palmer K. Pediatric emergency assessment tool (PEAT): a risk-adjustment measure for pediatric emergency patients. Acad Emerg Med. 2001;8(2):156–62. [PubMed: 11157292]
Nelson KG. An index of severity for acute paediatric illness. Am J Pub Health. 1980;70:804–7. [PMC free article: PMC1619610] [PubMed: 7416339]
McCarthy PL, Sharpe MR, Spiesel SZ, Dolan TF, Forsyth BW, DeWitt TG, et al. Observation scales to identify serious illness in febrile children. Pediatrics. 1982;70(5):802–9. [PubMed: 7133831]
Hewson P, Poulakis Z, Jarman F, Kerr J, McMaster D, Goodge J, et al. Clinical markers of serious illness in young infants: a multicentre follow-up study. J Paediatr Child Health. 2000;36(3):221–5. [PubMed: 10849220]
Hewson PH, Gollan RA. A simple hospital triaging system for infants with acute illness. J Paediatr Child Health. 1995;31(1):29–32. [PubMed: 7748685]
NICE. National Institute for Clinical Excellence: Feversh illness in children – assessment and initial management in children younger than 5 years. 2007.
Van den Bruel A, Aertgeerts B, Bruyninckx R, Aerts M, Buntinx F. Signs and symptoms for diagnosis of serious infections in children: a prospective study in primary care. Br J Gen Pract. 2007;57:538–546. [PMC free article: PMC2099636] [PubMed: 17727746]
Bleeker SE, Derksen-Lubsen G, Grobbee DE, Donders AR, Moons KG, Moll HA. Validating and updating a prediction rule for serious bacterial infection in patients with fever without source. Acta Paediatr. 2007;96(1):100–4. [PubMed: 17187613]
Oostenbrink R, Moons KG, Derksen-Lubsen AG, Grobbee DE, Moll HA. A diagnostic decision rule for management of children with meningeal signs. Eur J Epidemiol. 2004;19(2):109–16. [PubMed: 15074565]
Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3:25. [PMC free article: PMC305345] [PubMed: 14606960]
Whiting PF, Weswood ME, Rutjes AW, Reitsma JB, Bossuyt PN, Kleijnen J. Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies. BMC Med Res Methodol. 2006;6:9. [PMC free article: PMC1421422] [PubMed: 16519814]
van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med. 2002;21(4):589–624. [PubMed: 11836738]
Van Houwelingen HC, Zwinderman KH, Stijnen T. A bivariate approach to meta-analysis. Stat Med. 1993;12(24):2273–84. [PubMed: 7907813]
Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005;58(10):982–90. [PubMed: 16168343]
Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001;20(19):2865–84. [PubMed: 11568945]
Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JA. A unification of models for metaanalysis of diagnostic accuracy studies. Biostatistics. 2007;8(2):239–51. [PubMed: 16698768]

9. Flow diagram

Image app7fu1
© 2012, Crown Copyright.

Included under terms of UK Non-commercial Government License.

Cover of Systematic Review and Validation of Prediction Rules for Identifying Children with Serious Infections in Emergency Departments and Urgent-Access Primary Care
Systematic Review and Validation of Prediction Rules for Identifying Children with Serious Infections in Emergency Departments and Urgent-Access Primary Care.
Health Technology Assessment, No. 16.15.
Thompson M, Van den Bruel A, Verbakel J, et al.
Southampton (UK): NIHR Journals Library; 2012 Mar.

PubMed Health Blog...

read all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...