2Methods

In this chapter we document a prospectively developed protocol that was used to conduct this evidence report. A core research team was assembled by the University of Alberta Evidence-based Practice Center (UAEPC). In consultation with the Agency for Healthcare Research and Quality (AHRQ) Task Order Officer (TOO) and the American Academy of Asthma Allergy and Immunology (AAAAI) representative, a Technical Expert Panel (TEP) was assembled to provide content and methodological expertise in the development of this report (Appendix A).

Literature Search and Retrieval

The research librarian, in collaboration with the core research team and TEP, developed search strategies designed to identify evidence relevant to each key question of the report.

For questions relating to the diagnostic test accuracy reviews (Key Questions D–1 to D–6), we systematically searched the following electronic resources: MEDLINE®, EMBASE, EBM Reviews - Cochrane Central Register of Controlled Trials, AMED, PsycINFO, PASCAL, CINAHL®, SPORTDiscus with Full Text, Academic Search Elite, Web of Science®, BIOSIS Previews®, PubMed, Scopus®, the Medion Database of Diagnostic Reviews - University of Maastricht, and Proquest Dissertations and Theses. The original searches were performed between November 5 and 28, 2008. On July 31, 2009, the, the searches were updated using the original search strategies in MEDLINE®, EMBASE, Cochrane Central Register of Controlled Trials, PASCAL, SPORTDiscus with Full Text, Web of Science®, BIOSIS Previews®, PubMed, and Scopus®.

Search terms were identified by reviewing search strategies of systematic reviews on similar topics and by looking at how potentially relevant studies were indexed in various databases. A combination of subject headings and text words was adapted for each electronic resource: (exerc* OR train* OR fitness OR physical OR athlete* OR sport*) AND (bronchoconstrict* OR asthma* OR antiasthma* OR wheez* OR “Respiratory Sounds” OR “Bronchial Spasm” OR bronchospas* OR “Bronchial Hyperreactivity” OR “Respiratory Hypersensitivity” OR (bronch* AND spasm*) OR (bronch* AND constrict*) OR (bronchial* OR respiratory OR airway* OR lung*) AND (hypersensitiv* OR hyperreactiv* OR allerg* OR insufficiency)) OR EIB OR EIA. Terms used to limit the searches to diagnostic studies included: diagnos* or predict* or probability or accuracy or sensitivity or specificity. Searches were limited to English language. Date restrictions were not applied. (See Appendix B for exact search strings).

Handsearches were conducted to identify literature from symposia proceedings from the following scientific meetings: AAAAI Annual Meeting (2007–2008), American Thoracic Society (ATS) International Conference (2008), British Thoracic Society (BTS) Winter Meeting (2008), Chest Meeting (2008), and the European Respiratory Society (ERS) Annual Congress (2008). We also searched the last 3 years (2006–2008) of conference proceedings of the American College of Sports Medicine (ACSM) and the Canadian Society for Exercise Physiology (CSEP).

For questions relating to the therapy reviews (Key Questions T–1 to T–6), a comprehensive search was run in the Cochrane Airways Register. The Register contains references to randomized controlled trials (RCTs) from systematic searches of bibliographic databases including Cochrane Central Register of Controlled Trials, Medline®, Embase, Cinahl®, AMED, PsycINFO, as well as handsearching of respiratory journals and meeting abstracts. The original search was performed on November 5, 2008. On August 4, 2009, the, the search was updated using the original search strategy.

Search terms were identified by reviewing search strategies of systematic reviews on similar topics and by looking at how potentially relevant studies were indexed in various databases. A combination of subject headings and textwords were used: (exerc* OR train* OR fitness OR physical OR athlete* OR sport*) AND (bronchoconstrict* OR asthma* OR antiasthma* OR wheez* OR “Respiratory Sounds” OR “Bronchial Spasm” OR bronchospas* OR “Bronchial Hyperreactivity” OR “Respiratory Hypersensitivity” OR (bronch* AND spasm*) OR (bronch* AND constrict*) OR (bronchial* OR respiratory OR airway* OR lung*) AND (hypersensitiv* OR hyperreactiv* OR allerg* OR insufficiency)) OR EIB OR EIA. We did not apply language or date restrictions. A study design filter was not required as all references in this database are RCTs. (Appendix B). The results from the literature searches were entered into a Reference Manager for Windows bibliographic database version 11.0 (© 2004–2005 Thomson ResearchSoft).

We handsearched abstracts for those years not yet available in the Cochrane Airways Register for the following conference proceedings: AAAAI (2007–2008), ATS International Conference (2008), BTS Winter Meeting (2008), Chest Meeting (2008), ERS Annual Congress (2008). We also searched the last 3 years (2006–2008) of conference proceedings of the ACSM and the CSEP.

To identify ongoing studies we searched ClinicalTrials.gov and ClinicalStudyResults.org. Studies were also identified by checking reference lists of included studies.

Study Selection

Our inclusion/exclusion criteria (Tables 4 and 5) were developed in consultation with the TEP.

Table 4. Inclusion and exclusion criteria for the diagnostic test accuracy review.

Table 4

Inclusion and exclusion criteria for the diagnostic test accuracy review.

Table 5. Inclusion and exclusion criteria for the therapy review.

Table 5

Inclusion and exclusion criteria for the therapy review.

We used a two-step process for article screening. First, two reviewers independently screened the titles and abstracts (when available) to determine if an article met broad inclusion criteria. Each article was rated as “include,” “exclude,” or “unclear.” The full-text of all articles classified as “include” or “unclear” were retrieved for formal review. Second, two reviewers independently assessed each study using a standard inclusion/exclusion form (Appendix C). Disagreements were resolved through discussion between the two reviewers or third party adjudication, as needed.

Assessment of Methodological Quality

We assessed the methodological quality of the diagnostic test accuracy studies using the Quality Assessment of Studies of Diagnostic Accuracy (QUADAS) tool.11 This tool comprises 14 items that assess several common sources of bias in diagnostic studies, including spectrum bias, selection bias, information bias, verification bias, misclassification bias, disease progression bias, and excluded data. For the purpose of this review, we chose to include only 13 of the QUADAS items. A priori, we determined that the question regarding incorporation bias did not apply because none of the index tests under review were incorporated in the reference standard. Each item in the QUADAS tool is scored as “yes,” “no,” or “unclear.” In addition, the source of funding (e.g., industry, government, other) for each study was recorded.159 Results of the quality assessment are presented in the text and a table for each of the individual key questions (D–1 to D–6).

We assessed the methodological quality of RCTs included in the therapy review using the Jadad12 scale and Schulz’s criteria to assess allocation concealment.13,14 Components of the 5‐point Jadad scale include randomization, double blinding, and reporting of withdrawals and dropouts. Allocation concealment was assessed and scored as “adequate,” “inadequate,” or “unclear.” In addition, the source of funding for each study (e.g., industry, government, other) was recorded.159 Results are presented in the text and a table for each of the individual key questions (T–1 to T–6).

The methodological quality of included studies was assessed independently by two reviewers. Decision rules regarding application of the quality assessment tools were developed a priori (Appendix C). Discrepancies were resolved through discussion between the two reviewers or third party adjudication, as needed.

Grading the Body of Evidence

For the diagnostic test accuracy reviews, we graded the available evidence for each key question using the GRADE system for rating the quality of evidence and strength of recommendations for diagnostic tests.15 We assessed the strength of the study designs, the quantity and quality of individual studies, and the consistency and precision of the results. We also assessed the indirectness of evidence. For all studies, the outcomes were measures of test accuracy (i.e., true positive, true negative, false positive, false negative), which are surrogates for patient-important outcomes. We determined that true positive and true negative results would improve outcomes that are important to patients. True positive results will lead to administration of effective, safe prophylactic treatment for EIB/EIA; true negative results will spare patients unnecessary treatment or further testing for EIB/EIA. We identified uncertainty about the consequences of false positive results which could lead to unnecessary treatment for EIB/EIA and the potential exclusion of individuals from participating in sports or working in occupations requiring heavy physical demands. We also determined that there was uncertainty about directness for false negative results related to delayed access to prophylactic and/or rescue treatments, ongoing symptoms of EIB/EIA, underperformance in physical activity, and reduced quality of life leading to avoidance of physical activities and sports. This uncertainty about directness resulted in a reduction in the quality of evidence. For each key question the quality of evidence was graded as high, moderate, low or very low.

For the therapy reviews, the strength of evidence for the primary outcome (maximum percent fall in FEV1 from baseline) was assessed for each key question using the EPC approach to grading the strength of a body of evidence.16 This approach assesses the evidence based on four domains: risk of bias, consistency, directness, and precision. We classified the strength of evidence as high, moderate, low, or insufficient.

Data Extraction

Data were extracted using standard forms (Appendix C) by one reviewer and checked for accuracy and completeness by a second reviewer. Data extracted for each study included details of study design and inclusion/exclusion criteria, details of the population, details of the index test and reference standard (for the diagnostic test accuracy reviews), intervention and comparator (for the therapy reviews), and results obtained for various outcomes. Reviewers resolved discrepancies in data extraction by consensus or in consultation with a third party.

Data Analysis

The following data assumptions were made and imputations performed to transform reported data into the form required for this report. Graph extraction was performed using CorelDRAW® 9.0 (Corel Corp., Ottawa, Canada). Means were approximated by medians, and 95 percent empirical intervals were used to calculate approximate standard deviations (SD). Because the majority of the included studies in the therapy reviews used a crossover design, standard errors of mean differences were either computed exactly using individual patient data (IPD) or imputed using an estimated within-patient correlation of 0.5.

Diagnostic Test Accuracy Review

Our reference standard was a standardized ECT- either using a treadmill or bicycle ergometer. Our threshold for a positive test on the ECT was a fall in FEV1 of 10 percent or more from a pre-exercise baseline.18 Thresholds of 15 percent and 20 percent were examined in sensitivity analyses. The threshold for a positive test for the methacholine challenge was a 20 percent or more drop in FEV1 at a provocative concentration of less than 8 mg/ml (PC20).18 For mannitol the threshold was a 15 percent drop in FEV1 at less than 635mg or a drop of 10 percent between consecutive doses.23 The threshold for a positive test for EVH, FRAST and sport/venue specific challenges was a fall in FEV1 of 10 percent or more. Where data were available we conducted sensitivity analyses using a different threshold dose for methacholine (less than 16 mg/ml). True positives, false positives, true negatives, and false negatives were recorded for each of the six index tests. These data were used to calculate sensitivity and specificity for each study. We present individual study results graphically by plotting the estimates of sensitivity and specificity and their 95 percent confidence intervals (95 percent CI) in forest plots. Where there were more than five studies, we plotted the sensitivity and specificity in receiver operating characteristic (ROC) space and present a summary ROC curve. Area under the curve (AUC) was estimated.

Because many of the studies presented IPD, we were able to use our thresholds for both reference and index tests for our primary and sensitivity analyses. Where no IPD were available and a different threshold was used by the study authors, we presented a qualitative summary of the results of the study.

Planned subgroup analyses included age (children [less than 18 years old] versus adult [18 years and older]), different thresholds for positive results, and patients with EIB versus EIA.

Therapy Review

For studies assessing therapy of the five treatments compared with a control or placebo, a mean difference (MD) was calculated for continuous variables and a risk ratio (RR) was calculated for dichotomous variables. Results are reported with accompanying 95 percent CI.

Meta-analyses were conducted using the random effects model where appropriate. The I2 statistic was used to assess heterogeneity.17 Planned subgroup analyses included age (children [less than 18 years old] versus adult [18 years and older]), severity of EIB/EIA as defined by the percent fall index on placebo (mild [less than 30 percent] versus moderate-severe [30 percent or more]), and patients with EIB versus EIA.

Sensitivity analyses were conducted to assess the robustness of the findings across study quality.160 When meta-analyses had sufficient studies, publication bias was tested visually using the funnel plot. All data pooling was performed using Review Manager 5.0 (The Cochrane Collaboration, Copenhagen, Denmark).

Peer Review

Eight experts in the field (Appendix A) agreed to peer review the draft report and provide comments. Reviewer comments were considered by the UAEPC in preparation of the final report. All peer reviewer comments and the UAEPC disposition of comments were submitted to AHRQ for assessment and approval.