Identification of studies

Studies were identified from the ISSG Search Filters Resource.⁶ The ISSG Search Filters Resource is a collaborative venture to identify, assess and test search filters designed to retrieve health-care research by study design. It includes published filters and ongoing research on filter design, research evaluating the performance of filters and articles providing a general overview of search filters. At the time of this project, regular searches were being carried out in a number of databases and websites, and tables of contents of key journals and conference proceedings were being scanned to populate the site. Researchers working on search filter design are encouraged to submit details of their work. The 2010 update search carried out by the UK Cochrane Centre to support the ISSG Search Filters Resource website was also scanned to identify any relevant studies that were not included on the website at that time.

We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator. The searches were carried out in 2010/11.

Inclusion criteria

The review included studies that reported the development and evaluated the performance of methodological search filters for health-care bibliographic databases. For pragmatic reasons, the review specifically focused on studies that developed and evaluated methodological search filters for economic evaluations, DTA studies, systematic reviews and RCTs. These study types are the ones most commonly used by organisations such as NICE to underpin their decision-making when producing technology appraisals and economic evaluations of health-care technologies and subsequent clinical guidelines. Publications prior to 2001 were excluded partly for pragmatic reasons but also because during this period search filters tended to be derived by subjective methods and because some of the filters had subsequently been updated or were now out-of-date because of changes in database indexing.

Exclusion criteria

Studies were excluded from the review if they:

• were available only in abstract form (e.g. conference abstracts)
• did not develop or revise a search filter
• did not report details of the methods used in developing the search filter
• did not evaluate search filter performance
• were published before 2001.

Data extraction

Data were extracted from selected studies using a standardised data extraction form to identify information regarding gold/reference standards, filter development/validation and performance measures reported.

Study details

Three studies included analyses of more than one search filter type: one study¹² included details of a diagnostic filter and a secondary (systematic review) filter and two studies^16,21 included details of both systematic review and RCT search filters. Thus, there were two studies examining economic search filters, seven studies examining diagnostic search filters, seven studies examining systematic review search filters and 10 studies examining RCT search filters.

The majority of the studies (n = 14)^{8–10,12–14,17–19,22,23,26,27,29} addressed the development of search filters for use with MEDLINE, 10 for the Ovid platform,^{8,9,13,14,17,19,22,23,27,29} three for PubMed^12,18,26 and one for DataStar.¹⁰ Six studies developed search filters for the EMBASE database,^{7,11,15,20,24,28} four for the Ovid platform,^7,15,20,28 one for DataStar¹¹ and one that used three different platforms (DataStar, Dialog and Ovid).²⁴ The remaining three studies developed search filters for the Cumulative Index to Nursing and Allied Health Literature (CINAHL),²¹ PsycINFO¹⁶ and the Latin American and Caribbean Health Sciences Literature (LILACS) database²⁵ respectively. The CINAHL and PsycINFO search filters used the Ovid platform whereas the LILACS database was searched using an internet interface.

Internal gold standards

A reference standard is a set of relevant records against which a search filter’s performance can be measured. In some studies the reference standard is used both to derive and to test a search filter. In these cases the standard is described as an internal standard.

Almost all of the studies used an internal standard to derive and/or validate the search filters. Only three of the 23 studies did not include an internal standard.^18,26,29 These studies tested the search filters against external standards (see External standards). Seventeen^{7–11,13,15–17,19–21,23–25,27,28} of the 20 studies that included an internal standard had derived this standard by hand-searching journals. The number of journals searched ranged from 2 to 161. In the other three studies^12,14,22 the internal standards were generated by a PubMed subject-specific search or from studies included in a number of systematic reviews, or from a database search [MEDLINE and the Cochrane Central Register of Controlled Trials (CENTRAL)]. One other study²⁴ used a search of EMBASE as well as hand-searching of journals to derive an internal standard. The size of the gold or reference standards varied from 58 to 1587 records. In three studies, the reference standard was initially split into two, with one set used to derive the filter and the second set used to internally validate the performance.^17,19,23

Inter-rater reliability in selecting studies for inclusion in the reference standard was assessed for almost all of the studies produced by the McMaster Hedges team^{7,8,13,15–17,20,21,23,28} and exceeded 80% in every case. In one McMaster Hedges team study,⁸ articles were independently assessed by two reviewers with disagreement being resolved by a third independent reviewer. Two studies quoted inter-rater reliabilities of 71%¹² and 81%²⁷ after articles were assessed by two reviewers. Two further studies^10,19 reported that articles were assessed by two reviewers, whereas one study¹¹ reported that articles were assessed by one reviewer with 10% of articles assessed by a second reviewer and one study⁹ reported that articles were assessed by three researchers with discrepancies resolved through discussion. None of these studies reported values for inter-rater reliability. The remaining four studies that derived internal standards^14,22,24,25 did not describe how the studies were selected.

Identifying candidate terms and combining them to create filters

In the 20 studies with internal standards, the internal standard records were used as a source for the identification of candidate search terms. Ten of these studies^{7,8,13,15–17,20,21,23,28} were carried out by the McMaster Hedges team and used essentially the same methodology for deriving search filters. This method involved the identification of index terms and text words from an internal standard of records as well as consultation with clinicians, librarians and other experts to add any other relevant terms. The individual terms identified were analysed for sensitivity and specificity and then terms with specified values of sensitivity and specificity were combined to create multiple-term search filters using the Boolean OR operator. The specified values for term inclusion varied for sensitivity and specificity from > 10% to > 75%. In one of the 10 studies²³ stepwise logistic regression was also used to try to optimise search filter performance. The use of logistic regression, however, did not result in better-performing search filters than those developed simply using the Boolean OR operator and therefore this approach was not used in any of the subsequent studies.

Another study²⁵ also identified terms from an internal standard and then combined terms with particular values for sensitivity, specificity and accuracy to derive multiple-term strategies to produce a maximally sensitivity strategy. Single terms with an individual sensitivity of > 20% and specificity and accuracy of > 60% were combined to give two-term strategies. Terms in the two-term strategies with sensitivity, specificity and accuracy of > 60% were then combined to give three- or four-term strategies. All terms in the three- and four-term strategies were then combined to give a maximally sensitivity strategy consisting of 10 terms. This final strategy was refined further by using the Boolean AND NOT operator to exclude single terms with zero sensitivity and high specificity. This increased the specificity of the final strategy while maintaining high sensitivity.

Five studies^{10,11,19,22,27} used bibliographic software to undertake a more formal frequency analysis of the terms in the internal standard. Two of these studies^10,11 carried out word frequency analysis for all of the records in the internal standard and then created search strategies by combining those terms that had the highest scores as determined by multiplying the sensitivity and precision scores. Two studies^19,22 used textual analysis of the internal standard records followed by discriminant analysis using logistic regression to determine the best terms to be included in the search strategy. The fifth study²⁷ also used frequency analysis to identify candidate terms for building a search strategy.

Previously published filters were used as a source of terms for four studies.^9,12,14,24 These strategies were then further developed by adding extra medical subject heading (MeSH) and text terms identified from the internal standard records. In one of these studies²⁴ the MeSH terms were first translated from a MEDLINE strategy into Emtree terms before adding additional Emtree terms and free-text terms identified from the internal standard records. This study also consulted experts for further suggestions. Individual terms were tested against the internal standard and those with a precision of > 40% and sensitivity of > 1% were added sequentially to develop the filter. Astin et al.⁹ also used the sequential addition of search terms to develop the search filter.

Internal validation performance measures

The performance of the search filters was tested against the gold or reference standard in 19 studies^{7–17,19–21,23–25,27,28} to test internal validity. Nine studies^{7,13,15–17,20,21,23,28} carried out by the McMaster Hedges team reported the results for single-term and combined-term search strategies, whereas the remaining study⁸ from this team reported only the performance of combination-term strategies. Studies reporting single-term strategies included between one and six single-term strategies whereas the number of combination strategies reported varied between four and 14. The performance of strategies was usually reported in terms of high sensitivity, high specificity or optimised balance between sensitivity and specificity. The other nine studies^{9–12,14,19,24,25,27} tested between one and eight filters, with some single-term strategies but mostly combination strategies. The focus of these search filters was to produce highly sensitive, highly specific or highly precise outcomes.

The performance measures reported for internal validation are presented in Table 6. Sensitivity was reported by all 19 studies, precision was reported by 16 studies and specificity was reported by 14 studies. Accuracy was reported by seven studies and the number needed to read (NNR) by four studies. Positive likelihood ratio (LR+) values and fall-out were each only reported in a single study. All of the performance measures were presented in tables with the exception of one study,²⁵ for which the results were presented in a figure that contained the full search strategy and values for sensitivity and specificity.

TABLE 6

Review A: performance measures – internal standards

External standards

Nine of the 23 studies used external standards to test or validate the search filters that had been developed or revised.^{9,10,17–19,22,26,27,29} For these studies, a reference standard that was different from the one used to derive the search filter was used. These studies included studies of diagnostic test, systematic review and RCT filters. Four studies^9,10,17,18 used hand-searching of journals to generate the external standard. The number of journals searched ranged from 1 to 161, resulting in between 53 and 332 records in the external standards. Two of these four studies^17,18 increased the numbers in the external standard by adding records from a search of either the Cochrane Database of Systematic Reviews (CDSR) or the Database of Abstracts of Reviews of Effects (DARE).

Four^22,26,27,29 of the other five studies that used external standards were of RCT search filters and one¹⁹ was of a systematic review search filter. Two of these studies^27,29 identified records for their standards by searching systemic reviews (one searched 61 reviews from the CDSR²⁹ and one²⁷ searched seven systematic reviews of cluster RCTs). Another study²⁶ searched for records in 11 journals in the CENTRAL database, generating 308 references. In the remaining RCT search filter study²² MEDLINE was searched to identify records that were assessed as being trials. In the study that examined a systematic review search filter¹⁹ models were tested using a validation data set and against a ‘real-world’ scenario using Ovid MEDLINE on compact disc, read-only memory (CD-ROM). The validation data set had been created from a hand-search of five journals. The results of this hand-search had been split into an internal test set (n = 256, 75%) and an external validation set (n = 89, 25%).

External validation performance measures

The performance of the search filters was tested against external standards in nine studies.^{9,10,17–19,22,26,27,29} The performance measures reported for external validation are presented in Table 7. All nine studies reported sensitivity and seven of the nine studies reported precision. Two studies^9,17 reported specificity and two^10,29 reported the NNR (described as ‘article read ratio’ in one article). Two studies^26,27 reported a single performance measure, that is, sensitivity only, three studies^18,19,22 reported two performance measures and four studies^9,10,17,29 reported three performance measures. The performance measures were again presented almost exclusively in tables, with one exception,²⁶ in which the performance measures were simply discussed in the text of the article.

TABLE 7

Review A: performance measures – external standards

Methods used to develop and validate search filters

A total of 23 studies were included in this review. In the majority of these studies an internal gold or reference standard was used to develop the search filter by identifying candidate terms and assessing performance. The way in which terms were chosen for inclusion, however, and how the combinations were determined varied. The internal gold standards were mainly derived from journal hand-searches although a few were derived by other methods (from a database search or studies identified from systematic reviews). Ten of the studies were produced by the McMaster Hedges team and these all used the same method of search filter development, for example through consultation with experts and use of their internal gold or reference standard. Five other studies made use of statistical methods for filter development. The use of statistical methods helps to make the process more objective rather than depending on human expertise. In a few cases, the search filter was not developed using a gold standard or reference standard but was adapted from a previous search filter. Only nine studies undertook external validation, that is, validation against a standard that was different from the one used to develop the filter. As this provides an independent assessment of filter performance, it provides a more rigorous assessment and gives a better indication of how a filter is likely to perform in the real world.

Reported performance measures

Across the 23 studies included in the review, eight different performance measures were reported; however, as precision and positive predictive value (PPV) are equivalent, there were actually seven different performance measures. The performance measures used for internal and external validation and their frequency of use are listed in Tables 6 and 7 respectively. The most frequently reported performance measures were sensitivity, precision and specificity respectively.

All studies reported sensitivity, reflecting the importance of this measure when determining the usefulness of a search filter. As the filters are used to identify relevant articles, it is important to measure the number of relevant articles retrieved by the filter compared with the total possible number of relevant articles. When carrying out a systematic review, in which it is important to identify as many relevant studies as possible, it makes sense to use a search filter with a high sensitivity value.

The performance measures of specificity and precision were the next most reported measures. It is important that a search filter rejects non-relevant articles and thus a high specificity is desirable. In a well-performing search filter a high specificity value would be desirable as well as a high sensitivity value, as there would not be much point in using a filter that retrieves lots of non-relevant articles as well as all of the relevant articles. The articles in the review often included search filters that were optimised for the best balance of sensitivity and specificity.

As precision measures the number of relevant articles as a proportion of all articles retrieved, the aim is to maximise the precision of a search filter. As sensitivity and precision are, however, inversely related, it is difficult to achieve both high sensitivity and high precision. The NNR is another way of reporting precision as it is calculated by dividing 1 by the precision value. This measure gives the number of articles that need to be read to find one relevant article and may, therefore, be more easily understood than precision, which is usually quoted as a percentage value.

The accuracy performance measure was used only in articles produced by the McMaster Hedges team. It provides a measure of the number of articles that are classified correctly as either relevant or non-relevant. The usefulness of this measure on its own, however, is unclear as a high accuracy value may be obtained when the specificity value is high but the sensitivity value is medium or low. In most cases the accuracy value is close to the specificity value and does not give an indication of the sensitivity value.

The other two performance measures that were found (LR+ and fall-out) each appeared in one article. These performance measures were reported in addition to sensitivity and either specificity or precision.

Presentation of performance measures

The most commonly used format for the presentation of performance measures used for single studies of search filters was tables. Only two studies of RCT filters did not present the performance measures in tables. One of these studies presented the search strategy and its performance measures in a figure whereas the other study simply discussed the performance measures in the text of the article. Thus, tables seem to be a popular and useful way of presenting performance measures. Often the results are ordered in tables according to one of the performance measures, for example sensitivity, thus making it easy to identify the most sensitive and the least sensitive search filter. The studies often presented the performance measures in a number of tables to allow ordering by different performance measures, for example tables ordered by sensitivity or specificity or precision. This makes it easier to select a search filter for a specific need, for example researchers involved in performing systematic reviews requiring very sensitive search filters could select the most sensitive search filters whereas busy clinicians who are simply looking for some relevant articles could select a filter with the highest precision.

Key findings

• Internal gold or reference standards were mostly derived by hand-searching of journals.
• Validation of filters was mostly carried out using internal validation.
• The most commonly used performance measures were sensitivity, precision and specificity.
• The majority of the studies presented performance measures in tables.

Identification of studies

Studies were identified from the ISSG Search Filters Resource.⁶ The ISSG Search Filters Resource is a collaborative venture to identify, assess and test search filters designed to retrieve health-care research by study design. It includes published filters and ongoing research on filter design, research evaluating the performance of filters and articles providing a general overview of search filters. At the time of this project, regular searches were being carried out in a number of databases and websites, and tables of contents of key journals and conference proceedings were being scanned to populate the site. Researchers working on search filter design are encouraged to submit details of their work. The 2010 update search carried out by the UK Cochrane Centre to support the ISSG Search Filters Resource website was also scanned to identify any relevant studies not at that time included on the website. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, due to serious illness of the principal investigator. The searches were carried out in 2010/2011.

Inclusion criteria

For the purpose of this review, methodological search filters were defined as any search filter or strategy used to identify database records of studies that use a particular clinical research method. A pragmatic decision was taken to include only studies comparing the performance of filters for RCTs, DTA studies, systematic reviews or economic evaluation studies. These study types are the ones most commonly used by organisations such as NICE to underpin their decision-making when producing technology appraisals and economic evaluations of health-care technologies and subsequent clinical guidelines.

Studies were selected for inclusion in the review if they compared the performance of two or more methodological search filters in one or more sets of records. Studies reporting the development of new methodological filters whose performance was compared with that of previously published filters were also included.

Exclusion criteria

Studies were excluded from the review if they:

• reported the development and initial testing of a single search filter that did not include any formal comparison with the performance of other search filters
• compared methodological search filters that had not been designed to retrieve RCTs, DTA studies, systematic reviews or economic evaluation studies
• compared the performance of a single filter in multiple databases or interfaces
• were not available as a full report, for example conference abstracts
• were protocols for studies or reviews
• lacked sufficient methodological detail to undertake the data extraction process.

Data extraction and synthesis

A data extraction form was developed by two reviewers (JH, CF) to standardise the extraction of data from the selected studies and allow cross-comparisons between studies. Details extracted included the methods used to identify published filters for comparison, the methods used to test filter performance and the performance measures reported. Data extraction for each study was carried out by one reviewer (JH) and verified by a second reviewer (CF). A narrative synthesis was used to summarise the results from the review.

Gold standards

In search filter research a gold standard or reference set is a set of relevant records against which a filter’s performance can be assessed. For example, a collection of records of confirmed RCT studies would be used when testing the performance of a methodological search filter designed to identify RCTs.

Studies included in this review used a range of techniques to identify and/or create a gold or reference standard against which to test the performance of multiple filters. One study did not use a gold standard;³³ instead, each of the filters was combined with single terms describing four topics (hypertension, hepatitis, diabetes and heart failure) and the retrieved studies were checked to confirm whether or not they were RCTs.

The size of the gold or reference standards used to test filter performance ranged from 33 to 1955 records. None of the studies included in this review reported whether or not they had carried out a sample size calculation when developing their gold or reference standard (a sample size calculation is a statistical process that determines the minimum number of records required for a gold standard to provide accurate estimates of performance). Four of the DTA filter studies^2,14,49,58 and one RCT filter study²² limited their gold standard to specific clinical topics.

Ten studies developed their gold or reference standards by hand-searching journals.^{10,15,17,19,23,25,56,57,61,63} The number of journals hand-searched ranged from 4 to 161. The time span covered by hand-searching varied from 1 to 23 years. All of the studies using hand-searching had specific criteria for the identification of the desired study type for inclusion in their gold or reference standard.

Of the 10 studies identifying their gold or reference standard from hand-searching journals, eight were studies in which the authors had developed new search filters and then compared those filters with existing filters.^{10,15,17,19,23,25,56,57} One study that created a reference standard from hand-searching journals created a ‘control set’ of records from the same group of journals that were not of the desired study design.⁵⁷

Five studies developed a gold or reference standard based on the studies included in systematic reviews [relative recall (RR) gold standard]^{2,14,48,49,58} and four studies used database searches to identify records to include in their gold standard.^22,56,58,59 The number of completed systematic reviews used as a source of gold standard records varied: one study used included studies from 27 systematic reviews,⁴⁸ one used included studies from two reviews,⁵⁸ one used included studies from seven reviews of DTA studies² and a fourth used studies included in a single case study review.⁴⁹ One study that developed a DTA study filter and compared it with published filters used the studies included in 16 reviews as the gold standard.¹⁴

Translation of filters

Search filters were developed using a range of different search platforms (or interfaces), including Ovid, PubMed or WebSPIRS for MEDLINE filters. Any study comparing the performance of filters may therefore need to ‘translate’ the filters from the syntax used in the original development interface to the syntax required by the interface used in the filter comparison.

Four of the studies included in this review did not translate or adapt the filters that were being compared because the filters had been developed in the same interface as was used in the performance comparison.^25,33,56,63 When one or more filter required translation, most of the studies comparing the performance of existing filters reported the complete details of the changes made so that the accuracy of the translation could be verified.^{2,48,58,59,61} In contrast, most of the studies reporting the development of new filters that included a comparison with existing filters did not mention the requirement to translate any of the filters or provide details of the translation, so it is unclear if valid comparisons were being made.^{10,17,22,23,57} The review of economic evaluation filters applied an exclusion strategy (animal studies and publication types such as letters and editorials, which are unlikely to be economic evaluations) to filters being tested in MEDLINE and EMBASE.⁵⁹

Methods of testing

Four of the filter studies that used included studies from systematic reviews as their gold or reference standard replicated the original searches when possible with the addition of the filters being tested.^2,48,49,58 None of the original searches incorporated a study method search filter.^2,48,49,58 A fifth study using references from systematic reviews as a reference standard combined the filters with ‘terms for deep vein thrombosis’ but did not specify what these terms were or if the original search strategy was used.¹⁴

The performance analyses carried out by Leeflang et al.⁴⁸ and Ritchie et al.⁴⁹ occurred after the original reviews (on which the gold or reference standard was based) had been undertaken and therefore attempted to recreate a ‘historical’ search. Ritchie et al.⁴⁹ noted a small discrepancy in the number of records retrieved between the original searches and the rerun searches, whereas Leeflang et al.,⁴⁸ who could replicate only 6 out of 27 reviews, did not provide details of any differences in the numbers of retrieved records. Using the complete reference standard from the original reviews, Leeflang et al.⁴⁸ tested whether those studies were captured by the filters being compared.

Two studies did not provide any information about whether the performance analysis had been undertaken concurrently with the reviews or at a later date.^14,58 The review by Whiting et al.,² which was published online in 2010 and to which we had prepublication access at the time of our study, recreated the original subject search and compared using the subject search alone with using the subject search combined with 22 other filters.

Four studies by the McMaster Hedges team at McMaster University used their internally developed database for testing filters, with the DTA, RCT and systematic review subsets acting as gold standards.^17,23,61,63 One of these studies did not undertake any new analysis but collated the results from previous publications that had used a common gold standard.⁶³

The economic filters study identified a gold standard by searching the NHS Economic Evaluation Database (NHS EED).⁵⁹ Published MEDLINE and EMBASE economic filters were then tested for their ability to retrieve these gold standard records from MEDLINE and EMBASE. Corrao et al.³³ had no gold standard but manually checked whether the records retrieved after applying the filters were RCT studies.

Studies that compared new search filters with existing filters can be divided into two groups based on the type of gold standard used to compare filter performance. One group used a reference standard that had not been used to develop the new filter strategy so that all of the filters in the comparison underwent external validation.^{10,17,22,23,57} In other words, the performance of all of the filters being compared was tested in a set of records that had not been used to develop any of the included filters. The other group of studies used the same reference standard that had been used in the development of the new filters, so that, although the new filters underwent only internal validation (filter performance was tested only on the one set of records that had also been used to develop the new filters), the comparison filters underwent external validation.^{14,15,19,25,56} The methodology used in the latter group risks introducing bias in favour of the new filters.

Performance measures reported

The most commonly reported performance measures in studies comparing the performance of search filters were sensitivity/recall and precision (Table 11). A total of 16 studies reported sensitivity/recall^{2,10,14,15,17,19,22,23,25,49,56–59,61,63} and 13 studies reported precision values.^{2,10,15,17,19,22,33,49,56,58,59,61,63} Specificity was reported in seven studies.^{15,17,23,25,57,61,63}

TABLE 11

Review B: measures reported in filter performance comparisons

In one study that did not use a gold standard or reference standard, sensitivity could not be calculated and instead the proportion of retrieved records that met the authors’ criteria for being a RCT was reported.³³ In another study the proportions of gold standard records retrieved and missed for each filter were reported.⁴⁸ When the original search strategy could not be replicated, this article reported the NNR.⁴⁸ Bachmann et al.¹⁰ reported the NNR for the filter that they developed but not the previously published filter that they used as a comparator. Whiting et al.² reported the NNR and the number of records missed from the reference set.

No studies comparing the performance of two or more existing filters reported accuracy values (the number of records correctly retrieved or correctly not retrieved as a proportion of all records). The study by Manríquez²⁵ reporting the development of a RCT filter for the LILACS database did report accuracy values for the new filter, as did the study by Wilczynski et al.¹⁵ for their newly developed DTA study filters. Additional measures reported in performance comparisons were:

• number of records retrieved⁴⁹
• retrieval gain (absolute and percentage variations in the number of citations retrieved)³³
• the proportion of articles missed per original review⁴⁸
• the proportion of articles not identified per year⁴⁸
• diagnostic odds ratio (DOR) (the odds of being truly relevant among the relevant divided by the odds of being assessed as relevant among the irrelevant)⁵⁷
• the number of relevant articles retrieved.⁵⁶

Confidence intervals surrounding performance results were reported by three of the studies that compared the performance of existing search filters.^2,61,63 Five of the studies comparing the performance of developed search filters with that of existing search filters reported confidence intervals.^{10,15,17,25,57}

Methods used to display performance comparisons/data

All of the studies included in the review displayed the results using a table format, with only two studies supplementing tables of results with graphical (non-tabular) displays of comparative data.^2,48 None of the studies reporting the development of new filters displayed comparative performance in a graphical format.^{10,14,15,17,19,22,23,25,56,57}

The majority of tables presenting performance comparison data displayed the filters in rows and performance measures in columns (an example is provided in Table 12). The results in the tables in all included studies were provided as percentages or proportions. Within tables, authors generally listed filter results in descending order by the measure of interest, for example decreasing sensitivity. Four studies reporting the development of a filter only included data on comparative performance in the text of the study report.^10,23,25,57

TABLE 12

Review B: example of a filter performance comparison table as commonly presented in the literature

Tables that did not list filter results in descending order by the measure of interest instead arranged results by:

• the databases in which the filters were tested^15,63
• strategy type (sensitive strategy, specific strategy, optimised strategy)^15,63
• filter criteria (sensitive, accurate, etc.)⁴⁸
• filter alone compared with a clinical subject strategy⁵⁸
• use or not of an exclusion strategy⁵⁹
• clinical topic considered in the performance testing^33,58
• subject search alone compared with the same subject search with each test filter²
• author or source of published filters^15,17
• descending order of cumulative precision or cumulative sensitivity.⁵⁶

Tables were also used to present information on the number of studies retrieved⁵⁸ and the specificity, sensitivity and precision of single terms.⁶³ One study that reported highest precision combined with sensitivity of > 69% showed the results of the filters meeting these criteria in a separate table.⁴⁹

Leeflang et al.⁴⁸ used a bar graph to display the average proportion of retrieved and missed gold standard records per filter tested (Figure 1). Whiting et al.² presented the overall sensitivity and specificity of each filter tested in a forest plot, including confidence intervals (Figure 2).

FIGURE 1

Review B: bar chart displaying the comparative performance of filters for DTA studies as published by Leeflang et al. Republished with permission of Elsevier from the Journal of Clinical Epidemiology, Use of methodological search filters to identify diagnostic (more...)

FIGURE 2

Review B: forest plot of overall sensitivity and precision for each filter in the study by Whiting et al. CEBM, Centre for Evidence Based Health; CRD, Centre for Reviews and Dissemination; HTBS, Health Technology Board for Scotland. Republished with permission (more...)

Reporting methods of comparison

It was difficult to assess the reliability of the methods used in studies comparing the performance of multiple search filters because the size of the gold or reference standard, the method of testing, the performance measures reported and the presentation of the results varied greatly across studies. In addition, among studies that developed new filters, the methodological detail provided on the comparison of filter performance between new and existing filters was limited.

The description of the methods used in studies reporting the development of new filters and studies comparing only published filter performance differed. Those developing new filters focused their methods section on describing the selection and combination of terms for use in the new filters, with only minimal detail provided in the sections dedicated to describing the performance comparison of the new filters and existing filters. The comparison was often secondary to the main analysis and suffered from a lack of transparency. In contrast, in studies in which the focus was on comparing the performance of multiple existing filters, the methods used in identifying and testing the published filters included in the study tended to be reported more fully.

Many filter development studies did not clearly explain how they had identified filters for inclusion in performance testing. Not reporting how filters were identified and whether or not they were developed in the same interface used for testing could have implications for reliability and bias within the studies. If studies do not report how the filters used in comparisons were identified, it is not possible to determine whether the filters were selected in an unbiased fashion or whether they might have been preferentially selected to suit the test environment. In this review, studies reporting the development and testing of one or more filters all found that the new filter performed better than the existing filters used as comparators. This makes it particularly important that studies clearly report how filters are selected and the comparison performed, as otherwise this could be a sign of bias in the results.

Details about the translation of published filters for different interfaces were lacking in many filter development studies. Generally, more details about methods of translation were provided in studies that reported filter performance comparisons separately from the development of new filters. Combined with the lack of information about the original interface used in the development of published filters, the lack of translation details in many filter development studies makes it almost impossible to determine the accuracy of any alterations. As incorrect or imprecise translation of a filter is likely to impact on the results retrieved, the lack of methodological detail provided is a cause for concern.⁶⁶

Almost all of the included studies used a gold or reference standard to test the comparative performance of developed and existing filters. This would seem to indicate that using a gold or reference standard to test and compare filter performance is widely accepted in the filter research community. The size of the gold or reference standard used, however, varied widely, from tens to thousands of records. It is possible that the size and content of the gold standard may have an impact on the performance measures recorded for a specific filter, and so it would be helpful if researchers could justify their choice, by, for example, reporting a sample size calculation.

Some of the studies included in the review used a single gold or reference standard for both developing a new filter and comparing the new filter with published filters. This could potentially introduce performance bias in favour of the new filter as the new filter undergoes only internal validation whereas the comparator filters undergo external validation. In other words, the new filter is tested only against the set of records from which it was developed, whereas the comparator filters are tested against a set of records that are different from the gold or reference standards that were used to develop them. When a filter is tested against the same set of records from which it was developed, it is likely that the filter will perform better than it might in a different sample of records.

Reporting performance measures

Sensitivity and precision appear to be considered the most useful measures of filter performance as they are the most commonly reported measures in the literature. As the same performance measures were reported in studies developing new search filters and studies reporting the comparative performance of existing filters, this is one area of methodological consistency between the two types of performance comparison study included in this review.

There is a suggestion, from the small number of studies included in this review, that there are some measures that are preferentially reported for DTA study filters, for example the NNR. Similarly to the metric ‘number needed to treat’ (NNT), the NNR reflects the number of retrieved records that need to be assessed to identify a relevant study. By reporting the NNR, studies seek to make it easier for searchers to determine how effective a filter will be in reducing the number of irrelevant records retrieved and therefore the relative reduction in time needed to identify relevant studies for inclusion or full-text retrieval.

The method used to present the results of filter performance comparisons was limited to tables, with only two studies presenting data graphically, perhaps reflecting the difficulties in presenting filter performance comparisons visually. Many of these tables were long and complicated, making interpretation of the results and the selection of an appropriate filter challenging. In most cases it would not be easy to identify the most suitable filter without reading several studies, including tables, in detail. A lack of time and search filter expertise potentially compounds the problem of selecting an appropriate filter based on performance data as they are currently reported in the literature.

Of the two graphics used in the included studies to present results, a design similar to a forest plot (see Figure 2) may prove attractive to searchers as it is a familiar format used in systematic reviews and meta-analyses. This design may also make it easier to identify visually the most precise, most sensitive and best-balanced filter. A further exploration of methods for graphically presenting filter performance comparisons would be useful for both researchers involved in filter performance research and searchers needing to identify a suitable filter for their project.

Limitations of this review

There are a number of potential limitations to this review. It was not possible to undertake a full systematic review because of time constraints. It was also not possible to review all filters for all study methods. The review was, however, focused on study types that were felt to be the key study designs of current interest in evidence-based health research (namely RCTs, DTA studies, systematic reviews or economic evaluation studies). Finally, research carried out on the performance of multiple search filters that has not yet been published or has been presented only at conferences was excluded from the review, possibly resulting in some alternative formats for the presentation of results being missed. Conference abstracts, however, would be likely to report even fewer methodological details than full articles included in this review.

Key findings

• The main measures of search filter performance reported in the literature are sensitivity/recall, precision and specificity.
• Filter performance comparison studies most commonly report highest sensitivity, highest precision and optimal/balanced filter strategies.
• Articles reporting the development of new search filters and a comparison with existing filters provide limited methodological details.
• Tables are the most frequently used method for reporting the results of filter performance comparisons but graphs may be more useful.

Conducting diagnostic test accuracy studies

Diagnostic test accuracy measures the ability of the diagnostic test being evaluated, the index test, to distinguish between patients with and patients without the targeted disease or condition.⁶⁷ The results are verified against the results of a reference standard in the same group of patients. The reference standard is independent of the index test and is usually the best available method to identify patients with the target condition.^68,69 When a comparator test is also under evaluation, the index and comparator test must be evaluated against the same reference standard and in the same population.⁶⁹ In the absence of a suitable reference standard a number of alternative methods have been proposed.^70–72

Test accuracy is not fixed and can vary between patient subgroups, with disease severity, in different clinical settings and with different test interpreters.⁶⁷ Several guidance documents describe how these variations in the design and conduct of diagnostic tests can lead to bias, resulting in substantial differences being observed between primary studies.^69,73–76 The effects of different types of bias have been estimated using empirical data.^76–79

As diagnostic tests do perform differently in different populations, the importance of testing in a suitable sample of patients receives much attention in the literature. The patient sample should be representative in terms of the disease severity of the target population for whom the test is intended, to avoid spectrum bias (i.e. the variation in the sensitivity and/or specificity of a diagnostic test when applied to people of different ages, genders, nationalities or specific disease manifestations).^69,73,75,80 Ideally, patients should be recruited consecutively or randomly in a single cohort and be unselected by disease state.⁷⁴ Case–control studies are likely to lead to bias because patients with and without the condition are recruited using different sets of criteria^69,73 and because they overestimate diagnostic accuracy.⁷⁷ Other main sources of bias relate to the unsuitability of the reference standard, how the reference and index tests have been undertaken, interpreter blinding and interpretation of the results.⁷⁹

Uncertainty around estimates of diagnostic accuracy decreases with increasing sample size⁷⁵ and it is recommended that sample size calculations should be undertaken during study planning to ensure that a reasonably precise estimate of test accuracy can be achieved.^81,82 Tables have been published to assist in determining the minimum sample size required⁸³ for a DTA study once the prevalence of the target condition in the population as well as the expected sensitivity have been determined. However, two reviews of DTA studies found that very few studies gave any consideration to sample size.^81,82

The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool has been developed to assist researchers to assess the quality of primary DTA studies^84,85 and as such provides a useful guide to the issues that should be addressed when undertaking a DTA study. The questions cover aspects of methodology that are thought to make a difference to the reliability of a study, such as the suitability of the patient sample and the reproducibility of the reference standard and index test. Poor reporting of DTA studies, however, can make applying the QUADAS tool difficult.⁷⁸ Since the searches for this review were undertaken, a revised version of the QUADAS tool has been published. The QUADAS 2 tool, which is applied in four phases, will, according to the publishers, allow for a more transparent rating of bias and the applicability of primary diagnostic accuracy studies than the original QUADAS tool.

Measuring diagnostic test accuracy

Contingency table

The primary outcomes of interest in DTA studies are the data required to populate 2 × 2 contingency tables presenting the presence or absence of the target condition or disease, as defined by the reference standard against the result of the index test (Table 13). From this all DTA measures can be derived.

TABLE 13

Review C: contingency table

Measures

Table 14 describes the measures of diagnostic accuracy that are commonly calculated, namely sensitivity, specificity, likelihood ratio (LR), DOR and predictive value.

TABLE 14

Review C: measures of diagnostic accuracy

Two statistical measures of diagnostic accuracy are traditionally used in a clinical setting: the true positive rate or the sensitivity of the test (the proportion of those with the disease who have an abnormal test result) and the specificity of the test (the proportion of those without the disease who have a normal test result). To rule out a diagnosis a test must have high sensitivity whereas to confirm a diagnosis a test must have high specificity.^69,73,80 Both measures are susceptible to spectrum bias^76,86 but are not directly influenced by prevalence.⁷⁶

The predictive value is the probability of the test correctly diagnosing patients. The PPV is the proportion of patients with a positive test result who are correctly diagnosed. Conversely, the negative predictive value (NPV) is the proportion of patients with a negative test result who are correctly diagnosed. Predictive values depend on the prevalence of the condition in the population being tested. When prevalence is high, it is more likely that a positive test result is correct and a negative result is wrong.^86,87

Likelihood ratios describe the performance of diagnostic tests and can be useful in a clinical setting. The ratio describes whether or not a test result usefully changes the probability that a condition exists. The LR+ is the probability of a person who has the disease testing positive divided by the probability of a person who does not have the disease testing positive. A LR+ of > 10 and a negative likelihood ratio (LR–) of < 0.1 are judged to provide convincing diagnostic evidence.⁸⁸ Their interpretation, however, depends on the clinical context.⁸⁷

The DOR is a summary measure of the diagnostic accuracy of a diagnostic test. It is calculated as the odds of positivity among diseased persons divided by the odds of positivity among non-diseased persons. When a test provides no diagnostic evidence then the DOR is 1.0.⁸⁹ This measure has a number of limitations. In particular, it combines sensitivity and specificity into a single value, hence losing the relative values of the two, and is difficult to interpret clinically.⁸⁷

Sensitivity and specificity are based on binary classification of test results (either positive or negative). Test measures, however, are often categorical or continuous and so a cut-off point must be defined to classify results as either positive or negative. As the threshold shifts, the sensitivity and specificity of a test will change, with an increase in one resulting in a decrease in the other. This trade-off at different thresholds can be presented graphically in a receiver operating characteristic (ROC) curve, describing the relationship between the true-positive value (sensitivity) and the false-positive value (1 – specificity), and can be used to identify a suitable threshold for clinical practice.⁶⁹ Figure 4 displays a sample ROC curve of test performance using different threshold values from ≥ 5 to > 25.

FIGURE 4

Review C: example ROC curve.

The Q* value is the point on the ROC curve where sensitivity equals specificity and can be used as a single indicator of overall test performance when there is no preference for maximising sensitivity (minimising false negatives) or specificity (minimising false positives) but can give misleading results if used to compare performance between tests.^69,90 Overall, diagnostic accuracy is summarised by the area under the curve (AUC) and ranges from 0.5 (very poor test accuracy and equivalent to chance) to 1.0.^69,87 The more accurate the test, the more closely the curve approaches the top left hand corner and has a value close to 1.0.

Whiting et al.⁸⁷ have undertaken an overview of the various types of graphical presentations that have been used in the DTA literature and describe other graphical displays that could be used to present DTA data. These include dot plots, box-and-whisker plots and flow charts (Figure 5).

FIGURE 5

Review C: example graphical displays for primary study data. (a) Dot plot; (b) box-and-whisker plot; (c) flow chart.

Dot plots are used for test results that take many values and display the distribution of results in patients with and without the target condition but do not directly display diagnostic performance. Box-and-whisker plots summarise the distributions of true-positive and true-negative groups by a continuous measure. Flow diagrams depict the flow of patients through the study, for example how many patients were eligible, how many entered the study, how many of these had the target condition and the numbers testing positive and negative.

Reporting of test accuracy results

The Standards for the Reporting of Diagnostic Accuracy Studies (STARD) statement⁶⁸ provides guidance on how DTA studies should be reported to provide transparency and allow the reader to assess the validity of a study. Full details on participants, method of recruitment, reference and index tests, statistical methods and results are required. Several predominantly small reviews of between 16 and 243 studies^91–98 have looked at the reporting of DTA studies and found poor description of the methods used. Studies either lacked completeness of reporting, with < 50% of studies reporting over half of the STARD items,^95,96 or lacked clarity, hence making assessment difficult.⁹⁷ These reviews concluded that the STARD statement seems to have resulted in little improvement in study reporting. Most of these reviews, however, included studies that were published prior to or soon after the STARD statement was published^91,92,98,99 and so it may be the case that insufficient time had elapsed to make a valid assessment.

Guidance documents provide few recommendations about which DTA measures should be reported. The choice of accuracy measures presented depends on the aims of a particular study and on who is likely to use the information. For example, LRs may be more useful in a clinical setting as they can be used to calculate the probability of disease for individual patients, whereas DORs are difficult to interpret clinically. US,⁷⁵ Australian⁷⁶ and UK⁶⁹ guidance suggests that the 2 × 2 contingency table together with sensitivity and specificity pairs and LR pairs should be presented, along with 95% confidence intervals.^75,76 The US Food and Drug Administration (FDA) also recommends that measures are reported both as fractions and as percentages.⁷⁵

There is some information about measures reported in the literature.¹⁰⁰ In a review of 90 DTA reviews,¹⁰¹ sensitivity or specificity was the most common measure used to report the results of primary studies (in 72% of reviews); predictive values were included in 28% of reviews; and LRs were included in 22% of reviews. In reviewing the reporting of DTA measures in primary studies, two studies have noted that sensitivity and specificity were reported in most studies, with ROC curves reported in less than half of the studies.^95,96

There is some evidence that studies rarely present diagnostic information graphically.^87,91,102 In a review of 57 primary studies,⁹⁹ 57% used graphical displays to present results. Dot plots or box-and-whisker plots were the most commonly used graphs in the primary studies (in 39% of studies) whereas ROC curves were displayed in 26% of studies.

Methods to compare and synthesise diagnostic test accuracy performance from primary studies

Several HTA organisations, in guidance for undertaking DTA evidence synthesis,^{69,76,90,103,104} recommend using the QUADAS tool or a modified version to assess the methodological quality of primary studies. Undertaking a formal assessment provides an indication of the degree to which the included studies are prone to bias^{100,102,105,106} and hence the reliability of the study results. A report from the Agency for Healthcare Research and Quality (AHRQ)¹⁰⁰ found that there had been a trend in recent years for an increasing number of DTA reviews to formally assess study quality.

Several organisations have developed guidance on carrying out systematic reviews of DTA studies^{69,75,76,90,102–104} and agree that analysis is more complex than for clinical effectiveness. Combining results from individual studies can be problematic because of the methodological variability (heterogeneity) found across the studies. In particular, combining test accuracy studies with heterogeneity can produce biased, and hence inaccurate, results.^{74,79,104,107,108}

It is recognised that variability among studies is to be expected. Some of the variability is due to chance, because many diagnostic studies have small sample sizes. The remaining heterogeneity may be the result of differences in study populations or differences in study methods or the result of variation in the diagnostic threshold adopted.⁷⁴ Several methods have been described to measure heterogeneity, using graphical plots and statistical tests.^36,76,109 Although it is recommended that such a thorough investigation be undertaken prior to meta-analysis,^{69,75,76,86,90,100,102–104} this is often not carried out. In a review of 189 systematic reviews,¹⁰⁹ only 32% investigated heterogeneity and the authors concluded that this underuse reflected uncertainty about the correct approach to adopt.

It is recommended that only studies using the same reference standard, including substantially similar patients and showing minimal heterogeneity should be synthesised by meta-analysis.^{69,74,76,90,104} When this type of complex analysis is undertaken it has been recommended that reviewers should enlist the specialist support of an experienced statistician in the field.^36,69,109 When it is not suitable to undertake meta-analysis a narrative approach should be adopted using graphical presentations, such as forest plots and ROC space plots,⁶⁹ to provide a visual overview of the results from the included studies.

Paired forest plots (Figure 6) can show the spread of estimated values for sensitivity and specificity for each study. Point estimates are shown as dots or squares and can be sized according to the precision of the estimate or sample size. Confidence intervals around the estimate are shown by horizontal lines either side of the point estimate. If meta-analysis is then undertaken, the pooled estimate is displayed as a diamond. ROC space plots (Figure 7) present the relationship between sensitivity and specificity, with each point representing the summary performance for each study.⁶⁹

FIGURE 6

Review C: example of a paired forest plot. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

FIGURE 7

Review C: example of a ROC space plot showing summary sensitivity and specificity. df, degrees of freedom.

When performance measures are pooled, separate meta-analyses of sensitivity and specificity data are both the simplest and the most useful approach.^69,104 Such an approach, however, assumes that all included studies are using the same threshold value. Summary ROC (SROC) curves are a form of meta-analysis in which the result is a ROC curve with each data point representing the paired estimate of sensitivity and 1 – specificity from the separate studies (Figure 8). Hierarchical and bivariate statistical models have been developed to estimate the SROC curve.^110,111 The SROC curve is a useful presentation when a threshold effect is observed. The curve provides a global summary of test accuracy and, as with a ROC curve, shows the trade-off between sensitivity and specificity at different threshold levels. It does not, however, provide a single statistic of overall test performance¹⁰⁴ and a review has indicated slow uptake of these newer methods.¹¹²

FIGURE 8

Review C: example of a paired SROC curve, comparing the accuracy of test 1 with that of test 2.

Other graphical methods that can be used to present data in a way that is useful in a clinical context have been suggested.⁸⁷ The two main methods are LR nomograms and the probability-modifying plot. These graphs enable the clinician to estimate the post-test probability of a patient having the disease, based on their pretest probability, when the LRs of tests are known.

Whiting et al.⁸⁷ reviewed the graphical presentation of diagnostic information in 49 systematic reviews. Just over half (53%) of the reviews used graphical displays to present the results. ROC plots were the most common type of graph and were included in 22 reviews (45%), whereas forest plots were used in 10 reviews (20%) to display individual study results. In another review of DTA reviews, Honest and Khan¹⁰¹ found that, when meta-analysis had been undertaken, pooled sensitivity or specificity was reported in 35 out of 60 (58%) reviews, pooled predictive values in 11 out of 60 (18%) reviews, pooled LRs in 13 out of 60 (22%) reviews and pooled DORs in five out of 60 (8%) reviews. SROC plots were reported in 44 out of 60 (73%) of the meta-analyses. Dinnes et al.¹⁰⁹ noted that, out of 189 systematic reviews included in their review, 30% had involved narrative analysis and, when meta-analysis had been undertaken, 52% statistically pooled data, 18% reported SROC plots and a further 30% employed both techniques.

Inclusion criteria

• Studies that report how clinicians choose between diagnostic tests and what factors influence their decisions.
• Screening programmes that provide criteria for the selection of screening tests.

Exclusion criteria

• Studies that report on any factors influencing test ordering decision behaviour without reference to test choice.
• Studies that consider the decision whether or not to order one particular test.
• Studies that report interventions designed to influence test ordering behaviour.
• Studies written in languages other than English.

Data extraction

For studies meeting our criteria, the following information was collected:

• research method(s) used to elicit data
• clinical discipline of participants and setting
• clinical condition or disease and diagnostic tests from among which clinicians made their choice
• factors implicated in clinicians’ choice.