So many filters , so little time : The development of a Search Filter Appraisal Checklist

Objectives To develop a tool to assess the quality of search filters designed to retrieve records for studies with specific research designs. Methods The InterTASC Information Specialists’ Subgroup (ISSG), a group of experienced healthcare information specialists, held consensus meetings to develop filter appraisal tools. The group developed a Search Filter Appraisal Checklist and a structured abstract using consensus methods and tested them using three published filters. Results A published appraisal checklist was assessed for suitability. The group developed a detailed Search Filter Appraisal Checklist that captured relevance criteria and methods used to develop and test search filters. The Checklist includes categorical and descriptive responses. A structured abstract accompanies the appraisal Checklist. Discussion The Checklist is a comprehensive appraisal tool to assist health sciences librarians and others choose between search filters. It reports filter design methods and search performance measures, such as sensitivity and precision. It also prompts filter developers by indicating which information on core methods needs to be reported to help librarians to assess filter suitability. The generalizability of the Checklist for non-methods filters remains to be explored.


Introduction
Search filters are developed to improve the efficiency and effectiveness of searching [1].They are typically created by identifying and combining search terms to retrieve records with a common feature [1].They can be expert-informed, research-based or a combination [1].
Information about the methods of filter development, along with the results of testing, is important to enable potential users to judge whether the filter may be relevant and reliable [1].
Over the last two decades research methods have been used increasingly to develop and test search filters, to make them more robust and reliable [1].Research-based search filters feature in bibliographic databases such as PubMed and others have been developed to assist with international study identification exercises for databases such as CENTRAL and DARE [2][3][4][5].Search filters are proliferating as librarians and researchers try to identify records reporting projects with specific designs to assist with evidence-based healthcare [2][3][4][5].For example, there are at least 8 search filters available for retrieving diagnostic test accuracy studies from MEDLINE [6].Even experienced health sciences librarians may be challenged to select appropriate filters and to advise researchers which, if any, to use for a particular search query.
In evidence-based health care many critical appraisal tools have been developed to assess the quality and relevance of research reports [7][8][9].The UK InterTASC Information Specialists' SubGroup (ISSG), which supports the research groups providing technology assessments to the National Institute for Health and Clinical Excellence (NICE) in the UK, identified the need for such a tool to help its members select from the search filters on its website [10].The ISSG decided to develop a tool to appraise search filters which would help their members, health sciences librarians and others to choose the most relevant filter for their needs.The ISSG was unable to identify an existing suitable tool and, therefore, decided to develop its own tool, the Search Filter Appraisal Checklist, described in this paper.

Methods
The Search Filter Appraisal Checklist was developed using consensus methods over three meetings of the ISSG during 2006 and 2007.ISSG members felt that, as a group of highlyskilled healthcare information specialists, they had the relevant skills to develop such a tool, having experience of publishing search filters, testing search filter performance, practising critical appraisal and developing checklists and structured abstracts.
Before the first meeting the ISSG members searched databases and their reference collections to identify existing tools.An ISSG draft checklist and a brief summary (see Figure 1) were designed.The checklists and the summary template were tested during the first meeting on a search filter by Zhang [11].The strengths and weaknesses of the alternatives were discussed.Key checklist concepts were agreed to be: • the focus and scope of the filter: limitations, generalizability and obsolescence; • the quality of the methods used to develop the filter, specifically how gold standards of relevant records were identified, how search terms were identified, how the strategy was developed, and how the filter performance was tested on the test gold standard (internal validity) and on separate validation gold standards (external validity).
It was agreed that a checklist should contain both categorical and descriptive information.A checklist should avoid numerical quality scores, for individual elements and the overall tool, because of known difficulties in assigning scores to individual dimensions of a tool and in interpreting a final combined score [12].
The dilemma of reporting adequacy should be addressed by wording comments to indicate that the assessment of the quality of the design of the search filter has to be made from the (sometimes limited) information provided in the report.The format of a checklist should be flexible to cope with the variety of search filter design methods.
Following the first meeting a second draft of the ISSG Search Filter Appraisal Checklist was developed and then discussed by email.
The summary, complementing the checklist, was re-addressed.Two alternative templates were developed: a one hundred word summary and a longer structured abstract (see Figures 2 and 3).The structured abstract template was designed to describe the filter objective, the methods used to develop the filter, key validation data, any reported limitations of the filter design and additional comments as appropriate.
ISSG members pilot-tested the checklist and abstracts at the second meeting using three different filters which had different methods of filter design [13][14][15].Two of the filters were from published articles with detailed methods sections and one was published on a website which reported little about its development.During the meeting the ISSG members discussed the usability, clarity, practicality and reproducibility of the draft Checklist and the two summary formats (Figures 2 and 3).
Following discussions the Checklist was revised and underwent a final round of feedback.
The final ISSG Search Filter Appraisal Checklist and structured abstract format were agreed by the ISSG at a third meeting in April 2007.

Results
One checklist was identified by searches: Michelle Jenkins' search filter appraisal checklist [1].
The consensus of the first ISSG meeting was that the Jenkins' checklist was helpful but not entirely suitable.It focused on determining generally whether filter design methods were reported.The checklist did not focus on filter design details or offer opportunities to extract data from the study.For example, the Jenkins' checklist asked "Do the authors report clearly how the filter performance was tested?", but did not ask what performance testing was undertaken or prompt the assessor to report performance data.The Jenkins' checklist asked some highly technical questions that might be difficult for some assessors to answer, for example, "Does the gold standard have sufficient power to allow statistically significant results?"Some of the questions, such as "Are the methods of search term derivation clearly described, and are they reasonable and likely to be effective?",were difficult to answer because they contained several elements.ISSG members agreed that the Jenkins' checklist was a helpful prompt, however, a more detailed checklist was required.
ISSG members agreed that the draft ISSG Checklist required further development.The stages of search filter design and purpose should be broken down into focused questions and the checklist should include data extracted from the publication.The members agreed that the ISSG Checklist should allow for narrative comment and that the brief summary (see Figure 1) was too brief to be a useful decision aid.
At the second meeting, the group agreed that the revised ISSG Checklist, tested against three filters, was an improvement.It captured relevance information more effectively.
Members felt that the revision addressed the issue that the assessor can only assess what the filter author reports.It achieved this by recording the reporting of the design but also by including prompts which reflect issues of design quality.These prompts should alert assessors to consider whether (unreported) alternative approaches might have been more suitable.
The ISSG felt that the revised checklist was flexible enough to capture the growing variety of methods reported in search filter design.It could capture information about multiple gold standards and validation testing activity.It also allowed an assessor to report performance comparisons against other filters, which strengthened the information available for deciding between filters.The checklist, however, still required work to capture information on how strategies were derived from the selected search terms.
At the second meeting the ISSG members also chose between summaries.The structured abstract was agreed to be more helpful than the one hundred word summary because it captured the filter objective, the main methods used to develop the filter, any key validation data, and any major limitations to the filter design.It provided space to summarise the strengths and weakness of the filter design.ISSG members agreed that the abstract was suitable for quick relevance assessment, with the Checklist offering the essential detail required for informed decision-making.
The final ISSG Search Filter Appraisal Checklist is shown in Table 1.Examples of completed Checklists are published on the ISSG website [16].An example structured abstract is shown in Figure 3.

Discussion
The Search Filter Appraisal Checklist is being used by ISSG members to appraise published search filters.Checklists are completed by an information professional, checked by an independent assessor, and edited by the website editor for consistency.Copies of Checklists will be sent to the original authors of the filters and feedback received will be published.
Completed checklists are published on the ISSG website [10].
The ISSG Search Filter Appraisal Checklist is designed to be comprehensive.Its structure follows the life-cycle of the process involved in developing a search filter from gold standard identification, search term selection, strategy development, testing and validation through to comparison with other filters.It may take time to complete but should provide clearer insight into the quality and suitability of a filter.
Health sciences librarians trying to decide between filters now have several tools.They can use the ISSG website to find appraisals of filters in the form of structured abstracts and Checklists.The abstract offers a rapid assessment of relevance and the Checklist offers more detailed information to assist with deciding whether a filter is useful.Alternatively, librarians can complete the blank Checklist themselves, to assess a filter of interest.The website and Checklist are also resources that librarians can recommend to relevant enquirers.
The Checklist is not exclusive.It does not 'reject' search filters which have been designed informally or which have not been tested or validated.It does, however, allow librarians and others to differentiate easily between evidence-based, validated filters and those of a less rigorous design.
Critical appraisal checklists serve several purposes.The clear breakdown of the reported methods in the Search Filter Appraisal Checklist is designed in the hope, shared with designers of other critical appraisal tools, that it will encourage filter authors, many of whom are librarians, to report detailed methods [17].In highlighting which methods to report to help readers assess the quality and relevance of a filter, librarians can also assist authors to achieve more transparent research reporting.
The Checklist focuses on search filters designed to retrieve studies with specific research methods (such as systematic reviews) or study type focus (such as diagnostic tests).Some of the Checklist's elements are likely to be applicable to search filters in other areas.Health sciences librarians may wish to explore the applicability of the Checklist beyond methods search filters.
There is scope to evaluate the performance of the Checklist, using independent assessors and a range of filters.Evaluations could assess ease of use, clarity, comprehensiveness and consistency.Since the Checklist was finalised the CADTH critical appraisal and ranking tool for search filters has been developed.It is less detailed than the ISSG Checklist and incorporates a score [18].A comparison of the two tools is a topic for further research.
Meanwhile, the ISSG Checklist is being used to assess filters on the ISSG website and is offered for evaluation.The ISSG would be grateful for feedback from health sciences librarians on its use.For each filter report the following information   The authors employed a pragmatic filter design approach.A set of candidate records was identified by searching MEDLINE and the authors derived terms from those records and tested out the precision of the strategy in MEDLINE. of 366 records was derived by hand searching 161 core health care journals in 2000.60% of the GS was used to derive the search terms and filters.The remaining 40% of the GS was used to validate the filters.Offers highly sensitive (92.47%), highly specific (99.36%) and best sensitivity and specificity compromise filters.The scale of the hand search is impressive, but the focus on one year may affect the future currency of the filter.

B. 4
What are the inclusion criteria for each GS? B.5 Describe the size of each GS and the authors' justification, if provided (for example the size of the gold standard may have been determined by a power calculation) B.6 Are there limitations to the gold standard(s)?Yes/No/Unclear B.7 How was each gold standard used?[ ] to identify potential search terms [ ] to derive potential strategies (groups of terms) [ ] to test internal validity [ ] to test external validity [ ] other, please specify B.8 Other observations.C. How did the researchers identify the search terms in their filter(s) (Were the search terms combined (using Boolean logic) in a way that is likely to retrieve the studies of interest?C.11 Other observations.D. Internal validity testing (This type of testing is possible when the search filter terms were developed from a known gold standard set of records).D.1 How many filters were tested for internal validity?

F. Limitations and comparisons . F. 1 F. 2 F. 4 G. 6
Did the authors discuss any limitations to their research?Are there other potential limitations to this research that you have noticed?F.3 Report any comparisons of the performance of the filter against other relevant published filters (sensitivity, precision, specificity or other measures).Include the citations of any compared filters.F.5 Other observations and / or comments.G.Other comments.This section can be used to provide any other comments.Selected prompts for issues to bear in mind are given below.G.1 Have you noticed any errors in the document that might impact on the usability of the filter?G.2 Are there any published errata or comments (for example in the MEDLINE record)?G.3 Is there public access to prepublication history and / or correspondence?G.4 Are further data available on a linked site or from the authors?G.5 Include references to related papers and/or other relevant material.Other comments.

Figure 1 . 1 A
Figure 1.Two brief summary formats tabled at the first consensus meeting

Figure 2 .
Figure 2. Example brief abstract tabled and rejected at the second consensus meeting

External validity testing (This section relates to testing the search filter on records that are different from the records used to identify the search terms).
E.