Searching for systematic reviews of the effects of social and environmental interventions : a case study of children and obesity *

s, Science Citation Index (SCI), Social Sciences Citation Index (SSCI), Arts & Humanities Citation Index (AHCI), ISI Science and Technology Proceedings, and Social Services Abstracts Specialist review registers Database of Promoting Health Effectiveness Reviews (DoPHER) and Database of Abstracts of Reviews of Effects (DARE) These databases mostly include reviews about the effectiveness of interventions. To be included on DARE, reviews must meet quality criteria [15]. Education Education Resources Information Centre (ERIC) Social policy Social Policy and Practice and Social Care Online Topic specific SPORTDiscus, Physical Education Index, and TRANSPORT Other 16 topic-specific Internet sites, C2-RIPE (Campbell collaboration website), Google Scholar, the British Library catalogue (ZETOC), and checking relevant bibliographies The Internet sites were identified through our scoping searches and previous work we had done [16]. All these searches used careful reviewer judgment during the searching process and so cannot be fairly compared to the database searches. * Some of the databases are freely available (MEDLINE via PubMed, HTA, DoPHER, and DARE), whilst most require subscription for access. Many academic libraries hold subscriptions and allow on-site access to databases. Details of databases can be readily found via Google or on academic library websites. Woodman et al. 142 J Med Libr Assoc 98(2) April 2010 included in our map and generated a total of 381 search results, making 3.4% of the records relevant (Table 3). The search of the Health Technology Assessment database (HTA) was the second most precise, as we included 2.3% of records generated by this search in our map. Although the PubMed search located the second highest absolute number of included systematic reviews (n57), it was a labor-intensive source to screen. The 7 relevant reviews only accounted for 0.2% of the 3,167 records generated by the PubMed search. In other words, to get to every relevant systematic review in the PubMed results, we had to sift through about 500 nonrelevant records. In contrast, when screening the source supporting the most precise search, DoPHER, we only had to sift through about 30 nonrelevant records for every relevant systematic review (Table 3). Given that our search dates were wider than our inclusion dates, it is possible that our analyses underestimate the precision of all databases and especially of the older, larger databases such as PubMed. Five sources supported moderately precise sources and gave a yield of between 1 and 2 relevant systematic reviews per 100 records generated. Two of these were results from databases focusing on health research: CINAHL and PsycINFO. Three came from sources focusing on social science and physical Table 2 Analyses carried out to assess the performance of each search source Name Description Calculation Precision For a source to support precise searches, it should provide a high number of included (relevant) systematic reviews in a relatively low number of total records. Less researcher time is needed to extract each relevant review in search results that have a higher precision. The number of included systematic reviews that were contained in the search results of one source divided by the total number of records generated


INTRODUCTION
Systematic reviews are an important part of the evidence base in public health.By locating and using all of the best available evidence, systematic reviews can overcome the potential for bias that is found in individual studies and provide reliable and accessible answers to questions about policy and practice.Systematic reviews also inform future research activity by identifying gaps and preventing duplication of effort.However, it can be difficult to know where to look for systematic reviews given the multiple research databases that index them and the fact that many are, at least initially, published as reports that may not be indexed in bibliographic databases.
There is little research investigating the best sources for locating systematic reviews.Research that has been done in the area focuses on the development of effective search terms and strategies (but not sources) for locating reviews, either in medical databases [1][2][3] Highlights N Specialist review registers including the Database of Promoting Health Effectiveness Reviews (DoPHER), Database of Abstracts of Reviews of Effects (DARE), and Cochrane Database of Systematic Reviews are likely to be time efficient and good places to start searching for reviews on the social and environmental determinants of health.
N Databases beyond the health arena did not yield many extra, relevant studies.This may inform the search choices for rapid evidence assessments or other resource-limited searches.

Implications
N The traditional health databases (PubMed, Psyc- INFO, CINAHL, and Health Technology Assessment [HTA]) should be searched for completeness.
N Supplementary searches (reference checking and Inter- net searches) remain important as they identified a substantial number of data not found by any other source.
N The choice of search sources should be informed by the type of evidence that is sought.Appropriate sources for locating primary and secondary evidence may be different, even when topic areas are similar.
or across a broader range of databases [4].During scoping searches, the authors only found one previous study evaluating search sources for locating reviews.The study described searches for reviews in four databases (MEDLINE, EMBASE, Cochrane Database of Systematic Reviews, and ''Best Evidence'') and concluded that results from EMBASE and MEDLINE were equally profitable for locating reviews [5].
The choice about which sources to search is difficult when the topic is multidisciplinary in nature, as is often the case in public health research [6].For example, researchers or policy makers who wish to identify systematic reviews that investigate the effects of social and environmental interventions (that is, targeting the wider determinants of health) will be interested in interventions that lie beyond the traditional health arena.Consequently, research is likely to be located in publications (and databases) that span a range of disciplines and topic areas.A methodological study by Ogilvie and colleagues investigated the productivity of different search sources across a range of disciplines, when searching for primary research on one type of social and environmental intervention.Ogilvie and colleagues' study found that most relevant research about interventions promoting a population shift from car use to active transport, with the aim of increasing physical activity, was not located in the familiar health or science databases [7].Instead, it was found in topic-specific databases.As far as we know, there has been no similar evaluation of search sources for systematic reviews.
The analysis of search sources presented here is based on a systematic descriptive map that aimed to describe the range, focus, and quality of existing systematic reviews investigating the effectiveness of social and environmental interventions to reduce obesity and overweight among children (referred to as ''the map'' in this paper) [8].Social and environmental interventions included those that aimed to alter the physical environment, social norms, technology, or the economy to impact eating behaviors, activity levels, weight status, or relevant attitudes.The map aimed to locate evidence from outside the traditional health arena and to bring together a wide range of research from health, transportation, physical activity, and food policy.

OBJECTIVES
The research sought to: 1. evaluate the precision of each search (identify the yield of systematic reviews that we included in our map per 100 records generated by each database) in order to measure how resource intensive it was to find and extract each relevant review from all the records generated 2. assess the relative and distinct contribution of each database source to the findings of our descriptive map in order to assess how necessary it was to conduct searches of databases that indexed literature beyond the health arena

Searches
In November and December 2007, we conducted systematic searches for a descriptive map of reviews on the effectiveness of social and environmental interventions to reduce obesity in children.The searches covered nineteen electronic databases, websites, and relevant bibliographies that were from the fields of health, social science, physical activity, and transportation and included specialist registers (Table 1) [8].We developed searches iteratively for each database and used a complex set of terms for the following concepts: (physical activity OR sedentary behaviour OR eating OR obesity) AND social and environmental AND review AND date (published since 1986)) Our searches were designed to be sensitive and wide-ranging, and our definition of ''review'' was very broad in order to locate evidence that might not be described as a ''systematic review'' or that did not use our ''social and environmental'' terminology.For example, our search terms for physical activity included those referring to activity, exercise, and sports generally; to active transport; to non-sedentary leisure activities such as gardening; and to children's nondirected ''play.''We used a combination of index terms and free text for each concept to ensure that our results were not biased by how well individual databases were indexed.Full details of the strategies for each database are available from the Evidence for Policy and Practice Information and Co-ordinating (EPPI)-Centre website [9].
We did not search the Cochrane Database of Systematic Reviews directly as this source is searched quarterly and uploaded to the EPPI-Centre's own specialist review database, the Database of Promoting Health Effectiveness Reviews (DoPHER), which was searched.The search strategy used to locate public health and health promotion reviews on the Cochrane Database of Systematic Reviews has been developed over many years and is highly sensitive [10].It includes exploded Medical Subject Heading (MeSH) terms for health education, health promotion, public health, preventive health services, preventive medicine, primary health, and primary prevention along with free text terms.We anticipated that DoPHER would include all relevant material listed in the Cochrane Database of Systematic Reviews.
With the aim of making our future searches more efficient, once the descriptive map was published, we retrospectively analyzed the usefulness of the search sources, and these analyses are presented here.

Inclusion criteria for systematic reviews
We included systematic reviews that investigated an obesity-relevant topic (broadly defined, see full report [8]); reviewed the effectiveness of interventions (broadly defined, see full report [8]); included the four-to-eighteen age group; and focused on ''social Searching for systematic reviews and environmental'' interventions (i.e., interventions targeting the wider determinants of obesity).Although we searched for reviews published since 1986, to restrict the dataset to manageable proportions, we only included reviews published in or after 1996.We applied the 1996 cut off at the first screening stage (title and abstract).At this stage, forty-five records were excluded on the basis of date alone (i.e., the abstract suggested that the review appeared to meet all our other inclusion criteria or had insufficient detail to tell whether all criteria where met).We found that there was often limited detail in abstracts about the methods used and interventions reviewed, and many of the full texts that we retrieved did not in fact meet our inclusion criteria relating to systematic review, age of target population, or social and environmental nature of the interventions.It is therefore likely that many of the forty-five records published before 1996 would also have been excluded, based on these criteria, had we had detail available in full text.
We defined reviews as ''systematic'' if they clearly stated their aims, search strategy, and inclusion criteria.We did not further appraise the quality of reviews.The records were divided between 3 reviewers who independently ''screened'' records on title, abstract, and, where records were potentially relevant, full text.''Screening'' is the process of systematically checking all the bibliographic records captured electronically against predefined inclusion criteria to identify relevant studies.For quality assurance, the first 5% of all titles and abstracts were screened by all 3 reviewers, who then compared results and recalibrated their screening technique to improve consistency.All 3 reviewers were on hand throughout screening to discuss and resolve any uncertainties.

Analysis
We grouped the search sources into seven broad categories, some based on topic, others based on the type of research indexed, such as the registers of reviews (Table 1).The grouping of the sources into categories was suggestive rather than definitive.We analyzed the different sources in three ways (Table 2).All analyses were carried out using EPPI-Reviewer data-management software [11].
When conducting the searches grouped in the ''other'' category, the reviewers used careful judgment to identify potentially relevant records.They were, therefore, partially screened for relevancy before being uploaded.In addition, researchers did not upload records that they knew had already been found by other sources.As a result, the ''other'' searches cannot be compared on equal terms with the database searches in terms of precision and sensitivity and are not included in Table 3.However, the ''other'' searches are comparable in terms of their contribution of unique reviews and are included in Table 4.

RESULTS
The original searches from databases generated 6,197 records, of which 697 were duplicates.We excluded 5,167 records on abstract and title and retrieved 333 full reports.We included a total of 32 relevant systematic reviews from all sources.For full details, see our map [8].

Which sources supported the most precise searches?
The source supporting the most precise search was the specialist review database, DoPHER.This source contained 13 of the systematic reviews that we The Internet sites were identified through our scoping searches and previous work we had done [16].All these searches used careful reviewer judgment during the searching process and so cannot be fairly compared to the database searches.
included in our map and generated a total of 381 search results, making 3.4% of the records relevant (Table 3).The search of the Health Technology Assessment database (HTA) was the second most precise, as we included 2.3% of records generated by this search in our map.
Although the PubMed search located the second highest absolute number of included systematic reviews (n57), it was a labor-intensive source to screen.The 7 relevant reviews only accounted for 0.2% of the 3,167 records generated by the PubMed search.In other words, to get to every relevant systematic review in the PubMed results, we had to sift through about 500 nonrelevant records.In contrast, when screening the source supporting the most precise search, DoPHER, we only had to sift through about 30 nonrelevant records for every relevant systematic review (Table 3).Given that our search dates were wider than our inclusion dates, it is possible that our analyses underestimate the precision of all databases and especially of the older, larger databases such as PubMed.
Five sources supported moderately precise sources and gave a yield of between 1 and 2 relevant systematic reviews per 100 records generated.Two of these were results from databases focusing on health research: CINAHL and PsycINFO.Three came from sources focusing on social science and physical The number of included systematic reviews that were contained in the search results of one source divided by the total number of records generated by that source.

Sensitivity
To assess the relative contribution of each source to our map, we calculated sensitivity.A source with high sensitivity would locate a high proportion of our 32 included reviews (even if other sources also located the same reviews).
The number of included reviews found by a source divided by the total number of included systematic reviews (n532).

Unique reviews
To determine whether it was necessary to search sources that indexed reviews outside of the tradition health arena, we calculated the number of ''unique'' relevant systematic reviews found by each source (i.e., only contained in the results from 1 source).Seventeen included reviews were located exclusively by 1 source (i.e., 17 unique reviews).Analyzing unique reviews allowed us to see the unique contribution made by the sources that primarily indexed health research and by those that focused on research areas other than health.
The number of unique records located by a source divided by the total number of unique reviews (n517).Searching for systematic reviews activity: Social Sciences Citation Index/Science Citation Index, Arts & Humanities Citation Index, and SPORTDiscus, respectively.One was a specialist review register: DARE.Nine of the 19 databases did not contain any records that we included in our map, although they generated a total of 884 records, all of which had to be screened.Many of these 9 databases focused on social policy, education, or social science research (Table 3).

How important was it to search beyond the traditional health databases?
Table 3 shows that, when combined, the search results from the 4 health databases contained almost half (n515) of the total relevant systematic reviews that we located.The specialist review registers located 17 relevant systematic reviews, more than half of all that we found.While these 2 sources made a large contribution, many sources did not uniquely locate any systematic reviews and so did not contribute anything to our findings.
Seventeen of the 32 included systematic reviews (53%) were unique (located by only 1 search source).Table 3 shows that 70% of these unique included reviews were found by the health databases and specialist review databases (29% and 41%, respectively).An additional 4 systematic reviews were unique to the results from our searching websites, library catalogs, and bibliographies of relevant research (classified as ''other'' in Table 1) (Table 4).
Apart from the specialist review registers, sources that had a focus other than health contributed little to our map.Table 4 shows that only one included systematic review would have been missed if we had limited our search to the health databases (which located fifteen unique reviews), the specialist review registers (which located twelve unique reviews), and website, library catalogs, and bibliographies of relevant research (which located four unique reviews).The one additional included systematic review that was not from these search sources was located in the results from Physical Education Index (Table 3).

Principal findings
Given their precision, the specialist review registers are likely to be most time efficient for researchers and decision makers looking for evidence in this area.The least time-efficient sources were the nine sources from social science, education, and social policy, which did not locate any of the systematic reviews included in our map but together generated 884 records (before de-duplication).
Although resource intensive to search (given the high numbers of records generated), the ''health' searches contained almost half (n515) of the 32 relevant systematic reviews.The results from searching the specialist review registers databases were both precise and made a high relative and unique contribution to our map.Four (13%) of the reviews we included were found only by searching websites, library catalogs, and relevant bibliographies (classified as ''other'').Limiting our searching to these 3 groups of sources (health, specialist review, and ''other'') would have located almost all (n531) of the 32 relevant reviews we found.Searches of 11 databases that focused on research outside the health arena contributed only 1 extra review to our map.

Strengths and weaknesses of this analysis
Our searches were designed to be extremely sensitive, and our inclusion criteria were broad.Therefore, it was likely that we found a large proportion of all relevant data that existed and that our results might be of use to many of those interested in the social and environmental determinants of health, regardless of their specific research question.Our quality assurance measures when screening (reviewer calibration through ''double screening'' and constant discussion) were designed to minimize reviewer bias and to promote consistency between reviewers when selecting relevant reviews.
However, our results were likely to be affected by the fact that some search sources would be a better ''match'' than others to our specific research question or topic.As might be expected, the source supporting most precise searches in our analysis was DoPHER, which was matched to our research question both in terms of the study type (reviews of effectiveness) and subject focus of the indexed reviews (social and environmental determinants of health).It also included sensitive searches of other highly relevant sources, as DoPHER is populated through regular searches of specialist databases of reviews of effectiveness, including the Cochrane Database of Systematic Reviews.Further reviews for DoPHER are identified through hand-searching and searches of relevant organizations' websites.Given the high relevance of DoPHER to our research question, it might not be such a rich search source for research questions outside the scope of effectiveness of interventions and/or public health.Development and maintenance of DoPHER is funded by the UK Department of Health to increase access to systematic and nonsystematic reviews of relevance to health promotion and public health.These results indicate that DoPHER is fulfilling this function.
Our results should be interpreted in the knowledge that the categories we used for the analyses are, in Woodman et al.
practice, not mutually exclusive.For example, Do-PHER could be considered both a ''specialist review register'' database and a ''health'' database.Search sources should be chosen in light of the research question; those whose scope overlaps with the research question in more than one characteristic are likely to be most relevant.
It is also possible that the discrepancy between our search dates (1986 onward) and inclusion dates (1996 onward) may have underestimated precision in all databases and especially in the older, larger databases, such as PubMed.
For the interests of this paper and in anticipation that DoPHER would include all relevant Cochrane reviews at the time of our searches, we retrospectively tested this assumption.Two of the included reviews were Cochrane reviews [12,13], which we found on 3 other sources: DoPHER, DARE, and PubMed.We retrospectively searched the Cochrane Database of Systematic Reviews using terms derived from our PubMed strategy, as both databases index research using the same MeSH.We analyzed the results in terms of sensitivity, precision, and unique reviews.The search yielded 523 records, 2 of which were reviews meeting all our inclusion criteria.These 2 reviews had been identified by DoPHER and other sources.The sensitivity of this search was 6.3% (comparable to our searches of HTA and PsycINFO), and precision was 0.4% (comparable to the large databases such as PubMed).We did not identify any unique reviews.
Whilst the Cochrane Database of Systematic Reviews was not searched directly for this review, we recommend that it should always be searched when trying to identify trials or systematic reviews, for a number of reasons.Whilst records of Cochrane reviews are included in sources such as PubMed, DARE, and DoPHER, there will always be a time lag between their appearance in the Cochrane Database of Systematic Reviews and in other sources.Uniquely, the Cochrane Database of Systematic Reviews also provides access to records of protocols for systematic reviews in progress.Furthermore, many countries now have free access to full-text Cochrane reviews that are not available via other sources.
Finally, like Ogilvie and colleagues, we only analyzed what we found in practice and not what could have been found had our search terms or inclusion criteria been different [7].Testing the performance of the search strategy on each database would involve manually checking whether any of the thirty-two reviews were indexed in each of the sources at the time searches were run and, if indexed, why these reviews were not identified by that particular search.As a large number of databases was searched, a comprehensive analysis would be prohibitively time consuming and be subject to issues involving the staged timing of adding citations, abstracts, and index terms to databases and subsequently uploading abstracts and indexing them.
On balance, these results, although not a definitive answer to searching for systematic reviews, will help to inform future searches, especially those seeking to answer research questions about the effectiveness of social and environmental interventions.Further analysis of substantive reviews and reviews of reviews, such as this one, may help build a picture from which a pattern can be seen and wider conclusions drawn.

CONCLUSIONS
Our analysis suggests that information scientists, librarians, and researchers who want to access the best available reviews on the effectiveness of social and environmental interventions should start by searching the specialist review databases, including DoPHER, DARE, and the Cochrane Database of Systematic Reviews.To achieve any level of completeness, the traditional health databases-such as PubMed, CINAHL, PsycINFO, and HTA-should also be searched.Supplementary searches, such as searching carefully selected Internet sites and checking the bibliographies of relevant research, remain an important source of systematic reviews in this area.
The results of this analysis suggest that it may not be necessary to search databases outside the traditional health arena to locate systematic reviews about the effectiveness of social and environmental interventions, at least in the area of obesity.To be truly exhaustive, searches of topic-specific databases outside the health arena are still required.Careful consideration should be paid to the type of research question and the extent to which it is matched to the research contained in these databases before they are ignored.
The results of our analysis may inform tools for quality assessment of ''rapid evidence assessments,'' which use systematic reviews to answer questions in a short time frame.Our analysis suggests that a search of the mainstream health databases, particularly PubMed, may reasonably form the backbone of rapid search strategies in this area, particularly if they are supplemented by searches of the specialist review registers databases, the Internet, and bibliographies.
Our findings did not match those of Ogilvie and colleagues in their analysis of search sources for their review on promoting a population shift from using cars to walking and cycling [7].Ogilvie and colleagues found that very few of their relevant studies were located by the mainstream health databases, and most were located by the topic-specific source TRANSPORT.As the scope of our systematic map covered many of the interventions reviewed by Ogilvie and colleagues (such as publicity campaigns to promote walking, financial incentives or taxes to reduce car use, and engineering measures to promote safety for cyclists and walkers), the differences between our findings suggest that sources that are very useful for locating primary research in this field are not necessarily the most useful for locating systematic reviews.
Centre health team on whose work and experience we drew when preparing this report.

Table 1
Seven source categories

Table 2
Analyses carried out to assess the performance of each search source

Table 3
Sensitivity and precision of each search source, grouped by broad category