Included under terms of UK Non-commercial Government License.
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Brush J, Boyd K, Chappell F, et al. The Value of FDG Positron Emission Tomography/Computerised Tomography (PET/CT) in Pre-Operative Staging of Colorectal Cancer: A Systematic Review and Economic Evaluation. Southampton (UK): NIHR Journals Library; 2011 Sep. (Health Technology Assessment, No. 15.35.)
The Value of FDG Positron Emission Tomography/Computerised Tomography (PET/CT) in Pre-Operative Staging of Colorectal Cancer: A Systematic Review and Economic Evaluation.
Show detailsSystematic reviews adopt an approach of searching extensively for studies to ensure that as much available evidence as possible can inform the review, and to minimise bias, including publication bias. Handsearching has formed part of the extensive search approach for systematic reviews of effects. Handsearching is the page-by-page examination of the entire contents of a journal issue or conference proceedings to identify eligible reports of studies for a review.1,2 Relevant information may appear in any part of a journal: articles, conference abstracts, news reports, letters, editorials, comments and other journal parts.
Guidance for conducting Cochrane systematic reviews of effects evidence and more recently for Cochrane diagnostic test accuracy studies recommends that handsearching should be considered to enhance the retrieval of relevant studies.1,2 Handsearching is viewed as potentially valuable because it might identify additional reports of diagnostic test studies compared with searches of databases alone.1,2 A systematic review of handsearching compared with database searches for randomised controlled trials (RCTs) has found that handsearching generates useful studies in addition to database searching.3 A similar review is not yet available for diagnostic test accuracy studies, but single studies report that database searches alone can miss potentially relevant studies. This may be because those studies are not recorded in the database or are recorded but are not retrievable with the strategy used, perhaps because of inadequate or no indexing.4–6 In addition, the variable quality of the reporting of diagnostic test accuracy studies has been noted by several authors and this is also likely to affect effective database retrieval.1 However, the evidence for the benefits of handsearching for diagnostic test accuracy studies for systematic reviews remains to be established.1
In the light of this sparse research evidence, we designed a complementary but distinct research project to exploit the opportunity to explore the value of handsearching to inform an imaging systematic review and to contribute to our understanding of the role of handsearching in the identification of reports of diagnostic test accuracy studies.
Aims and objectives
The aim of the sub-study was to investigate the contribution of handsearching to the identification of studies to inform a diagnostic test accuracy review.
The objectives of the handsearch project were:
- to identify a range of relevant journals and handsearch them to identify diagnostic test accuracy studies, and more specifically diagnostic test accuracy studies of fluorine-18-deoxyglucose (FDG) positron emission tomography (PET)/computerised tomography (CT) imaging in colorectal cancer (CRC)
- to assess the relative yield of handsearching compared with database searching in identifying reports of diagnostic test accuracy relating to FDG PET/CT imaging in CRC
- to assess the costs of handsearching compared with those of database searching.
Methods
The first stage of the project was to identify high-yield journals to be handsearched for studies. A systematic review of handsearching studies to identify RCTs has reported several methods that have been used to identify and select journals to handsearch.3 The most frequently mentioned approach is to identify which journals offer the highest yield of relevant studies for a specific review question and then to select a number of those journals that provide the largest number of studies as the candidate journals to handsearch. This approach is also endorsed by the Cochrane handbook for systematic reviews of diagnostic test accuracy.1 Information retrieval advice in other subjects has also recommended this selection method.7,8 High-yield journals were identified by selecting reviews of FDG PET for CRC retrieved by a search of Ovid MEDLINE in April 2009 and collecting the references of the studies included in those reviews. The references were obtained, de-duplicated and grouped by journal. From this frequency list, the 10 highest yielding journals to which we had electronic access were selected.
The high-yield journals were handsearched by accessing them in electronic form on their journal home sites. Journal issues from 2005 to June 2009 were handsearched.
Each article in each issue of each of the 10 journals was assessed for relevance by reading the title, abstract and as much of the paper as required to determine whether or not the paper reported the results of a diagnostic test accuracy study and in particular whether or not it focused on the diagnostic accuracy of FDG PET/CT imaging for CRC. In cases in which the journal issue contained conference abstracts in themed groups, only the sub-sections relevant to CRC or FDG PET were handsearched. Details of the journal, issue, time spent handsearching per issue and diagnostic test accuracy papers identified were recorded in an Excel (Microsoft Corporation, Redmond, WA, USA) spreadsheet.
The results from the handsearches were compared with the results of database searches conducted for the FDG PET/CT systematic review for the same years. The relative unique yields for handsearching and for database searching, and the overlap, were calculated. The average time spent searching per candidate study identified was calculated. The database searches and record assessment and selection were carried out independently from the handsearches.
Three handsearchers conducted the handsearch (MC, PL and JG). They participated in a 2-hour training session followed by several rounds of selection decision checking. An inter-relater reliability exercise was conducted to estimate the level of agreement between the two high-volume handsearchers (MC and JG) using a randomly selected 10% sample of journal issues.
In the final stage of the research, the studies included in the FDG PET/CT review were investigated to assess how far the journals in which they were published overlapped with the candidate journals that were handsearched. The journals in which the included studies were published were noted and the number of diagnostic test accuracy studies identified in each was collected. The two highest yield journals for included studies were then handsearched to identify any diagnostic test accuracy studies published in the period January 2005 to June 2009 inclusive.
We also investigated the effect of removing the studies from the three more general reviews that contributed to the original frequency list to see if the order of candidate journals would have changed. We also analysed the disciplines of the candidate journals.
Results
The MEDLINE search for systematic reviews of FDG PET/CT for CRC identified 187 potentially relevant reviews. From these, six reviews were selected and their included studies were collected.10–15 A total of 216 records of individual diagnostic test accuracy studies were sorted by journal name, and the journal names that appeared most frequently are shown in Table 1. Ten of the highest yielding 11 journals were selected for handsearching; the International Journal of Colorectal Diseases, which appears in the 10 highest yield journals, was not available to search free of charge through our institutional subscriptions.
A total of 573 journal issues from the 10 journals selected were handsearched. This process took 185 hours, not including the quality-checking process. The inter-rater reliability between the two high-volume handsearchers on record selection on a 10% random sample of 53 journal issues was found to have a Kappa value of 0.614 (p < 0.001), which falls just within the range of substantial agreement (0.61–0.8).
Through handsearching, 25 candidate diagnostic test accuracy records were identified (see Appendix, Table 4). Records were identified from full article, mixed full and abstract and abstract only journals. One candidate record was identified per 7.4 hours of handsearching. The time excludes the quality-checking process.
When the 25 records were checked against the database search results, seven of the handsearched records identified had not been identified by the database searches. However, once the full paper had been assessed, none of the seven studies fit the systematic review inclusion criteria. Some of the records were abstracts from the British Journal of Surgery and these were checked (searching on first and last authors) in MEDLINE for full reports, but none was identified at that time.
In addition to identifying diagnostic test accuracy studies of FDG PET/CT, this research also identified 855 potential diagnostic test accuracy studies for other conditions: 4.6 studies per handsearch hour.
The database searches for studies of FDG PET/CT had identified 30 studies that met the eligibility criteria. The journals in which these studies had been published were assessed for overlap with the journals that had been handsearched (Table 2). The 30 studies that met the FDG PET/CT review eligibility criteria appeared in 24 different journals, and three of these had been handsearched. None of the 21 non-handsearched journals had appeared as high-frequency journals in the original frequency analysis of the journals in which 200 relevant diagnostic test accuracy studies of FDG-PET had been published. The handsearch identified the one included study that was in a handsearched journal in the date range searched.
The database searching and record selection process (excluding quality checking processes) took 126 hours and comprised the following activities:
- 20 hours spent searching, testing and downloading, de-duplicating and writing up
- 25 hours spent checking MEDLINE for full reports of conference proceedings
- 45 hours for selection of records from the database searches
- 36 hours spent checking the references of review articles and primary studies.
This represented one eligible study per 4.2 hours of searching and selection.
The two journals that published the highest number of included studies for the current review were then handsearched: the Journal of Nuclear Medicine and the European Journal of Nuclear Medicine and Molecular Imaging. The total time taken to handsearch the two additional journals (including the construction and population of the proformas) was 5.6 days. The European Journal of Nuclear Medicine and Molecular Imaging yielded a greater number of FDG PET/CT studies than the Journal of Nuclear Medicine (15 vs 3), although the total number of potentially relevant diagnostic test accuracy studies for all conditions was similar (108 vs 103). The mean time spent searching to identify a potentially relevant study in the two journals was 2.2 hours for FDG PET/CT and 0.2 hours for any diagnostic test accuracy study.
Of the 18 records identified by handsearching, six records had not already been retrieved by the database searches. All six records were conference abstracts. The records were assessed for relevance and checked in MEDLINE for full publications. One record was rejected because it was a study of separate PET and CT machines, three records had been published as full reports and are included in the reviews and two records do not seem to have full publications yet.
An analysis of the broad discipline categories of the journals that published diagnostic test accuracy studies is presented in Table 3.
The highest number of diagnostic test accuracy studies was reported in imaging journals, followed by surgical journals and then colorectal journals.
Discussion
We had expected that the handsearch would identify reports that had not been retrieved by the electronic searches. Although handsearching did identify publications that we had not previously encountered, it did not yield unique relevant diagnostic test accuracy studies. This result may be attributable to several factors. First, it may be a tribute to the sensitivity of the systematic review database searches and reference checking of reviews conducted for the project. It may also be the case that FDG PET/CT studies are relatively easy to retrieve in a sensitive way because the terms used to describe FDG PET/CT in abstracts and titles are reasonably standard. The search strategy development was informed by previous investigations of optimal searching for FDG PET conducted by Mijnhout et al.16–17 and modified by a member of this team (MD) to maximise sensitivity.
There is little overlap between the journals that were handsearched and the journals in which the included studies were published: only three journals. The high-yield journal list used to identify journals to handsearch emerged from FDG-PET reviews. The journals in which the included studies selected for this review were published include some that did not yield any articles in previous reviews. This seems to indicate that FDG PET/CT studies are highly scattered in journals in many disciplines, and that it is difficult to predict where researchers may publish nuclear imaging diagnostic test accuracy studies and where handsearching may be most fruitful. This exercise may also indicate that the frequency analysis approach to identify journals for handsearching may not help when searching for imaging studies. It may be better to handsearch later in the review when included studies are emerging rather than basing the handsearch selection on previous reviews.
There is one area in which this handsearch was not comprehensive. Some journal issues contained conference abstracts. Where conference abstracts were grouped by session theme, only the relevant groups of abstracts were handsearched: sessions on CRC or imaging FDG-PET. This means that relevant abstracts may have been missed if they were categorised into different sessions from those we searched. In addition, non-FDG-PET diagnostic test accuracy conference abstracts in other sessions will also not have been captured.
It is possible that the choice of reviews to mine for included studies biased the journal frequency table. We investigated this by assessing the effect of removing some of the more general of the original reviews from the journal list, to see if this would have raised to prominence some lower-frequency journals from the original list. The reviews by Bipat et al.,10 Halligan et al.12 and Purkayastha et al.14 consider techniques other than FDG PET and might, in retrospect, be deemed less pertinent to the final topic of this review. The references from those three reviews were removed from the initial journal frequency list to investigate the impact on high-frequency journals. The revised frequency list still ranked seven of the journals originally identified for handsearching in the top 10 (see the appendix, Table 5 for details). When the revised frequency list was compared with the journals in which the included studies for this review were published, the overlap was still four journals. Neither the revised or original frequency list highlighted the European Journal of Nuclear Medicine and Molecular Imaging as a candidate for handsearching, despite it yielding the highest number of included studies in this review.
In comparison with the other ‘high-yield’ journals searched, the European Journal of Nuclear Medicine and Molecular Imaging would appear to be an extremely good source of diagnostic test accuracy studies focusing on FDG PET/CT in CRC. Searching the two high-yield journals arising from the database searches had a substantial impact on the number of diagnostic test accuracy studies identified by handsearching. The total number of candidate records increased by approximately 70% (from 25 to 43) and the time spent handsearching to identify one candidate record fell by approximately 30% (from 7.4 to 5.2 hours).
The analysis of the journals by broad discipline shows that the publication of FDG PET/CT diagnostic test accuracy studies is widely scattered. The journal categories were very broad and approximate, with no overlap between categories allowed, and thus may not reflect the exact nature or encompass the true diversity of the journals. For example, the journal Radiology is not specific to pure imaging applications as it also reports studies relating to treatment. However, the exercise does illustrate that any search for diagnostic test accuracy studies for imaging cannot be easily limited to specific disciplines, and means that identifying journals to handsearch in this topic is problematic.
Conclusion
Handsearching is time-consuming and expensive. In this review, handsearching did not yield additional unique studies relevant to FDG-PET in addition to database searching and reference checking. Explanations for the low yield may be that the database searches were highly sensitive and FDG PET/CT studies tend to be consistently described and hence are easier to retrieve. The value of handsearching to identify studies of less clearly defined or reported diagnostic tests remains to be investigated. It was frequently difficult, during the preliminary identification of potentially relevant studies, to identify whether or not a paper was reporting a diagnostic test accuracy study from the abstract alone. Abstracts often allude to diagnosis or diagnostic issues, but only on reading the full paper does it become apparent that diagnostic test performance measures are not reported.
Handsearching did yield conference abstracts that may be useful clues or prompts to look for later published studies. Conference abstract publication rates may also give insight into publication bias in diagnostic tests.
Identifying the highest yield journals to handsearch for imaging studies may be difficult because of the scatter of studies over journals from many disciplines. The approach to identifying candidate journals to handsearch for imaging studies and diagnostic test studies more generally may require further investigation. Inevitably, in a topic in which there is publication in many different journals, the database and related searches are likely to provide the highest and most efficient yield of study reports.
Handsearching yields reports of many other diagnostic test accuracy studies in addition to those for the topic of interest, and this may be a useful byproduct of handsearching for a specific issue. It would be particularly valuable for future reviews in all topics if those studies identified by handsearching are contributed to the Cochrane Register of Diagnostic Test Accuracy Studies. As the Cochrane Register grows the need for handsearching should diminish.
References
- 1.
- de Vet HCW, Eisinga A, Riphagen II, Aertgeerts B, Pewsner D. Cochrane handbook for systematic reviews of diagnostic test accuracy, Version 0.4. Cochrane Collaboration; 2008. Chapter 7: Searching for studies.
- 2.
- Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions, Version 5.0.1. Cochrane Collaboration; 2008. (updated September 2008)
- 3.
- Hopewell S, Clarke M, Lefebvre C, Scherer RW. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007;2:MR000001. [PMC free article: PMC7437388] [PubMed: 17443625]
- 4.
- Whiting P, Westwood M, Burke M, Sterne J, Glanville J. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008;61:357–64. [PubMed: 18313560]
- 5.
- van der Weijden T, Ijzjermans CJ, Dinant GJ, van Duijn NP, de Vet R, Buntinx F. Identifying relevant diagnostic studies in MEDLINE. The diagnostic value of the erythrocyte sedimentation rate (ESR) and dipstick as an example. Fam Pract. 1997;14:204–8. [PubMed: 9201493]
- 6.
- Vincent S, Greenley S, Beaven O. Clinical evidence diagnosis: developing a sensitive search strategy to retrieve diagnostic studies on deep vein thrombosis: a pragmatic approach. Health Info Libr J. 2003;20:150–9. [PubMed: 12919278]
- 7.
- Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health. 2005;27:388–91. [PubMed: 16311247]
- 8.
- Torgerson C. Systematic reviews and meta-analysis. London: Continuum International Publishing; 2003.
- 9.
- Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med. 2002;21:1625–34. [PubMed: 12111923]
- 10.
- Bipat S, Glas AS, Slors FJ, Zwinderman AH, Bossuyt PM, Stoker J. Rectal cancer: local staging and assessment of lymph node involvement with endoluminal US, CT, and MR imaging – a meta-analysis. Radiology. 2004;232:773–83. [PubMed: 15273331]
- 11.
- Bipat S, van Leeuwen MS, Comans EF, Pijl ME, Bossuyt PM, Zwinderman AH, et al. Colorectal liver metastases: CT, MR imaging, and PET for diagnosis – meta-analysis. Radiology. 2005;237:123–31. [PubMed: 16100087]
- 12.
- Halligan S, Altman DG, Taylor SA, Mallett S, Deeks JJ, Bartram CI, et al. CT colonography in the detection of colorectal polyps and cancer: systematic review, meta-analysis, and proposed minimum data set for study level reporting. Radiology. 2005;237:893–904. [PubMed: 16304111]
- 13.
- Huebner RH, Park KC, Shepherd JE, Schwimmer J, Czernin J, Phelps ME, et al. A meta-analysis of the literature for whole-body FDG PET detection of recurrent colorectal cancer. J Nucl Med. 2000;41:1177–89. [PubMed: 10914907]
- 14.
- Purkayastha S, Tekkis PP, Athanasiou T, Tilney HS, Darzi AW, Heriot AG. Diagnostic precision of magnetic resonance imaging for preoperative prediction of the circumferential margin involvement in patients with rectal cancer. Colorectal Dis. 2007;9:402–11. [PubMed: 17504336]
- 15.
- Wiering B, Krabbe PF, Jager GJ, Oyen WJ, Ruers TJ. The impact of fluor-18-deoxyglucose-positron emission tomography in the management of colorectal liver metastases. Cancer. 2005;104:2658–70. [PubMed: 16315241]
- 16.
- Mijnhout GS, Hooft L, van Tulder MW, Deville WL, Teule GJ, Hoekstra OS. How to perform a comprehensive search for FDG-PET literature. Eur J Nucl Med. 2000;27:91–7. [PubMed: 10654153]
- 17.
- Mijnhout GS, Riphagen II, Hoekstra OS. Update of the FDG PET search strategy. Nucl Med Commun. 2004;25:1187–9. [PubMed: 15640777]
Appendix
Table 5 compares the numbers of studies retrieved from each journal depending on whether studies from six or three reviews were assessed. The revised frequency list ranked seven of the journals identified for handsearching (from the original list; marked ***) in the top 10. A further journal lost its top 10 ranking, while the three remaining journals failed to make the list. There was little change when considering the list of journal sources of studies included in the review (marked +). Neither the revised or original frequency list highlighted the European Journal of Nuclear Medicine and Molecular Imaging for handsearching, despite it yielding the highest number of included studies. The revised frequency list also failed to highlight Abdominal Imaging, the second highest yielding journal.
- FDG PET/CT as an add-on imaging test versus routinely used imaging modalities fo...FDG PET/CT as an add-on imaging test versus routinely used imaging modalities for pre-operative staging in patients with primary, recurrent or metastatic colorectal cancer: a handsearch study - The Value of FDG Positron Emission Tomography/Computerised Tomography (PET/CT) in Pre-Operative Staging of Colorectal Cancer: A Systematic Review and Economic Evaluation
Your browsing activity is empty.
Activity recording is turned off.
See more...