The evaluation of rapidly evolving health technologies to inform policy decisions is a challenge for those conducting systematic reviews. There is debate as to whether data from unpublished studies available only as conference abstracts and presentations should be included in high-quality systematic reviews of evidence.
Inclusion of unpublished data from conference abstracts and presentations could assist in the generation of a more comprehensive data set. However, conference abstracts and presentations are difficult to locate as they are poorly or not indexed in standard bibliographic databases typically searched when conducting systematic reviews. In addition, overall quality of reporting in conference abstracts and presentations may be inadequate, and data reported in these sources may not be complete and may be inconsistent with those reported in subsequent full publications.
The objectives of this research were to assess:
- the extent of use of data from conference abstracts and presentations in health technology assessments (HTAs) provided as part of the National Institute for Health and Clinical Excellence (NICE) appraisal process
- the ability to judge the methodological quality of trials from conference abstracts and presentations
- the consistency of reporting major outcomes between conference abstracts and presentations and subsequent full-length publications
- the effect of inclusion or exclusion of data from conference abstracts/presentations on the meta-analysis pooled effect estimates
- the timeliness of availability of data from abstracts/presentations and full articles in relation to the development of technology assessment reports (TARs).
Evidence for this research was obtained from:
- a survey of technology assessment reviewgroups (TAR groups): conducted of all seven TAR groups in the UK to identify current policy and practice regarding identification, inclusion and assessment of conference abstracts and presentations for TAR reports
- an audit of published TARs: included all NICE TARs published between January 2000 and October 2004 to identify the extent of use of conference abstracts and presentations
- case studies of selected TARs: included TARs of rapidly evolving technologies that identified and included trial data from conference abstracts and presentations and included a quantitative analysis.
Analyses of the results of the survey and audit are presented as a descriptive summary and in a tabular format. Data extracted from abstracts and presentations and subsequent full publications included in the case studies are presented descriptively and quantitatively. Sensitivity analyses were carried out to compare the effect of inclusion of data from abstracts and presentations on the meta-analysis pooled effect estimates by including data from both abstracts/presentations and full papers, and data from only full publications, included in the original TAR. These analyses were then compared with meta-analysis of data from trials that have subsequently been published in full.
All seven TAR groups completed and returned the survey. Five out of seven groups reported a general policy that included searching for and including studies available as conference abstracts and presentations. Five groups responded that if they included data from abstracts/presentations they would carry out methodological quality assessment of studies from abstracts/presentations using the same assessment tools as for full publications, and would manage the data from these sources in the same way as fully published reports. All groups reported that if relevant outcome data were reported in both an abstract/presentation and a full publication, they would only consider the data in the full publication. Conversely, if data were only available in a conference abstract/presentation, all but two groups reported that they would extract and use the data from the abstract/presentation.
In total, 63 HTA reports for NICE were identified. In 20 of 63 TARs (32%) explicit statements were made with regards to inclusion and assessment of data from abstracts/presentations. Thirty-eight (60%) identified at least one randomised controlled trial (RCT) available as a conference abstract or presentation. Of these, 26 (68%) included trials available as abstracts/presentations.
About 80% (20/26) of the 26 TARs that included RCTs in abstract/presentation form carried out an assessment of the methodological quality of such trials. In 16 TARs full reports of these trials were used for quality assessment where both abstracts/presentations and subsequent full publications were available. In four TARs it was clearly stated that formal quality assessment was not possible for the trials that were available only as abstracts/presentations, and in one TAR trial quality could not be fully assessed; however, trials were not excluded from the review on the basis of methodological quality.
Twenty-three of 63 TARs (37%) carried out a quantitative analysis of results. Of these, ten (43%) included trials available as abstracts/presentations in the review; however, only 60% (6/10) of these included data from abstracts/presentations in the data analysis of results.
Thirteen TARs evaluated rapidly evolving technologies and only three of these identified and included trial data from conference abstracts/presentations and carried out a quantitative analysis where abstract/presentation data were used. These three TARs were used as case studies.
In all three case studies, the overall quality of reporting in abstracts and presentations was generally poor. In one case study, this was more apparent in the conference abstracts compared with the online conference presentations, possibly because of limited space available in abstracts. In all case studies abstracts and presentations failed to describe the method of randomisation or allocation concealment. Overall, there was no mention of blinding in 66% (25/38) of the abstracts and in 26% (7/27) of the presentations included in case studies, and one presentation (4%) explicitly stated use of intention-to-treat analysis.
Results from one case study [drug-eluting stents (DES) review] demonstrate discrepancies in data made available in abstracts or online conference presentations. Not only are discrepancies evident between these sources, but also comparison of conference abstracts and presentations with subsequently published full-length articles demonstrates data discrepancies in reporting of results.
Sensitivity analyses based on one case study (DES review) indicated a change in significance of effect in two outcome measures when only full papers published to date were included. In terms of direction of effect, only using data from full papers published to date would not have altered the direction of any of the results when compared with those published in the original review. If conference abstracts and presentations were excluded from data available at the time of the original review, the direction of effect, and hence the conclusions of the review, would not have changed substantially, except in one of the ten results.
There are variations in policy and practice across TAR groups regarding searching for and inclusion of studies available as conference abstracts and presentations. There is also variation in the level of detail reported in TARs regarding the use of abstracts/presentations. Therefore, TAR teams should be encouraged to state explicitly their search strategies for identifying conference abstracts and presentations, their methods for assessing these for inclusion, and where appropriate how the data were used and their effect on the results.
Comprehensive searching for trials available as conference abstracts/presentations is time consuming and may be of questionable value. However, there may be a case for searching for and including abstract/presentation data if, for example, other sources of data are limited. If conference abstracts/presentations are to be included, the TAR teams need to allocate additional time for searching and managing data from these sources.
Incomplete reporting in conference abstracts and presentations limits the ability of reviewers to assess confidently the methodological quality of trials. Where conference abstracts and presentations are considered for inclusion in the review, the TAR teams should increase their efforts to obtain further study details by contacting trialists.
Where abstract/presentation data are included, reviewers should discuss the effect of including data from these sources. Any data discrepancies identified across sources in TARs should be highlighted and their impact discussed in the review. In addition, there is a need to carry out, for example, a sensitivity analysis with and without abstract/presentation data in the analysis.
Recommendations for research
There is a need for research into the development of search strategies specific to identification of studies available as conference abstracts and presentations in TARs. Such strategies may include guidance with regard to identification of relevant electronic databases and appropriate conference sites relevant to certain clinical areas.
As there are limited case studies included in this report, analyses should be repeated as more TARs accrue, or include the work of other international HTA groups (e.g. the Canadian Coordinating Office for Health Technology Assessment, the Blue Cross Blue Shield Association, the Swedish Council for Technology Assessment in Health Care and Australian HTA) to support the findings.
Dundar Y, Dodd S, Dickson R, Walley T, Haycox A, Williamson PR. Comparison of conference abstracts and presentations with full-text articles in the health technology assessments of rapidly evolving technologies. Health Technol Assess 2006;10(5). [PubMed: 16487455]
NHS R&D HTA Programme
The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly influence key decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care. HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key component of the 'National Knowledge Service' that is being developed to improve the evidence of clinical practice throughout the NHS.
The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. 'Health technologies' are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.
The HTA Programme commissions research only on topics where it has identified key gaps in the evidence needed by the NHS. Suggestions for topics are actively sought from people working in the NHS, the public, service-users groups and professional bodies such as Royal Colleges and NHS Trusts.
Research suggestions are carefully considered by panels of independent experts (including service users) whose advice results in a ranked list of recommended research priorities. The HTA Programme then commissions the research team best suited to undertake the work, in the manner most appropriate to find the relevant answers. Some projects may take only months, others need several years to answer the research questions adequately. They may involve synthesising existing evidence or conducting a trial to produce new evidence where none currently exists.
Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a National Clinical Director. TARs bring together evidence on key aspects of the use of specific technologies and usually have to be completed within a short time period.
Criteria for inclusion in the HTA monograph series
Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.
Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.
The research reported in this monograph was commissioned and funded by the HTA Programme on behalf of NICE as project number 04/05/01. The protocol was agreed in May 2004. The assessment report began editorial review in July 2005 and was accepted for publication in August 2005. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors' report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.
The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme, NICE or the Department of Health.
Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Peter Davidson, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein
Managing Editors: Sally Bailey and Sarah Llewellyn Lloyd
Y Dundar,1,* S Dodd,2 R Dickson,1 T Walley,1 A Haycox,1 and PR Williamson2.
NIHR Journals Library, Southampton (UK)
Dundar Y, Dodd S, Dickson R, et al. Comparison of conference abstracts and presentations with full-text articles in the health technology assessments of rapidly evolving technologies. 2006. In: NIHR Health Technology Assessment programme: Executive Summaries. Southampton (UK): NIHR Journals Library; 2003-.