Evidence review group approaches to the critical appraisal of manufacturer submissions for the NICE STA process: a mapping study and thematic analysis

Kaltenthaler E, Boland A, Carroll C, et al.

Publication Details


The National Institute for Health and Clinical Excellence (NICE) single technology appraisal (STA) process was set up as a rapid way to appraise new technologies for use within the NHS in England and Wales and has been in place since 2005. Manufacturers present clinical effectiveness and cost-effectiveness evidence in their submissions to NICE. Evidence Review Groups (ERGs) are given the task of critically appraising manufacturers' submissions (MSs) as part of this process. However, little guidance has been provided by NICE on how to do this.


The aims of this study were to review the methods currently used by ERG teams to critically appraise MSs within the NICE STA process and to provide recommendations on approaches that could be considered in the future. An additional aim of the study was to assess what has happened in the STA process so far, particularly in relation to timelines and decisions.

There were five primary objectives:

  • to provide a map of the STA process to date
  • to identify current approaches to the critical appraisal of MSs by ERGs
  • to identify recurring themes in clarification letters sent to manufacturers
  • to provide recommendations for possible alternative approaches to be used in the critical appraisal process
  • to revise the current ERG report template.


In order to map the STA process to date, data for each STA were collected from the NICE website. A mapping spreadsheet was developed to collect data on 22 predefined variables related to timings and outcomes. Simple descriptive statistics were used to analyse the data.

Thirty completed STAs were identified for thematic analysis of ERG reports and clarification letters. In the case of the ERG reports, data on key elements of the MSs, the processes undertaken by ERGs and the strengths and weaknesses of the MSs were extracted. A framework of a priori themes was developed. Data were extracted, coded and analysed according to a framework approach.

Only 21 of the 30 STAs had clarification letters available. These were examined and data extracted using a set of codes to cover report quality, systematic review methods and clinical and economic issues.

The current ERG report template was modified and sent to the current ERG teams for comment. All comments were considered and formed the basis for further revisions to the template.


Ninety-five STAs were included in the mapping exercise. These included all STAs identified by NICE up to, and including, March 2009. Nearly all necessary information for this mapping exercise was available from the NICE website. STAs did not appear, on the whole, to be completed within the suggested timelines between final scope and final appraisal determination (FAD), although it was difficult to determine the exact timeline recommendations within the NICE process guide. Incremental cost-effectiveness ratios (ICERs) reported by manufacturers were consistently lower than those estimated by the ERGs. An appeal was undertaken in 22% of the STAs against a 'no' decision and 32% of appraisals were either suspended or terminated. Suggested changes to the NICE website include the need for consistency in placing documents, a guide for the public regarding where specific documentation is kept and the use of unique identifiers for each STA topic.

The ERG reports highlighted the strengths and weaknesses of MSs to the STA process. Thematic analysis of these data offered a means of clarifying and describing these aspects of the submissions. This analysis generated the five themes of 'process', 'reporting', 'satisfaction of objectives', 'reliability and validity of findings' and 'content'. 'Process' concerned how the various relevant methodologies had been applied in the performance of the review, analyses or modelling; 'reporting' how well these processes had been described or justified in the submissions; 'satisfaction of objectives' how far the submission complied with or addressed the scope or decision problem; and 'reliability and validity' how far the findings of the submission were affected by uncertainty or bias. These themes were also inter-related. The adequacy of reporting in the submission influenced the assessment of the processes being conducted, and both the content and the conduct of the review or modelling directly affected the reliability and validity of a submission's findings. In the same way, a submission's success or failure to address the objectives set by the scope or decision problem affected the external validity of the submission. The STA process may be improved if manufacturers address these issues more carefully in their submissions.

Points from clarification letters were analysed and presented in four main categories: report quality, systematic review methods, clinical data analysis and economic data analysis. The majority of clarification points related to the economic data analysis and covered issues such as inconsistencies between the clinical and economic sections of the submission, queries regarding sources of data and their use in the economic analysis, queries about modelling decisions, data queries and requests for additional analyses.

The ERG template was modified, based on revisions suggested by the seven ERG teams. The new template will be trialled for 6 months.


This report presents the first independent mapping exercise of the NICE STA process to date. Nearly all data were obtained from the NICE website; therefore, any errors in the data on the website will be reflected in the mapping analysis presented in this report. Missing data for the mapping exercise do limit the generalisability of the findings. As many of the STAs included in the mapping exercise were still ongoing at the time of data extraction (August 2009) and even 'completed' STAs have been changed since then, this report presents a 'snapshot' of data available at that time.

The thematic analysis of the ERG reports used validated methods and multiple reviewers to check and verify analyses. Only the first 30 ERG reports were included in the thematic framework analysis. Critical appraisal methods used by the ERG teams may have developed over time and may not be accounted for in these analyses. There may be some misinterpretations in these analyses as only documentary analysis was used to collect data. As ERG teams gain more experience in developing their reports and build up expertise, they may use different critical appraisal methods. Analysis is limited to what is reported in the ERG reports.

The analysis of the clarification letters provides an overview of the commonalities of the queries put forward in the letters and, as such, provides direction for the ERGs, NICE and the manufacturers. Scoping workshops are now part of the STA process and may have a bearing on how manufacturers produce their submissions. Clarification letters are now developed using a more structured approach. More recently, there have been changes made to the MS template. These changes may impact on the issues of concern in current clarification letters, which will not be reflected by this analysis.


Proposed changes to the NICE website would enable easier access to relevant information and ensure that the process for accessing information is even more transparent. Guidance suggested for manufacturers will help to ensure that more appropriate submissions are received in the future, while recommendations provided for ERG teams will help to guide teams and train new members to ensure reporting methods are transparent. Finally, changes to the ERG report template will ensure that the reports have less repetition and that it is easier to find relevant information.

Suggested research priorities include the need for an investigation into what Appraisal Committee members deem to be the most relevant and appropriate information to be included in ERG reports. A more in-depth analysis of approaches used by ERG teams to critically appraise MSs is also needed. Finally, research is needed to identify the most appropriate methods for reviewing utility data and other model parameters, as well as review methods to inform indirect comparisons.


Funding for this study was provided by the National Institute for Health Research Health Technology Assessment programme.


  • Kaltenthaler E, Boland A, Carroll C, Dickson R, Fitzgerald P, Papaioannou D. Evidence review group approaches to the critical appraisal of manufacturer submissions for the NICE STA process: a mapping study and thematic analysis. Health Technol Assess 2011;15(22). [PubMed: 21561569]

NIHR Health Technology Assessment programme

The Health Technology Assessment (HTA) programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care.

The research findings from the HTA programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’.

The HTA programme is needs led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of projects.

First is the commissioned route. Suggestions for research are actively sought from people working in the NHS, from the public and consumer groups and from professional bodies such as royal colleges and NHS trusts. These suggestions are carefully prioritised by panels of independent experts (including NHS service users). The HTA programme then commissions the research by competitive tender.

Second, the HTA programme provides grants for clinical trials for researchers who identify research questions. These are assessed for importance to patients and the NHS, and scientific rigour.

Third, through its Technology Assessment Report (TAR) call-off contract, the HTA programme commissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring together evidence on the value of specific technologies.

Some HTA research projects, including TARs, may take only months, others need several years. They can cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence, undertaking a trial, or other research collecting new data to answer a research problem.

The final reports from HTA projects are peer reviewed by a number of independent expert referees before publication in the widely read journal series Health Technology Assessment.

Criteria for inclusion in the HTA journal series

Reports are published in the HTA journal series if (1) they have resulted from work for the HTA programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this issue of the journal was commissioned by the HTA programme as project number 08/228/01. The contractual start date was in April 2009. The draft report began editorial review in June 2010 and was accepted for publication in November 2010. As the funder, by devising a commissioning brief, the HTA programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley CBE

Series Editors: Dr Martin Ashton-Key, Professor Aileen Clarke, Professor Chris Hyde, Dr Tom Marshall, Professor John Powell, Dr Rob Riemsma and Professor Ken Stein

Associate Editor: Dr Peter Davidson