U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

O’Haire C, McPheeters M, Nakamoto E, et al. Engaging Stakeholders To Identify and Prioritize Future Research Needs [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Jun. (Methods Future Research Needs Reports, No. 4.)

Cover of Engaging Stakeholders To Identify and Prioritize Future Research Needs

Engaging Stakeholders To Identify and Prioritize Future Research Needs [Internet].

Show details

Appendix FDescriptions of Stakeholder Engagement Methods


Includes in-person, phone and group interviews based on a series of questions related to a topic of interest to generate ideas, elicit feedback and/or answer a set of pre-determined questions.

One-on-one interviews (including telephone)

Stakeholders are engaged individually, either formally or informally. In formal interviews, a one on one interview guide (e.g. semi-structured interviews) may be used for evaluation. Informal procedures are flexible and sometimes include ad-hoc conversations that can allow you to gather additional information from stakeholders.

Semi-structured interviews

Generally used to engage individuals or a very small group (two or three individuals), semi-structured interviews are conducted with an open framework which allow for focused, conversational, two-way communication. Structured questions are designed and phrased ahead of time and are typically asked of during all interviews. Questions are also created during the interview, allowing both the interviewer and the person being interviewed the flexibility to probe for details or discuss issues.

Focus/working groups

A planned discussion in a small (4 to 12 members) group of stakeholders facilitated by a skilled moderator. It is designed to obtain information about preferences and opinions in a relaxed, non-threatening environment. The topic is introduced and, in the ensuing discussion, group members influence each other by responding to ideas and comments. The moderator may use some predetermined questions as prompts to encourage discussion or to return the conversation to the intended focus of the discussion.


A questionnaire is a means of eliciting the opinions, beliefs and attitudes of some sample of individuals. As a data collection instrument, it is structured or unstructured. A questionnaire is usually concise with a preplanned set of questions designed to yield specific information to meet a particular need for research information.

Citizens’ juries

Used to elicit views of members of the public about a variety of health and other issues. Based on the principles of “deliberative democracy” and active citizenship, they aim to promote decision-making based on process of ‘careful consideration,’ debate, and respect for different viewpoints. They bring together diverse members of the public as jurors who are given information relevant to the issue under debate by “expert witnesses,” (innovators, patients, health care policy-makers, and clinicians) and the discussion has a facilitator or moderator present to guide the process. The session can include small and large group priority-setting exercises based on actual examples of technologies under consideration for assessment by local and national bodies. The end result is often a written report authored by the jurors, which can also be in the form of a questionnaire with juror responses.

Town meetings

Individuals residing in a specific geographic area are invited to a public meeting to discuss issues relevant to their community. Often, this meeting is announced by the local media and attended by residents as well as other individuals including state and local officials, health care providers, researchers, manufacturers, and topic experts. In general, everyone is offered the opportunity to speak in a relaxed environment, the meetings are often loosely organized and used to identify and a broad list of research topics/interests. Voting to prioritize research items may also occur.


Meetings or conferences for discussion of a topic, especially one in which the participants both form an audience and make presentations.

Nominal group technique

Structured problem-solving or ideas-generating activity in which individuals’ ideas are gathered and combined in a face-to-face, nonthreatening group environment. The process is intended to promote creative participation in group problem -solving. Each member of the group is invited to express their opinions that are used to generate a list of priorities. Members may be asked to vote or rank priorities from the list either formally or informally. The voting process may occur multiple times. Nominal group technique is designed to promote the free exchange of opinions and the generation of a list of priorities in a structured and nonhierarchical discussion forum (maximizes creative participation and ensures balanced output while utilizing each participant’s experience and expertise to reach consensus on complex topics). The purpose is to provide structure to a group discussion when the group is facing the challenge of reaching agreement on complex topics. Examples of variations in use of the technique include:

Example 1

  1. Introduction to technique
  2. Individual generation of ideas—each participant writes their ideas
  3. Sharing of ideas—every member is asked to present their ideas and brief explanation
  4. Voting/ranking

Example 2

  1. Survey of members to get participants ideas
  2. Removal of duplicates
  3. Structured/facilitated groups discussion all members of nominal group for each participants list with explanations, resolve ambiguities
  4. Voting/analysis

Delphi technique

The Delphi technique uses a series of consecutive questionnaires to determine the perceptions of a group of individuals. The Delphi method allows respondents to communicate their opinions anonymously. Each questionnaire is considered a round. The method is often used to prioritize research/topics. For example, Hauck1 and colleagues conducted the following study to identify research priorities of clinic staff working with the community:

  1. Round 1: This questionnaire was used to create a list of five important questions relating to future research in care for children in this community. Content analysis was used to analyze and summarize the responses and develop the second questionnaire. All issues were discussed, assigned a general category and then described as a research topic.
  2. Round 2: The clinical staff was asked to prioritize the research ideas/suggestions using a 7-point Likert- type response format, with one indicating a low priority and seven, the highest priority.
  3. Round 3: The top 10 research topics were identified. Both clinicians and clients were asked to rank the topics identified.

Modified Delphi technique

The modified Delphi technique is similar to the Delphi technique in terms of both intent of engagement (i.e., to predict future events and to arrive at consensus) and the procedure by which research priorities are determined (i.e., a series of rounds with selected experts). The major difference between the two is that Round 1 of the Modified Delphi technique already includes preselected research topics; in contrast, the Delphi technique uses the first round of questionnaires to elicit research topics. The pre-selected research topics used in Round 1 of the Modified Delphi technique may be drawn from various sources including reviews of the literature, and interviews with key informants.2


Value of information (VOI)/expected value of information (EVOI)

VOI/EVOI provides a methodological framework that explicitly considers the uncertainty surrounding the decision of a health care system to adopt a health technology. Specifically, using existing evidence, this method focuses on the likelihood of making a wrong decision if the technology is adopted. The value of additional research is based on the extent to which further information will reduce this decision uncertainty. VOI/EVOI includes the following:

  1. Estimating the effective population that may benefit from additional evidence, including time horizons for different technologies and incorporating this uncertainty in the estimates of value of information
  2. Estimating the value of information for correlated parameters
  3. Estimating the over value of information based on estimates of the value of information for patient subgroups
  4. Presenting the value of information and the value of full implementation of guidance on use within the same framework of analysis


Scoping study

Literature review of published and grey literature, followed by focus group and interview consultations: scoping studies aim to map key concepts underpinning a research area and the main sources and types of evidence available. They include a literature review and consultation phase that may be used to (a) examine the extent, range and nature of research activity, (b) determine the value of undertaking a full systematic review, (c) summarize and disseminate research findings, or (d) identify research gaps in the existing literature. For example, a scoping study may start with a literature review followed by a series of focus groups and key informant interviews to prioritize research.

Concept mapping

This is an approach particularly designed for facilitating consensus in the understanding and organization for various concepts. Mapping is based on multivariable statistical analyses in which statements produced during a brainstorming session are grouped in weighted clusters. Cluster mapping includes the following five stages: focusing on the question, brainstorming session, rating and sorting statements, data analysis and interpretation of maps.

References: Appendix F

Hauck Y, Kelly RG, Fenwick J. Research priorities for parenting and child health: a Delphi study. J Adv Nurs. 2007;59(2):129–139. [PubMed: 17584304]
Custer R, Scarcella J, Stewart B. The Modified Delphi Technique—A Rotational Modification. Journal of Vocational and Technical Education. 1999;15(2)


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (571K)

Related information

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...