NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

O’Haire C, McPheeters M, Nakamoto E, et al. Engaging Stakeholders To Identify and Prioritize Future Research Needs [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Jun. (Methods Future Research Needs Reports, No. 4.)

Cover of Engaging Stakeholders To Identify and Prioritize Future Research Needs

Engaging Stakeholders To Identify and Prioritize Future Research Needs [Internet].

Show details

Executive Summary


National attention has recently focused on engaging stakeholders to create a more patient-centered research agenda to improve health care decisionmaking in the United States.1,2 Specifically, patients and physicians seek to apply current research to their treatment decisions, and they require high-quality evidence that directly addresses their information needs.3 In response to this need, there is growing interest among the research community to engage the public in the research development process. Consumer involvement has been shown to positively impact research and its applicability to the public,4 and the Agency for Healthcare Research and Quality (AHRQ) has emphasized the importance of considering the patient perspective during all aspects of comprehensive comparative effectiveness reviews (CERs).

Engaging stakeholders to identify and prioritize future research for CERs is a new process, and the methods to delineate future research needs are not well developed. Evidence-based Practice Centers (EPCs) have engaged individuals and organizations outside of the EPC program in their reviews, especially in the early phases for input on topic development and topic refinement.5 Recently, some EPCs have initiated new activities engaging stakeholders to identify and prioritize future research needs following completion of CERs.

To develop guidance for a systematic approach to identifying future research needs for the EPCs, AHRQ requested that the Oregon and Vanderbilt EPCs conduct this project to better understand the various methods for engaging stakeholders in defining and prioritizing research needs. In response to this request, the Oregon and Vanderbilt EPCs conducted a methods project to identify and describe methods used in engaging stakeholders to prioritize research needs, evaluate the tradeoffs of different engagement methods, describe methods for analyzing stakeholder input, and to describe the experiences of EPCs who engaged stakeholders for their future research needs pilot studies. This is one of a series of methods projects to inform and systematize AHRQ’s guidance related to identification and prioritization of future research needs.


This project consisted of three phases. Phase 1 involved a scan of the published literature for methods of stakeholder engagement to set research priorities. Phase 2 involved key informant (KI) interviews of individuals and organizations (i.e., researchers and policymakers) who engaged a broad range of stakeholders to identify research topics and prioritize research needs. Phase 3 involved interviews of EPCs that engaged stakeholders for their future research pilot studies in 2010.

Phase 1: Literature Summary

We searched PubMed/MEDLINE (OVID), PsychINFO (OVID), CINAHL (EBSCO), Cochrane Consumer Group, LocatorPlus (NLM catalog), and Sociological Abstracts (CSA) from inception until August or September 2010. We reviewed relevant literature to identify methods and processes used to engage stakeholders in identifying and prioritizing research.

Phase 2: Key Informant Interviews

Individuals known to have expertise engaging a broad range of stakeholders (i.e., consumers, clinicians, and policymakers) were invited to be KIs. We also asked them to recommend additional individuals and/or organizations for this project. Interviews were conducted between June 16 and July 9, 2010. Two investigators interviewed KIs about their experiences engaging stakeholders, the types of stakeholders engaged, the methods used to engage them, the processes used to set research priorities, and the impact of stakeholder engagement. After the interviews were completed, investigators analyzed interview transcripts to identify common themes.

Phase 3: EPC Interviews

We evaluated EPC experiences engaging of stakeholders to identify research gaps and prioritize future research needs for the 2010 pilot projects in two ways: (1) we reviewed EPC proposals to assess the methods proposed by EPCs to engage stakeholders in future research needs development and (2) we conducted semistructured interviews with EPC directors and staff at the conclusion of the pilots to identify lessons learned about the stakeholder engagement process. We received copies of each EPC’s protocol for their Future Research Needs pilot project and extracted the information specific to methods for stakeholder engagement to prioritize future research. We then sent letters of inquiry to all EPC directors asking them to participate in brief, semistructured interviews to discuss their experience of engaging stakeholders in future research development. A list of potential questions was developed to guide the conversation.


Phase 1: Literature Summary

We identified 56 studies that provided information about methods for engaging stakeholders specifically for the purpose of prioritizing future research. An important first step for evaluating stakeholder engagement is to have a common terminology; however, the literature lacked consistent terminology, definitions, or structured processes for stakeholder engagement. For the purposes of this report, stakeholders are individuals or organizations who have an interest, personal or professional, in the topic. We developed operational definitions of the major categories of stakeholders based on definitions, terms, and stakeholder categories reported by researchers.

Important considerations for stakeholder engagement included the intended purpose for engagement, the explicit identification of stakeholder groups, the distribution of the constituency (i.e., equal participation by group compared with other), the complexity of the topic, and the nature of the feedback that investigators want to see. The literature informed common practices around identification of stakeholders, recruitment of stakeholder representatives, and methods to prioritize and inform future research. Snowball sampling, which involves identifying stakeholders through referrals from others, was one of the most common methods used to identify stakeholders. However, because individuals may be more likely to recommend people who are similar to themselves, it is possible that bias could be introduced. While many studies did not provide details regarding the effectiveness of recruitment strategies, postal and electronic mailings were the most commonly reported contact methods regardless of stakeholder group. Studies frequently used a mixed-methods approach for research prioritization, combining in-person venues with quantitative prioritization processes. Studies used in-person methods to clarify stakeholder questions and to allow for a deeper understanding of stakeholders’ viewpoints, whereas they used quantitative methods such as voting, Delphi, questionnaire, or other explicit rating processes to arrive at final priorities.

Phase 2: Key Informant Interviews

To determine the methods used to engage stakeholders to prioritize future research and to identify important issues in engaging stakeholders and analyzing stakeholder input, we conducted a series of 13 interviews with a broad range of KIs from academic institutions, practice-based research networks, nonprofit organizations, and other organizations devoted to health care issues on local and national levels in the United States, Canada, and Europe.

Defining Stakeholders

Even among individuals with extensive experience engaging stakeholders, stakeholder terminology was unclear or inconsistently applied. The interviews emphasized the lack of consistency and clarity in this process.


KIs used several methods to recruit stakeholders that differed according to topic, organization, and the stakeholder groups engaged. Often, they recruited multiple groups of stakeholders for engagement. KIs most commonly used existing contacts and telephone or e-mail communication to identify and recruit stakeholders. Additional methods used to recruit consumers included use of the media, and Internet/Web. In addition, KIs noted that followup was essential to recruitment and stakeholder engagement.


KIs engaged stakeholders using a variety of methods, including e-mail, conference calls or Webinars, focus groups, citizens’ juries, workshops conferences, and the modified Delphi technique. They often used multiple methods (mixed methods) to engage the same stakeholder groups. The two most common methods that were used by KIs to engage stakeholders were one-to-one meetings (in person or by phone) and working group/committee/council meetings. According to KIs, the advantages of in-person meetings included: (1) ensuring a time of focused stakeholder attention to the subject, (2) fostering relationships among stakeholders, and (3) promoting a better understanding of the each other’s needs and priorities through discussion.

KIs identified engagement strategies that were particularly effective for different stakeholder groups. For policymakers and research funders, KIs preferred to either engage them by phone, one-to-one meetings, or in small meetings. KIs differed in their views about the appropriate context for engagement of consumers and frontline clinicians. For clinicians, KIs used one-to one meetings, e-mail, conference calls, committee meetings, and conferences. For consumers, KIs used methods similar to that of other stakeholder groups; however, the motivation and needs of consumers were seen to be quite different. KIs reported that it took a substantial amount of time to engage this group, and warned that often researchers underestimate the amount of time, patience, and attention that is required to engage consumers.


Specific Considerations for Engagement of Stakeholder Groups

There was considerable variation in the value KIs placed on involving consumers in their research prioritization processes. Some KIs advocated strongly for their inclusion and felt that input from these stakeholders was critical for setting a relevant research agenda. They noted that consumers were not only capable of understanding the research process, they linked researchers to the community and made important contributions during topic identification, study design, and manuscript preparation. In contrast, other KIs excluded consumers or dismissed their contributions because they lacked research experience, scientific knowledge, or technical expertise to meaningfully contribute to health care discussions or setting a research agenda.

Policymakers. KIs preferred to policymakers by phone, with one-to-one interviews, or in small meetings. Two KIs stressed the importance of having “off-the-record” discussions with policymakers so that they felt comfortable asking questions about research without representing a specific constituency or agency. Several KIs mentioned that it is important to understand the pressing time demands that these policymakers encounter on a daily basis.

Researchers. Engaging researchers as stakeholders can be challenging because though they are familiar with this process, they may not be accustomed to this type of collaboration.


Prioritization ranged from little or no formal process to a structured, formal prioritization process that consisted of multiple rounds of voting/ranking, with at least one vote occurring in person. An initial voting/ranking process was often used by KIs to limit the number of research topics discussed and prioritized at a subsequent in-person meeting. KIs reported that stakeholders used multiple criteria to prioritize research including: funding availability, current resources available, the burden of disease, community resonance, policy “shelf life,” and in the case of systematic reviews, having adequate primary research to conduct a systematic review.

Considerations in Working with Stakeholders

Regardless of the method used to engage stakeholders, KIs emphasized that leveraging contacts, establishing long-term partnerships, and building trust and credibility were critical. Bringing diverse stakeholders together may require as much attention to group dynamics as to the presentation of research topics and prioritization criteria. Several KIs reported that stakeholders’ busy schedules and competing demands for their time are common and having flexible engagement methods can help to ensure that critical stakeholders are engaged. In addition, KIs commented that individuals frequently encounter both logistic and financial barriers to providing input as stakeholders. KIs offered the following advice to foster stakeholder relationships: engage stakeholders early, find common ground, and foster partnership, understanding, and respect.

Phase 3: EPC Interviews

Review of Protocols

We reviewed the EPC future research pilot protocols and found that EPCs proposed a range of approaches to engaging stakeholders. Among these approaches included two who intended to engage the support of the Effective Health Care Scientific Resource Center and two who intended to work with the Eisenberg Center in engaging stakeholders. Most centers intended to use group calls, with two EPCs using a formal Delphi process.

Semistructured Discussions with EPC Directors and Staff

We interviewed representatives from the eight EPCs that conducted the future research pilots about their experiences engaging stakeholders during those projects. Although these pilot projects were intended to develop and compare a range of approaches to developing future research documents, the EPCs had remarkably similar experiences and methods for engaging stakeholders. A defining feature associated with reporting fewer challenges in engaging stakeholders was the currency of the report itself. Reports published more recently and/or by the EPC doing the future research had a natural constituency from which to draw potential stakeholder members. These teams generally drew upon the investigators and the Technical Expert Panel (TEP) members from the original evidence report to participate in or suggest other individuals for the stakeholder group.

Stakeholder Constituencies

EPCs included between six and 30 stakeholders for their research gaps pilot projects. Target groups for stakeholder engagement included primary researchers, methodologists, patient/consumer advocates, policymakers, funders, payers, industry (for devices), and patients themselves. Of these, all EPCs targeted active researchers, with representation from the other groups varying. One EPC had three distinct stakeholder groups: authors of the report, a group of local experts (considered internal), and a group of stakeholders outside of their institution (external). Several EPCs identified particular barriers to engaging funders. EPCs reported that the need for high levels of clearance/approval for individuals to participate was ultimately time prohibitive. All EPCs discussed the question of whether and how to engage patients or consumers in the process of identifying research gaps and future research needs.

Methods for Identifying Stakeholders

All EPCs used a combination of approaches for identifying potential stakeholders, most of which began with individuals who were known to the EPC investigators, either because they were involved in the report for which future research recommendations were being developed or through professional relationships. EPCs did not select stakeholders at random, but based selections primarily on availability and convenience, knowledge, and personal contacts. Their most common approach, along with e-mail or Web-based prioritization, was group conference calls. One EPC presented a Webinar to bring stakeholders up to speed. EPCs varied in the numbers and types of contacts that occurred with stakeholders, including letters, one-to-one orientation calls, group calls, survey tools, e-mails, and Webinars.

Challenges Identified

EPCs routinely identified three main challenges, including timing, restriction on number of stakeholders, and limited availability of Federal stakeholders. The requirement that the project take place very quickly and during the summer made it difficult to coordinate schedules and likely resulted in an inability to engage some stakeholders, particularly academic researchers. Engaging more than nine nonfederal citizens with a structured survey on behalf of the United States Government would require a lengthy clearance process through the Office and Management and Budget. The time needed to obtain this clearance was prohibitive to timely completion of this project, so the number of stakeholders engaged was kept to nine or less. This limited number was generally perceived as too few to capture the breadth of views of researchers and patients necessary for identifying research gaps. However, the EPCs reported not knowing what number would be optimal. Finally, the difficulty in engaging Federal stakeholders, primarily at the National Institutes of Health (NIH), was largely due to the need for permissions that took too long and resulted in several EPCs not including NIH representatives.


From the literature and the KI and EPC interviews, we identified the following considerations regarding stakeholder engagement to prioritize research:

  1. Use of consistent terminology and definitions is important throughout the stakeholder engagement process, from planning to implementation to evaluation.
  2. The intent of stakeholder engagement is important in determining the methods of engagement. In-person methods are helpful for brainstorming, identifying topics, clarifying issues, and eliciting a deeper understanding, while quantitative methods such as voting and Delphi are best applied to prioritize research. A flexible approach using mixed methods may be required to engage multiple stakeholder groups. We summarized the tradeoffs for engagement methods in this report to assist programs in their selection of feasible and effective methods for their project.
  3. Engaging stakeholders early and maintaining relationships are important to building trust and credibility.
  4. Assurance that representative samples of stakeholders are engaged is critical to the credibility and legitimacy of the stakeholder engagement process and results.
  5. When multiple stakeholder groups are engaged, special consideration should be given to how to weight input from each group. The issue of weighting should be addressed at the outset during recruitment because it will affect the composition of stakeholder recruitment lists, and it should also be considered when synthesizing and analyzing stakeholder input.

Additionally, we used results from all three phases to identify the following recommendations for stakeholder engagement to prioritize future research needs, including a proposed reporting checklist for EPC reports (Table A):

Table A. Recommendations for stakeholder engagement to identify and prioritize future research.

Table A

Recommendations for stakeholder engagement to identify and prioritize future research.


Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. Washington, DC: The National Academies Press; 2009.
Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Washington, DC: U.S. Department of Health and Human Services; 2009.
Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290(12):1624–1632. [PubMed: 14506122]
Staley K. Exploring Impact: Public involvement in NHS, public health and social care research. Eastleigh, U.K.: INVOLVE; 2009.
Hickam D, Gordon C, Curtis P, et al. AHRQ Effective Health Care Program Evidence-based Practice Centers Assessment. Rockville, MD: Agency for Health Care Research and Quality; 2009.


Related information

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...