Stakeholder Panel

We considered seven stakeholder categories to build a panel representing the full range of stakeholders who may use research evidence in health care and public health decisionmaking.

1. Patients and the public: This group represents current and potential consumers of patient-centered health care and population-focused public health. This group also includes caregivers, family members, and patient advocacy organizations, all of whom address the interests of consumers.
2. Providers: This group includes individuals (e.g., nurses, physicians, mental health counselors and other providers of care and support services) and organizations (e.g., hospitals, clinics, community health centers, community-based organizations, pharmacies, EMS agencies, skilled nursing facilities, schools) that provide care to patients and populations.
3. Purchasers: This group includes employers, the self-insured, government, and other entities responsible for underwriting the costs of healthcare.
4. Payers: This group represents insurers, Medicare and Medicaid, individuals with deductibles, and others responsible for reimbursement for interventions and episodes of care.
5. Policymakers: This group includes organizations such as the White House, the U.S. Department of Health and Human Services, Congress, States, professional associations, and intermediary groups that collate and distribute information to policymakers.
6. Principal investigators, researchers, and research funders.
7. Product makers: This group represents drug and device manufacturers.

These categories are not necessarily mutually exclusive. In concept, each individual represents at least one key type of health care and health care research decisionmaker. Any single person or entity may have several roles and may be responsible for different types of decisions. For example, some health care purchasers are also payers, and conversely, some payers also provide care. Patients and their advocates may be providers or employers with policymaking responsibilities, and so on. In addition, each of these seven stakeholder types may be focused on applying CER at the patient level or at the population level. Patient-level decisions include questions pertaining to what treatment would be best for a given patient at a particular time. Population-level decisions include questions pertaining to what services, resources, policies or other alternatives are best for groups of patients and entire communities that are connected by practice setting, geography, clinical domain or other cluster. To be patient centered, decisions made about groups of patients must recognize both the diversity of needs across populations and the heterogeneity of individuals within populations.

Identification and Invitation of Individual Stakeholders

We compiled a list of potential stakeholders from three sources: (1) those individuals who were or had been invited to be a member of the sleep apnea report’s Key Informant panel or Technical Expert Panel, or a peer reviewer; (2) a previously compiled list of stakeholders assembled for a 2010 Stakeholder Forum on Comparative Effectiveness Research for the National Institutes of Health Clinical and Translational Science Awards program; and (3) a list of stakeholders compiled in 2010 for the AHRQ Effective Health Care Program. We also solicited recommendations for stakeholders from selected government agencies, professional organizations, and other representative bodies. We selected people who would potentially fall within one or more of the stakeholder categories for either obstructive sleep apnea (OSA) diagnosis or treatment. With the assistance of our local domain expert, we initially selected the most promising individuals based on their perceived interest in the topic, their level of previous participation in discussions on the topic, and their fit into the stakeholder categories.

Individuals who met criteria to be a stakeholder were contacted directly by email with a brief description of the project, an invitation to be a stakeholder for one or both stakeholder panels (diagnosis and treatment), and the Executive Summary of our sleep apnea CER. Potential stakeholders were also telephoned, as necessary, to solicit their interest. Other individuals on our list were contacted specifically requesting suggestions for appropriate stakeholders. These people, in turn, were contacted by email. In addition, through Internet searches and focused searches in MEDLINE, we found other potential stakeholders or individuals who could suggest stakeholders. In particular, we contacted senior level administrators at various governmental, nongovernmental, and professional organizations, including but not limited to, the National Association of Community Health Centers, the Veterans Administration, Blue Cross Blue Shield, Centers for Medicare and Medicaid Services, Academy Health, the National Heart, Lung, and Blood Institute, the Office of Minority Health, and the Centers for Disease Control and Prevention. Patients were solicited from our Key Informants list, personal contacts, and an Internet search of sleep apnea advocacy, support, or discussion groups. Selected potential stakeholders were invited to participate in both Future Research Needs Prioritization projects on OSA (diagnosis and treatment), which were run concurrently. All stakeholders completed a standard disclosure of interest form.

We also contacted manufacturers from the list of companies that were sent Scientific Information Packets for the original sleep apnea report. These companies were asked to provide potential FRN topics with rationales, but were not invited to be stakeholders.

Introduction of Process to the Stakeholder Panel

Along with an invitation letter, we distributed the executive summary and the Future Research section of the original CER¹ to the invited stakeholders. The original Key Questions, summary of evidence table, and the Implications for Future Research sections in the executive summary were highlighted. The purpose of the FRN project and expectations for the input from the stakeholders were outlined clearly in the invitation letter.

We scheduled teleconferences to allow general introductions and for the EPC to explain the purpose and process of the FRN topic development process, after compiling the stakeholder panel.

Iterative Process To Identify Future Research Needs Topics

The EPC reviewed the CER Future Research section and, from this, developed a series of FRN topics. We wrote a brief rationale statement for each. These formed the initial list of FRN topics.

After stakeholders submitted their disclosure of interest statements, we invited them to submit FRN topics to the EPC. For each FRN topic, stakeholders were asked to provide a brief rationale (maximum 250 words) considering the four dimensions of need as listed below under Approach to Prioritization. We reviewed submitted FRN topics, and planned to categorize each topic into one of three categories: “definitely relevant to Key Questions,” “not relevant to Key Questions,” or “unclear.” We planned to combine duplicate or similar FRN topics together into one topic. For FRN topics categorized as “unclear,” we asked the stakeholders to provide additional information or clarifications. All topics were distributed to stakeholders. Throughout the stakeholder panel discussion period, stakeholders were invited to submit new FRN topics.

Use of Microsoft^® SharePoint

Two separate Microsoft SharePoint Web sites were created to: (1) host the FRN discussion; (2) submit additional topics for discussion; and (3) nominate topics for future research. SharePoint resources were provided through the Tufts Clinical and Translational Science Institute, of which the Tufts EPC is a member. One Web site was created for the diagnosis topics discussed in this document and a separate Web site for the parallel treatment discussion. SharePoint was chosen as it offered stakeholders the flexibility of an asynchronous online discussion forum. Additionally, the secure Web site provided Tufts staff the most control, in terms of site content, structure and functionality, when compared with other publicly available discussion platforms. The secure, password-protected Web site was housed behind the Tufts firewall and accessible to stakeholders via invitation only. The SharePoint Web site also served as a platform for Tufts EPC staff to post project announcements, a welcome video, and important reference documents. These documents included instructions to stakeholders on how to navigate and use the Web site, an FRN project overview, the Executive Summary and Key Questions of the sleep apnea CER, lists of participating stakeholders, initial FRN discussion topics, common abbreviations and acronyms, announcements, and reference documents. The documents on the Web site were also emailed to all stakeholders. In addition to the EPC staff and the stakeholders, the AHRQ Task Order Officer was given access to the Web site.

The FRN topic discussion boards were the primary feature of the SharePoint Web site. Discussion boards were prominently placed front and center on the Web site’s homepage (Figure 2). Links to individual discussion boards were also strategically placed on the Web site’s main navigation toolbar for direct access from any page on the Web site.

Figure 2

SharePoint homepage: Diagnosis.

Each FRN topic was uploaded as a separate “discussion board.” These were structured to allow stakeholders to comment on both the original FRN topic as well as other stakeholder comments (Figure 3). For their discussions, stakeholders were asked to consider four dimensions of need related to the proposed topic (see Appendix A and the Approach to Prioritization section, below). Stakeholder participation was monitored and discussions were moderated daily by Tufts EPC staff to ensure appropriateness and relevance of all comments. The discussion boards were initially scheduled to be open for a 2-week period. During the open period, stakeholders were regularly encouraged (two to four times per week) to contribute to the discussion boards as well as submit additional topics for discussion. Stakeholders were encouraged to go through all topic areas, and provide comments and feedback on all topics on the Web site.

Figure 3

Sample discussion thread: Diagnosis.

We planned to use the SharePoint Web site to nominate FRN topics using a process similar to topic submission described above. This strategy was revised due to limited stakeholder engagement with the SharePoint Web site (as described in the Results section).

At the conclusion of the project, all online discussions and email communications were archived for transparency purposes.

Approach to Prioritization

The stakeholders were asked to consider four dimensions of need related to the proposed topic. These four dimensions come from the AHRQ Effective Health Care Program Selection Criteria (Appendix A). These dimensions and the Effective Health Care Program guidance on them were described in detail in the lead up to topic submission, discussion and selection of FRN topics.

• Importance
• Desirability of Research/Duplication
• Feasibility
• Potential Impact

EPC program staff evaluated a fifth dimension, Appropriateness, after submission of initial FRN topics.

After the close of the online discussion, the stakeholders were asked to identify up to 10 FRN topics that were of highest priority and that met the Effective Health Care Program selection criteria. The original plan was to conduct this nomination step on the Web site, but as described in the Results section, nomination was conducted by email and individual phone calls. After nomination, the EPC grouped similar topics into overarching topics and edited the names of the title for clarity and consistency. Based on the stakeholder nomination, the EPC categorized the overarching topics into four groups:

• High-priority FRN topics: Clearly of interest to stakeholders (based on high levels of nominations). A consensus of stakeholders expressed that these topics are of high priority. We aimed for about five topics, but we used natural breaks in the rankings of the topics, rather than strictly defining the numbers of topics in this category. For these topics, the background, stakeholder discussion, and study design considerations are fully elaborated in the Results section.
• Second-tier FRN priority topics: Of interest to a substantial number of stakeholders, but lacking a consensus that these were high-priority topics. For these topics, only a summary of the stakeholder discussion is presented.
• Other topics: Of relatively little interest to stakeholders. These include the remaining topics (see the next category for exceptions) that were nominated by few if any stakeholders. Topics in this group are listed in the Results section without further discussion.
• Does not meet the Effective Health Care Program Appropriateness criteria. The EPC will move any topics that do not meet these minimum requirements for inclusion in the FRN report into this category for potential further discussion. These topics are not explicitly listed in the Results.

Approach to Research Question Development and Considerations for Potential Research Designs

For each high-priority FRN topic, we considered the range of study designs that would best address the topic. We did this taking into account the PICOD criteria (Population, Intervention, Comparator, Outcomes, study Design). For each topic, we described our assumptions about the most appropriate PICOD criteria, in particular describing the advantages and disadvantages of various potential research designs. We specifically considered the feasibility of the research questions focusing on potential sample size, time, and recruitment issues. For selected topics, we consulted with our local domain expert, who was also a stakeholder.

To determine candidate study designs, the feasibility of the study designs, and sample size calculations, we followed the structure laid out in the Future Research Needs document Framework for Considering Study Designs for Future Research Needs.⁴

Briefly, candidate study designs will differ across types of FRNs. Effectiveness or efficacy of treatments can be most definitively addressed in randomized controlled trials (RCTs), and secondarily in well-conducted nonrandomized comparative observational studies. In contrast, eliciting patient preferences can be meaningfully performed with nonexperimental designs (e.g., in a survey or focus group). Furthermore, observational studies may be most appropriate to enhance generalizability and determine effectiveness, as opposed to efficacy alone. Each final FRN topic was assessed as to context of the research question and a determination as to whether evaluation of efficacy or effectiveness is of greater need. This informs the choice of study design.

When a simple RCT (or diagnostic test study) was deemed to be an appropriate study design to address a FRN topic, we performed sample size calculations using standard formulae for a two-sided chi-squared test at the 0.05 level of significance. We assumed an allocation ratio of 1:1, no loss to followup, no crossover between treatments, and no sequential monitoring. For studies with dichotomous outcomes, we determined a range of reasonable control rates (event rates in the comparator arm) and relative effects (risk ratios) and based on these calculated a range of scenarios and minimum sample sizes. For studies with continuous outcomes, we range of reasonable mean differences in effect size (e.g., hours of sleep) between arm and standard deviations of the differences and based on these calculated a range of scenarios and minimum sample sizes. For continuous outcomes we also a conservative range of equivalence (the range between the smallest mean difference that would be clinically significant and the largest difference that could be reasonably expected). Where possible, these assumed values were derived from the existing evidence. All power calculations were set at 90 percent power to detect a significant result. We report all our assumptions.