We used a deliberative process to identify and prioritize research questions relevant to the evidence gaps identified in the recently completed draft CER on PT for knee pain secondary to OA.14 Figure 3 illustrates the eight steps used to accomplish the objectives of this project.

Figure 3 describes the flow of the project in 8 steps. Step 1 is to identify the evidence gaps from the CER. Step 2 is to form and orient the stakeholder panel. Step 3 is to translate research gaps into researchable questions. Step 4 is to elicit stakeholder feedback on those questions via conference calls and emails. Step 5 is to revise the list of research gap questions based upon the stakeholder feedback. Step 6 involves having the stakeholders prioritize the revised set of research questions, higly prioritized questions are deemed research needs. Step 7 involves the EPC team identifying research design considerations for these research needs. And Step 8 is to develop the FRN report.

Figure 3

Project flow. CER=Comparative Effectiveness Review; PICOTS=population, intervention, comparison, outcome, timing, and setting

Engagement of Stakeholders

We recruited panel of stakeholder’s panel with diverse perspectives relevant to the topic. We followed guidance on stakeholder engagement for recruitment and communication.15 We sought to recruit stakeholders who were actively interested in PT treatments for patients with knee pain secondary to OA and who wished to help shape future research priorities. We identified potential stakeholders via several means. We sought recommendations from the CER project team, including select Key Informants and Technical Expert Panel members. We also identified stakeholders who were serving on panels from related Agency for Healthcare Research and Quality (AHRQ) FRN projects or who were listed in the Effective Health Care Contacts Database.16 Research representatives were national experts familiar with evidence-based medicine and aware of the obstacles faced in conducting well-designed research from rheumatology, orthopedics, and PT. We invited representatives from organizations supporting or conducting relevant research, including the National Institute of Arthritis and Musculoskeletal and Skin Diseases, the National Institute on Aging, the American Physical Therapy Association, and others, as well as policy and payer representation from the Centers for Medicare and Medicaid Services and the Center for Disease Control and Prevention. We engaged providers and consumers, including representation from the Arthritis Foundation, because the decisional dilemmas faced by these groups are critical to identifying and prioritizing research questions. Many stakeholders were also involved in the CER process as Key Informants, Technical Expert Panel members, or peer reviewers. This made engaging them as stakeholders challenging due to the overlap in timing with the FRN project and finalization of the CER.

Handling Conflicts of Interest

We collected disclosures of conflicts of interests from all stakeholders. Disclosed interests did not bar any stakeholders from participation, but allowed the Evidence-based Practice Center (EPC) to evaluate contributions based on possible conflicts. Stakeholders used a Web-based survey to rank specific topical research questions during the prioritization exercise, thus researchers and funders were blind to the others’ stated opinions.

Refinement of Research Questions

We provided members of our stakeholder panel with a preliminary set of research questions prior to conference calls. During conference calls, we sought stakeholder input to further refine the research questions (i.e., organization and wording of the questions, identification of additional research questions, and elimination of research questions with limited clinical value). To facilitate this input, we provided stakeholders in advance with background materials, including the draft CER executive summary and the Effective Health Care Program Selection Criteria. We conducted two conference calls with available stakeholders in February and March of 2012. A total of 14 stakeholders participated in the calls. All participants provided input on the calls. We circulated summaries of group calls to all participants, including two additional stakeholders not able to participate in the conference calls. We invited stakeholders to clarify or supplement the call summaries or to suggest additional research questions in response to the call summaries, and several did so via email. We revised the preliminary questions based upon these discussions and email communications. The revised set of questions moving on to the prioritization phase is listed in Appendix A.


We and our stakeholders evaluated the revised set of research questions according to specified criteria. The Effective Health Care Program Selection Criteria provided a starting point (Appendix B), including Appropriateness, Importance, Feasibility, Redundancy, and Potential Impact. The Appropriateness and Importance criteria are de facto met since PT treatments for patients with knee pain secondary to OA was accepted as an AHRQ topic.

We addressed the Redundancy criteria by conducting a search for ongoing and recently completed research using ClinicalTrials.gov. CER authors also updated the bibliographic database search for relevant newly published studies in December 2011 and incorporated these findings into the final CER. We conducted a precise search of recently published studies addressing aspects of identified research questions through May of 2012. The search strategies appear in Appendix C. We attempted to match identified recent and ongoing studies with revised research questions.

We then asked stakeholders to rank the research questions focusing on their potential impact criteria (i.e., the likelihood that addressing the research gap question would inform clinical practice and policy). We developed a Web-based survey using SurveyMonkey to collect stakeholder prioritization of the research gap questions.17 A subset of 14 stakeholders (fewer than 10 were non-Federal employees) were invited to rank research questions identified via the stakeholder conference calls. The subset of stakeholders was chosen from the broader set to assure representation from all major viewpoints. These stakeholders numerically ranked their top three of seven methodological research questions, and their top four of 11 topical research questions.

Stakeholder rankings were weighted according to their assigned numerical ranking. If a stakeholder assigned a question the number one priority, that question received four points; number two ranking – three points; number three ranking – two points; and number four ranking – one point. We identified natural breakpoints in the weighted rankings that separated high, moderate, and low priority research questions. Highly prioritized research questions were considered research needs. We disseminated results of the forced ranking procedure to all engaged stakeholders for review and comment prior to preparing the final report.

We then evaluated the feasibility criteria for research needs. We framed feasibility in terms of anticipated research designs. For example, factors that affect the feasibility of conducting randomized controlled trials include the sample size needed for the outcome, the size of the available pool of potential subjects, followup duration, willingness to randomize, and applicability issues. In contrast to randomization and applicability, observational studies face feasibility issues related to measuring study variables using different data sources and unobserved variables that create risk of bias.

Research Design Considerations

We generated research design considerations for identified research needs. For methodological research needs we provided context and described resources and research design considerations potentially useful to researchers, facilitators, and funders of this type of research. For topical research needs we highlighted the relevant PICOTS (population, intervention, comparison, outcome, timing, and setting) element(s), provided context, described related ongoing research, and discussed potential research designs. Because more than one research design can be applied to an individual research need, we discussed the advantages and disadvantages of different options. These discussions were guided by a recent AHRQ report describing frameworks for evaluating research designs in FRNs.18 We did not consult with stakeholders for input on research design considerations.