What are the most important unanswered research questions on rapid review methodology? A James Lind Alliance research methodology Priority Setting Partnership: the Priority III study protocol

Background: The value of rapid reviews in informing health care decisions is more evident since the onset of the coronavirus disease 2019 (COVID-19) pandemic. While systematic reviews can be completed rapidly, rapid reviews are usually a type of evidence synthesis in which components of the systematic review process may be simplified or omitted to produce information more efficiently within constraints of time, expertise, funding or any combination thereof. There is an absence of high-quality evidence underpinning some decisions about how we plan, do and share rapid reviews. We will conduct a modified James Lind Alliance Priority Setting Partnership to determine the top 10 unanswered research questions about how we plan, do and share rapid reviews in collaboration with patients, public, reviewers, researchers, clinicians, policymakers and funders. Methods: An international steering group consisting of key stakeholder perspectives (patients, the public, reviewers, researchers, clinicians, policymakers and funders) will facilitate broad reach, recruitment and participation across stakeholder groups. An initial online survey will identify stakeholders’ perceptions of research uncertainties about how we plan, do and share rapid reviews. Responses will be categorised to generate a long list of questions. The list will be checked against systematic reviews published within the past three years to identify if the question is unanswered. A second online stakeholder survey will rank the long list in order of priority. Finally, a virtual consensus workshop of key stakeholders will agree on the top 10 unanswered questions. Discussion: Research prioritisation is an important means for minimising research waste and ensuring that research resources are targeted towards answering the most important questions. Identifying the top 10 rapid review methodology research priorities will help target research to improve how we plan, do and share rapid reviews and ultimately enhance the use of high-quality synthesised evidence to inform health care policy and practice.


Introduction
Systematic reviews summarise existing research and use structured and 'explicit methods to identify, select and critically appraise relevant studies, and collect and analyse data from these studies' 1 . Systematic reviews, therefore, provide a robust synthesis of existing research for a given topic and are typically seen as the cornerstone of research evidence for informing health policy and practice decision-making. However, systematic reviews require significant resources (time, expertise, funding) to conduct and can take up to two years to complete 2 . While full systematic reviews can, and have been, completed rapidly, rapid reviews have emerged more commonly as a form of evidence synthesis in which certain steps of the systematic review process are omitted or simplified to produce information more efficiently within limited resources to inform healthcare decisions 2,3 .
Evidence suggests that rapid reviews are increasingly being commissioned, and the results used to inform decision making by policymakers and funders as a more resource efficient alternative to conventional systematic reviews 4,5 . The coronavirus disease 2019 (COVID-19) pandemic has further increased the demand for rapid reviews due to the need to provide time-sensitive information to decision-makers 6 .
Although growing demand has led to a steady rise in rapid review publications generally since 2010 3,7 , albeit often poorly reported with many important details not included 8 , there are challenges in identifying rapid reviews commissioned specifically by organisations as they are often not published in peer review journals 3 . This may, in part, be due to the suggestion that commissioned rapid reviews are considered more relevant in the context of narrow questions that address a specific question that may be of relevance to that commissioning body alone, rather than broader questions of wider relevance that may require the focus of a conventional systematic review 5,9 .
Despite an evident rise in the commissioning and conduct of rapid reviews, there is an absence of high-quality evidence underpinning some decisions about how they are planned, done and the findings shared. There is also limited evidence on the relative impact of different simplifications or omissions, for example, restricting searches to published material or by date/language or conducting single reviewer screening or data extraction 10,11 , on the validity and application of findings 3 commonly used in rapid reviews. Furthermore, there is no universally accepted definition of a rapid review 3,6,12,13 and debate exists regarding the use of the word 'rapid', with several different synonyms previously identified within the literature 7,10 . These debates have been further complicated in the era of the COVID-19 pandemic where it has been highlighted that rapid reviews cannot be categorised simply based on the time taken to complete a review given that well-resourced conventional systematic reviews have been conducted to a high standard, rapidly.
Overall, rapid reviews are 'poorly understood, ill-defined diverse methodologies supported by limited published evidence' 3 , with a need for further research to establish a robust methodological evidence base 14 . Without such an evidence base, the validity, appropriateness and usefulness of rapid reviews are undermined 2 .
Research prioritisation plays a key role in minimising research waste by ensuring that research resources are targeted towards questions of the most potential benefit 15 . Research prioritisation involves identifying, prioritising and obtaining consensus on research needs and questions that are relevant and important to all relevant stakeholders for that topic 16 . The James Lind Alliance (JLA) is a non-profit organisation that brings multiple stakeholders, including patients, carers and clinicians, together in an equal, transparent and evidence-based Priority Setting Partnership (PSP) to determine the most important evidence uncertainties or unanswered research questions. Although commonly focused on knowledge gaps surrounding the effects of treatments, the approach has been broadly applied to other areas such as diagnosis, prevention, and more recently, to identify methodology uncertainties in recruitment and retention within clinical trials 17,18 .
This Priority III PSP aims to identify the top 10 unanswered research questions about how we plan, do and share rapid reviews, as identified and prioritised by contributors drawn from key stakeholder groups, including patients, the public, reviewers, researchers, clinicians, policymakers and funders. Evidence Synthesis Ireland is conducting the Priority III PSP in collaboration with the JLA. Evidence Synthesis Ireland is an all-Ireland initiative funded by the Health Research Board and the Health and Social Care, Research and Development Division of the Public Health Agency in Northern Ireland. It aims to make evidence syntheses better designed, conducted, reported, and more usable within health care policy and clinical practice decision-making by patients, the public, health care institutions and policymakers, clinicians and researchers.

Protocol
Ethical approval Ethical approval was granted by the National University of Ireland Galway Research Ethics Committee (reference: 20-Apr-02).

Amendments from Version 1
Protocol version 2 contains changes made in response to reviewers feedback on version 1.

This includes:
-Clarification on aspects such as the geographical spread of the steering group and participants generally in light of constructive comments from the reviewers.
-Grammatical errors identified by reviewers have been updated.
-Explanation on the rationale behind publishing the protocol prior to dissemination.
Any further responses from the reviewers can be found at the end of the article REVISED James Lind Alliance PSP process The study employs a PSP based on the methods of the JLA and will be reported following the REporting guideline for PRIority SEtting of health research (REPRISE) criteria 16 .
Following the JLA Guidebook for PSPs 19 , this PSP will take place in seven stages and build on modified JLA guidance 17,18 and previous PSPs in recruitment and retention within clinical trials 17,18 .
The seven stages of the project are: establishing the steering group, identifying and inviting potential partners, gathering uncertainties, data processing and verifying uncertainties, interim priority setting; final prioritisation workshop; and disseminating findings.
This protocol was submitted prior to the consensus workshops and published after completion of the final prioritisation workshop. There are several factors that contributed to the late submission of the protocol. We had to balance the evolving nature of the PSP processes and having sufficient clarity on processes for inclusion in a protocol. We also encountered delays due to useful discussions with F1000/ HRB Open Research on affiliation of public partner authors and technical problems with submission attempts prior to the consensus workshop.

Establishing the Steering Group (6 months-January 2020).
The Priority III PSP will be led and managed by an international Steering Group who will coordinate, oversee and guide the PSP activities. The primary roles of the Steering Group will be to discuss and agree on the scope and remit of the project, enable access and reach to key stakeholder groups, contribute intellectually towards the study methods and interpretation, and ensure that all perspectives are captured and meaningfully included throughout.
The Steering Group will also determine decisions about the processes used throughout as the project progresses. Membership of the Steering Group will include individuals and representatives from organisations, including patients and the public, reviewers, researchers, clinicians, policymakers and funders. The Steering Group will comprise of up 25 members across stakeholder groups. In line with the international relevance of the PSP topic, steering group members will be drawn from a wide geographical spread. Potential members will be approached to participate via email. Patient and public participants will be paid for their time spent on Steering Group activities. Other participants (researchers/ reviewers, clinicians, policy makers, funders) will not be paid.

Identifying and inviting potential partners (7 months-April 2020).
Partners are organisations, groups and individuals impacted by how rapid reviews are planned, done and shared. They will commit to supporting the PSP, promoting the process and encouraging their represented groups or members to participate 19 . Steering Group members will identify and engage additional appropriate partners through a process of peer knowledge and consultation, using the Steering Group members' respective networks. Organisations and individuals who can reach and advocate for key stakeholder groups will be invited to participate with the PSP as partners. We have not set a minimum or maximum number of partners we would like to recruit. Partners will be organisations or groups representative of diverse stakeholder perspectives (patients and the public, reviewers, researchers, clinicians, policymakers and funders) who will commit to supporting, participating and promoting the PSP among their stakeholder groups. As far as possible, the partners involved will seek to represent the interests of all stakeholders. All partners will be asked to confirm that they agree to support the PSP.

Gathering uncertainties (1 month-October 2020). The Rapid
Review Methodology PSP will gather uncertainties from patients and the public, reviewers, researchers, clinicians, policymakers and funders. This will be undertaken by conducting an initial survey with all relevant stakeholders. The survey will identify unanswered questions or uncertainties about how we plan, do and share rapid reviews. The survey will be designed and piloted by the Steering Group members. The survey will be created using QuestionPro 20 software and hosted on the Evidence Synthesis Ireland website. Participants will be asked to give their explicit consent to take part in the survey using yes/no questions. The survey will contain four open-ended questions. Three of the questions will focus on different stages of the rapid review process, and participants will be asked to answer as few or as many as they wish. The fourth question asks for any additional questions or comments on the rapid review process. The four questions are; Participant demographic data will also be collected to monitor responses of different stakeholder groups and help refine and target the promotion of the survey towards under-represented groups if necessary. In line with previous methodology-focused PSPs 17,18 , the survey will be open for four weeks.
The survey will be advertised and distributed via the Steering Group and the PSP Partners. Specifically, members will distribute the survey via their networks, mailing lists, newsletters and social media. We do not have an a priori determined sample size and instead will distribute the survey widely with a view to reaching as wide an audience as possible. The survey and all other study materials can be found as Extended data 21 .

Data processing and verifying uncertainties (5 months -November 2020).
The initial survey consultation process is expected to produce substantial 'raw' questions and comments indicating stakeholders' areas of uncertainty. The survey data will be downloaded from QuestionPro 20 . These raw questions will be categorised and refined by the National University of Ireland, Galway research team (CB, CH, DD) into summary questions that are clear, addressable by research, and understandable to all. Similar or duplicate responses will be combined where appropriate. Questions will be considered out of scope if they do not relate to planning, doing and sharing the results of rapid reviews within healthcare. We defined healthcare a priori as being related to the "treatment, control or prevention of disease, illness, injury or disability, and the care or aftercare of a person with these needs (whether or not the tasks involved have to be carried out by a health professional)" 22 . Questions will also be considered out of scope if they are asking for information or advice, or being too broad, unclear, unrelated or off topic. Questions deemed to be out of scope will be compiled separately and made available for future use upon request.
The Steering Group will exercise oversight on data analysis and processing to ensure that the raw data are being interpreted appropriately. The summary questions will be developed in a reflective manner that is understandable to all audiences. The process will be conducted transparently to ensure that the finalised questions can be traced back to the raw response data.
Each in-scope question will be checked against existing sources of evidence to determine which questions remain unanswered. A question will be verified as unanswered if a synthesis gap is apparent following a search of all relevant systematic reviews published within the past three years. We judged a review to be systematic when it involved explicit methods to search, select, critically appraise and synthesise individual studies. If a systematic review has been conducted to answer any of the questions completely, the quality of that systematic review will be appraised using the AMSTAR 2 23 tool to help inform decisions about the extent to which the question has been answered. If, following use of the AMSTAR 2 tool, a systematic review conducted to answer a question completely is identified as being of high quality (between 8-11), the question will be deemed as answered. Evidence checking will be completed by one researcher (CB) and verified by one other researcher (DD) Difference of opinion will be resolved by consultation with a third researcher if necessary.
Existing sources of evidence will be identified through a search of the PubMed bibliographic database for systematic reviews published from 2018 to the time of searching (2021) using a search strategy developed specifically for use in this project with the support of an experienced information specialist (see Table 1 for search strategy and limits applied). Due to limited resources (time and unavailability of translator) grey literature and publications not available in English will be excluded.
The JLA Question Verification Form, which describes the process used to verify question uncertainty, will be completed.
In line with JLA guidance, the Question Verification Form includes details of the types and sources of evidence used to check uncertainty. This form will be published on the JLA website to enable stakeholders to understand how the PSP decided that these questions are unanswered and any limitations of this process.
The Steering Group will be asked to review and refine, as appropriate, the final list of summary questions for inclusion in the interim survey.

Interim priority setting (1 month-April 2021).
Interim priority setting is where the long list of questions is reduced to a shorter list that can be taken to the final priority setting workshop. This shortlist is aimed at a wide audience and will be administered using an electronic survey (QuestionPro software 20 ) and hosted on the Evidence Synthesis Ireland website. It will be designed and piloted by the Steering Group members. In this survey, stakeholders will be asked to prioritise summary  ) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND ("literature search*" OR Information Storage and Retrieval)) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND ("plain language summar*" OR "lay summar*")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND "strength of evidence")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") "METHODS"[SH] AND "study selection")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND "data extraction")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND "quality assessment")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND "critical appraisal"))) NOT Protocol[TI]) Limit 1 Limited to 'review' and 'systematic review' Limit 2 Limited to three years prior to search taking place Limit 3 Limited to English questions developed from the initial survey in order of importance. This interim priority setting process will help reduce the long list to a short, manageable set of approximately 20 indicative questions that are clear, addressable by research and understandable. The survey will be open for four weeks and will be advertised and distributed like the initial survey via the Steering Group and the PSP Partners. Participant demographic data will be collected to monitor response rates from different stakeholder groups to help refine and target the promotion of the survey towards under-represented groups.
Questions prioritised in the interim survey will go forward for final prioritisation at a stakeholder workshop(s). Where the interim prioritisation does not produce a clear ranking or cut off point, the Steering Group will decide, and report, which questions are taken forward to the final prioritisation.

Final prioritisation workshop (2 days-May 2021).
The final priority setting stage will involve two half-day virtual workshops facilitated by the JLA to ensure transparency, accountability, fairness and appropriate representation. Based on the methods of the JLA, these workshops would ordinarily take place in person, but due to the Priority III study being conducted under restrictions contingent upon the global COVID-19 pandemic, the workshops will be held virtually. With input from the Steering Group, up to 24 contributors drawn from key stakeholder groups (patients, public, reviewers, researchers, clinicians, policymakers and funders) from a wide geographical spread, will be recruited to participate in a day of group discussion, plenary sessions and ranking exercises to determine the top 10 questions for research. Four virtual breakout rooms will be used to facilitate smaller group discussions. All participants will declare their interests, enabling a diverse group of stakeholders to exchange knowledge, perspectives, and experiences and inform the decision-making process. A maximum of four Steering Group members will be invited to participate in the workshops, with a steering group member in each breakout room to participate and answer any questions about the Priority III process. Additional Steering Group members who wish to attend will do so in an observer capacity. The outcome of the workshop will be consensus on a top 10 list of research priorities of unanswered questions about rapid review methodology.

Disseminating findings.
The results of the study will be disseminated through the JLA and Evidence Synthesis Ireland websites. The Steering Group will identify additional appropriate audiences to engage when sharing the results, such as researchers, funders, and patient and clinical communities. The steering group will also identify opportunities to collaborate and contribute evidence to answer the top 10 list of research priorities.

Conclusion
The Priority III study will identify a top 10 of unanswered research questions regarding rapid review methodology, guided by an adaptation of the James Lind Alliance PSP process.
The findings of the study will contribute to minimising research waste in rapid review methodology, ensuring that research resources will be used to answer questions on this topic that have been prioritised by key stakeholders internationally, including patient and the public, reviewers, researchers, clinicians, policymakers and funders.

Study status
At the time of submission, all stages up to the dissemination of findings had been completed. There are several factors that contributed to the submission of the protocol at this time: In relation to their response to my second point about the use of AMSTAR 2: the authors clarified that they will use a rating of 8-11 to classify a systematic review as high quality but I am a little perplexed as to how this will work in practice as AMSTAR 2 has 16 items and is not intended to generate an overall score. However, I am happy for the authors to clarify this in any resulting publications of results.
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: I have experience in conducting both systematic reviews and rapid reviews. I have also been involved in the development and assessment of methodologies for rapid reviews.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Despite the increasing need and use of "rapid reviews" during the COVID-19 pandemic, the evidence behind these abbreviated reviews themselves, is rather scant. The authors, who are presumably members of the James Lind Alliance (JLA), propose a research prioritization study (Priority Setting Partnership), which consists of two stakeholder surveys, followed by a consensus workshop to identify the top 10 unanswered questions regarding rapid reviews.
The steps described in the paper include: Establishing a steering group (6 months) to oversee the PSP. The steering group includes members across stakeholder groups and are reasonable for the project activities. 1.
Identifying and inviting potential partners (7 months). The authors state that there is no minimum or maximum number of partners. Patient and public partners will be reimbursed (the authors do not state how much). No minimum or maximum number of participants are identified.

2.
Gathering uncertainties (1 month -Oct 2020) from patients and the public, reviewers, researchers, clinicians, policymakers and funders (by conducting an initial 4 question survey with all relevant stakeholders).

3.
Data processing and verifying uncertainties (5 months) -raw questions will be categorized and refined into summary questions (downloaded from QuestionPro and then categorized and refined by a "NUI Galway" research team.

4.
Interim priority setting ( 1 month -April 2021) -long list of questions is reduced to shorter list. Stakeholders will be asked to prioritize summary questions.

5.
Final prioritization workshop (2 days -May 2021) -two half day virtual workshops of up to 24 key stakeholder members to participate in discussions (group discussion, plenary sessions, raking exercises, virtual breakout rooms). The outcome of the workshop will be consensus on a top 10 list of research priorities of unanswered questions about rapid review methodology.

6.
Disseminating findings -results will be shared through JLA and Evidence Synthesis Ireland websites 7.

Major Concern
The paper is written in the future tense, but all steps except for 7 above, have already been completed per the authors. Throughout my reading, I am anticipating that this is a future study that has yet to be conducted. It comes as a surprise to me that at the very end, in the "study status," the authors reveal all the steps except the final one (disseminating findings), ○ have been completed.

Minor Concerns
Minor grammatical error: "Evidence suggests that rapid reviews are increasingly being commissioned, and the results used, to inform decision making by policymakers …." Please remove the comma after "results used." The sentence should read: "Evidence suggests that rapid reviews are increasingly being commissioned, and the results used to inform decision making by policymakers…" ○ Presumably the authors of the paper are on the steering group? This is not clearly stated.
○ NUI Galway needs to be spelt out for readers unfamiliar that it represents National University of Ireland, Galway.

○
The authors do not state how much the partners will be reimbursed.

○
The time frames for the initial steps listed above are inconsistent. Some are written to only include the duration, but others include both duration (e.g., 1 month, 6 months, 2 days) and the month (Oct 2020, April 2021, May 2021). My recommendation would be to stick with one approach and be consistent (e.g., include the time period with actual dates because the duration can be implied).

Summary
This paper describes a proposed research project that consists of a series of focus groups and stakeholder meetings to define the most critical 10 research questions regarding rapid reviews. It is well-written with minimal grammatical and syntax errors, but I am unclear why it was written in the future tense. The authors describe a proposed study, when in fact, all steps of the project have been conducted except for the last one. When I get to the final section of the paper and realize the proposed steps have been completed, my natural questions include; How did the sessions go? Who was recruited to participate? How many were recruited and what was the breakdown? How much were the participants reimbursed for their time? What did the questions reveal? What are the next steps? But these questions are unanswered by the authors because the paper is written as a proposed project, despite the steps having already been completed. This issue needs to be resolved before I am fully comfortable with accepting in the current state.

Recommendation
Thank you for the opportunity to review this paper. My recommendation would be to convert this into a dissemination paper, such as the PRioRiTy (Prioritising Recruitment in Randomised Trials) publications the authors cite (references 17 and 18). If one of the purposes of the project is to conduct research on the top 10 list of research priorities of unanswered questions about rapid review methodologies AND then disseminate those findings, then the authors may want to consider converting their paper into one that serves the latter purpose.

Is the study design appropriate for the research question? Yes
Are sufficient details of the methods provided to allow replication by others?

Are the datasets clearly presented in a useable and accessible format?
Not applicable Claire Beecher, Evidence Synthesis Ireland and Cochrane Ireland, Galway, Ireland

Reviewer Comment 1
The paper is written in the future tense, but all steps except for 7 above, have already been completed per the authors. Throughout my reading, I am anticipating that this is a future study that has yet to be conducted. It comes as a surprise to me that at the very end, in the "study status," the authors reveal all the steps except the final one (disseminating findings), have been completed.

Response 1
Thank you for raising this point. This protocol paper gives much more detail on the methods than will be contained in the results paper hence the importance of this protocol paper.
There are several factors that contributed to the submission of the protocol at this time: The evolving nature of the PSP process and the decisions that needed to be made by the steering group as the study progresses through to the later stages.

1.
Discussions with F1000/ HRB Open Research on affiliation of public partner authors and the time to agree same.

2.
Attempts were made to be submit the protocol prior to the completion of the workshop but due to an issue with submission process, this was not possible until after the workshop had been completed (issue rectified by HRB Open Research).

3.
Although the protocol would ideally have been submitted at an earlier stage, this was not possible given the above factors.
We have retained future tense in keeping with protocols papers but have added the following to the paper to be more transparent: 'This protocol was submitted prior to the consensus workshops and published after completion of the final prioritisation workshop. There are several factors that contributed to the late submission of the protocol. We had to balance the evolving nature of the PSP processes and having sufficient clarity on processes for inclusion in a protocol. We also encountered delays due to useful discussions with F1000/ HRB Open Research on affiliation of public partner authors and technical problems with submission attempts prior to the consensus workshop' (4 th paragraph under heading 'James Lind Alliance PSP process')

Reviewer Comment 2
Minor grammatical error: "Evidence suggests that rapid reviews are increasingly being commissioned, and the results used, to inform decision making by policymakers …." Please remove the comma after "results used." The sentence should read: "Evidence suggests that rapid reviews are increasingly being commissioned, and the results used to inform decision making by policymakers…"

Response 2
Thank you very much for highlighting this, it has now been amended.

Reviewer Comment 3
Presumably the authors of the paper are on the steering group? This is not clearly stated.

Response 3
Not all co-authors are members of the steering group. All co-authors have been included based on their contribution as identified using the CREDiT Taxonomy.

Reviewer Comment 4
NUI Galway needs to be spelt out for readers unfamiliar that it represents National University of Ireland, Galway.

Response 4
Thank you very much, this has now been updated.

Reviewer Comment 5
The authors do not state how much the partners will be reimbursed.

Response 5
This has not been stated as there is no reimbursement for partners.

Reviewer Comment 6
The time frames for the initial steps listed above are inconsistent. Some are written to only include the duration, but others include both duration (e.g., 1 month, 6 months, 2 days) and the month (Oct 2020, April 2021, May 2021). My recommendation would be to stick with one approach and be consistent (e.g., include the time period with actual dates because the duration can be implied).

Response 6
Thank you for pointing this out -this has now been updated as suggested (please note some overlap in timeline included between establishing steering group and identifying and inviting partners as these processes ran concurrently for a limited time).

Reviewer Comment 7
This paper describes a proposed research project that consists of a series of focus groups and stakeholder meetings to define the most critical 10 research questions regarding rapid reviews. It is well-written with minimal grammatical and syntax errors, but I am unclear why it was written in the future tense. The authors describe a proposed study, when in fact, all steps of the project have been conducted except for the last one. When I get to the final section of the paper and realize the proposed steps have been completed, my natural questions include; How did the sessions go? Who was recruited to participate? How many were recruited and what was the breakdown? How much were the participants reimbursed for their time? What did the questions reveal? What are the next steps? But these questions are unanswered by the authors because the paper is written as a proposed project, despite the steps having already been completed. This issue needs to be resolved before I am fully comfortable with accepting in the current state.

Response 7
Thank you for raising this feedback. Response to the point has been provided in point 1.

Reviewer Comment 8
Thank you for the opportunity to review this paper. My recommendation would be to convert this into a dissemination paper, such as the PRioRiTy (Prioritising Recruitment in Randomised Trials) publications the authors cite (references 17 and 18). If one of the purposes of the project is to conduct research on the top 10 list of research priorities of unanswered questions about rapid review methodologies AND then disseminate those findings, then the authors may want to consider converting their paper into one that serves the latter purpose

Response 8
Thank you very much for your time reviewing this protocol. We do take on board your feedback and although ideally the protocol would have been submitted at an earlier stage, this was not possible given the factors that have been listed above.
This protocol paper gives much more detail on the methods than will be contained in the results paper hence the importance of this protocol paper. We have retained future tense in keeping with protocols papers but have added the following to the paper to be more transparent: 'This protocol was submitted prior to the consensus workshops and published after completion of the final prioritisation workshop. There are several factors that contributed to the late submission of the protocol. We had to balance the evolving nature of the PSP processes and having sufficient clarity on processes for inclusion in a protocol. We also encountered delays due to useful discussions with F1000/ HRB Open Research on affiliation of public partner authors and technical problems with submission attempts prior to the consensus workshop' (4 th paragraph under heading 'James Lind Alliance PSP process') Competing Interests: No competing interests were disclosed.
Reviewer Report 11 October 2021 I was rather surprised to reach the end of the protocol and find out that the research had already been completed, except for the dissemination of the findings. It made me wonder what then the purpose of peer review was, except perhaps to ensure accuracy of reporting. While I imagine that there are good reasons for the protocol being submitted at this late stage, I think it would be more transparent (and helpful for the reader) that this is made clear from the beginning, along with the reason. Thus, I suggest that the protocol should be written in the past tense, where applicable, and dates added for all stages.
The inclusion of the process for checking that questions were indeed still unanswered is a very valuable inclusion in the methods -I look forward to seeing the results of this process. I hope that the authors can include in the publication of the results the questions that are already answered and the sources of the existing evidence. I do have some concerns about the methodology here and suggest that the authors include a justification of their choice of methods or further clarification: The quality of the systematic reviews will be appraised using AMSTAR 2 but there is no description in the protocol as to how the score will be used in deciding if the question has been answered, e.g. does it need to have a minimum quality?
○ Only PubMed will be searched and only reviews published in English will be included. It is not clear how many reviewers will be involved in the quality assessment and why non-English and grey literature is not being included. If the authors were to evaluate their own review process using AMSTAR 2 I expect that it would not score highly. Thus, the authors should provide greater clarity and a justification for their choice of methods for this part of the protocol.

○
In the paragraph headed 'Data processing and verifying uncertainties (5 months)' I noticed that the scope of the priority setting exercise is limited to the healthcare setting. This is a very important limitation but no explanation is given for why. Why not include public health or health system questions, which are equally (if not more) important for rapid reviews?
It is not clear from the protocol if there was any attempt to include the views of people from lowto middle-income countries, which may be able to offer a broader perspective of the unanswered questions. Can this be clarified?
When defining rapid reviews (in the abstract and introduction) I suggest noting that an important distinction from systematic reviews is that rapid reviews are generally conducted with the needs of decision-makers in mind -see ○ have also been involved in the development and assessment of methodologies for rapid reviews.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 05 Nov 2021 Claire Beecher, Evidence Synthesis Ireland and Cochrane Ireland, Galway, Ireland Reviewer Comment 1 I was rather surprised to reach the end of the protocol and find out that the research had already been completed, except for the dissemination of the findings. It made me wonder what then the purpose of peer review was, except perhaps to ensure accuracy of reporting. While I imagine that there are good reasons for the protocol being submitted at this late stage, I think it would be more transparent (and helpful for the reader) that this is made clear from the beginning, along with the reason. Thus, I suggest that the protocol should be written in the past tense, where applicable, and dates added for all stages.

Response 1
Thank you for raising this point. This protocol paper gives much more detail on the methods than will be contained in the results paper hence the importance of this protocol paper.
There are several factors that contributed to the submission of the protocol at this time: The evolving nature of the PSP process and the decisions that needed to be made by the steering group as the study progresses through to the later stages.

1.
Discussions with F1000/ HRB Open Research on affiliation of public partner authors and the time to agree same.

2.
Attempts were made to be submit the protocol prior to the completion of the workshop but due to an issue with submission process, this was not possible until after the workshop had been completed (issue rectified by HRB Open Research).

3.
Although the protocol would ideally have been submitted at an earlier stage, this was not possible given the above factors.
We have retained future tense in keeping with protocols papers but have added the following to the paper to be more transparent: 'This protocol was submitted prior to the consensus workshops and published after completion of the final prioritisation workshop. There are several factors that contributed to the late submission of the protocol. We had to balance the evolving nature of the PSP processes and having sufficient clarity on processes for inclusion in a protocol. We also encountered delays due to useful discussions with F1000/ HRB Open Research on affiliation of public partner authors and technical problems with submission attempts prior to the consensus workshop' (4 th paragraph under heading 'James Lind Alliance PSP process')

Reviewer Comment 2
The inclusion of the process for checking that questions were indeed still unanswered is a very valuable inclusion in the methods -I look forward to seeing the results of this process. I hope that the authors can include in the publication of the results the questions that are already answered and the sources of the existing evidence. I do have some concerns about the methodology here and suggest that the authors include a justification of their choice of methods or further clarification: The quality of the systematic reviews will be appraised using AMSTAR 2 but there is no description in the protocol as to how the score will be used in deciding if the question has been answered, e.g. does it need to have a minimum quality?
○ Only PubMed will be searched and only reviews published in English will be included. It is not clear how many reviewers will be involved in the quality assessment and why non-English and grey literature is not being included. If the authors were to evaluate their own review process using AMSTAR 2 I expect that it would not score highly. Thus, the authors should provide greater clarity and a justification for their choice of methods for this part of the protocol.

Response 2
Thank you for this comment. Yes, in the final paper all results and original documents used will be presented via a link to Open Science Framework to ensure transparency of this process.
The following text has been included in relation to how the AMSTAR 2 score would be used in deciding if a question is answered: 'If, following use of the AMSTAR 2 tool, a systematic review conducted to answer any of the questions completely is identified as being of high quality (between 8-11), the question will be deemed as answered.' (3 rd paragraph under heading 'Data processing and verifying uncertainties').

○
It is stated that "Evidence checking will be completed by one researcher (CB) and verified by one other researcher (DD) Difference of opinion will be resolved by consultation with a third researcher if necessary" (3 rd paragraph under heading 'Data processing and verifying uncertainties').
○ Thank you very much for highlighting this omission. A rationale for non-English and grey literature not being included has been added to the text (4 th paragraph under heading 'Data processing and verifying uncertainties').

Reviewer Comment 3
In the paragraph headed 'Data processing and verifying uncertainties (5 months)' I noticed that the scope of the priority setting exercise is limited to the healthcare setting. This is a very important limitation but no explanation is given for why. Why not include public health or health system questions, which are equally (if not more) important for rapid reviews?

Response 3
We wish to clarify that we had a relatively broad definition of healthcare which we defined a priori as being; "treatment, control or prevention of disease, illness, injury or disability, and the care or aftercare of a person with these needs (whether or not the tasks involved have to be carried out by a health professional)". This definition has been adapted from the UK Department of Health and Social Care guidance document titled 'National Framework for NHS Continuing Healthcare and NHS-funded Nursing Care'.
We have added this to the paper (1 st paragraph under heading 'Data processing and verifying uncertainties').

Reviewer Comment 4
It is not clear from the protocol if there was any attempt to include the views of people from low-to middle-income countries, which may be able to offer a broader perspective of the unanswered questions. Can this be clarified?

Response 4
The views of people from low-to middle-income countries are very important in each stage of the Priorty III stages.
Reference to the importance of representation from underrepresented groups has been highlighted by the statement "Participant demographic data will also be collected to monitor responses of different stakeholder groups and help refine and target the promotion of the survey towards under-represented groups if necessary" within the gathering uncertainties and interim priority setting headings/ stages.
The final prioritisation workshop stage has been updated to highlight that workshop participants will be drawn from a wide geographical spread including representation from low-middle income countries.

Reviewer Comment 5
When defining rapid reviews (in the abstract and introduction) I suggest noting that an important distinction from systematic reviews is that rapid reviews are generally conducted with the needs of decision-makers in mind -see

Response 5
Thank you for this feedback, however we do feel that the point you have raised is addressed in the first two paragraphs of the introduction. For example, we say (paragraph 1 & 2 of introduction), "While full systematic reviews can, and have been, completed rapidly, rapid reviews have emerged more commonly as a form of evidence synthesis in which certain steps of the systematic review process are omitted or simplified to produce information more efficiently within limited resources to inform healthcare decisions. Evidence suggests that rapid reviews are increasingly being commissioned, and the results used to inform decision making by policymakers and funders as a more resource efficient alternative to conventional systematic reviews….."

Reviewer Comment 6
Third paragraph of introduction: 1) in the first sentence it is not clear what is 'often poorly reported', and 2) the last sentence doesn't make sense to me. Consider revising both.

Response 6
Thank you very much for this feedback-both sentences have been updated (paragraph 3 of introduction)

Response 7
Thank you very much for highlighting this, it has now been amended (paragraph 1 under heading 'Gathering uncertainties') The paper provides a well-structured and readable overview of a project intended to develop a better understanding of the strengths and limitations of "rapid reviews," and presumably provide a framework for better standardizing this form of evidence synthesis.
I note that according to the authors "At the time of submission, all stages up to the dissemination of findings had been completed." Normally I would expect the material in the current paper to be included in the final publication, rather than in a separate protocol paper that seems to have been written essentially contemporaneously with the final work. I will not second-guess this decision, but will instead comment as if the project were not now largely complete.
There are a few places where I believe that the reader's understanding of the project could be improved by including either tables or supplementary material to improve the specificity of the document. The document states that: "Membership of the Steering Group will include individuals and representatives from organisations, including patients and the public, reviewers, researchers, clinicians, policymakers, and funders. The Steering Group will comprise of up 25 members across stakeholder groups." Either the membership of this Steering group should be provided, or the types of groups and organizations that were contacted to participate should be provided. Were these groups restricted to Ireland, or to Europe, or to English speaking countries, or to advanced economies? It seems likely that the ultimate product coming from a Eurocentric steering committee will be different than that which involves individuals from Asia, Africa, and South America. On the other hand, it would be difficult to adequately represent researchers, clinicians, policymakers, and funders across a wide geographic distribution.
The areas of expertise associated with steering group members (statistics, survey methodology, etc.) should be described in more detail.
Similarly, the nature of the "partners" should also be better defined, as should a bit about the total number of stakeholders expected to participate. The final report should include information on the nature and number of stakeholders that were contacted and who declined to participate.

Is the rationale for, and objectives of, the study clearly described? Yes
Is the study design appropriate for the research question? Yes

Claire Beecher, Evidence Synthesis Ireland and Cochrane Ireland, Galway, Ireland
Reviewer Comment 1 I note that according to the authors "At the time of submission, all stages up to the dissemination of findings had been completed." Normally I would expect the material in the current paper to be included in the final publication, rather than in a separate protocol paper that seems to have been written essentially contemporaneously with the final work. I will not second-guess this decision, but will instead comment as if the project were not now largely complete.

Response 1
Thank you for raising this point. This protocol paper gives much more detail on the methods than will be contained in the results paper hence the importance of this protocol paper.
There are several factors that contributed to the submission of the protocol at this time: The evolving nature of the PSP process and the decisions that needed to be made by the steering group as the study progresses through to the later stages.

1.
Discussions with F1000/ HRB Open Research on affiliation of public partner authors and the time to agree same.

2.
Attempts were made to be submit the protocol prior to the completion of the workshop but due to an issue with submission process, this was not possible until after the workshop had been completed (issue rectified by HRB Open Research).

3.
Although the protocol would ideally have been submitted at an earlier stage, this was not possible given the above factors.
We have retained future tense in keeping with protocols papers but have added the following to the paper to be more transparent: 'This protocol was submitted prior to the consensus workshops and published after completion of the final prioritisation workshop. There are several factors that contributed to the late submission of the protocol. We had to balance the evolving nature of the PSP processes and having sufficient clarity on processes for inclusion in a protocol. We also encountered delays due to useful discussions with F1000/ HRB Open Research on affiliation of public partner authors and technical problems with submission attempts prior to the consensus workshop' (4 th paragraph under heading 'James Lind Alliance PSP process')

Reviewer Comment 2
There are a few places where I believe that the reader's understanding of the project could be improved by including either tables or supplementary material to improve the specificity of the document. The document states that: "Membership of the Steering Group will include individuals and representatives from organisations, including patients and the public, reviewers, researchers, clinicians, policymakers, and funders. The Steering Group will comprise of up 25 members across stakeholder groups." Either the membership of this Steering group should be provided, or the types of groups and organizations that were contacted to participate should be provided. Were these groups restricted to Ireland, or to Europe, or to English speaking countries, or to advanced economies? It seems likely that the ultimate product coming from a Eurocentric steering committee will be different than that which involves individuals from Asia, Africa, and South America. On the other hand, it would be difficult to adequately represent researchers, clinicians, policymakers, and funders across a wide geographic distribution

Response 2
We do understand the relevance of the membership of the steering group; however we haven't included detailed information on the steering group here as information on the areas of expertise associated with steering group members will be described in detail in the results paper that presents the outcome of each of the seven stages of the priority setting process.
Thank you for raising this point-we confirm that the steering group members are drawn from a wide geographical spread-this has now been highlighted within the text (2 nd paragraph under heading 'Establishing the Steering Group'). Given the international relevance of the topic, as you have highlighted, it is very important that Priority III steering group members were not restricted to Ireland, Europe, English speaking countries, or advanced economies.

Reviewer Comment 3
The areas of expertise associated with steering group members (statistics, survey methodology, etc.) should be described in more detail.

Response 3
Thank you very much for suggesting. We haven't included detailed information on the steering group here as information on the areas of expertise associated with steering group members will be described in detail in the results paper that presents the outcome of each of the seven stages of the priority setting process.

Reviewer Comment 4
Similarly, the nature of the "partners" should also be better defined, as should a bit about the total number of stakeholders expected to participate. The final report should include information on the nature and number of stakeholders that were contacted and who declined to participate

Response 4
Thank you very much for suggesting this. We have now added further information on the nature of partners (first line under heading 'Identifying and inviting potential partners').
We agree fully that the final report should include information on the nature and number of stakeholders that were contacted and who declined to participate.