Q13Describe disadvantages of using framework

FormStage CompletedDescription of DisadvantagesJHU Response
BFRNWe found that applying this framework to all potential combinations of PICOS for the <XXX> FRN project yielded more than 1000 research gaps. This was due to the large number of populations, settings, and intervention/comparisons and the overall poor quality of the existing literature. The framework is much more practical when there is a manageable number of potential PICOS combinations (e.g. FRN for <XXX> project).We have added text to the instructions suggesting that teams discuss prior to the use of the framework whether to, and how to, lump or split. For instance, it may be more manageable to abstract gaps by class of intervention and comparison.
CSRThe overlap with GRADE is less helpful as it is not clear how the gap will assist in the judgments of the SOE. For example a gap in research design….not all issues can be addressed with trials….so not sure how to make this link.
Also I think the list of reasons for developing a gap should be expanded. I found I used B2 very often to provide a reason for the gap….and the recommendations are widely varied. Thus some categories are not discriminating enough.
We did not consider the process of identifying gaps as a way to assist in making judgments about SOE. The framework was designed to leverage work completed, if SOE was assessed.
We do not see how B2 could be made more specific. It is to be selected if the body of evidence was considered at high risk of bias (this may be for a number of reasons, but is aggregated across the studies). Use of the framework does not preclude providing more details.
DSRInvolvement of different stakeholder groups may be not representative; information about ongoing studies may be incomplete ; no full representation of the NIH, other funding agencies; the role of industry is unclearWe have clarified at the beginning of the instructions the purpose for the framework – to identify and characterize gaps from systematic reviews. How to solicit stakeholder involvement and prioritize gaps is beyond the scope of this work.
EFRNThe gaps are not clearly conveyed by the table. The statement in the instructions that “other elements will be apparent from the key question” does not seem to be accurate to us.
At the FRN point it is almost too late; too difficult to use. It may have been more helpful during the CER.
We have added some discussion of this, including examples, under Characterization of Research Gaps in the instructions.
We agree that there are different challenges in applying the framework retrospectively versus while completing a systematic review.
We have added text in instructions providing some suggestions about how to proceed if doing so retrospectively.
FFRNNot all gaps are equally important, so it is not an efficient use of time to be required to complete this chart for every gap. Suggest that only the critical research gaps be prioritized for the chart.
It may be too cumbersome for readers to understand. Many of the codes may need to be listed for each gap. It is not clear that using the codes, as opposed to a narrative description, will make the gaps easier or more efficient to understand.
It is unclear how one would determine the ‘most critical’ gaps without first systematically identifying and characterizing the gaps, such as through use of this framework or other method.
The codes and worksheet were developed to aid in abstraction. The future research needs section of the systematic review, or future research needs document, would present the gaps. Our previous report provided a suggested presentation format (also another EPC has produced a report on how to present gaps).
GSRThe key disadvantage I see is that it may replicate work already done. This may be less of an issue if it was done alongside preparing the results, as was suggested in your instructions. I did it after the review was complete so I found it fairly redundant, as much of this information was already in SOE or summary tables. I'm not sure that it highlighted any issues that were not already known, i.e., very few studies providing data for the same comparisons & outcomes. So it could add a lot of work without providing much additional insight.We would hope that a team could leverage the work done in completing SOE but we take your point that it could also be redundant. We think this will depend on the team, the specific review, and the timing of applying the framework.
HFRNTo some degree it can be overly constraining and it really doesn't work well for a review topic on which there is very little available. In this case, the overwhelming gap is that much more research needs to be done, period. Trying to specify at the level of the framework is not yet possible or appropriate. Also, the framework is not ideal for methodologic issues.We agree that the framework may be too granular to use for questions for which, essentially, the entire question is a gap. We have added some text about these sorts of decisions to the instructions

From: Appendix E, Detailed Analysis of Evaluation of the Use of the Research Gaps Framework by Evidence-based Practice Centers (EPCs)

Cover of Framework for Determining Research Gaps During Systematic Review: Evaluation
Framework for Determining Research Gaps During Systematic Review: Evaluation [Internet].
Robinson KA, Akinyede O, Dutta T, et al.

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.