The impact of panel composition and topic on stakeholder perspectives: Generating hypotheses from online maternal and child health modified‐Delphi panels

Abstract

1. INTRODUCTION

Multistakeholder engagement is crucial for conducting health services research; it helps ensure that key stakeholder perspectives inform the research process and its outcomes. ¹ Patients, caregivers, clinicians, researchers, payers, purchasers and policy‐makers are key stakeholders ² whose engagement can positively impact all stages of the research process. ^{3 , 4} Nonetheless, multistakeholder engagement is challenging due to logistical difficulties, power imbalances and stakeholders' capacity to participate meaningfully. ⁵

One way to conduct multistakeholder engagement is to convene a Delphi panel. ^{6 , 7 , 8} Delphi‐based methodologies that combine iterative rounds of questions with feedback on intermediary panel results were designed to more objectively develop group consensus. ^{9 , 10} The Delphi method is based on the idea that exposure to alternative perspectives improves the quality of the final responses, which are used to determine the existence of consensus. Delphi‐based methodologies provide a useful approach for measuring whether and how participants' perspectives change. ^{11 , 12}

Modified‐Delphi methodologies that start with a survey, proceed with feedback on and an in‐person, telephone or virtual discussion of initial survey results, and end with participants revising their original survey responses offer stakeholders an opportunity to directly engage with each other, which is absent in traditional Delphi panels. ^{8 , 13 , 14 , 15 , 16} Online modified‐Delphi (OMD) approaches are particularly useful engagement techniques because they allow for large‐scale (50+ participants) anonymous engagement, which is not possible in modified‐Delphi panels that meet in‐person. The requirement of in‐person discussion limits the panel size to 9–20 participants. ^{13 , 14} While the online method has clear benefits, little is known about the contextual factors such as panel composition or topic that might affect the outcomes of multistakeholder engagement.

Research suggests that stakeholder perspectives vary, with patients and clinicians, for example, having different perceptions of research priorities, treatment preferences and harm‐benefit tradeoffs. ^{17 , 18} Although patients' voices may be dominated by clinicians', ¹⁷ true consensus in multistakeholder initiatives may not be achieved without directly exposing stakeholders to the perspectives of other groups. While patients may be more comfortable sharing their perspectives with peers and, therefore, could be more satisfied with engagement in homogeneous panels, participants in mixed panels could change their positions after being exposed to the alternative perspectives, which is key for developing true consensus in multistakeholder panels. ¹⁹ Although it is possible to imagine how the outcomes of a multistakeholder engagement might vary depending on its topic, we are not aware of previous studies that directly addressed this question in the context of modified‐Delphi panels.

This paper advances methods for conducting online multistakeholder panels using a modified‐Delphi approach by exploring the impact of panel composition and topic on stakeholder judgments and uses the results of this analysis to generate empirically grounded hypotheses for future research. We use the propensity and meaningfulness of response changes after stakeholders receive statistical feedback and discuss their original responses with others as a measure of panel impact on individual stakeholder judgments. To reach the study goals, we use the data from three OMD panels that engaged patients and professionals around the severity of maternal and child health outcomes linked to gestational weight gain. ^{20 , 21} We treat outcomes of different levels of severity as proxies for different panel topics. Our findings have practical and methodological implications for assembling multistakeholder panels and contribute to ongoing scholarly debates about the impact of feedback ²² and the nature of consensus‐building in Delphi panels. ^{11 , 12 , 23}

2. METHODS

3. RESULTS

3.4. Model results

Table 3 shows the results of the mixed‐effects logistic regression predicting response changes. Looking at all outcomes shows that patients in the mixed panel (odds ratio [OR] = 1.5, confidence interval [CI] = 0.9–2.3) and professionals in the homogenous panel (OR = 1.4, CI = 0.9–2.1) were about 40%–50% more likely than patients in the homogeneous panel to change their ratings. These differences, however, were only marginally significant and only for patients. Moreover, panel composition was a significant predictor of response changes for medium and low severity outcomes, but not high severity outcomes. For medium severity outcomes, patients in the mixed panel (OR = 2.1, CI = 1.2–3.9) and professionals in the homogeneous panel (OR = 1.7, CI = .9–3.1) were more likely to change their ratings, compared to patients in the homogeneous panel. Moreover, patients and professionals in the mixed panel were more likely than patients in the homogeneous panel to change their answers about low severity outcomes (OR = 2.7, CI = 1.2–6.1 and OR = 1.9, CI = 0.9–3.9, respectively).

Table 3

Results of mixed‐effects logistic regression models predicting response changes

Predictors	All outcomes (N = 1461)	High severity outcomes (N = 528)	Medium severity outcomes (N = 530)	Low severity outcomes (N = 403)
	OR (95% CI)	OR (95% CI)	OR (95% CI)	OR (95% CI)
Patients in a mixed panel	1.5*	0.7	2.1**	2.7**
	(0.9–2.3)	(0.4–1.2)	(1.2–3.9)	(1.2–6.1)
Professionals in a mixed panel	1.1	0.9	0.9	1.9*
	(0.7–1.6)	(0.5–1.7)	(0.4–1.9)	(0.9–4.0)
Professionals in a homogeneous panel	1.4	1.0	1.7*	1.6
	(0.9–2.1)	(0.6–1.9)	(0.9–3.1)	(0.8–3.3)
Study participation was satisfying	0.8	1.2	0.5**	0.8
	(0.5–1.3)	(0.6–2.3)	(0.3–1.0)	(0.4–1.7)
The charts helped me understand how my responses compared to those of other participants	0.7	1.4	0.5	0.4*
	(0.3–1.5)	(0.6–3.7)	(0.2–1.5)	(0.1–1.2)
Round 2 discussion changed my perspective on the study topics	1.4	1.2	1.5	1.6
	(0.9–2.1)	(0.7–2.1)	(0.9–2.6)	(0.8–2.9)

Open in a separate window

Note: Patients in a homogeneous panel are a reference group. We control for demographic characteristics, such as race and age for all models. Models were clustered at the participant level. Coefficients for constant are excluded. Values presented in this table are odds ratios (OR) and robust 95% confidence intervals (CI).

^** p < .05

^* p < .1.

Those satisfied with their participation were less likely than their less satisfied counterparts to change ratings on medium severity outcomes (OR = 0.5, CI = 0.3–1.0), whereas those who felt that charts helped them understand how their responses compared to those of others were less likely to change their ratings on low severity questions (OR = 0.4, CI = 0.1–1.2). We note that small sample sizes led to imprecise estimates.

Figure 1 shows the marginal effects of the logistic regression predicting response changes, which provide additional support to our modelling results. Briefly, patients in the homogeneous panel had the lowest probability of changing their responses (50%) when looking at all outcomes together. Participants had the lowest probability (below 50%) of changing their responses on high severity outcomes. Patients in the mixed panel rating high severity outcomes had the lowest predicted probability of changing their responses (38%), whereas patients in the mixed panel rating low severity outcomes had the highest predicted probability of modifying their responses (72%).

Open in a separate window

Figure 1

Marginal effects of the logistic regression predicting response changes

Table 4 shows the results of the mixed‐effect logistic regression predicting the meaningfulness of response changes. Panel composition was a significant predictor of meaningful response changes only for questions about low severity outcomes. Relative to patients in the homogeneous panel, patients in the mixed panel and professionals in the homogeneous panel were more likely to meaningfully change their answers (OR = 2.5, CI = 1.2–5.3 and OR = 1.9, CI = 0.9–3.8, respectively). The difference between professionals and patients in homogeneous panels was only marginally significant.

Table 4

Results of mixed‐effects logistic regression models predicting meaningful response changes

Predictors	All outcomes (N = 1461)	High severity outcomes (N = 528)	Medium severity outcomes (N = 530)	Low severity outcomes (N = 403)
	OR (95% CI)	OR (95% CI)	OR (95% CI)	OR (95% CI)
Patients in a mixed panel	1.4	0.9	1.4	2.5**
	(0.8–2.3)	(0.4–1.7)	(0.7–2.8)	(1.2–5.3)
Professionals in a mixed panel	0.99	0.7	0.9	1.5
	(0.6–1.6)	(0.4–1.5)	(0.4–2.1)	(0.7–3.4)
Professionals in a homogeneous panel	1.4	1.0	1.6	1.9*
	(0.9–2.2)	(0.5–2.0)	(0.8–3.0)	(0.9–3.8)
Study participation was satisfying	0.7	0.9	0.5*	0.9
	(0.4–1.3)	(0.4–2.3)	(0.3–1.0)	(0.4–2.0)
The charts helped me understand how my responses compared to those of other participants	0.5	1.0	0.5	0.3**
	(0.2–1.2)	(0.4–2.4)	(0.2–1.6)	(0.1–1.0)
Round 2 discussion changed my perspective on the study topics	1.1	0.9	1.2	1.4
	(0.7–1.8)	(0.5–1.8)	(0.7–2.1)	(0.7–2.8)

Open in a separate window

Note: Patients in a homogeneous panel are a reference group. We control for demographic characteristics, such as race and age for all models. Models were clustered at the participant level. Coefficients for constant are excluded. Values presented in this table are odds ratios (OR) and robust 95% confidence intervals (CI).

^** p < .05

^* p < .1.

Although perceived usefulness of charts reduced the likelihood of meaningful response changes on low severity outcomes (OR = 0.3, CI = 0.1–1.0), participation satisfaction made participants marginally less likely to change their responses by 10 or more points on medium severity outcomes (OR = 0.5, CI = 0.3–1.0).

Figure 2 shows the marginal effects of the logistic regression predicting meaningful response changes. As in previous models, patients in the homogeneous panel and professionals in the mixed panel had the lowest probability of changing their responses meaningfully (34% for both groups). Looking across the outcome severity levels, the lowest predicted probability of a response change of 10 or more points was observed for high severity outcomes. Professionals in the mixed panel rating high severity outcomes was the group with the lowest probability of meaningful response changes (22%). Patients in the mixed panel rating low severity outcomes was the group with the highest probability of changing responses meaningfully (54%).

Open in a separate window

Figure 2

Marginal effects of the logistic regression predicting meaningful response changes

4. DISCUSSION

We analysed the impact of panel composition and topic on the presence and meaningfulness of response changes in OMD panels using the data on the severity of maternal and child health outcomes linked to gestational weight gain as a case study. Our results show that professionals and patients rating the seriousness of maternal and child health outcomes changed more than half of their original responses and that the majority of response changes (563 of 818, 69%) were meaningful. This finding suggests that the exposure to and the discussion of the perspective of other participants affect individual judgments about outcome severity.

In contrast to previous research that suggested that personal characteristics of OMD panellists were not associated with response changes, ¹² our results showed that participant background matters and that patterns of response changes are different for pregnant and postpartum women and maternal and child health professionals. It is worth noting, however, that the previous study focused on patient and caregiver panels on different topics, included a different set of participant background measures and did not look at the meaningfulness of change.

Our results also illustrate heterogeneity in the impacts of panel composition on response changes and their meaningfulness based on panel topic, operationalized as outcome severity. While we saw some patterns, we cannot, with certainty, say that certain types of stakeholders are more likely than others to change their responses on certain topics. Nonetheless, our study design provides a unique opportunity to generate hypotheses to be tested in future research.

First, we hypothesize that participants in OMD panels, regardless of their stakeholder group, are more likely to change their responses for certain preference‐sensitive topics, such as those where there is a range of viable alternatives or perspectives. In our study, the likelihood and meaningfulness of response changes were affected by the nature of maternal and child health outcomes considered. Moreover, participants had the lowest probability of changing their responses and doing so meaningfully while rating outcomes deemed ‘severe’ by participants. There is little if any debate that infant death and stillbirth are serious health outcomes. Therefore, it is not surprising that our participants rated these outcomes highly in Round 1 and that their perspectives did not change greatly after the discussion round. At the same time, individual judgments about less severe outcomes, such as gestational diabetes and SGA birth, were more affected by the perspectives of other participants. For example, while some patients may have initially focused more on the short‐term impact of gestational diabetes thinking that it resolves after delivery, exposure to a professional perspective may have brought to the fore concerns about risks of complications during delivery and increased risk of Type 2 diabetes after child‐bearing years.

Second, we hypothesize that, regardless of the panel topic, stakeholders may be more likely to change their responses in different panel types: Patients may be more likely to change their responses and to do so meaningfully in mixed panels, whereas professionals may be more likely to do so in homogeneous panels. The fact that patients generally had the highest probability of changing their responses in the mixed panel may be an illustration of collaborative learning, which takes place in diverse stakeholder panels that focus on important issues. Stakeholders can learn from the perspective of a different group and change their responses based on new ideas they may not have considered. ²⁶ Indeed, patients may be eager to learn not only from the experiences of other patients but also from professionals who have specialized expert knowledge of the topic. To illustrate, engagement with professionals could help patients learn about the potential burden of undergoing treatments for complications caused by a problem that patients may not have otherwise considered severe enough to worry about.

Although unexpected and somewhat methodologically undesirable, professionals in our study were more likely to change their responses and to do so meaningfully in the homogeneous panel. This finding, however, is not too surprising given that shared decision‐making often suffers from ‘selective paternalism’—a situation where healthcare professionals step outside of shared decision‐making to choose a course of action they think would work best for their patient, but that discounts hearing an alternative patient perspective. ²⁷

Third, we hypothesize that the association between panel composition and response change may vary according to the topic. Panel composition may play a bigger role in panels on somewhat less serious or consequential, but still important health topics. In our study, marginal effects of panel composition varied by the outcome type: While patients in the mixed panel were more likely to change their responses and to do so meaningfully on medium and low severity outcomes, the reverse was true for high severity outcomes. Professionals in the homogeneous panel generally had higher probabilities of changing their responses and doing so meaningfully than professionals in the mixed panel. This was true across topics with one exception: Professionals in the mixed panel had a higher probability of changing their responses on low severity outcomes. Although these findings support our hypothesis that patients and professionals learn under different circumstances, it offers an important nuance—mutual learning may happen in diverse panels, but only for certain types of outcomes. While previous research shows that there are no statistically significant differences between patient/caregiver and clinician/research experiences with OMD panels or their willingness to use OMD in the future, ¹⁹ this study suggests that mixed panels may promote mutual learning in multistakeholder panels on certain topics.

Our study has important limitations. Our analysis was limited to three OMDs that engaged pregnant and postpartum women and health professionals on the topic of maternal and child health outcomes. We note that the women who participated in our study were highly educated. Therefore, the patterns of findings may be different in panels that engage different stakeholders and/or focus on other topics. Moreover, not all study participants answered the same questions twice or provided responses to satisfaction questions, which limited our sample size. Nonetheless, attrition is common in OMD panels, and our participation rates were higher than in other panels. ²¹ Finally, this paper relies solely on the rating data and has not looked at how the content of online discussion comments affects response changes. Future research should test our hypotheses in OMD panels conducted with different types of participants and on different topics and analyse the impact of the discussion content on response changes in different panel types.

5. CONCLUSIONS

We recognize that our study cannot provide conclusive answers to our research questions. That is why we consider our study results, and the empirical data it relies on, as the necessary basis for formulating evidence‐informed hypotheses about panel composition and topics that should be tested in future research. Nonetheless, we believe that our results offer a number of practical recommendations, presented in Figure 3, which can help panel designers assess possible threats to achieving valid, reliable panel conclusions and encourage them to consider how panel design considerations may affect their conclusions.

Open in a separate window

Figure 3

Five practical recommendations for online modified‐Delphi organizers

CONFLICT OF INTERESTS

Dmitry Khodyakov is a leader of the ExpertLens team. ExpertLens is a RAND‐developed online modified‐Delphi platform used to collect data in this study.

AUTHOR CONTRIBUTIONS

All authors contributed to the study conception and design. Dmitry Khodyakov led the data collection, designed the analytic procedures and drafted the paper; Sujeong Park conducted all analyses, created all tables and figures and helped draft the paper; Jennifer A. Hutcheon and Lisa M. Bodnar obtained funding, designed the overall study, advised on the analytic procedures, reviewed and revised earlier drafts of the manuscript; Sara M. Parisi participated in the data collection and reviewed and revised earlier drafts of the manuscript. All coauthors reviewed and approved the final version of the manuscript

Supporting information

ACKNOWLEDGEMENTS

Notes

Khodyakov D, Park S, Hutcheon JA, Parisi SM, Bodnar LM. The impact of panel composition and topic on stakeholder perspectives: generating hypotheses from online maternal and child health modified‐Delphi panels. Health Expect. 2022;25:732‐743. 10.1111/hex.13420 [PMC free article] [PubMed] [CrossRef] [Google Scholar]

REFERENCES