Discussion

A O’Mara-Eves; G Brunton; D McDaid; S Oliver; J Kavanagh; F Jamal; T Matosevic; A Harden; J Thomas

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

O’Mara-Eves A, Brunton G, McDaid D, et al. Community engagement to reduce inequalities in health: a systematic review, meta-analysis and economic analysis. Southampton (UK): NIHR Journals Library; 2013 Nov. (Public Health Research, No. 1.4.)

Community engagement to reduce inequalities in health: a systematic review, meta-analysis and economic analysis.

Show details

Contents

< Prev Next >

Chapter 9Discussion

Summary of evidence

Although there are many ways in which terms relating to community engagement are used, they can all be understood as being situated beneath the ‘engagement’ umbrella, moving from terms that suggest limited amounts of engagement (‘information’ and ‘consultation’) towards ‘development’, ‘participation’ and ‘empowerment’. This final concept, ‘empowerment’, has its roots in concerns about social justice and movements promoting social and structural change, and is held as the ideal in models describing continua of community engagement. Although discussed in relation to community engagement to improve health, those advancing empowerment as a strategy often do so from a wider sociopolitical perspective, in which health is only one aspect of change that people’s empowerment can achieve. Critically, true community empowerment needs to begin within the community; this is at odds with many of the studies in this review, which often start from an academic or health service/systems perspective.

A number of models of community engagement have been advanced, which usually suggest that ‘empowerment’ is the ideal form. This is both because it is considered socially desirable and equitable, and because it addresses some of the social determinants of ill health and thus will also result in improved health and reductions in health inequalities. Taking a different strategy, but still rooted in a concern that interventions should be appropriate, accessible and sensitive to the needs of their target population, some studies in this review have evaluated the effectiveness of utilising peers or lay people to deliver the intervention. These studies often do not seek to empower, nor sometimes even to involve communities in intervention design, but achieve a degree of engagement nevertheless; existing models of community engagement often do not encompass this strategy. We therefore conclude that the following may be a useful heuristic to use when thinking about the range of approaches to community engagement:

Theories of change for patient/consumer involvement. This is engagement with communities or members of communities in strategies for service development, in which empowering individuals enhances their engagement with service professionals to effect sustainable changes in services. The need for ongoing investment will depend on the nature of the changes made; ongoing partnership is not necessary for sustaining changes, but can benefit subsequent changes.
Theories of change for peer-/lay-delivered interventions. Services engage communities, or individuals within communities, to deliver interventions. The aim of empowering people by enhancing their skills is to effect sustainable change amongst themselves and their peers. Although the individual behaviour changes sought may be sustainable, the intervention needs ongoing investment from services for subsequent generations.
Theories of empowerment to reduce health inequalities. When people are engaged in a programme of community development, an empowered community is the outcome sought by enhancing their mutual support and their collective action to mobilise resources of their own and from elsewhere to make changes within the community. An empowered community can do much to sustain its own efforts.

We compared the effectiveness of interventions based on these different theories of change in the meta-analysis of effectiveness data (see Chapter 5, RQ3: Which approaches to community engagement are associated with improved health outcomes among disadvantaged groups? How do these approaches lead to improved outcomes?). The results suggest that peer-/lay-delivered interventions tend to have larger effects than interventions based on empowerment or patient/consumer involvement, although this trend did not significantly explain variation in the effectiveness across studies. We propose that this association is likely to be confounded by other factors such as intervention intensity and exposure (peer-/lay-delivered interventions tend to be more intense, one-on-one or small group interventions than other intervention types). For such models, we might expect to see large effects over a narrow range of outcomes, as opposed to empowerment models that might have smaller effects over a broader range of health and social outcomes. Unfortunately, there were insufficient data to adequately test these relations.

Promisingly, overall, our analyses suggest that community engagement interventions for disadvantaged groups are effective in terms of health behaviours, health consequences, participant self-efficacy and perceived social support. A small number of studies also suggest that interventions can improve outcomes for the community and engagees. Only one subgroup – interventions targeted at participants on the basis of disadvantage according to their place of residence (rural or inner city) – did not have a pooled effect that was statistically significantly different from a null effect. This suggests that community engagement interventions work across a wide range of populations and intervention characteristics.

There is evidence from a small group of studies to suggest that intervention effects are still present long after the intervention has finished, although effects generally are smaller than at post test. Moreover, improvements in health behaviours are significantly related to increases in self-efficacy. These findings suggest that benefits occur across different domains and can be sustained, although more evidence is required to determine whether this is accurate across different conditions and contexts.

The meta-analysis also identified trends in the effectiveness of interventions that can be considered when designing future interventions, such as the intervention setting and duration. We were, however, unable to detect any clear predictors or moderators of intervention effectiveness.

Relatively few outcome evaluations are accompanied with robust process evaluations. The process evaluations that were available to synthesise gave a taste of what might have been available had more studies undertaken this type of evaluation alongside their examinations of effectiveness; and we recognise that there will have been other process evaluations that we did not synthesise, because they did not accompany outcome evaluations. We found that the diversity in conceptualisations, definitions and operationalisations of community engagement made it difficult to identify an appropriate data extraction tool for processes.

We found that community-designed or -delivered interventions, or culturally relevant programme materials, were linked to acceptability, which authors suggested influenced programme success. Successful partnerships and efforts to build relationships between partners appear to influence programme outcomes, and paying community members and participants can influence participation. Some coalitions were able to win external funding, helping the programmes to be sustainable beyond initial funding periods; in some cases, uncertainty over long-term funding had adverse impacts on outcomes. Intervention timing, frequency and duration and extent of an intervention influence outcomes. Intervention type (e.g. media events as opposed to one-to-one counselling) can affect accessibility or ‘reach’. Good project management and specific, adequate, ongoing training and support for engagees impact on implementation, and good relationships between engagees and professionals providing an intervention are important for programme implementation.

We examined 210 papers that were identified as having some discussion of economic issues,⁷¹^–⁷⁵^,⁷⁸^–⁸⁰^,⁸³^,⁸⁴^,⁸⁶^–⁸⁹^,⁹²^,⁹⁵^,⁹⁷^–⁹⁹^,¹⁰¹^,¹⁰⁴^–¹⁰⁸^,¹¹⁰^–¹¹⁴^,¹¹⁶^–¹²¹^,¹²³^,¹²⁴^,¹²⁷^–¹³⁶^,¹³⁹^–¹⁴²^,¹⁴⁴^–¹⁵¹^,¹⁵³^,¹⁵⁵^–¹⁶⁴^,¹⁶⁷^–¹⁶⁹^,¹⁷¹^–¹⁸⁴^,¹⁸⁶^,¹⁸⁸^–¹⁹²^,¹⁹⁵^,¹⁹⁷^,¹⁹⁸^,²⁰¹^,²⁰⁴^,²⁰⁷^,²⁰⁸^,²¹⁸^,²²⁰^,²²¹^,²²³^–²²⁶^,²²⁸^,²³⁰^–²³²^,²³⁹^,²⁴⁰^,²⁴²^–²⁴⁵^,²⁴⁸^,²⁵²^,²⁵³^,²⁵⁸^,²⁶¹^,²⁶⁶^,²⁷²^,²⁷³^,²⁷⁹^–²⁸²^,²⁸⁴^,²⁸⁶^,²⁹⁴^,³⁰⁰^,³⁰²^,³⁰⁸^,³¹¹^,³¹²^,³¹⁸^–³²³^,³²⁹^,³³²^,³³³^,³³⁵^,³³⁸^–³⁴⁰^,³⁴³^–³⁵⁷^,³⁵⁹^,³⁶¹^–³⁶⁵^,³⁶⁷^–³⁶⁹^,³⁷²^,³⁷⁵^,³⁷⁷^,³⁷⁸^,³⁸⁰^,³⁸¹^,³⁸⁴^–³⁹³^,³⁹⁷^,³⁹⁹^,⁴⁰⁰^,⁴⁰⁴^,⁴⁰⁵^,⁴⁰⁷^,⁴¹⁰^,⁴¹²^,⁴¹⁴^,⁴¹⁶^,⁴¹⁹^,⁴²⁰^,⁴²⁶^,⁴²⁷^,⁴⁸⁸^–⁴⁹³ although in many instances this was limited to a cursory mention of costs or a call for subsequent economic evaluation. In total, 58 studies or linked papers included some substantive discussion of costs⁷³^,⁷⁹^,⁹²^,⁹⁵^,⁹⁸^,¹⁰¹^,¹¹⁰^,¹¹¹^,¹¹⁶^,¹¹⁹^,¹²⁴^,¹²⁹^,¹³⁶^,¹³⁹^,¹⁴⁰^,¹⁴²^,¹⁶²^,¹⁶⁴^,¹⁶⁹^,¹⁷²^,¹⁷⁶^–¹⁷⁸^,¹⁸⁶^,²⁰¹^,²⁰⁴^,²⁰⁷^,²⁰⁸^,²²⁰^,²²³^,²²⁴^,²²⁶^,²³⁰^,²³⁹^,²⁴⁸^,²⁵²^,²⁵³^,²⁵⁸^,²⁶¹^,²⁸⁴^,³⁰²^,³²⁰^,³²⁹^,³⁴⁰^,³⁴⁶^,³⁵¹^,³⁵⁵^–³⁵⁷^,³⁵⁹^,³⁶⁵^,³⁷²^,³⁷⁸^,³⁸⁰^,³⁹³^,⁴⁰⁵^,⁴²⁰^,⁴²⁶ and 38 provided some breakdown of the resources needed to implement interventions,⁸⁰^,⁹⁵^,¹¹¹^,¹²^,¹¹⁶^,¹²⁸^,¹²⁹^,¹³⁶^,¹³⁹^,¹⁶²^,¹⁶⁴^,¹⁸²^,²⁰⁷^,²²⁰^,²²²^,²²³^,²³¹^,²³²^,²³⁹^,²⁶¹^,²⁸⁴^,³⁰⁰^,³⁰²^,³¹¹^,³¹⁹^,³²⁰^,³²²^,³²⁹^,³⁴⁰^,³⁴⁴^,³⁵¹^,³⁶¹^,³⁶³^,³⁸⁰^,³⁸⁴^,³⁹⁹^,⁴⁰⁰^,⁴⁰⁴ although in many cases this was partial information, often focusing on training and interaction components of community engagement strategies but typically not providing information on the time commitments of personnel, lay health workers and other volunteers. A total of 21 economic evaluations were identified, either as standalone documents in our review database or reported in other papers.¹¹⁶^,¹³⁶^,¹⁴⁰^,¹⁶²^,¹⁶⁴^,²⁰¹^,²²⁰^,²²³^,²²⁴^,²²⁶^,²⁶¹^,³²⁰^,³⁴⁰^,³⁵⁵^,³⁶⁵^,³⁷²^,³⁷⁸^,³⁹³^,⁴⁹⁸^,⁴⁹⁹^,⁵⁰⁷^,⁵¹⁰ Only eight of these studies were identified as being potentially relevant via NHS EED.¹³⁶^,²⁰¹^,³⁴⁰^,³⁷²^,³⁷⁸^,³⁹³^,⁴⁹⁸^,⁵⁰⁷

The evaluations are thinly spread across health topic areas and concentrate on traditional peer-/lay-delivered projects, which are considerably easier to evaluate because of their focus on individual interventions. Only three looked at empowerment-centred models.¹³⁶^,³²⁰^,³⁵⁵ Some individual studies suggest a good economic case for specific community engagement interventions, for instance the Expert Patient Programme in England for which the cost per QALY gained would fall below £20,000 in nearly all circumstances.³⁷⁸ Looking at models centred on empowerment, an economic analysis focused not on health outcomes but on a reduction in crime reported strong net benefits from the Neighbourhood Warden Scheme in England.³⁵⁵

However, the limited number of economic studies identified, in many cases with significant quality concerns, means that it is impossible to reach any general conclusion on the cost-effectiveness of community engagement models per se. Moreover, community engagement is not often evaluated as an ‘adjuvant’ mechanism to usual means of engagement; thus, it is difficult to isolate the costs and effectiveness of a community engagement component as distinct from other aspects of the intervention. Careful thought needs also to be given to the appropriateness of the comparator used, particularly given the dominance of US literature in our review.

The contribution of this study

How this review contributes to knowledge

The ability to compare different models of community engagement and their underlying theories of change is a major contribution of this work. In Chapter 4 we outlined a variety of models that can broadly be classified as having utilitarian (health systems) and social justice (ideological) rationales. To the best of our knowledge, no systematic review has synthesised evidence representing such a broad spectrum of community engagement models that span the utilitarian–social justice divide. This has enabled us to directly compare the effectiveness of different models, consider the implementation and resource implications of different models and ultimately produce a conceptual framework that embraces the diversity of approaches. Importantly, this allows us to consider whether different community engagement approaches might be more effective under different circumstances, rather than constraining our thinking to models that conform to specific underlying theories.

Several process evaluations noted that the quality of the relationship and power sharing between partners influences a programme’s success. Successful partnerships were those that are able to both sustain projects beyond the end of a particular programme and recognise that they had served their purpose and come to a natural end. Swainston and Summerbell³ reviewed the effectiveness of community engagement approaches and methods for health promotion interventions. They described a range of specific approaches suitable for health promotion and, like Popay et al.,⁴ noted that power struggles between government, providers and community as well as project devolvement and short-term funding and lack of appropriate space were barriers to community engagement. The authors of these systematic reviews have acknowledged that the strength of their findings is limited because of the small numbers of, and a lack of rigour in, included studies.

In terms of costs, both our review of process evaluations and economic analysis of cost and resource data suggest that financial recognition of participants’ and engagees’ time helps. Other study authors noted that engagees could save government service costs by advocating and intervening with and on behalf of participants. Further, efforts to seek external funding by engagees may have extended programme existence or resulted in their mainstreaming to statutory services. Conversely, a lack of adequate funding identified in some studies limited evaluation efforts.

The effect of implementation varies: the timing, frequency, duration and extent of an intervention are all suggested to influence intervention outcomes. Agency philosophy may influence implementation itself. In other situations, interventions implemented more intensively (i.e. one-to-one vs. many people at once) are effective but have less ‘reach’. The available data allowed only the analysis of the impact of duration on outcome effect sizes in the meta-analysis (see Chapter 5, Duration of intervention).

Good management and support of staff, especially community engagees who train to provide an intervention, impacts on programme success. Specific, comprehensive and ongoing training of community engagees who provide the intervention contributes to programme success, as does engaging similar, empathetic, supportive community members. Further, the accessibility of a programme is influenced by the nature and extent of the interactions between community members (‘engagees’) and participants (i.e. who is interacting, how and where). South et al.⁵³² undertook a mixed-method evaluation of lay health worker involvement in public health roles. The importance of training issues, support and retention schemes was described, as was the idea that participants could benefit directly from the process, as well as the populations they represent benefiting indirectly. The relationships between lay workers and professionals were highlighted as an important factor in mediating the success of a strategy, as was the relationship between the community participant and the community he or she represents.

Our review of the economic literature adds considerably to previous reviews of economic evaluations in this area. We have been able to collate these evaluations by type of community engagement, which may aid in future analysis of their strengths, weaknesses and potential adaptation to different contexts and settings. However, only 10% of studies included an analysis of both incremental costs and effectiveness. At first glance this may appear surprising, given the growing importance of economic arguments in public health policy, of which there has been a substantive growth in economic literature in recent years.⁵³⁵ However, this is consistent with many of the reviews commissioned by NICE on the economic evidence for specific public health interventions, which typically identify a modest body of economic literature. A key observation is that low priority appears to be given to collecting resource and cost data as part of evaluations. When looking at actions that may place great weight on unpaid inputs, it is important to assess their opportunity cost, something that still is rarely carried out. This can have a major impact on reported cost-effectiveness ratios. There was little incorporation of equity concerns into economic evaluations, for instance by considering cost-effectiveness from the perspective of population subgroups.

Terminological challenges

As many authors have observed, the concept of ‘community engagement’ suffers from a bewilderingly large number of inconsistent and partially conflicting definitions.¹⁴^,⁴⁴⁸^,⁴⁵⁷ We have not redefined these, nor added a new one to the already extensive catalogue; rather, we have sought to understand the rationales behind some of the more significant definitions and what they mean in practice, and to characterise their differences in terms of their different theories of change. This report will, we hope, complement existing definitions by helping readers to understand what those differences mean in terms of how health outcomes are to be achieved. Our aim is to aid future evaluations and evidence syntheses by suggesting that, rather than focusing on the overarching heterogeneous concept of community engagement, we may be better served by identifying the key characteristics of interventions and how these relate to their underpinning theories of change.

Comparing different models for reducing inequalities in health

The variety of intervention strategies reviewed here speaks to debates about how best to reduce inequalities in health.⁴^,¹⁵ Many interventions have been shown to improve the health of disadvantaged groups. Those with the largest effect size estimates tend to be those that are targeted at a narrow range of health outcomes and which typically employ a peer/lay delivery approach to community engagement. Those interventions that have taken an empowerment approach – and consequently aim to improve a broader range of outcomes, which may include benefits for the engagees and the community – tend to exhibit smaller effect size estimates.

However, we should not then conclude that the simpler, more specific interventions should be recommended over the others. The significant correlation between self-efficacy and behaviour change shown in Chapter 5 (see Correlation between outcome types), plus the positive impacts on other outcomes, including social support, engagee outcomes and community outcomes, identified in the evidence base, are important to understanding different models for reducing health inequalities. This is because these findings support the view that community engagement interventions may be able to impact on a range of outcomes – not just health behaviours and health consequences. If theories on the ‘virtuous circle’ are accurate, then such changes become mutually reinforcing.

To this end, we argue that the impact of an intervention should not be understood as simply being summarised in one or two effect size estimates; rather, the benefits of an intervention should be considered in terms of the combined effects across a range of relevant outcomes. The total effect of an intervention – which considers the effect on direct and indirect beneficiaries, on proximal and distal outcomes and on primary and secondary outcomes – might be useful for comparing different interventions that take different approaches. It is possible that the multiple smaller benefits of empowerment models might be as meaningful as one larger benefit from peer-/lay-delivered models, although this remains to be tested.

Reducing gaps and gradients

The proposed social gradient in community health characteristics²⁸ might mean that more disadvantaged communities benefit less from a community engagement intervention than other, less disadvantaged communities. It is analogous to the old saying that ‘the rich get richer while the poor get poorer’. Following from this, there is arguably a danger that a universal community engagement intervention may increase inequalities in health by having a proportionately greater impact on more advantaged people.

In the meta-analysis (see Chapter 5) we tested whether there was a systematic difference in the effectiveness observed for universal compared with targeted interventions. As a consequence of our inclusion criteria, the ‘universal interventions’ in this review have been delivered to a sample of predominantly people from any of the PROGRESS-Plus categories of inequality (i.e. at least 60% of the sample). The difference between ‘universal’ and ‘targeted’ interventions in this review lies in the explicit focus of the authors of the original studies on targeting a particular PROGRESS-Plus group, compared with delivering an intervention to a sample that just happens to be mostly constituted of people from a PROGRESS-Plus categorisation. This distinction – which was driven by our decision to include only studies with a majority of participants from any PROGRESS-Plus group – means that we are unable to satisfactorily address the issue of targeted compared with universal interventions in its strictest sense. In essence, our analysis in Chapter 5 tested whether interventions with an intended aim of targeting the population are similarly as effective as interventions with a more general aim, regardless of the actual composition of the sample/population. The findings suggest a (statistically non-significant) trend towards interventions with universal aims being more effective than interventions aimed at targeted PROGRESS-Plus groups. This gives us some clues about reducing the social gradient, but it does not address the issue directly.

We are therefore unable to address the dilemma articulated in the Marmot report²⁸ and outlined in Chapter 1:

If the focus were on the very bottom and social action were successful in improving the plight of the worst-off, what would happen to those just above the bottom, or at the median, who have worse health than those above them?
p. 16

This is not to say that the review does not have anything to contribute to the debate about gaps and gradients. As we sought evaluations of interventions that aimed to improve the health of disadvantaged groups, and found ample evidence that it is possible to impact on a wide range of outcomes, it follows that such interventions that are targeted at those with the poorest outcomes will be able to reduce gaps and gradients.

Interventions to achieve universal improvements in health are required, but ideally they should have a disproportionately large impact on those currently with the poorest outcomes. A recent rapid review of reviews on public health interventions (not specifically involving community engagement) found that certain types of interventions show some evidence of increasing inequalities between socioeconomic status groups (e.g. media campaigns, workplace smoking bans) whereas others decreased health inequalities (e.g. structural workplace interventions, provision of resources). Exploring this further with a focus on community engagement public health interventions would help to address the gaps and gradients dilemma.⁵³⁶

Methodological discussion

This type of meta-epidemiological work is challenging from a methodological perspective. Most systematic reviews aim for a certain amount of homogeneity in order to combine like with like. Such reviews are appropriate to answer specific questions about the balance of benefit and harm attributable to a given intervention, but can answer only fairly narrow questions. In this review, we were asking a very broad question, but one of interest to policy, practice and research. Heterogeneity in its broadest sense is a given – both conceptual and statistical – as the aim is to understand how different approaches to community engagement (itself a heterogeneous concept) work in different situations.

This type of heterogeneity changes the nature of the questions that can be asked (and answered). So instead of asking whether a specific intervention works with a specific population and outcome, we are able to examine the outcomes of different types of strategy of engagement across a range of outcomes (and subject areas). This is a potentially powerful use of research synthesis, although it does have inherent challenges and limitations that are explored below.

The novel search strategy

We adopted an unusual approach to searching that differs from that of other systematic reviews in terms of its emphasis, as we prioritised a different location to identify eligible studies. Searches for relevant studies for inclusion in systematic reviews usually proceed along familiar lines, with sensitive, exhaustive database searches carried out, many thousands of titles and abstracts screened for potential relevance, a few hundred (or sometimes more) papers retrieved for full-text appraisal and the final number of included studies being selected once the full texts of papers have been checked. When other, related systematic reviews are found, they are usually used as a source of primary studies. This process is often supplemented with citation checking, author and expert contact and other searches for ‘grey’ literature (e.g. using Google or specific relevant websites).

We took a different approach in this review because previous experience has shown us that it is often very difficult, if not impossible, to consistently, systematically and reliably identify community engagement approaches and PROGRESS-Plus groups on the basis of study abstracts; therefore, the full text of nearly all citations retrieved through database searching would need to be retrieved to check whether they were suitable for inclusion. This is simply because (as discussed in earlier chapters) both concepts encompass a wide range of terms and, in the case of community engagement, the extent of community involvement may not be clear from the abstract of a paper.

Bearing in mind the above problems of using the titles and abstracts of studies to identify PROGRESS-Plus populations and community engagement approaches, we therefore decided to utilise existing systematic reviews in a way that is unusual, at least in our experience. As systematic reviews usually contain detailed and structured summaries of the studies that they include (often in the form of extensive tables in appendices), we took the view that these summaries would be a useful source of potentially relevant studies, and would often contain more pertinent information for our purposes than is present in many abstracts. As is detailed in Appendix 1, we therefore cast a very wide net for systematic reviews and then retrieved the full text of these and screened their included studies for studies that were potentially relevant to us. We also supplemented this search within reviews with other searches detailed in Chapter 2.

This strategy was far more successful than we had expected. Bearing in mind our somewhat stringent eligibility criteria – that studies needed to have evaluated, using a controlled trial, a community engagement intervention in a PROGRESS-Plus group – we had expected our task to be akin to finding a needle in the proverbial haystack. This assumption was supported by the observation that previous reviews of community engagement initiatives have not found many studies that have a focus on disadvantaged groups and report health or community engagement outcomes³^,⁴^,²⁵ (it should be noted that these reviews focused on social determinants of health rather than disadvantaged populations to explore the issue of health inequalities, and so are indicative, rather than directly comparable). In contrast, we found hundreds of relevant studies. The eligibility of many of the studies that we have included could not have been ascertained from their abstracts – and, possibly more importantly, the non-eligibility of most studies could not have been established using this method either. We therefore conclude that we would not have been able to find the range of studies that we did using traditional searching methods [unless we had identified and retrieved many thousands (possibly 10,000+) full-text reports]. This approach has also allowed us methodically to examine studies in a consistent way to identify whether they contain relevant economic data or citations to supplementary economic analyses. One limitation, however, is that we may have missed useful information contained in modelling studies or in individual studies that somehow were not included in a systematic review. By supplementing the search of reviews with a search of the TRoPHI database of trials, we hoped to minimise this risk, but we acknowledge that some studies might have been missed. (As an indication of how difficult it is to construct a standard search for community engagement – without attempting to identify PROGRESS-Plus groups – only eight¹⁷⁹^,¹⁹⁸^,²³⁵^,²⁷⁶^,³³⁵^,³⁴⁹^,³⁸⁰^,³⁹¹ of the 361 papers in the map used these terms in their title or abstract: ‘community participation’ or ‘community engagement’ or ‘community empowerment’ or ‘community mobilisation’ or ‘community partnership’. We therefore recommend that future systematic reviews adopt similar search strategies for complex areas such as this.)

Epistemology

This was a challenging project to tackle conceptually, as we had to both aggregate and configure⁵⁸ concepts and findings from a diverse pool of studies and analyse them across a number of different, and at times competing, dimensions. The development of our conceptual framework was both a finding of and a solution to some of the epistemological challenges posed by our attempts to understand such an uneven field in a coherent way. For example, arguments put forward to promote empowerment begin in a different place from those on the utility of peer-/lay-delivered interventions; and there is a considerable difference between the theory around empowerment and the ways in which this principle is operationalised in practice. To some extent this is inevitable, given the range of studies that we included. Most controlled trials of community engagement interventions are designed by academics who are interested in testing a particular hypothesis; they are usually unable to engage in the sort of community development/empowerment programmes that are called for in the theoretical literature.

Issues in assessing process evaluations

We identified some challenges in our methods of assessing the process evaluations. Although we found using a data extraction tool for community engagement methods by Popay et al.⁴ excellent for identifying process issues around power, ownership/responsibility and practices of engagement, it was less able to capture simpler issues relating to, for example, public participation or intervention content. Turning to our own process evaluation data extraction tool that had been previously developed for public health intervention evaluations (see Appendix 4), we found that, although this captured data on methodological rigour, it focused more generally on process issues and did not always capture the complexities of processes. For example, embedding cultural values in an intervention may be partly about acceptability, partly about consultation and collaboration and partly about the quality of the programme materials. We believe that these challenges reflect the wider epistemological tensions in looking at community engagement, which also appear to subsequently influence evaluation methods.

We also noted that very few of the interventions involved the community in evaluation planning, conduct or dissemination. Thus, we are unable to say whether interventions are effective in terms valued by community members.

In general, process evaluations undertaken by study authors were of low to medium quality in terms of their efforts to reduce the risk of bias and their usefulness to this review. A clear link to outcomes was not always seen. For example, authors could report high participation rates but did not clearly link this finding to the programme’s success (or its lack of success). Further, authors appeared to go beyond the findings when drawing conclusions about effectiveness, for example when authors found that staff turnover was low they concluded that this was because of management practices but did not directly measure community engagees’ or participants’ opinions about those specific management practices.

More can be done to examine the impacts of financial incentives on the success of community engagement strategies, as well as the impacts of different levels of payment for peers involved in delivering community engagement interventions. It may also be helpful to use data from process evaluations to look at how cash flow and issues of long-term sustainability may impact on the relative effectiveness of interventions. It may be the case that an effective community engagement mechanism in fact fails because of some of these contextual issues.

Issues in assessing costing, resource and economic evaluations

We have noted that costs, when reported, are often not disaggregated, meaning that is difficult to disentangle the costs of the community engagement elements of an intervention from the costs of all other aspects of the intervention. It was also noticeable that few studies distinguished between the costs of conducting research and the routine costs of delivering a community engagement intervention.

When studies do report costs, most concentrate on costs of training and staff time, with little attention paid to the contribution of unpaid individuals or to in-kind contributions. It is important for both costing analyses and economic evaluations to measure all of the resource use associated with any intervention. Volunteer time is not a free resource, even if individuals are unpaid. The opportunity costs of volunteering should be consistently recognised in all studies so that more meaningful comparisons can be made. One potential way in which to capture some of these data may be through the use of self-report logs for volunteers that record not only time spent engaging with end users but also time spent in empowerment-related actions and/or in deliberations with other stakeholders. Tools do exist for this purpose, as highlighted in the study by Andersen et al.,⁴⁷⁹ which went to great lengths to accurately value the contributions of volunteers and other in-kind contributions to an initiative to increase the uptake of breast cancer screening. The same can also be said of measuring the opportunity costs of the time of communities more generally involved in the development and sustainability of actions. This is not an easy issue to deal with but the intensity of a community’s contribution may have a substantial bearing on overall impact and ideally it needs to be recorded and valued.

Few of our included economic studies included any form of modelling, an approach that can be helpful in assessing some of the long-term costs and benefits of actions. This can be of critical importance in public health in which the health consequences of lifestyles and health behaviours may take many years to be seen. Such models could also be used to place a value on some of the long-term benefits that might be realised outside the health sector, for example the benefits to the economy of improved educational performance in children, which were measured in one study of an Experience Corp working with school-age children.⁴⁸¹ It is likely, however, that we have missed some economic analyses that make use of simulation models bringing together data from different effectiveness studies and attaching costs, but we did not identify many such studies within NHS EED.

There is some scope for modelling work retrospectively synthesising cost information with effectiveness data, but again a challenge here is isolating the community engagement element of interventions. Community engagement is not often evaluated as an ‘adjuvant’ to existing interventions. Both the broadness of the definition of community engagement and the need for the synthesis of effectiveness data to focus on general community engagement concepts rather than individual interventions, coupled with the limited amount of costing data we have been able to identify, has meant that we moved away from our initial intention to model the potential cost-effectiveness of specific community engagement mechanisms as part of this review.

Another important issue for future work is to take more account of the implications for equity in economic evaluations of community engagement interventions. Subgroup analysis of the cost-effectiveness of health-promoting interventions has been recommended as a minimum in this respect.⁵³

One fundamental conclusion from our economic analysis is the need to routinely collect data not only on costs but also on resource use, including those resources required to initiate, develop and potentially sustain funding and support for an effective intervention, as part of any evaluation process. This will allow for economic evaluation to be conducted, even if this is carried out retrospectively. A second fundamental research recommendation therefore is to encourage evaluation funders to build in economic analysis into their terms of reference. This is also fundamental in helping to strengthen the case to policy-makers to invest in any intervention. Modelling can be carried out only on a case-by-case basis rather than for a generic community engagement mechanism.

Issues in interpreting statistical findings

Significant statistical heterogeneity was expected in this review and indeed the exploration of this heterogeneity was part of its design. When operating across such a wide range of topics, populations and intervention approaches, however, there is a disjunction between the conceptual heterogeneity implied by asking broad questions and the methods for analysing statistical variance that are in our ‘toolbox’ for answering them.

First, analysing the variance ‘explained’ by specific subgroups of studies according to our conceptual framework rarely reached accepted standards for statistical significance. This is inevitable, however, because conceptual homogeneity was never achieved through such a subdivision: each type of approach to engagement was observed across populations, topics, outcomes and a wide range of other unknown variables; we would therefore never reach the position of being able to say that the studies within a given subgroup differed only because of sampling error/variance (or that any of our subdivisions was the only way of partitioning the studies present). In other words, potential confounding variables or interactions amongst variables made it difficult to disentangle unique sources of variance across the studies. Second, the use of statistical significance testing in meta-analysis has itself been questioned as lacking a sound statistical basis.⁵³⁷^,⁵³⁸ Although defending the practice, Mark Lipsey⁵³⁹ states that the magnitude of effect size estimates should be given greater weight in meta-analysis than the results of tests for statistical significance (and observes that if such statistical testing is wrong for meta-analysis, then it is almost certainly incorrect for most social scientific research).

In the context of our analysis these debates have a clear relevance, because statistical tests for significance are unlikely to yield statistically significant findings because of complex heterogeneity in the data set. We are therefore left with an interpretive challenge: do we adhere strictly to the p > 0.05 convention before accepting that a given subgroup analysis is meaningful?; or do we place more importance on the magnitude of the differences in effect size estimates between subgroups? In this review, we have attempted to plot a path somewhere between the two extremes. We have tested and reported statistical significance, but have also drawn tentative conclusions from the directions and magnitudes of effects whether or not standard statistical significance had been achieved.

A further issue for the meta-analysis in this review relates to the comparators used in the evaluations. In the vast majority of interventions synthesised in the meta-analysis (118 out of 131, 90%), interventions were compared with a comparison condition that differed from the intervention in more ways than just community engagement. For example, the comparison condition might be a completely different intervention, or a waitlist/delayed treatment control condition. In contrast, a study in which the only difference between the treatment conditions was the presence or absence of community engagement might, for example, compare peer-led with non-peer-led health education using the same programme materials. The lack of a ‘pure’ comparator in most community engagement interventions in this review could cloud our interpretation of the findings. Although we conducted a sensitivity analysis of this issue in Chapter 5 (see Sensitivity analysis) and found no difference between studies with ‘pure’ comparators and studies with contaminated comparators, we are unable to conclude definitively that community engagement is the ingredient necessary for intervention success. More evaluations are required in which community engagement is the only difference between comparison conditions to determine the added value of community engagement.

Publication and evaluation bias

The literature utilised in this systematic review is necessarily partial and contingent on many factors that are outside our control. For example, previous research has shown us that research that has statistically significant ‘positive’ results is:

more likely to be published (publication bias)
more likely to be published rapidly (time lag bias)
more likely to be published in English (language bias)
more likely to be published more than once (multiple publication bias)
more likely to be cited by others (citation bias).⁵⁴⁰

Together, these biases mean that there is a danger that any systematic review gives a distorted view of research carried out because there is a systematic bias in favour of identifying positive findings. For this reason, systematic reviews include ‘grey’ literature and carry out exhaustive searches to find the less well-known research studies (and, by implication, possibly those that have less positive results). Even though this review has benefited from the extensive searches carried out in a large number of other systematic reviews, has conducted its own sensitive searches and has carried out statistical tests for publication bias (see Chapter 5), there is no way of knowing for certain the extent to which it has suffered from publication bias. This is a ‘known unknown’, and we can note only that its existence is possible but unquantifiable.

We know a little more about other biases that may be affecting this review, in particular what we will term evaluation bias. Some of the largest and best-known community engagement evaluations have been discussed in the conceptual framework in Chapter 4, but are absent from the map and subsequent analyses because of the design of their evaluations (which did not meet the inclusion criteria for this review). This is a common problem when reviewing interventions that may be implemented at the community level, as their evaluations sometimes do not have an adequate counterfactual, and so conclusions regarding causality are open to question. Because we wanted to ensure that the statistical analyses that we carried out were defensible, we adhered to standard practice in this regard and included only evaluations with a control group (and our checks for other risks of bias are detailed in Chapter 5). We ensured, however, that the theoretical bases of these interventions were not lost from the review by including them in the development of our conceptual framework.

An additional source of evaluation bias may be the synthesis of process evaluations (see Chapter 5). This was a small subset (< 10%) of the studies in the meta-analysis, the only ones with an integral process evaluation. Although an analysis of processes is a vital and important component in any evaluation, the fact that such a small number included a process evaluation means that there may be other important process measures that were not present in these studies. Further, it is difficult to estimate whether the studies with reported process evaluations are representative of the other studies in the meta-analysis. There is a risk that the interventions covered by these process evaluations are not the most significant, interesting or generalisable, but are simply those that happened to have sufficient resource for, and priority given to, an evaluation of their processes. To some extent we have mitigated any bias by including a discussion of the synthesis of relevant process evaluations carried out elsewhere.⁴

Reflections on our position as reviewers

The discussion on empowerment in Chapter 4 leads inexorably to our questioning of our position in this research, as our conceptual framework – and the judgements we make regarding what is within the scope of our review – are inextricably linked to our take on what community engagement is (or should be) and, in a related way, what research is (or should be). If Woodall et al.⁴⁶³ are correct in their argument that community empowerment risks losing its radical edge through association and elision with its related terms, does our review to some extent contribute to this? Moreover, does our broad understanding of engagement – encompassing empowerment, minimally participatory approaches and peer/lay delivery of interventions – weaken the arguments advanced by those promoting empowerment as the ultimate form of engagement?

Such questions relate to how we see the purpose of research. Some have argued that neutral research does not and cannot exist; and that research should be openly ideological, pursuing political, ethical or aesthetic goals (for a discussion of this see reference ⁵⁴¹). Although we may have sympathy with some of these aims, we feel that they seek to redefine what research is: the pursuit of humanly useful knowledge. For example, the issue of inequalities in health did not achieve its current prominence simply through political lobbying; it required a strong evidence base that could not be dismissed as simply being ideologically driven. In the same way, those making decisions to reduce such inequalities need to be informed by an evidence base that helps them to develop policies and implement programmes that stand the best chance of achieving their aims. Thus, rather than advocating for particular community development approaches, or for public health approaches that engage communities in strategy or delivery, we advocate strengthening the evidence, whatever the purpose or approach for engaging communities, with that evidence being framed in terms valued by the communities and people seeking to work with them.

Strengths and limitations of this review

Strengths

To our knowledge this is the only review to have examined systematically the theory, practice, outcomes and economics of using community engagement to improve the health of disadvantaged groups. The studies it includes were identified as the result of a careful, unbiased and systematic search strategy, which gives us confidence that the conclusions we have drawn reflect the state of knowledge in this area accurately.

We were surprised at the consistency of effects we found across topic areas and intervention approaches. Although this made it more difficult than we had expected to draw conclusions about particularly promising approaches, the breadth of our review gives us confidence in drawing conclusions about the relative effects of interventions amongst disadvantaged groups.

Limitations

The evidence base that we drew on

Although based on extensive and rigorous searches, we cannot rule out the possibility that relevant research was missed, either because of deficiencies in our searches or because of publication bias; to some extent this is a weakness inherent in all review work and is a limitation of unknown size or impact. We are also aware that, because we prioritised evaluations that enabled us to ascertain the balance of benefit and harm that might be ascribed to a given intervention, that is, controlled trials, the range of interventions that we could include resulted in some of the largest evaluations being excluded. Again, the impact of this limitation is difficult to ascertain, as without an adequate counterfactual it is hard to identify causal effects (see also Chapter 5, Sensitivity analysis and Risk of bias across studies).

It is also important to note the date and language criteria that we applied when deciding which studies to include. The cut-off date for inclusion was set at 1990. We wanted to ensure the currency of the evidence base (evaluation methods and definitions of health inequalities and engagement have changed much over recent years), and the specific date of 1990 was set to be in line with previous relevant reviews (e.g. Popay et al.⁴). It is difficult to assess the extent to which we missed out on relevant evidence because of this date limitation, but we believe that the other topic and methodological limiters are likely to have led to the exclusion of much of the pre-1990 evidence anyway.

Language was set to English only; this was a pragmatic, resource-driven decision. Only four reviews were excluded on the basis of language, and no primary studies were excluded on this basis. Although it is possible that relevant foreign-language evidence was missed, it is unlikely that the small number of missed studies would change the findings of the review; a methodological evaluation of language bias in meta-analysis concluded that, across 303 meta-analyses, ‘excluding trials published in languages other than English has generally little effect on summary treatment effect estimates’ (p. 115).⁵⁴² In summary, although date and language limiters should be emphasised when considering the generalisability of the findings, we have no reason to believe that they would systematically affect the evidence base in such a way as to alter the conclusions of the report.

Deviations from the protocol

We deviated from our protocol in five ways. These deviations were due to the surprisingly large number of relevant studies that our search strategy identified. As well as changing some of the points of consultation, this meant that the analytical strategy needed to change slightly.

First, we planned to contact authors of the studies included in the review to gain data on implementation issues. We did not do this for two reasons (although we were in touch with some authors): first, there was not the time available to track down and contact authors from 361 papers, most of whom were based in the USA, and so any effort we did spend on this would have been only partial and would risk not being representative of our data set; and, second, a large number of studies presented process information – some informally – which informed our thinking, and there were sufficient formal process evaluations for us to identify the barriers and facilitators reported in Chapter 6. In addition, we spoke to several people who are actively involved in implementing community engagement interventions (including, but not limited to, members of the project advisory group), and so the practitioner perspective has been consulted and incorporated in this review.

Second, we planned to examine ‘exemplars’ of effective approaches to consider relative resource requirements in the economic analysis. In the event, we examined a far greater number of studies than originally planned. We did this for two reasons. First, it quickly became clear that the range of approaches to community engagement found meant that the number of exemplars would be either too small to represent the data set or too large. Second, and as reported in Chapter 7, the proportion of studies reporting useful resource information was very small; we therefore took the view that it would be better to examine as many studies as possible to make good this deficit.

Although the level of data identified on implementation challenges and resources used for community engagement would have been bolstered through these exemplars, this would not have fundamentally changed our conclusions given the heterogeneity of interventions identified in this review. The review reinforces the importance of prospectively collecting data on resource use and costs alongside analysis of effectiveness. As time passes it becomes ever more difficult to retrospectively obtain accurate information on resource use, particularly for those more ‘hidden’ contributions such as time spent building informal networks and local political/administrative support to implement and sustain an intervention. We aim to conduct follow-up work in which we will collect some information on resource use in both comparable ongoing UK-based projects and those that have been recently completed. There are also potential opportunities, which have arisen out of workshops and meetings with the project advisory group, to liaise with other ongoing initiatives looking at community engagement.

Third, our protocol stated that we would assess the risk of bias in outcome evaluations using a tool developed in previous EPPI-Centre reviews.⁵⁵ This tool predated the Cochrane risk of bias assessment⁵⁴ and, as the two tools assess similar biases, we decided to opt for the more recent Cochrane tool to bring our work in line with current systematic review practice. We also stated that we would exclude studies from the meta-analysis that did not meet a minimum level of quality. In the event, such studies were excluded at the inclusion/exclusion stage of the review (and so were not in the analysis by default); in the meta-analysis, we conducted a sensitivity analysis to see whether differential effects due to methodological characteristics of the studies were detectable.

Finally, the original title has changed from ‘Can specific approaches to community engagement help to reduce inequalities in health; for whom, under what circumstances, and with what resources? A mixed-methods evidence synthesis’ to meet the PRISMA guidance on clearly signposting systematic reviews and meta-analyses.

Small number of studies conducted in the UK

It is important to recall that interventions conducted in the UK were reported in only 26 studies (8%) – most of the located trials were undertaken in the USA (268 studies, 84%). Similar proportions were observed for the subsets of studies included in the meta-analysis and the economic analysis. Given that most of the studies were not conducted in the UK, the applicability of the findings of this review to the UK context needs to be considered.

We employed two methodological decisions to help maximise the relevance of the findings to the UK context. First, we focused on health issues that are more critical for disadvantaged groups in the UK by categorising studies based on the policy objectives and priority areas identified in the Marmot Review.²⁸ Second, we specifically limited the scope of the review to OECD countries. However, it is clear that OECD countries still differ from each other dramatically in terms of cultural, social, economic and political structures and climates.

A factor that is likely to be particularly relevant to the transferability of findings is the availability of universal health care in the form of the UK’s NHS. Although broadly similar health-care systems funded through taxation or social health insurance are to be found in most OECD countries, this is not the case in all countries, most notably the USA. In the USA, although some groups have de facto access to universal health care, such as the over 65s through Medicare, the state Medicaid systems that provide access to health care for those on low incomes vary enormously in terms of both services provided and levels of entitlement. Individuals whose incomes are not low enough to qualify for Medicaid and who cannot afford to pay for private health-care insurance may find themselves without any adequate health-care coverage. The issue of relatively high costs of health care and gaps in coverage in the USA mean that issues related to some people (particularly those with a low income) seeking or obtaining adequate health care and advice may be very highly prominent in that country, whereas these issues are less evident in the UK for most of the population, other than for undocumented and new migrant groups.

It is also important to be mindful of the differences in funding for community development and engagement projects between the USA and the UK. In the USA, there are sources of funding available specifically for outreach projects for minority communities both from the federal government, for instance through the Office of Minority Health, and at state level. There are also funds available from the Indian Health Service for engagement with the Native American population. It is therefore possible that community engagement interventions designed to make health care available (e.g. by bringing health care to the home, or by providing free health services or resources) might be particularly successful in the USA where the context is very different and a mandate to invest in demonstration projects to reduce disparities in health status is in place.

We suggest that, through community engagement, interventionists should conduct a thorough needs assessment to ensure that any planned intervention will meet the specific needs of the community. This will ensure that interventions are not transplanted to the UK simply because they were successful in the USA, without consideration of local needs and health system structures.

In terms of the content and mechanisms of the interventions, there are no particular reasons to believe that the underlying theories of change should not operate similarly across countries. For example, cultural adaptations in themselves are likely to be transferable, as long as the adaptations are appropriate for the cultural context. Different levels of formal education or literacy between the countries might mean that certain aspects of some interventions might be more or less successful in the different countries – this should be considered in intervention design and is likely to be ameliorated through community engagement.

Regarding the populations included in the review, race/ethnicity clearly differs across countries. The 2011 UK census⁵⁴³ reported that most of the 56.1 million residents of England and Wales belonged to the white ethnic group, with 14% non-white residents. The largest minority ethnic group was Indian (2.5%), followed by Pakistani (2%) and African (1.8%). The 2011 census also highlights growth in ‘other white’ ethnic groups, in particular because of inward migration from Poland, which constituted the second largest group of foreign-born people in the UK in 2011. In contrast, according to the 2010 US census,⁵⁴⁴ roughly 27.6% of the total population of the USA were from non-white racial groups and 36.3% considered their ethnicity to be non-white. African Americans constituted 12.6% of the US population, and 16.3% of the total population identified themselves as having Hispanic or Latino ethnicity. As such, the ethnic minorities in the UK are quite different from those in the USA. It is, however, unclear whether cultural or biological differences between the ethnic groups that are most prominent in the UK (e.g. Indians, Pakistanis) and those that are most prominent in the USA (e.g. African Americans, Latinos) would influence levels of engagement or impact on the effectiveness of different intervention types.

Despite the differences between the countries in the review, we argue that community engagement interventions are inherently well suited to deal with the issue of transferability. By including members of the community in intervention identification, design and/or delivery, the interventions can be appropriately adapted to meet the differing needs in specific contexts. It is also critical that evaluations of community engagement interventions build in process analysis to help identify specific factors that are needed to deliver an intervention in a specific cultural context and to aid in adaptation to other contexts. We have highlighted a lack of detailed documentation of resource and cost data in most evaluations; this information is also crucial when considering whether it is feasible to implement an intervention in a different setting where the pre-existing infrastructure and human resources may be very different and where different mechanisms for engagement may need to be used.

Copyright © Queen’s Printer and Controller of HMSO 2013. This work was produced by O’Mara-Eves et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Included under terms of UK Non-commercial Government License.

Bookshelf ID: NBK262804

Contents