Limitations of the research
Our research has used rich cross-sectional and longitudinal survey and time series data sources compiled by the NDC NET to develop an innovative approach to the evaluation of the social and health impacts of different approaches to CE implemented within the same policy context, that is, implemented in the 39 local programmes that made up the national NDC regeneration initiative. However, there are several limitations to the data and methods that we used.
The outcome measures available to us were more limited than anticipated because the form of some measures in the MORI survey meant that we were unable to use them (e.g. measures of income) and because of constraints imposed by the size of the MORI longitudinal panel. Additionally, available measures of the experience of engagement and of community control and influence, and data for the effectiveness work, were particularly limited. On the costing side the main limits arose from inconsistencies in the volume and quality of available documentation, the retrospective nature of documentary data and recollections from a small number of interviewees. On the effectiveness side, for example, as noted above, we were not able to capture quantitatively some dimensions of the experience and impact of engagement expressed qualitatively by interviewees. Additionally, although there were some positive findings from the cost-effectiveness analyses, there was an almost equal number of negative findings and there was no obvious metric for combining them.
We were also aware of other limitations commonly associated with the use of secondary data sources: the fact that they were generated for other purposes; the often inadequate meta-data describing how the data have been collected and from whom; and their precise coding under different circumstances. In particular, some of the data sets (e.g. HES) have confidentiality restrictions; others (e.g. MORI) arise from the limitations of self-report data such as socially stratified reporting biases.
Bias may also have been introduced into our findings through residential mobility. Some of our results are based on repeat cross-sectional studies measuring change over time in different people; other analyses assess within-person change over time. The use of cross-sectional data reduces the problem of attrition found in the longitudinal data set but means that differences in the sample and residential mobility could contribute to the observed trends over time. To address this we undertook sensitivity analysis, excluding those who had recently moved and adjusting for a range of demographic characteristics to make the samples as similar as possible. For the longitudinal data, we compared results with and results without respondents who had recently moved. This comparison suggests that resident mobility effects seem to have been limited, at least in terms of the main outcomes of interest for this project. In all of our analyses we also adjusted for a range of demographic characteristics to help compensate for differences in the samples.
Findings from our longitudinal and cross-sectional analyses do not always agree. Although the sensitivity analyses with and without movers do not point to substantial bias when movers and non-movers are included, they do not throw much light on why this is the case. Numerous other studies show that longitudinal and cross-sectional approaches to estimating change frequently yield different estimates and different conclusions about the size and direction of change. Although longitudinal data are typically considered more precise within epidemiology, it is not necessarily the case that either of the two MORI data sources provides stronger evidence of impact in this study. Changing the profile of the resident population (e.g. through mixed-tenure redevelopment) was an explicit aim of some of the local NDC programmes, and between 2002 and 2008 there was substantial population turnover in some areas. The NET did attempt to follow up people moving out of NDC intervention areas but this was too expensive to pursue. Differences in the way that the health and social cohesion outcomes were treated in the longitudinal and cross-sectional approaches may also have contributed to different results across data sources. In the longitudinal analysis we analysed latent factors of social cohesion, which take better account of the measurement error associated with combining multiple items of trust in the community, trust in local services, mental health, etc. Despite the difficulties of interpretation introduced by using both longitudinal and cross-sectional data in the same study, this triangulation undoubtedly helps evaluations such as ours to capture more of the complexity of impacts associated with interventions such as the NDC in general and the approaches to engagement in particular.
As described in Chapters 3 and 7, alongside the secondary data that we used we also collected new primary qualitative data from interviews with residents of NDC areas and past workers. Although these data provide important insights into the processes of engagement, discussed further in the following section, the heavy reliance of our CE typology on the recollections of interviewees is a potential weakness. These recollections could have become distorted over time, particularly in NDC areas adopting a type D approach to engagement. These areas all involved redevelopment initiatives, which were more likely to have stalled after the economic problems of 2007 slowed the economy and in particular private sector house building. Additionally, being restricted to those residents and past NDC workers who were easiest to contact would have inevitably biased our sample of interviewees to those who were most engaged in the NDC (positively or negatively). Documents, another key source of data for the typology, also have problems of bias, typically being written to meet the requirements of particular audiences.
A final limitation was that we faced the usual challenges of developing a theoretically robust approach to conceptualising ‘community engagement’, challenges that are well documented in existing literature. As in all qualitative research, another team may have made very different decisions on this front.
New insights from the study
Our evaluation of NDC approaches to CE has sought to answer five research questions:
Which approaches to CE effectively engage which social groups in NDC populations?
Do different approaches to CE have different health and social outcomes for NDC populations?
Does the association between these outcomes and NDC approach to CE vary across groups defined by age, ethnicity, gender and material circumstances?
Do different approaches to CE have any impact on the health gap between NDC areas and areas from across the socioeconomic spectrum?
Does the approach to CE help to explain any of the differential outcomes of local NDC programmes identified in our previous research?
We also undertook an exploratory economic evaluation of the cost-effectiveness of different approaches to engagement in the NDC areas.
In seeking to answer these questions our evaluation has provided new methodological insights relevant to the evaluation of complex social interventions and new evidential insights into the pathways that might be expected to link CE in decision-making to positive social and health outcomes, and the factors that may support or undermine engagement processes. The study also provides insights into the processes of public engagement in evaluative research in the public health field, a field in which public engagement in research is relatively underdeveloped.
Methodological insights
To be able to answer research questions 2–5 the study design hinged on two key theoretical assumptions. First, in this and our previous evaluation we have conceptualised the NDC initiative, its component local programmes and their different approaches to CE, as holistic interventions with the potential to reduce health inequalities by addressing the social determinants of these inequalities. We therefore wished to evaluate these interventions as a whole, rather than evaluating the myriad and disparate projects and activities of which they are formed. However, this very diversity leads to the second theoretical assumption underpinning our work: that, although NDC local programmes all shared the same function in terms of the outcomes that they were seeking, the form of these programmes and the approaches to CE that they adopted would be different. This variation lies at the heart of our evaluation design as it can be exploited to compare the outcomes achieved by different types of local NDC programmes and different approaches to CE. Before we could do this, however, we had to develop a typology of CE approaches taken by the 39 NDC local programmes that would illuminate potential pathways to impact/change. Our attempt to develop this typology was therefore a crucial, and novel, component of the evaluation, without which the comparative analysis would not be possible. Some members of our advisory group warned us that this was an extremely difficult task that we might not be able to achieve.
Fortunately, we did produce a typology that made sense to NDC residents and practitioners who had been involved in NDC implementation in their area as well as nationally. Its development hinged on the elaboration of a theory of change linking CE to positive social and health (equity) outcomes. As noted in the introduction, the theory of change underpinning CE in the NDC initiative emerged from the SEU in the Cabinet Office. This theory, which has been criticised,50,51 assumed that engaging communities in developing and delivering local programmes would overcome problems of social exclusion and promote social cohesion, hence reducing crime and incivilities, and would also make services more responsive to local needs and hence increase access and effectiveness.48 However, we have not sought to evaluate whether or not the NDC initiative achieved its policy aims as reflected in the Cabinet Office theory (the purpose of the NET study). Instead, our typology and evaluation were underpinned by theories of community control/empowerment and the relationship to health inequalities and their social determinants, as set out in . Invariably there are overlaps between these different theories and community concepts, for example more cohesive and extensive social networks may enhance individual self-esteem and reduce postcode stigma, so increasing people’s mental health and their capacity to engage in other spheres of life, including the labour market, education, etc.
In particular, we were mindful that, within NDC local programmes, CE could also be operating at different levels. In this respect, CE type D approaches, which tended to engage communities around institutional agendas, might be expected to have more limited and narrower effects, for example on the quality of public services. In contrast, NDC approaches that sought to enable community control to develop at multiple levels within wider systems were arguably more likely to achieve positive impacts on a broad range of outcomes including psychosocial outcomes, public service quality, social cohesion/networks and material circumstances, and ultimately on population health and health inequalities.
The development of the typology also took into account evidence on contextual factors that may negatively impinge on these theorised pathways from CE to positive social and health outcomes. In the context of disadvantaged neighbourhoods, for example, stigma may lead to disengagement from ‘identity’-based collective action, undermining solidarity and social support.84 Building a sense of belonging within neighbourhoods may serve to strengthen networks and sources of local support, but on the other hand engagement that promotes exclusivity and homogeneity could conceivably exacerbate or maintain social divides in neighbourhoods.85
Using findings from analyses of documentary sources and qualitative interview transcripts together with valuable input from our public advisers we developed a typology of NDC approaches to CE that reflects these theoretical understandings of empowerment and the contextual factors that either support or undermine empowerment processes. It seeks to differentiate between strategic goals, underlying values, structures, processes and relationships (between residents and between residents and local agencies) that together had more or less potential to enhance and sustain residents’ control/influence over the local NDC programmes. Including a temporal dimension to our typology was practically challenging and we were also reliant on reported experiences of engagement rather than having the opportunity to see it being ‘enacted’ in real time (e.g. through non-participant observation). Notwithstanding this, understanding and tracking change proved critical in enabling us to characterise the longevity of CE values within local programmes and the extent to which changes in the values and practice of engagement were influenced by programme, political and economic factors.
The final typology consists of four different NDC approaches to engagement that sit on a spectrum, rather than representing discrete ‘types’ of CE. At one end of this spectrum we identified an approach to CE that was explicitly committed to empowerment, establishing and sustaining resident control over the design and implementation of actions aimed at improving people’s lives and the area in which they lived (CE type A). NDC areas allocated to CE type A were characterised by more facilitative and stable leadership than other areas and aimed to enable trust to be built between residents and agencies. They also invested more in the ‘participative’ capacity of individuals and the wider community. At the other end of the CE spectrum were NDC areas with a strongly instrumental approach to engagement (CE type D). Here, engagement was encouraged as a means to achieve externally set priorities. Structures and processes for engagement were focused on the immediacy of redevelopment issues, with very little if any emphasis on empowerment.
Type B and type C CE lie between these two extremes, but there is an important difference between them. NDC areas allocated to CE type B started out with a strong commitment to empowerment and resident-led change (i.e. a type A approach) but over time this gave way to a more instrumental approach, partly in response to pressure from central government to deliver ‘quick wins’. In this context it is possible that some residents would have felt let down by the failure of the NDC to deliver on the original ‘promise’ of active engagement. In contrast, type C NDC areas had a more balanced approach from the beginning, aiming for greater empowerment of local residents alongside a more instrumental approach, engaging local people in delivering externally imposed priorities such as those linked to housing redevelopments. NDC areas were therefore allocated to a CE type relative to each other and so there were inevitably overlapping characteristics. In particular, and importantly, although CE type A is argued to have the strongest emphasis on empowerment, NDC areas characterised as having type C and type B approaches to engagement also sought (at least initially) to be resident led and empowering.
We discuss the results of our analyses of the outcomes associated with these different approaches to CE in the following section, but from a methodological perspective we believe that the process of developing the typology and its use in this kind of innovative study design represents a methodological advance that will be informative for future evaluations of natural policy experiments at the community/local area level.
Another methodological advance made by our study was in the use of different data sets to triangulate the results and the construction of different socioeconomic comparators, to compare NDC outcomes with those of groups across the social spectrum. As we have already noted, the interpretation of the findings from our impact analyses using the different data sets was sometimes conflicting and presented conceptual and practical difficulties. These need to be taken into account in the planning of future evaluations using this kind of triangulation but these differing results also serve to illuminate the complexity of the likely impact of complex social interventions such as the NDC. Our previous study of health and social outcomes associated with different types of local NDC programme illustrated the value of constructing socioeconomic comparators from across the social spectrum. Comparing the performance of the NDC areas with the performance of areas with high, medium and low levels of deprivation provided evidence of a NDC ‘effect’ on the social determinants of health inequalities. Although we found no evidence in the current study that this positive performance of the NDC was explained by the approaches to CE, this methodological innovation in the construction of comparators could have value in similar evaluations of complex area-based interventions.
Finally, although the results of our economic evaluation were disappointing, the methods used in an attempt to measure and value the indirect costs of CE provide some useful insights and lessons for future work in this neglected area of research.
Insights into the impact of community engagement on health inequalities and their social determinants
Our study has provided no firm evidence that any one approach to CE was more successful than the others in engaging more or different social groups, or that the different approaches to CE had differential impacts on health inequalities or their social determinants. However, notwithstanding the lack of firm evidence, the pattern of impacts that has emerged from our research is consistent with the ‘theory of change’ underpinning our CE typology. For example, residents in NDC areas adopting a type A approach to CE and to a lesser extent those with type B and type C approaches were slightly more likely to participate in NDC events or to have any role within the NDC. There was also a tendency for areas with type A and type B approaches to CE to have better outcomes in relation to participation, trust, control/influence, social cohesion and mental health than areas with the strongly instrumental type D approaches to CE. Some aspects of cohesion and trust improved in type D areas relative to areas with other CE approaches but type D areas were the only ones in which residents’ ‘sense of control’ deteriorated over time. Residents of type D areas were also less likely to feel that the NDC programme had improved their area and were less likely to experience improvements in mental health, particularly compared with type A areas.
It is plausible that the engagement processes put in place in areas with a type A approach to CE would help build trust and greater cohesion within these communities, as these NDC initiatives reached out to different groups in the areas and brought them together. It is also plausible that greater interaction with, and influence over, the NDC initiative (as was particularly the case in areas with type A and type C approaches to CE) would lead to an increase in the number of residents linking the NDC initiative to perceived improvements in their area. The data on expenditure on community development (which was significantly higher in areas adopting type A and type B approaches to CE than in areas adopting type C and type D approaches to CE) and the qualitative findings support these speculations.
In answer to our fourth and fifth research questions, we found no consistent evidence that the type of CE approach had differential impacts on different socioeconomic groups in the population, neither did it help to explain the better performance of the type 2 NDC programmes, identified in our previous study. Similarly, the findings of our economic analyses were inconclusive. There are almost as many negative as positive scores in the effectiveness results, making the calculation of cost-effectiveness a rather arbitrary exercise.
There are several plausible reasons why we failed to find a significant effect of CE approach, beyond the possibility that CE has no impact in the NDC initiative. The first is the small numbers in some of the subgroup analyses. Second, the types in our typology may not differentiate sufficiently well between CE approaches – type A, for example, may have only appeared to be taking a more empowerment approach, while actually taking a more mixed approach more akin to the type B and type C approaches. Third, it is possible that none of the NDC areas achieved the depth of CE that would make an impact on the outcomes that we were measuring. Fourth, the official control areas selected by the NET were less than perfectly matched (most were more advantaged than their NDC counterpart); they had much smaller sample sizes in the MORI survey; and there may have been other regeneration initiatives involving CE taking place in control areas, even if they were not involved with the NDC initiative. Fifth, there may have been strong countervailing forces in operation over the period of the NDC initiative, in light of which even stopping outcomes from deteriorating may be seen as an achievement, rather than the more ambitious aim of bringing about improved outcomes on the factors that we measured. The statistical analyses of the routine and survey data sets cannot answer questions of why and how certain outcomes did or did not occur, but our qualitative studies did help to shed light on these questions.
Insights about the implementation of the New Deal for Communities and other area-based initiatives
Our qualitative interviews with residents and practitioners involved in the roll-out of the NDC initiative revealed processes that helped or hindered the implementation of the NDC initiative, and influenced the CE approach adopted and whether or not it could be sustained over time. In this section we suggest some lessons for the setting up and design of community-based interventions based on our findings.
An important source of tension within the programme was linked to power dynamics at a local level, particularly between agencies and residents. Conflict between local actors is not necessarily negative as within areas with type A approaches to CE it could be an indicator of resident influence developing within decision-making structures. However, in areas with type D (instrumental) approaches it was more likely to result from a lack of clarity or disagreement about the degree of influence that residents held or should have with regard to strategic decisions. This issue is not unique to the NDC programme as attention to power dynamics is too frequently neglected in initiatives aimed at engagement/empowerment.86
A critical tipping point for NDC approaches to CE appeared to have occurred during the early years of the programme when local partnerships were potentially at their most fragile. Ultimately, the pressure to deliver ‘early wins’ and the initiative’s top-down performance system spawned a disempowering environment at odds with a more gradual development process needed for true empowerment to evolve.57 Our research suggests that those NDC local programmes that retained their commitment to a resident-led approach over time were able to draw on organisational ‘resources’ that protected or enabled this ethos (e.g. positive working relationships between staff and residents, stability in staff teams and strong leadership committed to empowerment values) or, in the words of one of our fieldwork participants (see Chapter 7), managed to ‘hold their nerve’ in the face of pressures to spend quickly to show results. Towards the end, few NDC partnerships appeared to have formalised plans for sustaining engagement after the programme ended although, again, it is unclear how far this was affected by the economic situation. There were, however, examples of residents coming under immense personal and financial pressures by the end of the programme, with one council even taking away the assets of the local community organisation because the funding for the NDC initiative had ended.
In NDC areas with type 1 local programmes, which prioritised major physical regeneration, including widespread demolition, and which often pursued more instrumental CE goals, the demands placed on staff to manage relations with residents during processes of planning, rehousing or redevelopment could overwhelm ‘community-building’ activities that they may have aspired to. Paradoxically, this radical physical transformation may have served to undermine the social cohesion and trust that it was intended to help build, when ‘new’ and ‘old’ communities failed to engage or when housing plans stalled mid-programme because of the economic downturn.
Finally, it is important to emphasise that our qualitative findings are based on the experiences of residents who engaged with the local NDC programme. Although some respondents reported on NDC programmes’ efforts to be inclusive and reach the wider community, engagement in NDC programmes was ultimately characterised by small numbers. In this regard, initiatives such as the NDC may potentially exacerbate inequalities if groups who experience the most powerlessness are not supported to participate equally or meaningfully,57,87 or if non-participation is misunderstood by proferssionals as stemming from a ‘lack of capacity’ rather than community members enacting non-participation as a ‘rational reaction based on their socio-cultural context (p. 600).88
Insights from public involvement in this community engagement research
Public involvement in public health research is relatively underdeveloped compared with public involvement in health services research. In this context our experience of engaging both residents of NDC areas (some of whom were active on NDC partnerships) and workers involved in NDC programmes in this research is a useful exemplar of the value of such involvement. These ‘experts by experience’ were active members of our national advisory group and provided support in a range of other ways including advising on accessing fieldwork sites and research participants, proofreading information sheets, testing our research tools before we embarked on fieldwork, taking part in research as participants and contributing to the interpretation of findings.
These forms of public engagement have proven to be an integral component of the success of the research. First, local NDC programmes had already come to the end of their official funding when the evaluation began, with many staff moving into new posts and with community organisations experiencing significant funding pressures. With no obvious route into NDC areas, we drew on the ‘local knowledge’ of our public advisers to track down residents and professionals responsible for delivering regeneration strategies. Second, the experiential knowledge that these advisers have brought to the research has also been important in helping us to interpret our findings. One of the most powerful examples of engagement was a workshop in May 2013 at which preliminary results were discussed with five public advisers. Their experiences helped to place our ‘research in context’, providing important information about the political/economic environments within which regeneration partnerships designed and delivered their programmes.
There were invariably limitations to this form of public involvement. Our advisers tended to be long-serving community activists. Although their expertise is of real benefit, the risk is that the views of those less likely to get involved are not accessed. Second, engagement was led on a day-to-day basis by researchers working on fixed-term contracts as part of a time-limited project, making it more challenging to sustain public involvement over the longer term. University systems unfamiliar with public involvement in research could hinder rather than support involvement when, on occasions, the expenses of members of the public were not promptly reimbursed or highly bureaucratic processes made it impossible to pay fees in a way that was acceptable to some public advisers.
Finally, as readers of this report will have appreciated, this research has involved the use of highly specialised statistical procedures. This has made it more difficult (although not impossible) to engage public advisers in the interpretation and technical writing up of the findings. As a result, this report is perhaps not as readable as it could have been. Looking ahead, our advisers will be invited to be involved in developing a lay version of this technical report and in helping us to shape the dissemination of our research findings. Through our qualitative fieldwork we have built up a new wider network of interested local professionals and residents to support the impact of the research on public health practice and policy. Lastly, as researchers we have had privileged access to the detailed accounts of nearly 50 residents and workers about their own experiences of regeneration and CE in one of the UK’s largest regeneration programmes. We wish to archive these transcribed interviews for future use and will be seeking additional funding for the archiving or for future use.
Research implications
Our research represents a serious attempt to advance theoretical and methodological thinking on how complex social interventions involving CE can best be conceptualised and evaluated for their health and health inequalities impact. This included innovative use of secondary data sources in order to distinguishing different types of CE and different ways in which interventions are operationalised in different contexts. We have also tested ways of constructing comparators from across the social spectrum and explored ways of costing in-kind contributions to engagement. This work now needs to be taken forward and refined.
The relative lack of statistically significant results at a population level is not unusual in this field. For example, two recent reviews89,90 of diverse ‘CE’ interventions found that, although trends in quantitative evidence suggested that different approaches could have different impacts, the lack of statistically significant results made it impossible to conclude that a particular approach was more ‘effective’ than another. This points to the need for research to explore the sources – theoretical, conceptual and methodological – of this ubiquitous uncertainty in evaluations of CE interventions in the public health field. It also points to the need for evaluative research to be more tightly focused on the effectiveness of engagement, running concurrently with engagement processes (i.e. factored into the programme from the beginning), and to the need for methodological research on measures of community control and influence and on the economics of CE.
Our research also points to the importance of designing evaluations of CE interventions that are able to enhance understanding of the shaping influence of context and implementation processes. We have argued that the patterns of impacts that we have identified, relating in particular to community cohesion, control/influence and mental health, are consistent with theoretical understandings of the potential positive impacts of empowerment approaches to community engagement. In particular, the divergent patterns we have identified associated with a type A empowerment approach and a type D instrumental approach fit with a theoretical position which posits that if people are engaged only around the system's agenda rather than their own,86 community engagement approaches will have relatively little positive impact and may actually undermine some dimensions of social cohesion and well-being.
We have shown that a majority of the 39 NDC areas began with a commitment to empowerment and a resident-led model of change but that over time many were diverted to a more instrumental approach. In research on other regeneration initiatives, Lawson and Kearns91 have highlighted how empowerment has been used as a label that has ‘enabled other stakeholders to legitimate what they wanted to do in furthering more important objectives’ (p. 78). Head61 similarly concludes that engagement has rarely demonstrated increased power sharing with communities. This issue is highly relevant to the NDC initiative. Other researchers evaluating the NDC have argued that the initiative was ‘a tightly controlled policy space (p. 358)51 that had negative impacts on engagement processes, as our qualitative research confirms. Interestingly, a recent realist review of the evidence on CE initiatives involving lay health workers similarly found that contexts in which professionals and/or agencies maintain control limited the ability of lay health workers to deliver culturally tailored support.92
Our study has not provided firm evidence of the effectiveness of empowerment models of engagement, but it does suggest the need for more careful application of theory to public health interventions that aim to utilise CE as a vehicle to deliver external objectives. As Hart and colleagues93 argue, instrumental engagement may even result in an amplification of disempowerment. In this context it is important to illuminate the processes underpinning both empowerment and instrumental approaches to engagement and their impacts. To do so requires greater theoretical sophistication in the development and evaluation of CE interventions.
Summary of recommendations for further research
The earlier section on new insights from the study highlights the new knowledge that this project has generated on the impact of CE on health inequalities and their social determinants. It has also provided pointers for the future design and implementation of area-based initiatives concerned with empowering communities and their evaluation. The research was hampered, however, by having to rely on secondary data that had been collected for a different purpose and which therefore lacked some of the necessary information for a more robust evaluation. Further research is urgently needed that builds on our insights from this work on the challenges of evaluation and what is needed to overcome them.
There is a need for evaluations of CE interventions that test intermediate links in a theory-based logic model, to measure, in particular, whether the interventions actually achieved their objective of an increase in community empowerment or whether they resulted in disempowering hard-pressed communities still further. Too many evaluations in the past have assumed that empowerment has been achieved and have gone straight to the measurement of outcomes. Such evaluations, however, without the measurement of what empowerment, if any, has been achieved, do not provide a true test of the impact of community empowerment on health-related outcomes and may be one reason why evaluative research on CE in health-related decision-making has failed to provide definitive answers on impacts.
Determining whether or not community empowerment has been achieved by the interventions under study requires the development of better measures of community empowerment/control and influence, and ways of measuring the costs and benefits of CE to enable economic evaluation. The measures available in the secondary data that were available to us were relatively crude and underdeveloped, and revealed an obvious research gap that needs to be filled.
The results of evaluations of what helps or hinders the process of CE need to be fed into the design and development of future initiatives/interventions aiming to develop and support CE in decision-making in the health field.
Future evaluations of CE interventions need to combine explorations of the social and health (equity) impacts with process evaluations. These process evaluations should seek to identify the shaping influence of the wider local and national context in which CE initiatives are developed and implemented. Measures of context in community settings are starting to be developed and need testing and refinement.
Our study was able to consider the impacts of health and social outcomes over a relatively short time period and, as we have already noted, this endeavour was limited by our reliance on secondary data sources. Future evaluations need to be designed to track the health, social and economic outcomes for residents of area-based interventions such as the NDC over a longer time period. It is crucial that residents are tracked even if they leave the area, as some of those who have the most positive outcomes may be the ones who move away from the area, precisely because the intervention has improved their socioeconomic circumstances. Too often in area-based evaluations, even if there is a longitudinal element, participants are lost to follow-up if they move out of the study area. We experienced this loss to follow-up in the MORI longitudinal data set and it also meant that numbers in the study were reduced, affecting statistical power. Studies with a strengthened longitudinal design would be able to deal much better with residential mobility bias and address some of the pressing questions in this complex evaluation field.
