• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of bmjBMJ helping doctors make better decisionsSearch bmj.comLatest content
BMJ. Mar 13, 1999; 318(7185): 711–715.
PMCID: PMC27784

Development and evaluation of complex interventions in health services research: case study of the Southampton heart integrated care project (SHIP)

Fiona Bradley, lecturer,a Rose Wiles, senior research fellow,b Ann-Louise Kinmonth, professor,c David Mant, professor,d and Madeleine Gantley, senior lecturere for the SHIP Collaborative Group

The development and evaluation of complex interventions within randomised controlled designs is a challenging area in health services research. The process usually entails a pilot phase to confirm the feasibility and potential effectiveness of the design before embarking on large and costly trials. However, the focus is often more on the study design and measures than on the theoretical base and extent to which the intervention can be appropriately applied. In this article, we use a case study to describe an approach to pilot work that addresses this gap.

Summary points

  • Interventions are often defined pragmatically and lack any clear theoretical basis, which limits generalisability
  • Implementation is rarely described, which limits understanding of why an intervention is or is not locally successful
  • Integration of qualitative methods within pilot trials can help interpret the quantitative result by clarifying process and testing theory
  • This approach defines three levels of understanding: the evidence and theory which inform the intervention, the tasks and processes involved in applying the theoretical principles, and people with whom, and context within which, the intervention is operationalised
  • A case study shows how this novel method of programme development and evaluation can be applied

Background

Compared with drug trials or trials of surgical procedures, the design and development of a health service intervention is highly complex. In practice such interventions are often defined pragmatically, according to local circumstance, rather than building on any specific theoretical approach.1 Even if an approach or technology can be clearly grounded in theory and evidence, it must still be operationalised and evaluated among specific practitioners and patients. There is thus a tension between evaluation of complex interventions and generalisability of results. Randomised trials alone can not tell us why an intervention was or was not successful, or whether the theory and evidence informing the intervention were appropriate or needed revision.

To clarify this, we propose three levels for defining a complex intervention: the evidence and theory which inform the intervention, the tasks and processes involved in applying the theoretical principles, and the people with whom, and context within which, the intervention is operationalised.

Case study

The Southampton heart integrated care programme brought together the principal researchers from the family heart and OXCHECK studies—two trials of primary care interventions for risk reduction in cardiovascular disease.2,3 Debate after publication of the trials highlighted the importance of focusing on the design and process of interventions as much as on the evaluation of outcomes, to understand the process of application of an intervention in a way that would allow success to be understood and replicated, and unsuccessful approaches to be abandoned.4

The comparatively disappointing results from primary preventive programmes for cardiovascular disease also moved the focus of research to secondary prevention programmes. The Southampton heart integrated care programme was such a programme. It was led by specialist nurses who coordinated and supported follow up care in general practice of patients who had a hospital diagnosis of myocardial infarction or angina.

Definition of levels

The table describes the three levels of intervention as applied to the Southampton heart integrated care programme. Level 1 summarises the theory and evidence underpinning, in this case, the choice of a target population, service provision, and management of behaviour change within the programme; it deals with the gap between evidence of efficacy and provision of treatments for people with established ischaemic heart disease.47 It also deals with best practice in enabling behaviour change among practitioners and patients, using guidelines and psychological models.1922 Level 2 defines the essential tasks and processes required for operationalisation in these areas, at a generalisable level. Level 3 defines who would do what locally—elements which are specific to a local setting.

Quantitative approach

The pilot trial was designed to assess the impact of the programme on lifestyle and cardiovascular risk. The two arms of the trial compared the new approach with the usual care of patients with myocardial infarction or angina.8 Overall, 597 adult patients (from all 67 general practices in Southampton and south west Hampshire) with myocardial infarction or angina were randomised to intervention (33 practices) or control (34) groups. Follow up was 90% complete. The intervention increased follow up in general practice at 4 months and 1 year, and improved attendance for rehabilitation. No important difference was, however, observed between the intervention and control groups in any of the primary outcome measures of cardiovascular risk.8

Integration of a qualitative approach

The trial quantified the effect on contacts in primary care, but not the quality of those contacts. It did not allow interpretation of negative findings at the three levels of individual people, processes, or theory. To understand how the intervention was delivered, whether some of the elements seen as generalisable by the research team were particularly important or problematic, and the appropriateness of underlying theory, a qualitative approach is necessary.

The methods and results from the qualitative part of the programme have been reported in detail.9,10 Twenty five patients from the intervention group and, where appropriate, their partners were interviewed in depth. Participants were selected on the basis of maximum variety sampling.11 Patients were interviewed twice: shortly after discharge from hospital and again around 3 months later. In addition, 22 practitioners (hospital and practice nurses) involved with the intervention were interviewed or participated in a focus group once. A semistructured schedule of topics was used to guide the interviews and focus groups. Interviews and focus groups were audiotaped, fully transcribed, and subject to thematic analysis to identify emergent themes. Analysis was based on the grounded theory approach, and preliminary analysis of early interviews was used to inform further data collection and analysis.12

Explanation of numerical results

Level 3: people and context

Early data from interviews were used to refine the programme. To optimise implementation of the approach, the findings were fed back to the development group 4 months into the 18 month intervention. Initial analyses concentrated on the experiences and understanding that patients, their partners, and providers had of the role of the liaison nurse, primary care team, and rehabilitation services. The analyses examined the way these individuals perceived the intervention as being implemented. This phase of the analysis showed that patients were confused about the nature of the rehabilitation programme, that progression through the system was often slower than practitioners led them to expect, and that conflicting messages about the need for follow up were sometimes provided by primary and secondary care.

These findings were used to optimise the intervention. One improvement involved each specialist liaison nurse recruiting patients from specific practices for which they were solely responsible, rather than from all practices as before. This change encouraged communication between liaison nurses and practice nurses and between practice nurses and patients.

Level 2: tasks and processes

The later in-depth phase of the qualitative analysis was concerned with the interviewees’ experiences, perceptions, and understandings over time after myocardial infarction or angina was diagnosed. The purpose of this analysis was to understand in more detail the tasks and processes needed to successfully operationalise the intervention. This analysis identified issues that may clarify the negative findings of the programme. For example, the focus groups for practice nurses suggested that in order for them to effectively follow up patients with established heart disease, greater sophistication in specification of essential tasks and processes was required than we had anticipated (box).

Suggestions from focus groups with practice nurses, about changes to their role in follow up of patients with ischaemic heart disease

1. Status within the primary healthcare team must be developed

2. Training must address knowledge and skills of cardiac assessment, and drug use and adherence, as well as facilitating behaviour change in relation to lifestyle

3. Opportunity must be given for nurses to give continuity of care

4. Improved integration at the primary-secondary care interface needs to take place, with secondary care staff clearly recognising the role of the practice nurse

Level 1: testing theory

The second purpose of the in-depth analysis was to confirm or question the evidence and theory on which the intervention was based. At this level, the qualitative inquiry also identified potentially important insights. For example, the findings suggest that understanding of heart attack as an acute but short term event will inhibit the adoption of subsequent long term lifestyle change, and that the intervention failed to address this possibility. There were several reasons. Initially patients described their astonishment at surviving a heart attack, which they had previously understood to be a fatal event, and therefore defined their own event as necessarily mild. In the period immediately after the heart attack, the information provided to patients by practitioners apparently encouraged this view of heart attack as a self limited episode from which complete recovery was probable, with little reference to the continuing underlying disease processes. At this stage lifestyle change seemed to be understood by patients as being linked to recovery in the short term rather than a long term preventive measure.

At a later stage patients’ understandings about heart attack were subject to change particularly in cases where experience of recovery did not reflect the information given. For example, information encouraged patients to believe that they would be able to have sex in 2-3 weeks, would be back to work in 6 weeks, and would be back to normal within 3 months, when this was often not the case.13 At this later stage, faith could be lost in “official” information from practitioners and evidence drawn instead from personal experience. Conflict between the two was associated with questioning the explanatory power of information from practitioners, and viewing the adoption of long term lifestyle change as action that would not guarantee protection from a further heart attack.

The programme thus found that patients’ understandings of heart attack are closely linked to their attitudes to the potential of lifestyle change to keep them well. The failure of the intervention to acknowledge that the occurrence of heart attack, the severity of heart attack, and the natural history of recovery from heart attack cannot be accounted for entirely by lifestyle seemed to be a central feature in patients’ understanding.9

Comment

It is increasingly recognised that using qualitative research methods can “reach the parts other methods cannot reach.”14 Qualitative methods can be used on their own as a preliminary to a quantitative study in order, for example, to establish meaningful wording for a questionnaire or to develop the elements of an intervention.15,16 They can be used to explain findings after quantitative research has been completed.17 Quantitative and qualitative methods are also used in parallel with substantive trials.18,23,24 In the pilot trial of the Southampton heart integrated care programme, we have moved a step further by explicitly integrating qualitative methods within a pilot trial design. We argue here that parallel application of qualitative methods in a pilot trial can contribute significantly and efficiently to both optimising and evaluating a new health services intervention.

The pilot randomised controlled trial tests a hypothesis formulated from existing theory and evidence before data collection, examines inputs (resource use) and outcome (effect size), and provides evidence of whether the approach is feasible within a specific locality and worth substantive evaluation. Complementary to this, the qualitative research is concerned with the perspectives of the people involved in delivering and receiving the intervention, and the context in which the data are produced. Analysis of these phenomena can provide information about the process of implementing an intervention and can lead to suggestions about logistical changes needed to optimise the completion of tasks and processes within a local context.

There is always a learning curve in applying new interventions, but it is usually hidden. Our approach makes it explicit and formalises it through systematic feedback, allowing both description and optimisation of the application of an intervention at local level. This begins to move intervention development in the direction of what industrialists call “evolutionary operations.”25 This process involves performing rolling analyses over time—integrating both quantitative and qualitative findings to systematically optimise a production process.

At the deeper level of analysis for emergent themes, qualitative research can help understanding of the process whereby particular outcomes come about. It can thus enable a more sophisticated definition of what needs doing, as was shown with the practice nurses.9 It can also examine and test the theoretical basis of an intervention and question or affirm the principles on which the tasks and processes have been based. In this case the qualitative analysis raised questions about the information on natural history that is currently presented to patients by practitioners. It may be that in an effort to minimise fears and anxieties, practitioners are inadvertently providing an over- optimistic view of the natural history of convalescence for some patients and minimising the chronic nature of the underlying disease. These theoretical ideas are open to future hypothesis testing.

In summary, an integrated quantitative and qualitative approach to developing and evaluating complex interventions in health service research is both efficient and generalisable. Within one pilot study it formalises the usually hidden learning curve of implementation and optimisation. It allows better judgment of transferability of potentially effective programmes to other settings for confirmatory trials. It helps interpret quantitative findings, and questions underlying theory and assumptions to better inform future hypotheses and intervention designs.

Table
Defining a complex intervention at three levels for health services research: case of secondary prevention of ischaemic heart disease

Acknowledgments

We thank the patients and staff of the participating general practices. The Southampton heart integrated care programme was coordinated by the Primary Medical Care Group, University of Southampton in collaboration with: the Institute for Health Research and Development and the Institute for Health Policy Studies, University of Southampton; General Practice and Primary Care Research Unit, University of Cambridge; Department of Medical Statistics and Evaluation, Imperial College School of Medicine, London; Medical Statistics Unit, London School of Hygiene and Tropical Medicine; Health Economics Research Group, Brunel University; Department of Psychology, University of St Andrews; Department of Cardiac Medicine, National Heart and Lung Institute; and Department of Community Health and General Practice, Trinity College, Dublin. Members of the Southampton Heart Integrated Care Programme Collaborative Group are DA Wood (steering group chair), FB (former medical coordinator), M Buxton, A Davies, K Done, K Enright, M Johnston, D Johnston, K Jolly (medical coordinator), A-LK, DM (principal investigator), S Sharp, H Smith, V Speller, S Thompson, D Waller, RW, and L Wright. Members of the parallel qualitative research group are A-LK (chair), M Blaxter, MG, J Robison, A Spackman, and RW.

Footnotes

Funding: Research and development national programme grant from the National Health Service Executive, with service support from Southampton and South West Hampshire Health Authority. RW was in receipt of a NHS South and West Region research and development research training fellowship and FB was in receipt of a European Union research training fellowship. Subsequent development of the ideas in this paper were not externally funded.

Competing interests: None declared.

References

1. Shepperd S, Jenkinson C, Morgan P. Randomised controlled trials and health services research (letter) BMJ. 1995;310:125–126. [PMC free article] [PubMed]
2. Family Heart Study Group. Randomised controlled trial evaluating cardiovascular screening and intervention in general practice: principal results of British family heart study. BMJ. 1994;308:313–320. [PMC free article] [PubMed]
3. Imperial Cancer Research Fund OXCHECK Study Group. Effectiveness of health checks conducted by nurses in primary care: final results of the OXCHECK study. BMJ. 1995;310:1099–1104. [PMC free article] [PubMed]
4. King’s Fund. The way forward following the publication of the OXCHECK and FHS studies. London: King’s Fund; 1994.
5. Action on Secondary Prevention through Intervention to Reduce Events Steering Group. A British Cardiac Society survey of the potential for the secondary prevention of coronary disease: ASPIRE (Action on Secondary Prevention through Intervention to Reduce Events) principal results. Heart. 1996;75:334–342. [PMC free article] [PubMed]
6. Whitford D, Southern A. Audit of secondary prophylaxis after myocardial infarction. BMJ. 1994;309:1268–1269. [PMC free article] [PubMed]
7. Bradley F, Morgan S, Smith H, Mant D. Preventive care for patients following myocardial infarction. Fam Pract. 1997;14:220–226. [PubMed]
8. Southampton Heart Integrated Care Programme Collaborative Group. Randomised controlled trial of follow up care in general practice of patients with myocardial infarction and angina: final results of the Southampton heart integrated care project (SHIP) BMJ. 1999;318:706–711. [PMC free article] [PubMed]
9. Wiles R. Patients’ perceptions of heart attack. Soc Sci Med. 1998;46:1477–1486. [PubMed]
10. Wiles R. Empowering practice nurses in the follow-up of patients with established heart disease: lessons from patients’ experiences. J Adv Nurs. 1997;26:729–735. [PubMed]
11. Patton M. Qualitative evaluation and research methods. Newbury Park, CA: Sage; 1990.
12. Strauss A, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. London: Sage; 1990.
13. Southampton heart integrated care programme. Information for patients. Southampton: Primary Medical Care Group, University of Southampton; 1994.
14. Pope C, Mays N. Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ. 1995;311:42–45. [PMC free article] [PubMed]
15. Johnson A, Wadsworth J, Wellings K, Field J. Sexual attitudes and lifestyles. Oxford: Blackwell Scientific; 1994.
16. Little P, Williamson I, Warner G, Gould C, Gantley M, Kinmonth A-L. Open randomised trial of prescribing strategies in managing sore throat. BMJ. 1997;314:722–777. [PMC free article] [PubMed]
17. Bloor MJ, Venters GA, Samphier ML. Geographical variation in the incidence of operations on the tonsils and adenoids: an epidemiological and sociological investigation. J Laryngol Otology. 1976;92:791–801. , 883-95. [PubMed]
18. Brannen J, Moss P. Managing mothers: dual earner households after maternity leave. London: Unwin Hyman; 1991.
19. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342:1317–1322. [PubMed]
20. Schwarzer R. Self-efficacy in the adoption and maintenance of health behaviours: theoretical approaches and a new model. In: Schwarzer R, editor. Self-efficacy: thought control of action. London: Hemisphere; 1992. pp. 217–243.
21. Prochaska JO, DiClemente CC. Stages and processes of self-change of smoking: toward an integrative model of change. J Consult Clin Psychol. 1983;51:390–395. [PubMed]
22. Rollnick S, Heather N, Bell A. Negotiating behaviour change in medical settings: the development of brief motivational interviewing. J Ment Health. 1992;1:25–37.
23. Oakley A. Social support and motherhood. Oxford: Blackwell; 1992.
24. Stott NCH, Rollnick S, Rees M, Pill RM. Innovation in clinical method, diabetes care and negotiating skills. Fam Pract. 1995;12:413–418. [PubMed]
25. Box GEP. Evolutionary operation: a method for increasing industrial productivity. Appl Stat. 1956;vi(2):81–101.

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Group
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...