Conducting a study across different NHS organisations

Conducting the study across eight NHS trusts was not a streamlined process. The hurdles we describe are familiar to every researcher undertaking multisite research in the NHS. Though the field researchers had NHS Research Passports, trusts varied in the approvals they required (e.g. honorary contracts, letters of access, licences to attend) and in the procedures for obtaining them. One site required a site-specific agreement to be drawn up which involved extensive liaison between the researchers, representatives of the study’s sponsoring organisation (University College London) and the contracts team of the trust in question.

Similarly, it was time-consuming to find and follow the local procedures for acquiring the necessary security passes and IT access privileges at each trust. Some trusts required that the researchers go on specific training before granting access to certain computer programs used to access parts of medical records (e.g. software for viewing magnetic resonance imaging scans). This occurred even where the training was not relevant to the researcher’s role – for example, in order to gain access to certain medication records, one of our researchers was obliged to undergo training on electronic prescribing, even though this was clearly not a task she was going to undertake.

Difficulties in reaching the planned sample size for the quantitative analysis

We originally recruited 10 MDTs in eight NHS trusts. The necessary ethical and R&D approvals were obtained by the chief investigator before the project start date to allow us to begin data collection immediately. For a number of reasons, however, it became apparent that these teams would not provide sufficient data for us to reach our target sample size of 3300 patients. Although we drew on the experience of all our clinical coapplicants, the idiosyncratic nature of MDT meetings soon became apparent: there were extensive differences in the number of patients discussed both within and between teams concerned with different chronic conditions. We encountered particular problems in identifying heart failure teams with sufficient patient throughput to allow us to achieve the necessary sample size in the required time frame. Having met with heart failure consultant leads across London we identified two teams, but with low patient throughput. We therefore identified a third heart failure team, which agreed to participate in the study. Unfortunately, despite our efforts and timely production of all relevant documentation, delays in the R&D procedures at the site meant that after 6 months of liaison it was no longer going to be possible to recruit the required number from that site. Instead, we compensated for the small numbers discussed in some teams by observing these teams for longer than originally planned and we also observed extra meetings at the three MDTs discussing the highest numbers of patients.

One of the mental health teams disbanded shortly into the data collection phase because of service-restructuring within their trust. We were unaware of any plans to disband when we started data collection. We therefore recruited another team. As there was not enough time to go through the process of acquiring R&D approval from a new NHS trust, we recruited a new team from a trust which had already approved the study in relation to another team. This allowed us to recruit the new team and begin data collection relatively quickly.

Another challenge related to the nature of discussions in different teams. In our clinical coapplicants’ experience, each patient discussion resulted in a management decision. This information was used to calculate the sample size required. While this did occur in some teams, in practice several teams did not make decisions in every patient discussion, particularly in mental health teams. This meant that, even where teams were discussing sufficient patients, they were not making enough decisions to allow us to reach the target sample size. We therefore recruited a fourth mental health team to compensate for this.

Another unanticipated and time-consuming task related to quality assurance of our data. We originally planned to audiotape the first six meetings observed for each team. However, because meetings often involved team members talking simultaneously, we found that it was necessary to listen to the recordings of all meetings observed to ensure accurate completion of the observation proformas (i.e. to make sure we had captured all decisions and other relevant information). This meant listening to all 370 meetings at least twice (once in person and once on the recording). We amended the participant information sheet to reflect this.

Because of the recruitment of additional teams, we needed to employ an additional research assistant to attend and collect data from these teams. We applied for and were awarded service support costs from the Central and East London Comprehensive Local Research Network to fund this.

The NHS National Research Ethics Service granted approval for these minor amendments to the study.

Challenges relating to the interviews

The need to extend the data collection period in order to achieve a sufficient sample size had consequences in terms of our ability to undertake the planned number of interviews. We considered focusing only on the professional interviews; however, on the advice of our funders and members of our steering group, we proceeded with both professional and patient interviews as planned, though with a smaller sample and with carefully targeted recruitment to retain a diversity of disciplinary perspectives in the professional interviews.

We were assisted in the conduct of some of the interviews with mental health patients by a medical student and two specialist registrar psychiatrists (currently under the supervision of one of our coapplicants). The specialist registrar psychiatrists performed additional analyses of these interviews, and these have resulted in a research article which is currently under review. We invested considerable time to ensure that the medical student and trainee psychiatrists were fully cognisant of the study. Crucially, we trained them in qualitative interview techniques and observed their technique in pilot interviews with volunteer patients identified by our coapplicant PPI representatives before they began data collection. We also helped them to obtain the necessary approvals and submitted a substantial amendment (concerning the additional researchers) to the Research Ethics Committee.

Recruitment of mental health service users for interviews proved very challenging for a number of reasons. We recruited patients who had been recently discussed at MDT meetings, purposively sampling for a diversity of age, sex and ethnicity. For each selected patient, we asked their key worker to contact the patient and, if the patient agreed, we then made direct contact with the patient to explain the study, respond to their questions and arrange the interview. We agreed exclusion criteria with MDT members. These were patients who were acutely unwell, who had been discharged and were no longer contactable, and who were in prison.

The MDT teams from which these mental health patients were drawn were undergoing restructuring. Though only one team disbanded during the observation period (no patients from this team were interviewed), the others were preparing for staff changes and morale was very low because many staff members were concerned about losing their jobs. Thus, the research was understandably not a high priority for them. The key workers were also very busy and did not always have time to speak to the patients they had agreed to contact about the study. Several key workers did not respond to our e-mail and voicemail reminders.

Even where patients had been approved as appropriate by their key workers, by the time they had contacted them to explain the study and let us know, many patients had experienced deterioration in their health or had moved address and were no longer contactable.

Several patients agreed to be interviewed during our initial conversation but we were subsequently unable to contact them. One patient agreed to participate but then rescheduled the date of the interview and subsequently changed their mind and declined. One patient did not attend two appointments that we had agreed for the interview and so we decided not to pursue them for a third appointment. Both of the mental health patients we did interview had rescheduled at least once.

Managing large qualitative data sets

The qualitative aspect of the study involved 73 interviews and observation of 370 meetings (6053 patient discussions and approximately 530 hours of audiotape). Managing this volume of data was challenging, in terms of confidential storage, data reduction, ensuring consistency of data analysis across researchers and reporting the results concisely. At one point we unexpectedly ran out of electronic storage space and had to reduce the size of our audio files. This had a detrimental effect on the sound quality. The large volume of data also significantly slowed the running of the analytic software.

We adopted a number of strategies to facilitate the processing and analysis of the data. We outsourced transcription of the interviews to an external company that was recommended by qualitative researcher colleagues and that signed a confidentiality contract. We did not outsource transcription of the audio recordings of the meetings because they were too complex, with large numbers of speakers and variable audio quality.

Ensuring that each of the three field researchers coded the data reliably was a learning process necessitating close and regular communication between the researchers. Each researcher had different expectations regarding a ‘normal’ number of codes, how broadly or narrowly to interpret the research question, how to divide the task between them and which data management software to use, if any. Each code is open to different interpretations and it was a slow and laborious process to agree unambiguous definitions which coders could use consistently.

However, we found that the benefits of performing the analysis as a group far outweighed the costs. We believe it improved the robustness of the analysis because we needed to scrutinise, challenge and defend our ideas throughout. It led to more accurate and standardised coding and gave us insights into the assumptions underlying our interpretations of events. It allowed us to learn from each other and provided more ideas for a richer conceptualisation. Finally, it allowed us to analyse far more data than an individual could have managed alone.

We began to code data from the different sources as they became available, rather than waiting for the data collection phase to be complete before beginning the analysis. Observation of the different teams finished at different times. Thus we began the basic inductive coding and selective transcription of the observational data for the first team when it was ready, rather than waiting until observation of all the teams had finished. This meant that one field researcher could work on the coding framework and feed back to the team before the next researcher used it on the next ‘batch’ of data. This avoided the potential problem of researchers simultaneously coding different sections of the data and then meeting to find that their coding frames were incommensurable or redundant. This phased approach also meant that we were already very familiar with many of the data by the time data collection was finished and we could proceed relatively quickly to the integration stages.

Conclusions

Studies of this nature, involving collaboration across multiple sites, are inevitably subject to factors that are beyond the control of the research team. Responding to these challenges requires a willingness to be flexible and responsive to unanticipated events. Maintaining an open dialogue with our funders throughout and proactively seeking the advice of our steering committee were also essential to successfully navigating these challenges.