• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jamiaJAMIA - The Journal of the American Medical Informatics AssociationInstructions for authorsCurrent TOC
J Am Med Inform Assoc. 2008 Sep-Oct; 15(5): 701–704.
PMCID: PMC2528031

A Near Miss: The Importance of Context in a Public Health Informatics Project in a New Zealand Case Study

Stewart Wells, MBChB, MPH(Hons) * and Chris Bullen, MBChB, MPH(Hons)

Abstract

This article describes the near failure of an information technology (IT) system designed to support a government-funded, primary care–based hepatitis B screening program in New Zealand. Qualitative methods were used to collect data and construct an explanatory model. Multiple incorrect assumptions were made about participants, primary care workflows and IT capacity, software vendor user knowledge, and the health IT infrastructure. Political factors delayed system development and it was implemented untested, almost failing. An intensive rescue strategy included system modifications, relaxation of data validity rules, close engagement with software vendors, and provision of intensive on-site user support. This case study demonstrates that consideration of the social, political, technological, and health care contexts is important for successful implementation of public health informatics projects.

Introduction

The application of public health informatics to primary care has significant potential to improve the delivery of population-specific health interventions. 1 In New Zealand, a secure national health intranet and wide use of patient care information systems (PCIS) in primary care 2 make public health interventions delivered via primary care feasible. One such intervention is the identification and management of hepatitis B virus (HBV) infection, a highly infectious disease associated with nearly 1 million deaths globally each year. 3 Although immunization programs have been successful at preventing infection in children, high rates of chronic HBV infection remain in some population groups. In New Zealand, Maori (indigenous New Zealanders), Asian, and Pacific Island immigrant populations have very high rates of HBV infection ranging from 5% to 13%, whereas among European New Zealanders, rates are approximately 0.4%. 4 This article describes a qualitative evaluation of issues and challenges associated with implementation of a distributed information technology (IT) system established to support primary care–based screening and management of HBV infection in high-risk populations in New Zealand.

Case Description

In 1998, the New Zealand Ministry of Health funded a consortium of public health and primary care providers to develop a program to test high-risk population groups to identify and monitor people found to have chronic HBV infection for complications, and to immunize those found to be nonimmune. An IT system (the Hep B System) was developed that enabled primary care providers to opportunistically screen high-risk patients using existing PCIS to request laboratory tests, receive results, and claim and receive payment electronically for screening, immunization, and counselling services, providing system business rules were satisfied (see Appendix 1, available as a JAMIA online document at www.jamia.org). Interaction was required between 2 regional databases, the national unique patient identifier (UPI) database and the primary care PCIS, which also needed modification. After patients had given consent, were registered, and had a UPI allocated, blood samples were taken and delivered to a regional laboratory for analysis. Results were sent electronically to the Hep B System database and also to the PCIS of the primary care provider with a clinical advice message. A successful claim, which resulted in payment to the provider, required the data elements in the Hep B System database to match those in the submitted claim. Anonymized aggregated data were also sent monthly to the New Zealand Health Information Service (NZHIS) data warehouse. After implementation, a range of problems emerged that threatened the viability of the project and substantial changes were required. By the end of the program 3-year period (2002), most problems had been resolved, and 270 general practitioners had screened nearly 90,000 people, immunized the majority of those found to be nonimmune, and counselled and monitored almost 6,000 people found to have chronic infection. 4 Further details of the program have been described elsewhere. 4

Methods

Because the boundaries between the system and its context were unclear before the research, a nonexperimental qualitative case study method was adopted that used an interpretive research paradigm. 5–7 Field research was conducted between November 2004 and June 2005, 12 to 18 months after project completion, by one of the authors (S.W.), an experienced academic general practitioner with expertise in PCIS but with no prior experience of the system. With ethics committee approval, we sampled for diversity with key informants including project, laboratory, and practice managers; personnel from IT development and support; PCIS vendors; and primary care providers. Semistructured interviews were conducted until data saturation occurred. Field notes were taken, and interviews were audiotaped and transcribed. Relevant project documents were reviewed, including an independent evaluation report undertaken for the Ministry during the project, the project director's monthly board reports, and an IT support report. Themes and concepts from these sources were systematically identified as they emerged from the data using the principles of grounded theory and refined using constant comparison. 8,9 Concepts were entered into a 2-dimensional graphic model and linked where indicated by the data. The final model contained 82 concepts with 106 relational links (see Appendix 2, available as a JAMIA online document at www.jamia.org). The co-author (C.B.) provided an independent validity check of concept identification and analysis.

Results

Interview Data

The main themes that emerged from the data included premature implementation, low primary care IT capacity, PCIS modification difficulties, identity management issues, participant characteristics, and crisis resolution strategies.

Premature implementation of the system was reported by many as being a major cause of problems. This was thought to have happened because of the protracted time that it took to establish an ethnically representative governance structure as mandated by the Ministry of Health. The development team requested that implementation be delayed because of insufficient time for development, but this was declined by the Ministry.

Low levels of IT capacity in the primary care sector were identified by almost all interviewees. The vendors and program development team for PCIS overestimated user IT skill and made incorrect assumptions about the adequacy of primary care IT systems. Problems included limited availability of computer terminals and printers within practices, inadequate practice networks and servers in larger practices, and limited access to telephone lines for internet connections because of shared telephone/fax lines. This was thought to have arisen because general practitioner program advisers were not typical users of the system. Program developers thought that there was a limited understanding of primary care work practices.

Multiple problems were reported surrounding the development of the modifications to the PCIS. These were slow to be delivered, they varied in ease of use, and some had compatibility problems for Macintosh users. The PCIS vendors complained that specifications changed often, that communication with project management was poor, and that funding for modification development was absent.

Problems with identity management were widespread, and led to several areas of difficulty. Naming conventions were not standardized among the databases. Participant registration was difficult because of stringent data entry requirements and multiple problems with managing UPIs. Many patients had multiple UPIs because of variable naming practices by participants and errors introduced by manual data entry or handwritten test requests and sample labels. The mechanism for assisting users with UPI problems was difficult to access.

Because there were strict data matching rules, many payment claims were rejected, an issue exacerbated by rejection of an entire batch of submitted claims without indicating the individual erroneous claim, and further compounded by limited user access to the participant tracking system. Consequently, a large backlog of pending and rejected claims quickly arose, and significant temporary paper records were generated to cope with this.

Certain participant characteristics were reported as being relevant, including recent immigration, poor English language skills, lack of UPIs, and residential mobility, one consequence of which was that some received more than the required number of immunizations.

After recognition that a crisis had arisen, the project team instituted a team-based problem-solving approach. The Hep B System database was modified to enable individual claim rejection, standardized naming conventions were put in place across the interacting databases, enhanced UPI access was enabled via a website, barcode specimen identification was promoted more aggressively, and data matching requirements were significantly relaxed. Dedicated IT support staff from primary care nursing backgrounds were recruited to liaise with vendors to facilitate PCIS development, provide on-site IT support for practice staff, and assist with clearing the claim backlogs.

Document Review

The independent program evaluation noted early anxiety regarding reliance on the national UPI system. Late in system development, laboratory system testing revealed deficiencies that necessitated a complete system rebuild. At the practice level, there was limited availability of terminals for submitting claims, insufficient coordination among practice staff to ensure successful submission of a claim, and PCIS knowledge was held by a single staff member. Practice training for the program assumed the providers had good IT and PCIS fluency, but in fact many needed basic training. The PCIS vendors were slow to build appropriate functionality into their systems and provided limited support, and some failed to test software updates before their release. Despite these problems, by July 2002 IT systems were functioning well and support teams were in place, with priority given to practices in areas with large numbers of Maori. Claiming assistance continued to be required for some practices throughout the program. By February 2003, 93% of screenings had a corresponding claim.

Discussion

Many of the findings of this case study are consistent with known IT system implementation difficulties. 10–12 Although it could be argued that these resulted from a unique combination of factors, this does not lead to an understanding of how these factors arose. Our analyses identified important external influences, which Heeks et al. 13 conceptualize as contexts: “those realities that determine the pre-implementation state,” and which are known to be especially important for health IT systems development. 10,11

One context in New Zealand is the environment provided by the national health information systems, including the central health data warehouse and the national UPI system. It was the opinion of the program developers that the mandated use of the UPI seemed to create many of the system problems because of the fundamental importance of an accurate UPI for identity management for a complex system. Although desirable from a system point of view, this may not have been essential, because relaxation of the data matching rules that occurred to enable successful claiming involved ignoring the UPI. The eventual high level of completed claims indicates that this was a successful strategy. Subsequent to the conclusion of program, in March 2003 an intensive duplicate resolution program of the national UPI system was instituted. 14

A second contextual factor was the primary care environment. In New Zealand, primary care is largely provided by unregulated private practices with a fee-for-service payment model, often with low operating margins. Provider morale at the time of this project was low because of years of health care system reforms, declining incomes, and increasing compliance requirements. 15,16 It was therefore not surprising that the program team found low levels of investment in primary care IT infrastructure and low IT skill levels. Overestimation of the skill level was thought to have occurred because project managers sought advice from primary care “superusers”. Such computer-literate “experts” are worse than novices at describing work processes, 11 which in primary care, are fluid with frequent interruptions. 17,18 It was therefore inevitable that introducing a novel algorithmic system with multiple mandatory components into such an environment would be problematic. 12

The political context was also relevant to this project. In New Zealand, health outcome disparities between Maori and European New Zealanders are seen as a consequence of colonization and institutionalized racism. The political climate before the development of this project required that these disparities be addressed. 19 Establishing ethnically appropriate governance was therefore necessary but unexpectedly lengthy. The project team reported that they were under pressure to show results despite the IT systems not being fully ready.

Many subsequent problems were considered to be a direct consequence of failure to test the system before deployment, a well-recognized cause of system failure. Although reports of political factors influencing IT systems are uncommon, a previous project has been described in New Zealand in which a recently deployed centralized payroll system for the Education Department was discontinued because widespread negative public and teacher perception of the system allegedly threatened success at upcoming elections. 20

In this case study, inadequate understanding of multiple preimplementation domains led to compounding implementation problems. These contexts would have been better understood had there been a coherent program of user involvement, 12 exploration of how ethnic groups would interact with systems before deployment, flexibility around timelines built into the project parameters, and a preimplementation pilot study. Complex systems frequently have unanticipated consequences, and the delays that occurred in this case indicate that the situation was more complex than expected. Without the power to modify project parameters, the project team was restricted in its capacity to develop solutions; thus problems were inevitably passed onto providers.

The successful response by the project team was thought to be attributable to an intensive team-based approach that addressed problems at multiple levels. However, the use of IT support personnel with combined nursing and IT skills was thought to be a key factor because these people were able to understand the nature of the problems at the primary care level, communicate with providers, provide targeted training, and assist with practical problems such as claiming.

Strengths our of analysis included good access to important data sources and a range of informants, the research methods and paradigm were appropriate to the research question, systematic data extraction and analysis was undertaken, the researcher was appropriately skilled to undertake the interviews, and there was a high degree of consistency between informants about the events and their explanations. Limitations include data collection more than 1 year after project completion and lack of systematic validation of informant responses.

Conclusion

This case report illustrates that social, political, technological, and health care contexts are critically important considerations for public health informatics projects. Failure to consider these can adversely affect implementation and ultimately project success. This project was rescued by a concerted team response that addressed the identity management issues, facilitated resolution of the IT deficiencies, and using an IT support team with combined clinical and IT skills, provided intensive support to providers. Strategies that may prevent similar situations from arising include thorough environmental analysis, adequate preimplementation testing, close attention to accurate identity management, appropriate and sufficient human resources to cope with unanticipated consequences, and flexibility of project parameters.

References

1. Yasnoff WA, O'Carroll PW, Koo D, Linkins RW, Kilbourne EM. Public health informatics: improving and transforming public health in the information age J Public Health Manage Pract 2000;6:67-75. [PubMed]
2. Didham R, Martin I, Wood R, Harrison K. Information technology systems in general practice medicine in New Zealand N Z Med J 2004;117:U977. [PubMed]
3. Namgyal P. Impact of hepatitis B immunization, Europe and worldwide J Hepatol 2003;39:S77-S82. [PubMed]
4. Robinson T, Bullen C, Humphries W, Hornell J, Moyes C. The New Zealand Hepatitis B Screening Programme: screening coverage and prevalence of chronic hepatitis B infection N Z Med J 2005;118:1-11Available at: http://www.nzma.org.nz/journal/118-1211/1345/. Accessed March 7, 2006. [PubMed]
5. Klein HK, Myers D. A set of principles for conducting and evaluating interpretive field studies in information systems MIS Q 1999;23:67-93.
6. Benbasat I, Goldstein DK, Mead M. The case research strategy in studies of information systemsIn: Myers MD, Avison DE, editors. Qualitative Research in Information Systems. London, England: Sage; 2002. pp. 79-99.
7. Orlikowski W, Baroudi J. Studying information technology in organizations: research approaches and assumptions Inform Syst Res 1991;2:1-28.
8. Glaser B, Straus A. The discovery of grounded theory: strategies for qualitative researchNew York, NY: Aldine de Gruyter; 1967.
9. Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ 2000;320:114-116. [PMC free article] [PubMed]
10. Southon G, Sauer C, Dampney K. Lessons from a failed information systems initiative: issues for complex organisations Int J Med Inf 1999;55:33-46. [PubMed]
11. Littlejohns P, Wyatt JC, Garvican L. Evaluating computerised health information systems: hard lessons still to be learnt BMJ 2003;326:860-863. [PMC free article] [PubMed]
12. Wears RL, Berg M. Computer technology and clinical work. Still waiting for Godot. JAMA 2005;293:1261-1263. [PubMed]
13. Heeks R, Mundy D, Salazar A. Why Health Care Information Systems Succeed or Fail. Working Paper Series; Paper No. 9: Institute for Development Policy and ManagementManchester, UK: University of Manchester; 1999.
14. New Zealand Health Information Service. National Health Index (NHI) Upgrade Programme: Available at: url:http://www.nzhis.govt.nz/moh.nsf/pagesns/270?Open. Accessed April 11, 2008.
15. Dowell AC, Coster G, Maffey C. Morale in general practice: crisis and solutions N Z Med J 2002;115:U102Available at: http://www.nzma.org.nz/journal/115-1158/102/. Accessed April 11, 2008. [PubMed]
16. Devlin N, Maynard A, Mays N. New Zealand's new health sector reforms: back to the future? BMJ 2001;322:1171-1174. [PMC free article] [PubMed]
17. Parker J, Coiera E. Improving clinical communication: a view from psychology J Am Med Inform Assoc 2000;7:453-461. [PMC free article] [PubMed]
18. Coiera E. Interaction design theory Int J Med Inf 2003;69:205-222. [PubMed]
19. Cullen C. Budget Speech and Fiscal Strategy Report 2000: New Zealand Treasury, 2000http://www.treasury.govt.nz/budget/2000/pdfs/spch-fsr00.pdf 2003. Accessed April 11, 2008.
20. Myers D. A disaster for everyone to see: an interpretive analysis of a failed IS project Accounting, Management, and Information Technology 1994;4:185-201.

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...