Testing the prospective evaluation of a new healthcare system
Abstract
Research into health ICT adoption suggests that the failure to understand the clinical workplace has been a major contributing factor to the failure of many computer-based clinical systems. We suggest that clinicians and administrators need methods for envisioning future use when adopting new ICT. This paper presents and evaluates a six-stage “prospective evaluation” model that clinicians can use when assessing the impact of a new electronic patient information system on a Specialist Outpatients Department (SOPD). The prospective evaluation model encompasses normative, descriptive, formative and projective approaches. We show that this combination helped health informaticians to make reasonably accurate predictions for technology adoption at the SOPD. We suggest some refinements, however, to improve the scope and accuracy of predictions.
INTRODUCTION
Previous studies on the problem of introducing health ICT to the clinical workplace indicate that the failure to understand the workplace has been a major contributing factor to the failure of many computer-based clinical systems [1] [2][3]. An example of a failed system is CERNER’s FirstNet, a patient record information system that was adopted in New South Wales hospitals. The system was costly, prone to serious errors (including linking incorrect details for some patients), and so was abhorred by clinicians [4]. Healthcare organisations need to envision future use when developing and implementing health information systems. Predictions must be shaped by context as described in the literature [5][6].
In previous work [7], we described four different approaches to modelling the flow of information in a healthcare context: normative, indicating how information should flow [8]; descriptive, indicating how information does flow now [8][9]; formative, indicating how information could flow [11]; and projective, indicating how information will flow with a specific new health information system [12]. We used the term prospective evaluation, to describe the combination of the four approaches into a process for envisioning future use. Figure 1 presents the six steps of the model. The model was used for the prospective evaluation of the proposed introduction of porterage dispatch software (software to dispatch porters to collect and transport patients within the hospital) into the Department of Diagnostic Radiology (DDR) in a major tertiary hospital.
Stakeholders potentially affected by the proposed dispatch software were porterage management, porters themselves, and the radiologists, radiographers, nurses, and administrative personnel in the different imaging areas. The evaluation method predicted a number of outcomes for the different stakeholder groups within the DDR, given the introduction of the new ICT. After the analysis was completed, representatives of each stakeholder group assessed the outcomes and expressed confidence in the process taken to arrive at the results.
Our experience applying the prospective evaluation model to the DDR indicates that there are further challenges to meet if the method is to succeed, as listed in our report [7]. In the present paper, we discuss the fifth challenge mentioned: the prospective evaluation method should be validated against actual outcomes. We did not have the opportunity to perform validation for the DDR, because it was out of scope of that project. However, we have since applied our model to the introduction of a medical record retrieval and display tool (for the purposes of this paper we call it “Retriever”) to an emergency department (ED) and a Specialist Outpatients Department (SOPD) and validated its predictions against actual outcomes. This paper presents the results of using the model to predict the effects of Retriever on an SOPD. As we will show, the prediction process is relatively accurate, though it requires some further adjustments.
BACKGROUND
The hospital in question was in a large regional hospital in a seaside district that caters for young families and retirees of modest means. The hospital had been selected by the state public health provider as a “pre-pilot” site for an early trial of the Retriever system.
The SOPD in the hospital offers patients ready access to a wide range of specialist medical and surgical services conducted by hospital staff. Consultants (attendings) specialise in an area of medicine and have sessions “clinics” during which they see their patients in the SOPD by appointment, through either the public or private healthcare system. Nurses within the SOPD may specialise in an area and assist the consultants in that area, but they generally work across two or more clinics which might include any of the following: medical, surgical, gynaecological, rheumatology, fracture, pre-admission and pre-operation assessment. The main two activities in SOPD are chart preparation (gathering relevant patient information and test results for a consultant) and running the clinic (the consultant sees patients on his or her list of appointments while the nurse handles patient flow, paperwork, and follow-up requests by the consultant.)
We studied the impact that Retriever might have on the SOPD. Retriever is a read-only application that allows healthcare workers to see consolidated patient information across a number of existing electronic systems. Retriever sources information from other databases, including pathology results, discharge summaries, patient demographics and admission/ discharge history, radiology results, medication profiles and adverse reactions, and operation notes. One of the main objectives of Retriever is to save time: with Retriever, clinicians no longer have to seek patient information from paper charts or from external laboratories or hospitals that are part of the statewide public health system. Seeking information from sources outside the hospital can be time-consuming, and the information difficult to obtain, especially when the process involves filling out and faxing Request For Information (RFI) forms to the healthcare provider who holds the required patient information. At present, the Retriever makes healthcare patient records available from the state public health systems: its jurisdiction does not reach to private health providers, private laboratories, or to general practitioners in the community, all of which have their own separate information systems at varying levels of maturity.
Within SOPD, we looked closely at chart preparation, where the nurse checks that each patient’s paper chart contains all the information that the consultant will require during the patient’s consultation. Charts are normally prepared the day before the clinic and the preparation includes determining why the patient is coming in and making diagnostic tests available for the doctor to review. Nurses access patient information by locating referral letters for the patient from other doctors, requesting delivery of the patient’s chart from the medical records department, faxing laboratories for pending test results, making phone calls to providers or to the patient themselves if test results are missing, and by sending notes or formal RFI forms to other hospitals. Nurses also carry out tasks such as making arrangements for patients with special needs (e.g. diabetics, or requiring a walker). SOPD leadership hoped that the Retriever would assist with chart preparation.
Nurses are required to chase up a considerable amount of patient information from sources both inside and outside the hospital. Information, and especially paper-based information, often goes missing. A consequence is that the consultant may need to proceed with a consultation without that information, have the nurse continue to chase up the information during the consultation, or postpone the consultation until the information has been retrieved, with inconvenience to all. We anticipated that Retriever would benefit chart preparation, given that Retriever would make much patient information available electronically through the one portal at any workstation.
In summary, our goals in the study were (1) before the implementation of Retriever (pre-Retriever) to model the impact of Retriever with our prospective evaluation process, and then (2) after the implementation of Retriever (post-Retriever) to examine the actual impact of Retriever to determine whether our predictions were accurate.
METHOD
We adapted the process used for the DDR study, which is presented in Figure 1. In what follows, we discuss how we approached each of the six stages of the model.
Participants
Approval was gained through the IRB of health district and hospital involved, and through our organization’s IRB. Participation in the study was voluntary and participants provided written informed consent. Participants consented to being videoed while working or being interviewed. Consents extended only to the participant, not to the patients or staff members with whom the participant interacted while participating. Video was occluded and (where needed) audio disconnected if unconsented personnel were likely to be caught by our recording.
Data Acquisition
Our initial goals had been for the data collection part of the pre-Retriever phase to take place over a couple of months before the introduction of Retriever. In the event, because of delays in the state health provider securing participation of the hospital in the Retriever evaluation trial, and because of the necessary IRB processes and legal agreements that had be completed after that, only around two weeks were available for collecting pre-Retriever data in the SOPD and ED before the Retriever trial was due to start.
Steps 1 and 2 of Figure 1 required an understanding of who the different SOPD stakeholder groups were and what their values were, and an understanding of the relevant work contexts in SOPD. To understand how SOPD functions without the Retriever in place—and later with Retriever in place—we conducted contextual interviews with staff members, and also observed them at work. A contextual interview (CI) is an interview that takes place in the workplace while the participant is working, allowing the participant to retain the work “context” as much as practicable while demonstrating processes and equipment they use. Observations (OBS) involved following participants while they carried out work responsibilities at their own pace, without stopping to answer detailed interview questions. Interviews lasted for up to two hours, whereas observations lasted for up to four hours. We asked a number of focused questions during both the pre-Retriever and post-Retriever phases. In our pre-Retriever study we conducted one CI with a medical officer, two CIs with nursing officers and two OBS with nursing officers. In our post-Retriever study, we conducted one CI with a medical officer and four OBS with nursing officers.
Step 1. Identify stakeholder groups and priorities/values
We asked each participating nurse and consultant to describe their work priorities and values. Because we did not have IRB approval to gather patient information directly from patients, we inferred patient priorities and values from direction questions of and discussions with our nurse and consultant participants. The objective was to generate a more comprehensive set of stakeholder groups actually or potentially affected by Retriever through, for example, chart preparation in the SOPD. We subsequently listed all the participants’ comments and categorized them, as shown in Column 1 of Table 2.
Table 2.
Predictions and verifications of changes in professional priorities/values of stakeholders with respect to introduction of new electronic patient information system. Patient stakeholder results omitted due to space limitations.
| Professional priorities/values | Modeling Phase Prediction – Retriever will have impact on priority/value with respect to chart preparation | Post-Retriever Phase Verification – Retriever’s impact on priority/value with respect to chart preparation |
|---|---|---|
| Registered Nurse/Enrolled Nurse | ||
| Flow | ||
| Avoid long waiting lines in waiting rooms | Retriever could show whether test results have been done and subsequently stop a patient from coming in unnecessarily. But, overall negative effect because consultants may become irritated at having to check yet another system. | Very minor positive effect. If patient pointed out that they have been in another hospital, nurse could make a note in the chart for doctor to check Retriever, making consultation quicker and hence, affecting waiting times. |
| Have unit operational after prolonged closures (e.g. holidays) | No change. | No change, as predicted. |
| Keep unit (clinics) running during unexpected medical emergency | Positive effect in that Retriever may alert to a patient’s condition. | No change, because Retriever is not normally used during an emergency. |
| Planning for load balancing and provider expertise | No change. | No change, as predicted. |
| Maintain order of appointments | Positive effect as ease of access to patient information may allow clinics to run more smoothly and it may be easier to inform colleagues of patient’s special needs | Very minor positive effect. May help get easier access to patient information, and hence run clinic appointments in order (i.e. no waiting for information) |
| Create culture of helping/teamwork | Over very minor negative effect in that easily accessible information may reduce need for interaction between staff, but it may be easier to inform colleagues of patient’s special needs. | Very minor positive effect. Retriever may be used to relieve other team members’ workloads, e.g. ringing around for patient information. |
| Leadership in clinic management | No change. | Very minor positive effect: if definition of leadership includes education, nurses could teach colleagues benefits of using Retriever for chart preparation. |
| Effective communication between staff members | Positive effect as Retriever makes it easier to inform colleagues about test results, etc. | Very minor positive effect as Retriever could assist in the communication between the nurse doing chart prep and the doctor running the clinic. |
| Information management | ||
| Have patient information ready in time for consultation | Positive impact as Retriever is a quickly accessible and comprehensive go-to source. | If nurses make a habit of using Retriever a minor improvement could happen. But, use of Retriever is limited to patient information from public facilities. |
| Ensure completeness of patient chart | Positive impact as Retriever is a quickly accessible and comprehensive go-to source. | If nurses make a habit of using Retriever a minor improvement could happen. But, use of Retriever is limited to patient information from public facilities. Nurses print patient information from Retriever, which defeats idea of electronic system. |
| Patient safety | ||
| Support patients physically | Positive effect in that physical support of patients easier if nurses are fully informed on patient’s history/conditions | No change as nurses would use other sources of info to find out about patient’s special needs. |
| Short waiting time for appointment | Positive effect in that there will be a shorter waiting time if patient information is easily accessible | No change – waiting time is the same. |
| Educate patient | No change. | No change, as predicted. |
| Patient Satisfaction | ||
| Maintain patient confidentiality | Positive. Patients may feel that confidentiality of current results is maintained better electronically, rather than through paper charts that lie around SOPD. | Potential minor negative effect on patient’s privacy and confidentiality if nurses do not log off after using Retriever. |
| Preserve patient privacy | Positive. Patients may feel that confidentiality of current results is maintained better electronically, rather than through paper charts that lie around SOPD. | Potential minor negative effect on patient’s privacy and confidentiality if nurses do not log off after using Retriever. |
| Provide emotional support | No change. | Very minor positive effect: if test results are immediately available to patients, it helps with emotional distress especially in difficult situations. |
| Maximise patient happiness | Positive. More informed, treatment will lead to patient happiness | Very minor positive effect: if test results are immediately available to patients, it helps with emotional distress especially in difficult situations. |
| Provide high quality treatment | Access to more complete info should lead to more informed treatment. Positive effect. | Positive. Very minor improvement as previous encounters could be related to present presentation. |
| Provide timely consultation | No change. | Very minor positive effect. Retriever may help nurses with decision-making and see patients faster if Retriever provided info about their condition. Depends on the severity of condition. |
| Reduce number of visits | Positive. Increased info may reduce need for patient to visit unnecessarily | No change. |
| Maintain order of appointments | Positive. Easy access to info makes it easier to maintain correct clinic order. | No change. |
| Consultant | ||
| Information management | ||
| Chase up patient information | Easy access to availability of results may improve timeliness of treatment. Positive effect. | Will be easier, though efficiency of the system will increase with practice. Minor positive effect. |
| National Waiting Times; Standards | ||
| See patients in clinic within recognised time scales | Minor positive effect. Easy access to results may improve timeliness of treatment. | No change. |
| Patient satisfaction | ||
| Short waiting time for appointment | No change. | No change, as predicted. |
| Timely assessment/treatment | Minor positive effect. Easy access to results may improve timeliness of treatment. | Retriever can reduce the number of RFI to other hospitals/providers. Minor positive effect. |
| Educate patient | No change. | No change, as predicted. Retriever does not help clinicians to educate patients on their conditions. Images better than text for that purpose – Retriever is text based. |
| Professional education of others | ||
| Enter REGs into training programs | No change. | No change, as predicted. |
| Educate MDs to look up reports electronically | Positive effect: simplicity of Retriever may encourage MDs to look up patient info electronically, especially if results are available. | Minor positive effect: putting a mark in the chart could encourage doctors to look up reports in Retriever. Also, Retriever has the potential to reduce dependency on patient’s chart. |
Step 2. Identify main work contexts (situations and functions)
Because of the short time frame over which we were collecting pre-Retriever data from SOPD, we were not able to sample systematically across different work situations (e.g., different clinics, different paces of work). We focused instead on capturing a broad array of work functions associated with chart preparation and running clinics.
We reviewed the audio-visual recordings of our interviews and observations, identified processes and contingencies in chart preparation and running clinics, and represented them in a graphical descriptive workflow model (see Figure 2 for an example). Chart preparation can be a complex process, as Figure 2 shows. From the workflow model we abstracted the principal work functions associated with preparing patient charts and running clinics, to use in future steps of the prospective evaluation process.
Step 3. Identify possible future worlds given new health ICT
The main work functions for a nurse preparing patient charts, including doing so while running a clinic, are shown in the “Work function” column of Table 1. The “without Retriever” columns shows the tools and resources used to perform those functions pre-Retriever. The “with Retriever” column show anticipated changes in resources available after the implementation of Retriever. Note that Retriever will be used mainly for gathering patient information from outside the hospital, which was described as one of the most time consuming work functions.
Table 1.
Work functions without Retriever and with Retriever in place: Retriever potentially helps five of nine functions.
| Work function | Without Retriever | With Retriever |
|---|---|---|
| Determines why a patient is coming in, i.e. specific reasons for patient’s seeing consultant today; the reason is stated in consultant’s last letter to GP. | Notes*/referral * E.g. GP letter | Notes/referral |
| Determine if tests were ordered, i.e. check letter to see if consultant requested that specific tests be done. | Notes/referral | Notes/referral |
| Determine if the tests are physically present, i.e. in charts. | Chart | Chart |
| Determine if physically absent ordered tests were actually done, i.e. check public electronic systems and fax private pathology/radiology to see if test results are available. If available, results get faxed back. May call patient. | IT*/fax/call * Existing electronic info system | IT/Retriever/fax/call |
| Request absent test results from labs, as above, but specific to tests done within hospital. | IT | IT/Retriever |
| Request absent test results from providers/hospitals. Again, as above, but specific to tests done outside of hospital. | Fax | Fax/Retriever |
| Arrange for missing tests to be done at hospital or elsewhere. | Administrative Officer/Admin system | Administrative Officer/Admin system |
| Make test result presence/absence/content evident to doctor for consult. | Place markers in chart | Place markers in chart, including where to look on Retriever |
| Make arrangements for patient’s special needs (escort, mobility, etc.). | Organise | View on Retriever/Organise |
Step 4. Construct current and possible future work scenarios with steps
For work situations in which Retriever may play a role, we constructed new workflows derived from the ones shown in Figure 2. The projected post-Retriever workflows are not shown in this paper, due to space limitations, but the essential changes have been captured in Table 1.
Step 5. For each future world, evaluate impact of change on each step and summarize findings with preferred combination rules
We drew up a matrix with work situations on the horizontal axis and priorities and values on the vertical axis, such as shown in Figure 1, Step 5. We assessed each cell, i.e. each priority/value with respect to each work situation, and noted whether Retriever would have an impact. We applied the following values:
| Value | Assigned when Retriever: |
|---|---|
| 0 | Will have no impact |
| +/− 0.25 | May have impact, including second order effects |
| +/− 0.5 | Will have noticeable impact |
| +/− 0.75 | Will have significant impact |
| +/− 1 | Will change the world |
The full quantitative matrix is too large to present here. However, we present our general predictions for each priority/value in Column 2 of Table 2. To arrive at each prediction, we took the mean value of the quantitative assignment across all work situations with respect to each priority/value.
Step 6. Display results
This stage involves displaying results using the ValuesViewer™ which is shown in Figure 3. The lower panel with small squares gives an overview of all predictions for all stakeholder groups. The upper panel provides a zoomed-in snapshot of the display. In the upper panel, from top to bottom, the four tiers show (1) SOPD, (2) Stakeholders, (3) Priority/value Categories and (4) Specific priorities/values within each category.
Verification step: Assessing accuracy of predictions in a post-Retriever phase
Five months after the pre-Retriever phase we returned to the SOPD to observe the effects that Retriever had on the stakeholders’ priorities and values. Participants were the same five clinicians as in the pre-Retriever phase.
In the post-Retriever phase we approached our data acquisition in a similar manner to the pre-Retriever phase. In the earlier phase we had identified work situations likely to be influenced and priorities and values likely to be affected. Therefore, in the post-Retriever phase we placed more emphasis on asking specific questions rather than simply observing the interaction of the participant with patient information. This approach was also needed given our relatively restricted time in the field – we did not have many hours over which to collect data, so any data would not have been representative. Our questions related directly to how Retriever may have changed SOPD work functions and how readily the work priorities/values of SOPD personnel were met. For example, the question “Does Retriever help to determine why the patient is coming in?” was asked to check the predictions that were made regarding the first work function listed in Table 1.
We verified each prediction by populating a second version of the matrix described in Step 5. This time, the quantitative assignments related to actual rather than predicted changes. A summary of verifications with respect to priorities/values is presented in Column 3 of Table 2.
Unfortunately, time worked against both the Retriever developers and our project timeline in terms of when Retriever was actually implemented. We were due to finish our project, including this study and a report, by the end of 2011. We were thus required to conduct our post-Retriever SOPD studies from the end of September to early October. However, Retriever had not been made available to the hospital for the pre-pilot testing until early September due to technical and other issues. The short period that had elapsed between implementation and testing/verification of the predictions meant that we could not assess the effect of full adoption of Retriever. Some participants reported that they had had problems accessing the system whereas others mentioned the lack of penetration to that point of Retriever implementation and training. The verifications of predictions made in Column 3 of Table 2 should be considered in the light of these factors, which limit the validity of the post-Retriever data. We assess the accuracy of our predictions in the Results section.
RESULTS
In this section, we assess our prospective evaluation process as a heuristic for envisioning the future use of health ICT by examining the accuracy and speed.
Accuracy
To assess the accuracy of our predictions, we calculated the difference between matrix cell values from the Verification Step and matrix cell values from Step 5 to arrive at a series of difference scores with range −2 to +2. The histogram of Figure 4 demonstrates that we accurately predicted many of the effects that Retriever would have on stakeholders’ priorities and values. Note that most priorities and values were accurately predicted as 0: in other words, the prediction was the same as the verification. The negative frequency values relate to predictions where we estimated too positive an impact of Retriever on stakeholders’ priorities and values. Recall that we predicted that Retriever would have a positive effect on many priorities and values. When we returned to the field in the post-Retriever phase, we observed the adoption of Retriever at a much earlier stage than originally intended, with relatively few participants making full use of it. Our predictions may well have been more accurate if they had been taken six months to one year after implementation.
Prediction accuracy. Negative bin values relate to overestimates in our predictions and positive bin values relate to underestimates. Range is −2 to 2.
There were also unexpected positive consequences of Retriever. For example, Retriever was useful for teamwork, as for example when nurses worked together to identify why a disoriented and confused patient presented to SOPD without an appointment. The nurses were able to identify the patient’s condition by using Retriever.
A method for improving the accuracy of the predictions would have been to have stakeholders verify our list of priorities and values and also to verify aspects of the work situations before we made predictions. Such a step might have helped us avoid missing a few important aspects of work functions. For example, the nurses relied heavily on SOPD consultants’ outgoing letters to GPs when performing chart preparation. The SOPD consultants’ letters contained most information that a nurse needs for chart preparation, yet the letters are not available on Retriever. Therefore, the SOPD consultants’ letters are often the nurses’ preferred source for that information, rather than Retriever, but that fact had not emerged strongly during the pre-Retriever phase. Another important point was that nurses said they would use Retriever for chart preparation if they knew that a patient had been to a different public facility. However, they would not have this information until the patient arrived at SOPD, which was a day after chart preparation. We were not aware of such details when making our predictions, and so did not fully take into account some key contingencies and information sources when nurses performed SOPD chart preparation, which resulted in a relatively slow adoption and in some cases non-adoption of the Retriever system.
Speed of performing prospective evaluation
We evaluated our time spent in the following activities: in the field, on data analysis (including video encoding), categorising priorities/values, generating workflows, extracting work situations, and making predictions. A rough estimate shows that during the pre-Retriever phase, we spent 20% of our time on data acquisition; 40% on data analysis; 6% on priorities/values; 20% on work situations; and 14% on predictions. The entire pre-Retriever and modelling process took approximately one month, with the verification stage taking an additional month. A more thorough process free of the time constraints we experienced could take twice the time, and still be cost effective. Overall, we suggest that the time invested, which does not include time to solicit participation, is well worth the effort. It is far more cost effective compared to introducing faulty ICT in health facilities.
Limitations
Limitations are the small number of participants, which did not include all stakeholders, and also that we conducted our study in a single setting. As noted, gathering more representative responses was challenging as implementation of Retriever was in its early stages. Another limitation is that some of the same researchers conducted Pre-Retriever and Post-Retriever analyses, which may have introduced bias in the validations of the model.
CONCLUSION
In this paper we used our six-stage prospective evaluation process (Sanderson et al., 2012) to assess the impact of a new electronic patient information system on a SOPD, and we evaluated the accuracy and completeness of the predictions. We found that the process led to reasonably accurate predictions that were appropriate for understanding the probable consequences of technology change in healthcare.
The major refinement we would suggest is to ensure there is time to have stakeholders evaluate the workflow models and the priorities and values elicited before performing the predictions in Step 5 of the process. A second refinement would be to automate some stages of the data analysis, which is the most time consuming aspect of the work. Since much of our data were collected in audiovisual form, we will investigate using speech-to-text systems for video encoding in the future provided they give us the accuracy that we seek. A third refinement would be to increase the number of stakeholders and clinical settings to obtain more representative predictions of the entire user community. Finally, a greater amount of elapsed time than we experienced between the prediction and verification steps would lead to more accurate predictions. It would allow users more time to become familiar with a new system and would allow stable new workflows to emerge. In summary, we believe our approach is feasible and useful. In future steps, we wish to apply the full process, including the verification stage, to a more controversial and potentially disruptive technology than the well-designed Retriever.
Acknowledgments
We thank the state health authority and the Nurse Unit Manager of the SOPD for gatekeeper access. We also thank the many hospital clinicians and staff who contributed to or participated in this study.