Rural providers ’ access to online resources : a randomized controlled trial

Methods: In this randomized controlled trial, twenty-eight health care providers in rural areas were stratified by occupation and region, then randomized into either the DynaMed or the AccessMedicine study arm. Study participants were physicians, physician assistants, and nurses. A preand post-study survey measured participants’ attitudes toward different information resources and their information-seeking activities. Medical student investigators provided training and technical support for participants. Data analyses consisted of analysis of variance (ANOVA), paired t tests, and Cohen’s d statistic to compare preand post-study effects sizes.

Health care practitioners regularly seek to incorporate valid information into their evidencebased decisions.Electronic information resources now provide easy access to current health information and summarized forms of evidence to support clinical decision making.

Access effects on health care
Health care providers who are affiliated with wellfunded institutions benefit from access to a variety of high-quality information resources to support their evidence-based practices.Practitioners who are not affiliated with academic health sciences centers rarely have access to these same resources due to This article has been approved for the Medical Library Association's Independent Reading Program ,www.mlanet.org/page/independent-reading-program.. Supplemental Appendix A, Appendix B, Figure 1, Figure 2, Figure 3, and Figure 4 are available with the online version of this journal.
prohibitively high licensing costs.Ely et al. report that not having access to easy-to-use, high-quality, current information can negatively affect sound clinical decision making.[1].
Patients in Isaac et al.'s study who were admitted to hospitals that had access to an electronic evidencebased resource experienced reduced length of stay and lower risk-adjusted mortality rates for prespecified conditions [2].That study was validated on a broader scale by a multicenter investigation on the utilization of information resources by practitioners [3].
Primary health care practitioners who are not affiliated with academic health sciences centers, particularly those who practice in rural or remote areas, often articulate the need for increased access to health information resources.The authors' literature search and review of 114 of the most relevant research articles about the information needs and information-seeking behavior of health care practitioners suggested that these professionals most value speed and accuracy.For example, they likely would prefer to use point-of-care (PoC) resources for quickly and accurately answering their clinical questions.

Desirability of point of care
PoC resources quickly guide physicians through the diagnosis, treatment, and management of commonly encountered clinical conditions.PoC resources can present compilations of highly authoritative, often evidence-based, information.Physicians can answer more questions and revise clinical decisions more often using these PoC resources [4].Meanwhile, many health care practitioners still rely on textbooks familiar to them from their professional training programs, despite their potentially dated contents [5][6][7][8][9][10][11][12][13][14].

Limited access in rural New Mexico
A randomized controlled trial involving public health practitioners across the rural state of New Mexico reported many barriers in accessing valued information resources [15,16].In another study, researchers in New Mexico set out to determine and analyze information needs of health care practitioners who were not affiliated with an academic center.Fifty-one interviews of rural physicians, nurses, physician assistants, and nurse practitioners showed both a need and desire for access to information resources [17].

Goals and hypothesis
This study sought to determine which rural health care providers found more useful in answering everyday clinical questions: electronic PoC DynaMed, which is more clinically oriented, or electronic PoC AccessMedicine, which consists primarily of a health sciences textbook collection.We hypothesized that free access to the explicitly clinical format of the PoC resource DynaMed would result in more extensive use than a baseline of zero over a six-month period than the electronic PoC textbook collection AccessMedicine.We felt this would be due to greater utility of DynaMed in clinical practice.We also predicted that clinicians using the clinically oriented PoC would express higher levels of satisfaction than those using the textbook-based PoC and that clinicians using these PoC resources would prefer using them over other information resources.

Study design
This randomized controlled trial involved twentyeight health care practitioners in mostly rural and remote areas of New Mexico over a six-month time frame with free access to only one of the two PoC resources.Participants in both arms of this pilot study completed a two-page survey when enrolling in the study and at the end of the six-month study period.The two electronic subscription PoC resources are described below.
n AccessMedicine offers fully searchable access to over seventy-five textbooks and texts familiar to practitioners.Some titles include Harrison's Internal Medicine, DeGowin's Diagnostic Examination, Goodman & Gilman's Pharmacological Basis of Therapeutics, and Williams Obstetrics.n DynaMed aims to synthesize and summarize the best available evidence for busy practitioners.Users can access contents via text searches or via clinical categories such as diagnosis, epidemiology, or treatment.

Recruitment and enrollment
A previous outreach study identified two rural areas of the state as having the most critical need for access to PoC electronic resources [17].These areas contain many underserved populations experiencing health disparities.With approval of the University of New Mexico Human Research Review Committee, we contacted and recruited potential participants identified by health care provider informants in these areas.
Potential participants could enroll in the study by clicking a link in the invitation email.Only physicians, nurses, physician assistants, and pharmacists who currently had no access to any PoC or electronic textbook collections were eligible to participate.Access to the information resource served as the only incentive to participate.

Study participants
We enrolled 28 participants who all (100% response rate) completed the survey prior to the intervention phase.We distributed this same survey to participants after the completion of the 6-month trial.A total of 23 participants completed the study.The 5 participants who did not complete the study did not appear to constitute a distinct subpopulation that might suggest a bias to the study.
Those 23 health care providers completing the study consisted of 10 nurses (43%), 7 nurse practitioners (30%), and 6 (26%) physicians.An analysis of the urban-rural distribution revealed a participation rate of 55% and 45%, respectively.Respondents were 70% (n¼16) female, with a median of 20 years of experience ranging from 1 to 38 years, and ranged from 32 to 64 years of age with a median age of 50 years old.Appendix A (online only) features the invitation and consent email, while Appendix B (online only) provides a copy of the preand post-intervention survey.

Allocation
The initial twenty-eight participants were stratified by occupation and region of the state.These subgroups were then randomized using the webbased Research Randomizer [18].We then allocated each participant accordingly into the DynaMed or the AccessMedicine enrollee arms.Online resources subscriptions were paid for by a research grant.

Training
Medical student members of the research team trained the practitioners on the use of the PoC resource, resolved account and connectivity issues, and were available by phone and email to support the practitioners during the six-month study period.The medical student researchers produced a short instructional video on how to access and use each participant's specific, assigned resource.The videos could be downloaded and saved for later reference.The videos helped to ensure that all participants had access to the resources and had equal exposure to training.Subsequent training and support took place via email and phone calls on an as-needed basis.Initially, medical student researchers contacted their designated participants weekly and then later biweekly.Each medical student trained and supported about seven participants on a specific resource.

Survey measures
Two major dependent measures gauged (1) use and (2) satisfaction over the six-month study period.The survey also reported use and satisfaction with eleven other potential sources of health information.Identical surveys were distributed via email pre-and post-intervention.
Usage.The first section of the survey asked respondents, ''During the past 3 months on average, how often did you use each of the listed resources to look for information to help support or make an actual clinical decision?''For each of 13 listed resources, most of which were free, respondents could select a range from 0 to 10þ times to record their usage.
Satisfaction.The second section of the survey asked respondents to rate the same 13 information resources with the question, ''Overall, how satisfied are you with each of the listed health information resources in helping you in your practice?''A ''0'' indicated ''Not at all satisfied,'' whereas a ''10'' indicated that the respondent was ''Completely satisfied,'' with responses 1 through 9 taken as intermediate levels of satisfaction.
Respondents also answered questions on the length of time needed to answer their clinical questions, the percentage of successful outcomes to searches for information, sufficiency of information access needed, favorite information resources for self or patients, and willingness to pay for information access.The final survey section requested respondent characteristic data.

Data analysis
The frequency of reported use of thirteen health resources (within subjects) were compared pre-vs.post-intervention (within subjects) by the two arms of the study with doubly repeated analysis of variance (ANOVA) measures.A second analysis employed the same design to compare health care provider satisfaction with the thirteen health resources.We used the Huynh-Feldt adjustment for both ANOVAs.Paired t tests compared pre versus post measures of the four secondary outcomes listed above (see ''Limitations'').Given that this was a pilot study, sample size and statistical power were not computed in advance, and statistical power was low overall.Cohen's d-statistic compared pre-versus post-effect sizes by study group by health resources.(Cohen's d helps evaluate the difference in effect size between two means.)Both tables present the means, and the four figures (online only) show the profile of use and satisfaction means across resources.The table of means includes Cohen's d effect sizes and standard deviation, while the four figures show clear views of the profiles of variation of use of and satisfaction with the health resources by DynaMed versus AccessMedicine users.

Vendor reports
We originally intended to include vendor usage data to complement survey data in this study but were unable to achieve this goal.AccessMedicine and DynaMed have distinct database platforms that function differently.The two vendors collect and report usage data differently, even though both vendors report usage in Counting Online Usage of Networked Electronic Resources (COUNTER) [19].

Changes in usage of health information resources
The reported use of the 13 health information resources by the 23 health care providers was subjected to a doubly repeated measures factorial ANOVA of the design of: Resource (13) (within subjects) X Time (pretest vs. posttest)(within subjects) X Provider Gender (between subjects) As expected, the Resource main effect was highly statistically significant, indicating that providers reported wide differences across the frequency of use of the 13 health information resources, ignoring time of assessment and study arm (F(5.7,120.0)¼24.69,P,0.001, partial g 2 ¼0.54,).In addition, the Resource X Time interaction effect approached statistical significance (F(6.3,126.5)¼1.96,P,0.08, partial g 2 ¼0.09).This interaction effect suggested that use of the 13 resources varied across time, depending on the specific resource.Finally, the provider main effect reached statistical significance (F(1, 21)¼5.21,P,0.04, partial g 2 ¼0.19).
This pilot study is quite underpowered in reporting the effects for each ANOVA conducted, but also describes pretest versus posttest effect sizes (Cohen's d, the standardized difference in means) to help interpret the results of this trial.Attempting to understand our preliminary study results by focusing on effect sizes is consistent with modern recommendations for analyzing and interpreting study results [20].
Table 1 shows the mean (SD) times providers reported use of each of the 13 resources at baseline (time 1) versus post-intervention (time 2).The table also reports the standardized difference between time 1 versus time 2 means (i.e., Cohen's d statistic).Figures 1 and 2 (online only) supplement the table data by providing a more discernable visual profile of providers' resource use in both study groups from baseline to post-intervention.
Providers participating in this study had no reported access to other PoC resources such as UpToDate at the outset, yet use of UpToDate appeared in some DynaMed arm responses.Examination of the d-values shows that providers who were given free access to DynaMed reported that use of DynaMed (d¼þ1.50)increased meaningfully but that use of AccessMedicine (d¼þ0.06)did not change.In addition, providers who were given access to AccessMedicine had no changes in use of UpToDate (d¼0.00) or DynaMed (d¼0.00), but those providers did increase their use of AccessMedicine (d¼þ0.82).Note that providers who were given access to DynaMed used all 3 for-pay sources less than once per week prior to the study, and those who were given access to AccessMedicine reported no use of these 3 for-pay resources at all.
During the study period, health care providers' reported use of various types of medical websites declined moderately for both DynaMed and AccessMedicine.

Overall reported usage
Overall reported usage of the 13 health information resources by the health care providers was stable from baseline to posttest for providers who were given access to DynaMed (38.58 to 38.25 times per week overall; d¼À0.02;P.0.78).In contrast, reported use declined for providers who were given access to AccessMedicine (25.14 to 18.91 times per week overall; d¼À0.33;P,0.10).
Note that overall use of the 13 health resources (i.e., sum of ratings of all resources) was much higher at both baseline and posttest for DynaMed providers compared to AccessMedicine providers (mean¼38.58versus mean¼25.14;P,0.01; d¼0.77).This indicates a possible randomization error at baseline due to the low numbers of participants.The pre-versus post-design of this study allows comparison of the pre versus post changes in use (and satisfaction) between the 2 randomized groups.That is, does the pattern of resources use change across time differently for DynaMed compared to AccessMedicine providers?

Satisfaction with health resources
Health care provider reports of satisfaction with the 13 health information resources were limited by the fact that providers did not report satisfaction ratings (0¼not at all satisfied to 10¼completely satisfied) for resources that they never used within the past 6 months.For the sake of computing a coherent ''full model'' of satisfaction with the 13 resources from baseline to post-intervention, however, we recoded ''never used'' to implicitly mean ''not at all satisfied'' (i.e., 0 on the 0-10 rating scale of satisfaction).This is based on the conservative assumption that people who reported ''never using'' a resource during the past 6 months were not using the resource because of low satisfaction.We observed that overall satisfaction with use of the 13 health resources (i.e., sum of ratings across all resources) was much higher at both baseline and posttest for DynaMed (mean¼46.00)compared to AccessMedicine (31.73;P,0.01; d¼0.94) among participating health care providers.The reported satisfaction with 13 health information resources of 23 providers were subjected to a doubly repeated measures factorial ANOVA of this design: Resource (13) (within subjects) X Time (pretest vs. posttest) (within subjects) X Provider Group (access to DynaMed vs. AccessMedicine) (between subjects) As expected, the Resource main effect was statistically significant, indicating that providers reported wide differences across their satisfaction with the 13 health information resources, ignoring time of assessment and study group (F(10.3,216.4)¼15.48P,0.001, partial g 2 ¼0.42).In addition, the Resource X Time interaction effect was statistically significant (F(12, 1339.3)¼3.23,P,0.001, partial g 2 ¼0.13).This interaction effect indicates that satisfaction with the 13 resources varied across time, depending on the specific resource.The Time X Resource X Group 3-way interaction was also statistically significant (F(12, 1339.3)¼1.92,P,0.04, partial g 2 ¼0.08).This interaction indicates that satisfaction with the resources varied across time in a pattern different for each of the two study groups.
Table 2 shows the mean (SD) satisfaction providers reported for each of the resources at baseline (time 1) versus post-intervention (time 2) by study group, along with Cohen's d statistic for the preversus post-measure difference.For providers exposed to DynaMed, satisfaction with all 3 for-pay web resources (UpToDate [d¼3.32],DynaMed [d¼9.48], and AccessMedicine [d¼0.96])increased substantially, but for those exposed to AccessMedicine, only satisfaction with AccessMedicine increased (d¼0.59).Figures 3 and 4 (online only) are in parallel to Table 2 but clearly show the profile of satisfaction means for the 13 health resources.

Reduced time to find information
A Time X Group ANOVA showed that 18 providers reported a reduction in the amount of time to find sufficient information when searching health resources from baseline (mean minutes¼27.68;SD¼23.69) to post-intervention (mean minutes¼19.61;SD¼18.71)(Cohen's d¼0.30) (F(1, 16)¼3.19,P,0.10, partial g 2 ¼0.17).Only the providers given access to AccessMedicine (pre-mean¼33.88versus post-  Importantly, the reported mean use of resources of about 32 times per week (derived from the sum of uses across all 13 resources, Table 1) multiplied times a reported 7 fewer minutes per use equals 224 minutes per week or almost 180 hours of time per 48week work year.This calculation indicates the upper possible yield of time savings, because some of the 32 resource uses per week reported on average by providers might involve use of 2 or more resources per attempt.Even if the average search involved 3 resources, that would still indicate a potential savings of 60 hours per work year per provider.These findings are consistent with earlier resource access studies [3,21].

Information provided by current access
A third Time X Group ANOVA showed that 23 providers reported an increase from baseline (mean¼4.00;SD¼1.80) to post-intervention (mean¼5.41;SD¼1.68) (Cohen's d¼0.78) in how much information they perceived that their access to health resources overall provided them on a scale from 0¼''Far too little information'' to 5¼''The right amount of information'' to 10¼''Far too much information.''Thus, providers reported somewhat too little access at baseline and about the right amount of information at post-intervention, showing a large statistically significant effect size change (F(1, 20)¼7.24,P,0.02, partial g 2 ¼0.27).The pattern of increase was consistent for both study groups (P,0.63).

Willingness to pay for resources
A final Time X Group ANOVA showed that 19 providers overall reported being willing to pay the same at baseline (mean¼$70.47;SD¼78.29) as at postintervention (mean¼$71.84;SD¼83.04)(Cohen's d¼0.02).While the pattern across time for both study groups was consistent (P,0.86), the groups were quite discrepant in the amount they were willing to pay for regular access to an online health information resource (DynaMed mean¼$102 versus AccessMedicine¼$29; d¼1.22), indicating a meaningful difference in general between providers in the 2 study groups at baseline.At postintervention, 5 providers were not willing to pay any amount, but 8 providers were willing to pay $100 or more, up to $250 per year.This was very similar to the distribution reported at baseline, indicating the intervention had no influence on this measure.

DISCUSSION
This study demonstrates the potential power and replicability [22][23][24][25] of using a randomized controlled trial design in making rigorous comparisons between information resources.This study offers a model with wide applicability to other types of information resources.
Certain validations and some surprises emerge when reviewing the results of this study.As predicted, health care providers increased their use of PoC resources for health information when provided with free access.Participants also showed moderate to small reductions and moderate increases in the use of some other resources, such as general Internet websites and other free medical websites, that providers had previously used.Interestingly, providers' reported use of PubMed/ MEDLINE also increased some.Providers who reported using three free comprehensive health resources reported substantial satisfaction with using them in helping their practices.
Only about one-third of the health care providers participating in this study reported regular use of those PoC resources, even though the resources had been provided to them free for the period of this study.In addition, providers overall reported a reduction in the

Table 1
Provider mean (SD) frequency of use of health information resources during past 6 months at pretest versus posttest when given access to DynaMed or AccessMedicine 130.001, partial g 2 ¼0.42;ResourceX Time, P,0.001, partial g 2 ¼0.13 Resource X Time X Group, P,0.04, partial g 2 ¼0.13

Table 2
Provider mean (SD) satisfaction with health information resources during past 6 months at pretest versus posttest when given access to