The new Medical Library Association research agenda: final results from a three-phase delphi study
Associated Data
BACKGROUND
The Medical Library Association's (MLA's) 2007 research policy statement [1] charged the MLA Research Section with defining the MLA research agenda on a recurring basis [2]. In 2008, the MLA Research Section's Research Agenda Committee used a group consensus technique known as the delphi method that involved both researchers and leaders to articulate MLA's twelve highest priority research questions. The MLA Board adopted these twelve questions as the 2008 MLA research agenda. A subsequent article reporting these top twelve questions includes detailed description of the committee's research methodology [3].
Following several months of deliberation and in consultation with the MLA Board of Directors, the MLA Research Section's Executive Committee directed the Research Agenda Committee to conduct a new study during 2011, implementing the adaptations recommended in the previous report [4], while upgrading the methodology to improve the answerability of the questions. Although only a few years had passed, it also seemed possible that rapid technological changes and emerging societal conditions would produce a different set of top-ranked research questions.
These deliberations led to important changes from the 2008 delphi study. The 2008 study had generated two questions that were so lengthy and sprawling in their subject coverage that they would have been nearly impossible to answer, requiring that the authors strictly enforce a sixty-word limit during 2011. While the 2008 study had merged the MLA leaders and all members of the MLA Research Section into a single study population, the authors decided against including all Research Section members during 2011, because Research Section membership might reflect support for the research endeavor or a desire to learn more about research rather than actual research experience. Those Research Section members with research experience would be difficult to identify easily.
A year after the 2008 study, the authors coincidentally learned of a concurrent 2008 delphi method process that a team of Swedish researchers implemented for defining the research agenda for all Swedish librarians [5]. The authors otherwise know of no other delphi method studies on defining a research agenda in librarianship, although, as indicated in the 2008 study report [3], this method has been used by other professions to define their research agendas.
METHOD
Phase One
The authors defined MLA leaders as individuals who were elected or appointed to offices at the national, section, or chapter levels or appointed to MLA national or chapter committees. The University of New Mexico Institutional Review Board approved the study (HRPO #11-127) on April 7, 2011. Using a comprehensive list of 581 such individuals with email addresses supplied by MLA headquarters, the authors invited the leaders to complete a brief survey via SurveyMonkey on April 27, 2001. Two reminders were sent, and the survey closed after 3 weeks. The instructions read: “Please submit to us…the single most important and answerable research question facing the medical library profession. Your research question must be sixty (60) words or shorter.”* On May 19, 2011, the authors closed the survey after a little more than a 3-week time period.† In a follow-up survey, respondents were asked to designate their MLA chapter name and type of leadership role.
Phase 2
The 298 individuals who had published research articles during 2008–2010 in the 4 leading English-language, peer-reviewed journals with the greatest concentration of health sciences library research articles—Health Information & Libraries Journal, Journal of Hospital Librarianship, Journal of the Medical Library Association, and Medical Reference Services Quarterly—constituted the phase 2 participants. These authors, plus MLA research award recipients for the years 2008–2010, were invited via SurveyMonkey to participate in phase 2 of the delphi study on June 6, 2011. Reminders were sent on June 13 and 15. This phase 2 survey closed on June 21. The survey arranged similar questions in close proximity, so that respondents could choose the most clearly stated version [6, 7]. The survey asked respondents to “Please select your top questions (up to 10) from those listed below on the basis of (1) importance of the question topic; and (2) the feasibility of answering the question in accordance with any known research methods.” This second segment regarding answerability represented an innovation from the 2008 study. The authors determined that in addition to the 2 aforementioned lengthy questions, another 3 of the final 12 questions that the 2008 study produced were too broad for a single researcher or small team of researchers to handle easily.
To facilitate selection by respondents, the questions were initially grouped by subject—collections, education of users, information access, outcomes/impact, professional issues, and value, using a modification of a subject classification that Crumley, Koufogiannakis, and Slater developed [8, 9]—to incorporate questions related to outcomes and value. The outcome/impact category was used for questions on the worth of library/librarian contributions made to the quality of educational, clinical, or research outcomes at their parent institutions; “professional issues” for questions on the education of librarians, roles in the institution, skills needed, and compensation; and “value” for questions on the financial worth or return on investment made by the library/librarian for the parent institution. The subject domain labels were removed prior to survey deployment to reduce the possibility that some researchers might focus only on questions on either favorite or familiar subjects.
Phase 3
On July 8, 2011, those MLA leaders who had previously participated in the first phase were invited to review the answerable questions selected by the respondents in phase 2 and vote for their top three research questions. After two reminders, the survey closed on July 22, 2011. The three-question guideline was based upon various probability scenarios ranging from uniform distribution of votes across all questions to an extreme clustering on just three questions by all phase 3 voters. The authors ranked every question receiving at least ten votes in descending order from the most votes received to the fewest number of votes to create a final list of questions.
RESULTS
Figure 1 (online only), showing the 2011 MLA research agenda process, summarizes the three phases of the delphi method and includes the numbers of research questions emerging from each phase. One hundred forty-one of 581 MLA leaders answered the initial survey with 1 question each, resulting in 140 viable questions. One hundred eight individuals responded to the second survey invitation to 298 researchers, who narrowed the list to 35 questions. Table 1 lists the final 15 high-priority research questions in descending order from the highest to the lowest numbers of votes from 115 of the 140 surveyed MLA leaders in phase 3. The full list of 140 initially submitted viable research questions and the 35 semifinalist questions resulting from phase 2 will be published elsewhere [10]. Tables 2 and 3 (online only) provide the distribution of MLA leaders by chapter and leadership role to demonstrate the occurrence of wide survey participation.
During the phase 2 survey period, only 1 participating researcher questioned the absence of subject categories. The authors miscalculated how long it would take some researchers to review and deliberate upon these 140 questions before selecting their top 10 answerable and important research questions. Several of the 108 participating researchers commented that it took them longer than predicted on the invitation message to complete this second phase and that the number of questions made this a tedious exercise. When the 140 initial questions submitted in phase 1 are broken into quartiles, the results suggest that researchers in phase 2 preferred those questions appearing earlier in the survey. Questions 1–35 received 482 votes, questions 36–70 received 216 responses, questions 71–105 received 139 responses, and questions 106–140 received 168 responses in phase 2. Future delphi studies could shuffle the order of the broad subject areas during survey distribution to minimize unintended primacy or recency forms of bias. Primacy bias in this context would cause people to lend greater importance to items toward the beginning of the list, whereas recency bias would cause people to lend greater attention to items toward the end of the 140 questions [11].
DISCUSSION
The final top-ranked fifteen research questions emerging from this delphi study (Table 1) offer a clear view of the research priorities of MLA leaders and researchers. These questions reflect a broad spectrum of subjects clustered around the six aforementioned modified subject categories. The final list of fifteen questions reflects a high level of anxiety with respect to the financial future of health sciences libraries. The 2008 research agenda did not reveal the same high level of anxiety about the future. While most health sciences librarians are keenly aware of the many valuable contributions that they make to their parent institutions, the broader economic recession apparently has deeply shaken many of our librarian colleagues' confidence about the future.
The final list of questions in Table 1 represents the results of a refined and replicable process. The questions are more succinct and answerable than the list generated in 2008 due to methodological adaptations. This list should move forward as the MLA research agenda with efforts to promote research in these areas and evaluate progress.
Limitations and lessons learned
The results of both the 2008 and the 2011 delphi studies were potentially influenced by concurrent social, political, and/or economic events. For example, during the 2011 study period, news reports of stock market problems and troubling reports of economic indicators such as unemployment data were prevalent. The US Congress also was engaged in a high-stakes political dispute over whether to raise the debt ceiling, creating considerable anxiety in the stock and commodity markets as well as in the general population, as reflected in public opinion surveys [12, 13]. All of these factors represent a threat to external research validity known as the “history” artifact [14] and probably explain the anxiety about the future reflected in a number of research questions.
Several lessons were learned from this research project. First, it was delayed by 6 weeks due to institutional review board approval processes. Phase 1 of the study ideally should have been deployed 2 months before the MLA annual meeting. Instead, it was launched at a time when many MLA leaders were distracted with their preparations for that meeting. Alternatively, the study could have been conducted during the fall without annual meeting or summer vacation distractions. Yet, the fall would dilute participation due to obligations caused by a busy time of year, particularly for academic health sciences librarians. Second, the authors also probably should have instructed the researchers in phase 2 to allow more than 10 minutes' time to complete their review of the 140 questions submitted by the MLA leaders. In the future, such researcher participants might be given an incentive such as a gift card, since several researchers complained that reviewing the lengthy list of questions in phase 2 proved tedious. All researcher participants nevertheless did complete the survey and cast their votes. Finally, as noted earlier, the arrangement of questions by subject might have introduced bias, since questions appearing near the beginning of the list tended to appear in greater proportion on subsequent lists.
The next steps
The delphi method described in this article produced a list of high-priority questions that were generated by MLA leaders, vetted by experienced researchers, and then voted upon again by MLA leaders. The 2011 delphi method consequently appeared to offer a practical method for building consensus on the top-ranked research questions. Table 1 offers researchers clear directions on the current MLA research priorities. Researchers undertaking any studies based on these questions inevitably will need to refine many of the questions further. The immediate tangible outcomes of the 2008 delphi study were use of the top-ranked questions for selection criteria for the Donald A. B. Lindberg Research Fellowship and publication of the 2008 study. In contrast to 2008, the current MLA Research Section's Research Agenda Committee will track and attempt to facilitate research efforts that address these high-priority questions.
The authors recommend that researchers first conduct systematic reviews to determine what research has already shown with respect to the top-ranked fifteen questions listed in Table 1. Systematic reviews might offer opportunities for students in graduate degree programs or participants in MLA's Rising Stars Program to partner with established practitioners who are interested in research to collaborate on joint projects. A wiki that tracks existing systematic reviews in library and information science offers thirty-four systematic reviews that can inform these colleagues as they compile these new systematic reviews [15]. The MLA Research Section will be the logical locus for coordinating these research projects.
The authors have inserted the research designs available for answering the questions at the highest level of evidence [16] in Table 1. At least eight of the fifteen questions can be answered by using a cohort study design with the library or the librarians serving as the exposures. At least four could potentially be answered using a consensus-building method such as the nominal group technique in a face-to-face setting or a delphi method asynchronously in a virtual setting. At least two and possibly three questions could employ a randomized controlled trial or other experimental research design. At least one question might be answered best with a participant observation methodology [17].
CONCLUSIONS
The delphi method described in this article adapted new features to improve the overall process. The fact that 140 MLA leaders and 108 researchers, 4 times the number in the 2008 study, participated in this delphi study suggests that the profession values the place of research in advancing their professional practices. The top-ranked questions in Table 1 will guide decision makers and researchers in defining which areas of systematic inquiry deserve the highest priority.
Electronic Content
Figure 1
2011 Medical Library Association research agenda process:Table 2
Medical Library Association chapter geographic representation of phase 1 MLA leaders:Table 3
Leadership roles as of May 1, 2011:Footnotes
IRPThis article has been approved for the Medical Library Association's Independent Reading Program <http://www.mlanet.org/education/irp/>. <http://www.mlanet.org/education/irp/>
ECSupplemental Figure 1, Table 2, and Table 3 are available with the online version of this journal.
*Nine leaders exceeded this 60-word limit, so the authors asked them to resubmit shorter questions. Eight leaders returned their shorter reformulated questions, bringing the total number of viable final questions to 140.
†The authors extended the initial two-week survey period to allow those non-responding MLA leaders, possibly distracted by the upcoming MLA annual meeting, more time to respond.