The impact of library instruction : do first-year medical students use library resources specifically highlighted during instructional sessions ? *

Methods: Students attended a library resources instructional session. Resources that pertained to the students’ assignment were demonstrated and discussed. The students created a report including citations from relevant literature. The citations were analyzed and categorized as: a resource discussed at the instructional session, a resource found on the course LibGuide, a library resource, course material, or some other resource. All citations were subcategorized as print or electronic. Results: Three years (2008–2011) of data analyzing 2,983 citations showed that 49.55% of all citations were from resources discussed during library instructional sessions; 21.86% came from resources with links on the course LibGuide; 77.51% were from library resources; and 90.68% came from electronic resources.


INTRODUCTION
Measuring the impact of library instruction is a goal of many reference and instruction librarians.Over the years, librarians at academic institutions have tried to document that their efforts in the classroom, virtual or otherwise, bring about positive results and outcomes.
Librarians have looked at use statistics, interlibrary loan statistics, and bibliographies; have conducted surveys; and have tested users before and after instruction in an attempt to measure their impact on user behavior.
During a meeting for ''Introduction to Human Disease'' (IHD) at the University of Illinois College of Medicine-Urbana (COM-U) in the summer of 2008, the course instructors asked: ''What types of materials are the students citing for their assigned coursework?''Library instructional sessions had been integral to the IHD curriculum for years, but little was known about the resources the students were actually using and citing for their assigned coursework.Further investigation was warranted.Through citation analysis, this study examined the resources used in assignments of first-year medical (M1) students enrolled in IHD at COM-U for three consecutive years starting in the fall 2008 semester.

LITERATURE REVIEW
Various studies have focused on the use of citation analysis as a means to determine library instruction effectiveness.Dykeman and King published a small study in 1983 that linked student performance with library instructional sessions.They reported that ''the experimental group [that received library instruction] produced papers which were better written and Highlights N Review of the literature shows that instruction about library resources accompanied by clear guidelines on scholarly research methods and resources provided by course instructors have the biggest impact on the quality of cited materials in student bibliographies.
N Graduate medical students cited library resources more frequently than nonlibrary resources for their assigned coursework.
N Graduate medical students cited materials and resources specifically highlighted during library instructional sessions for their assigned coursework.

Implications
N Instruction about library resources tailored to medical students' assignments impacts the resources that the students consult and cite for their assignments.
N Using the information gathered from citation analysis to modify teaching plans can impact the effectiveness of library instructional sessions.
contained more subject-pertinent research material'' [1].In their 1986 publication, Kohl and Wilson found that ''bibliographic instruction taught as a cognitive strategy did increase the quality of student bibliographies to a statistically significant degree'' [2].Hovde, in 2000, found that ''students were, in fact, using the networked databases in which they had received instruction'' [3].
Davis and Cohen authored an oft-cited longitudinal study in 2001 that charted the effect of the web on undergraduate citation behavior.They wanted to see whether the move of academic libraries to online resources changed the quality and quantity of resources that students used for their coursework.Despite workshops held for the class that they were studying, they found ''a significant decline in the mean use of scholarly materials'' [4].Davis and Cohen argued for instructing students to critically evaluate resources and establishing guidelines for acceptable citations in course assignments [4].
In 2002, Davis wrote a follow-up to the 2001 study that included data from term papers gathered in 2000 and compared the data to previously reported data from 1996 and 1999.The results of the first study prompted the professor of the undergraduate course that Davis was investigating to stress the importance of finding quality scholarly research, hoping that verbal encouragement from the professor and instruction by librarians would suffice.However, the wording of the assignment was not changed.Students were not given minimum requirements, guidelines, or examples of scholarly resources.The follow-up study found the scholarliness of bibliographies continued to decline despite the encouragement of the professor and librarians.Davis concluded that ''librarians are not the entire solution.Professors, if they wish to see an improvement in the resources cited by students, will have to provide more clearly defined expectations in their assignments'' [5].
Davis wrote again in 2003, reporting a study that compared data from 1996, 1999, 2000, and 2001.He noted that the professor of the course was concerned that sources students consulted for their papers were becoming less scholarly.Therefore, the professor changed the wording of the assignment and implemented guidelines for acceptable reference sources.This change resulted in the 2001 bibliographies being quite different from previous years.The number of citations per bibliography increased, and, more importantly, the number of scholarly citations increased as well.Davis concluded that ''From the perspective of the library, minimum scholarship guidelines affirm the value of a library's collection and lend more relevance to library-mediated instruction'' [6].
In 2004, Robinson and Schlegl authored a study that intended to build on Davis and Cohen's original publication from 2001.Their aim was to test the generalizability of Davis and Cohen's work.Robinson and Schlegl analyzed bibliographies from papers submitted in 2002 by undergraduate students who received no intervention.These papers served as a baseline for the interventions conducted in the second term.They then analyzed bibliographies from two sections of the same class of the winter 2003 term.One section received library instruction, as well as encouragement from the instructor, to include scholarly sources in their research.The other section of the class received the same library instruction but was required to include a certain amount of scholarly resources or receive an academic penalty.Robinson and Schlegl found that instruction and encouragement had little effect on the quality of student research, but instruction and penalty did have significant impact.One finding was ''scholarly citations as a percentage of all citations increased from 72 percent in the control group to 74 percent with instruction-only; but, with instruction-and-penalty they increased to 86 percent'' [7].Their conclusion was: ''the quality of student research can be improved by librarian instruction backed up by clear standards and enforceable penalties'' [7].
Hurst and Leonard also examined the effect of library instruction on students' use of library resources.They investigated three sections of the same class, two of which received library instruction, while the third did not.Hurst and Leonard found that students who received library instruction cited a wider variety of resources, used a higher number of library resources, and cited more journal articles [8].They concluded that ''Students are more likely to use and cite scholarly resources when they have been shown how to use and access the databases'' [8].
These papers, to some degree, showed positive correlation between library instruction and the quality of resources cited in students' bibliographies with a focus on undergraduate students.Few studies were found in the literature that used citation analysis to investigate graduate students' response to library instruction.Long and Shrikhande conducted a study that sought to improve information-seeking behaviors of undergraduate and graduate business students [9].Bennett and Brothen analyzed papers of graduate students enrolled in organizational management and human services and used the findings of their study as a tool to develop an instructional program for two disciplines at their institution [10].The primary aim of this study is to show the impact of library instruction on medical student citation behavior, a topic that is not currently well represented in the literature.

Setting
The IHD course utilized case-based instruction that included 5 clinical cases distributed over the academic year.Those cases were presented as unique virtual patients in a software program that was developed by 1 of the course instructors.Around 150 M1 students are enrolled in IHD each year.A class was divided into 5 groups of about 30 students apiece.Each group was the ''teaching experts'' of one clinical case.Furthermore, the groups were divided into 8 teams.

Rafferty
Each of the teams was charged with teaching a small group of their peers (about 15 ''non-expert'' students) specific learning objectives associated with their case.Each team created a report with citations that covered the learning objectives.Altogether, 120 reports were given to the library for analysis.One hundred eighteen reports contained citations; 2 reports failed to include works cited and were excluded from the study.
The library was invited to conduct twenty-minute training sessions during course instruction time roughly two weeks prior to the due date of each report.Determining and choosing quality resources was discussed during the sessions.General navigation of AccessMedicine, MD Consult, and ACP Medicine was demonstrated in each session, but the majority of the time spent was focused on locating certain case-specific items found in those resources.A total of fifteen library instructional sessions were held (five per year) during this study.Eighty-five casespecific resources such as certain electronic books and e-book chapters were demonstrated during the instructional sessions.Teaching plans were modified each year, based on data gathered from citation analysis.If a resource was not discussed or demonstrated at the instructional session yet received a large number of citations and was determined to satisfy a learning objective for a case, oftentimes it was added to the teaching plans for the following year.The librarian informed the students that the citations on their reports would be analyzed and that the analysis had no effect on their grade for the course or their relationship with the university library.
The library also uses a LibGuide online course guide, which directs students to specific resources and materials for IHD.The library instructional sessions emphasized using the course guide as a starting point for conducting research for IHD.The course guide had a separate page for each clinical case.Each of those pages displayed a link to AccessMedicine, MD Consult, and ACP Medicine, which were discussed at the instructional sessions.The pages also included case-specific links to certain e-books and e-book chapters (when possible), between four and six recent e-journal articles, links to online catalog records for print materials, and, for some cases, links to outside resources freely available on the Internet.Like the teaching plans for the instructional sessions, content on each page of the course guide was updated every year, based on suggestions and trends discovered from the citation analysis.{

Reports
The teams' reports were due approximately two weeks after library instruction took place.The course instructors dictated that reports should not exceed twelve pages or be written at anything smaller than twelve-point font.Rules about margins or line spacing were not given.Students were not given stipulations governing a minimum or maximum number of citations for their reports.They were simply told that literature used to prepare the report should be referenced.The content of the reports was solely driven by the learning objectives for each case.The first three cases of the year had six learning objectives apiece, while the last two cases had seven each.An example of a learning objective was: ''Explain the function of a loop diuretic (e.g., furosemide) on the normal kidney.Hypothesize as to what the effect of a loop diuretic would be on a kidney blocked at the level of the ureters'' [11].

Categories for citation analysis
After the reports were turned in to the IHD instructors, copies were delivered to the library for analysis.Every citation from each report was transcribed to a spreadsheet, so they could be categorized and quantified.The citations fell into the following categories: & The citation was from a resource that was discussed during the instructional session.& The citation was from a specific resource or material found on the IHD course guide.& The citation was from any resource accessible through the library.& The citation was from course material such as PowerPoint presentations, lecture notes, or prepackaged course notes.& The citation did not fall into any of the aforementioned categories and was categorized as an ''other resource.''Each category was subcategorized as being from a print or electronic resource.If a resource was available in both formats, the construction of the citation determined subcategorization when possible.Anecdotal evidence from the instructional sessions and reference desk informed the researcher that students preferred electronic resources to print.Therefore, if the construction of a citation held no clues about the format of the resource, it was subcategorized as ''electronic.''A citation could be eligible for multiple categories.For example, navigation of Harrison's Principles of Internal Medicine Online found in AccessMedicine was demonstrated during the instructional session and a link to the e-book was found on the IHD course guide.Any citation from this e-book was marked as: (1) discussed during the instructional session, (2) found on the IHD course guide, and (3) a library resource.On the other hand, a cited e-book that was not discussed or demonstrated at the instructional session or found on the course guide would be marked as a library resource.Furthermore, general navigation of the e-books found in MD Consult was demonstrated at the instructional session, and links to specific ebooks were included on the course guide.In this case, a cited e-book found in MD Consult but not found on { The precise number and type of resources linked from the course guide is unknown.This information was not recorded, and the way the content was displayed on the course guide did not allow the links to be listed and link hits to accrue in the built-in statistics function of the LibGuides system.
The impact of library instruction the course guide would be marked as discussed during the instructional session and a library resource.

RESULTS
A total of 2,983 citations were analyzed from 118 reports.The mean number of citations per report was 25.24.The maximum number of citations in a single report was 59, and the minimum was 11, resulting in a range of 48.The standard deviation was 8.11.
Figure 1 displays the average percent of all citations by category.Students from all 3 years of the study heavily cited library resources (on average, 77.51% of all citations).This statistic answers the original question posed by IHD instructors.Students were mostly citing library materials for their assigned coursework.On average, nearly half (49.55%) of all citations came from some resource discussed or demonstrated at a library resources instructional session.Around 22% (21.68%) came from resources linked from the IHD course guide.
The percent of citations from resources discussed or demonstrated at instructional sessions increased from 43.92% in the first year of the study (2008/09) to 54.51% in the second year (2009/10), then fell slightly to 50.52% in 2010/11.However, a chi-square analysis (with a critical value for alpha 0.05 with 2 degrees of freedom was 5.991) showed the observation was not statistically significant (X 2 57.17).The percent of citations from resources found on the IHD course guide steadily increased over the 3-year study from 19.40% to 25.63%.Here, a chi-square analysis showed statistically significant results (X 2 55.708).

DISCUSSION
This study set out to examine the citation behavior in reports written by first-year medical students.Like the course investigated in Davis and Cohen's study [4] and then Davis's first follow-up study [5], IHD does not have clearly defined guidelines for the students to follow regarding the sources they consult for their assignments, which might have been expected to negatively impact the number of library resources that the students cited, including resources presented at the instructional sessions.However, analysis showed that 77.51% of all citations in IHD student reports came from some library resource.There are 2 likely explanations for this outcome: (1) the instructional sessions had a highly positive impact on student research behavior and (2) the study population was composed of students in professional programs.Graduate students have presumably conducted some research in their undergraduate studies.Their knowledge about where to find resources and the types of acceptable resources to consult when doing research likely increased the use of library resources found in this study as compared to use by undergraduate populations typical of prior research.
Hurst and Leonard found that library instruction when tied to specific assignments had a positive impact on student citation behavior [8].This study showed a similar result: 49.55% of all citations came from resources presented at instructional sessions, indicating that library instruction had an impact on the students' research habits.
In Davis and Cohen's initial study and then Davis's follow-up studies, no indication was given of how or if library instructional sessions changed in content or format over the many years of those studies [4][5][6].In this study, changes made to teaching plans from one year to the next typically had a positive effect on the number of citations coming from the resources demonstrated at library instructional sessions.The percent change for items discussed at an instructional session, while not statistically significant, was in the direction that might have been predicted given annual modifications to lesson plans.Links on the IHD course guide were also modified each year, based on the citation analysis.These modifications resulted in a steady increase of citations from resources found on the course guide.

Limitations
During this study, all M1 students received the library resources instructional training; hence there was no control group with which to compare results.Additionally, the instructional sessions focused on library resources that presumably were highly relevant to student success for each assignment.Therefore, it could be argued that students would likely have cited those resources for their assignments regardless of training.Informing the students about the study might have elicited the Hawthorne effect [12].In other words, because the students knew their bibliographies were being analyzed to see if presented resources were being cited, students would be more likely to cite those presented resources to comply with the study.This could have artificially inflated the number of library resources that the students cited.Finally, citations were rechecked for categorization accuracy against the author's teaching plans after the third year of the study.While it is possible the teaching plans were followed exactly, that was probably not the case.Therefore, some citations might have been inaccurately categorized because a resource in the teaching plan might not have been demonstrated during the instructional session for one reason or another (a technical glitch in accessing the resource or time constraints prevented the demonstration, for example).It is not likely this affected a large percentage of citations thereby dramatically changing the results, but it is a possibility that should not be ignored.

Future research
To be able to judge the impact of the library instructional sessions, it is important to determine if the students choose the resources and materials for their research because of the information they gain during the instructional sessions.While citation analysis tells what resources and materials the students cite for their research, it does not tell if the students cite those resources because of the instructional sessions.Including a control group-one that does not receive any library instruction-would provide data for comparison.Additionally, surveys that ask specific questions about the helpfulness of the instructional sessions and IHD course guide, coupled with citation analysis, may shed more light on this subject.
Content on the IHD course guide was reorganized for the 2011/12 academic year.The reorganization now allows the built-in LibGuides statistics functions to track every link on the guide.Analysis of citations from materials and resources highlighted on the IHD course guide compared to web page statistics may more accurately indicate the usefulness of the IHD course guide.Brief three-to-four-minute screen-casts demonstrating navigation to specific resources to address certain learning objectives were added to the IHD course guide for each case for the 2011/12 academic year as well.Questions about the usefulness of the screen-casts coupled with analysis of the web page statistics may indicate the usefulness of this feature.

CONCLUSION
Citation analysis is a valid means by which to gauge the effectiveness of library instruction.Analysis shows graduate medical students cited materials and resources specifically highlighted during library instructional sessions for their assigned coursework.
Over the three-year span of the study, the percentage of all citations coming from resources highlighted during the instructional sessions or found on the IHD course guide shows the impact that the library had on the students' work.

Figure 1
Figure 1Average percentage of total citations in student reports by category*