• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jmlaJournal informationSubscribeSubmissions on the Publisher web siteCurrent issue of JMLA in PMCAlso see BMLA journal in PMC
J Med Libr Assoc. Jan 2007; 95(1): 81–83.
PMCID: PMC1773045

Analysis of user messages to MedlinePlus.gov

MedlinePlus, the National Library of Medicine's (NLM) consumer health web site, debuted in October, 1998. Providing information to consumers via a web site was a new direction for NLM, and MedlinePlus staff have continuously monitored email feedback from users to measure its effectiveness and user satisfaction [1]. When users select the “Contact Us” link from any page on MedlinePlus, they access a feedback form. Forms feed into the Siebel customer relations management software program [2]. Each message receives a unique identifying “ticket” number and flows into a queue for triage.

Feedback can be anonymous; if the user desires a reply, he or she must supply an email address. All feedback with a return email address receives an automatic response saying that NLM has received the communication and that a response will come within four working days, if one is required. NLM customer service staff respond to messages as needed, using a knowledge base that includes hundreds of standardized answers to frequently asked questions. They assign tickets about MedlinePlus to a group of MedlinePlus staff, who reviews the responses and may provide additional responses to users if required.

This paper examines the feedback messages that NLM receives about MedlinePlus to explore what users tell NLM about the site and to compare their feedback with observations from published studies of health web site users.

METHODS

All messages (N = 3,061) originating from the “Contact Us” link on MedlinePlus pages during March and July of 2005 were examined. These months were chosen because historically they have had high (March) and low (July) usage as determined by Internet log files. The messages generally consisted of two subsets:

  1. General reference questions about diseases, conditions, or treatment, or how to use MedlinePlus to find information about them.
  2. Compliments, complaints, questions, or suggestions relating directly to the content of specific MedlinePlus pages.

Table 1 shows the number of reference questions submitted through MedlinePlus. Given the large number of reference questions, a random sample for each month was analyzed, with sample numbers generated by an online randomization program [3]. Each reference question received one of the following investigator-assigned categorization labels: directory assistance, general medical reference, purchase drugs or financial assistance, question about current condition requiring professional advice, reference question or drug/laboratory test, request for collaboration, and request for materials or homework assistance. Table 2 (online only) lists examples of each. Each comment received one of the following investigator-assigned categorization labels: collaboration, complaint about content, compliment, request for materials or license, link to me, correction or suggestion for improvement, technical or access, and other. The categories were defined prior to coding. Table 3 (online only) lists examples of each.

Table thumbnail
Table 1 Data from Siebel: Reference questions from MedlinePlus, March and July 2005

RESULTS

As noted, messages broadly fell into two types: general reference questions and comments about MedlinePlus itself. There were 3,061 reference questions submitted to MedlinePlus during the study period. After randomization and excluding duplicate messages, mis-categorizations by the customer service staff and unintelligible communications, there were 261 messages appropriate for analysis (Table 1). With regards to the comments about MedlinePlus, after excluding duplicate messages, mis-categorizations by customer service staff, and uninterpretable communications, there were 284 queries in March and 207 in July for a total of 491 messages appropriate for analysis (Table 4; online only).

Figure 1 (online only) summarizes the content of the sample of the reference questions originating from MedlinePlus for both months. Over one-third (103/ 261, 39%) of reference questions originating from MedlinePlus concerned the current condition of the writer, family member or friend. An additional 13% (33/261) concerned drug or laboratory tests. Although these questions might have been about the user's health, the submitter did not specify that the question was personal. Thirty-two percent (83/261) were standard general medical reference questions: facts, statistics, and requests for sources of information.

Figure 2 (online only) shows the distribution by category of MedlinePlus comments for the study period. Nearly one quarter (22%, 111/491) of the feedback about MedlinePlus consisted of requests from organizations to add a link to their web sites. Most requests came from commercial companies or sites that did not meet MedlinePlus's quality guidelines [4]. Three requests from professional or voluntary organizations with quality consumer information resulted in the addition of links to MedlinePlus.

The next largest category of feedback was compliments (19%, 94/491). These came from both members of the public and health professionals. There were 32 complaints about MedlinePlus content, representing 7% (32/491) of messages. In the two months, there were two complaints about the reading level of the information, one that material was too simple, and the other that material was too difficult. There was one complaint about conflicting information (from general drug information page; specific sources not specified). Because MedlinePlus is a portal to information from other organizations and licensed content within MedlinePlus, many complaints related to links to specific web sites or to content MedlinePlus licenses.

There were 78 suggestions for improvement or corrections to MedlinePlus, accounting for 16% (78/491) of comments. Many suggestions for improvement related to licensed content, which NLM forwards to the appropriate licensed content providers. NLM also implements suggestions for additional features whenever feasible.

DISCUSSION

Previous studies note that consumers seeking health information on the Internet are generally satisfied with what they find; however, Ybarra also found that substantial proportions of users wanted more information but did not know where to find it or were concerned with the quality of the information [5]. Even when users are satisfied with the information they find, medical librarians and others are concerned on their behalf [6, 7]. Among the concerns mentioned in the literature are the trustworthiness of the information [810], completeness of the information [11], the reading level of the information [1214], and the existence of conflicting information [12, 15].

It is probable that many of the reference questions originating from MedlinePlus pages stem from incompleteness of the information on these pages or information that is insufficient to fully address the question or information need of the user. However, it is also unlikely that a web site could provide answers to questions about individual cases, unless a health care professional provided that information. The “Contact Us” page on MedlinePlus states, “We cannot respond to questions about your individual medical case, provide second opinions or make specific recommendations regarding therapy. Address those issues directly with your healthcare provider” [16]. As librarians, MedlinePlus staff cannot provide this service.

MedlinePlus users who provide feedback about specific content are generally satisfied with what they find. They make suggestions for improvements and point out major and minor errors (broken links, typographical errors and misspellings) but many also provide feedback to thank NLM for the web site. The proportion of MedlinePlus users who actually contact NLM is small. In March 2005, 5.4 million unique users visited MedlinePlus; in July 2005, 4.9 million unique users visited the site [17]. The feedback from MedlinePlus during those months represents less than .001% of users.

CONCLUSION

Overall, the communications that originate from MedlinePlus pages reflect the wide variety of users visiting the site. The majority of communications from users are from those who need reference assistance or advice from a health care professional to answer their specific health questions. The messages about specific MedlinePlus content are generally positive. Messages from MedlinePlus that are reference questions probably reflect the need for more information or explanation of the content. However, they also tend to be focused on a personal situation and are generally not of a nature that would be answerable by content on a general consumer health portal. Health information professionals can point consumers to good web sites and assist in their use, but even the best portals are not a substitute for medical advice.

Supplementary Material

Table 2:
Table 3:
Table 4:
Figure 1:
Figure 2:

Acknowledgments

The author would like to acknowledge Eve-Marie Lacroix, Joyce E. B. Backus, and Barbara Rapp, of the National Library of Medicine, for assistance with this article.

REFERENCES

  • Lacroix E. How consumers are gathering information from MEDLINEplus. Sci Commun. 2001.  Mar; 22(3):283–91.
  • Siebel ebusiness Version 7.5
  • Randomizer. [Web document]. [cited 1 Feb 2006]. <http://www.randomizer.org>.
  • MedlinePlus Quality Guidelines. [Web document]. [cited 1 Feb 2006]. <http://www.nlm.nih.gov/medlineplus/criteria.html>.
  • Ybarra ML, Suman M. Help seeking behavior and the Internet: a national survey. Int J Med Inform. 2006.  Jan; 75(1):29–41. [PubMed]
  • Crespo J. Training the health information seeker; quality issues in health information web sites. Library Trends. 2004.  Fall; 53(2):360–74.
  • Fox S, Rainie L. Vital decisions: how Internet users decide what information to trust when they or their loved ones are sick. [Web document]. [cited 1 Feb 2006]. <http://www.pewinternet.org/reports/toc.asp?Report=59>.
  • Eysenbach G. Infodemiology: the epidemiology of (mis)information. Am J Med 15. 2002.  Dec; 113(9):763–5. [PubMed]
  • Huntington P, Nicholas D, Gunter B, Russell C, Withey R, Polydoratou P.. Consumer trust in health information on the web. Aslib Proceedings. 2004;56(6):373–382.
  • Frické M, Fallis D, Jones M, and Luszko GM. Consumer health information on the Internet about carpal tunnel syndrome: indicators of accuracy. Am J Med. 2005.  Feb; 118(2):168–74. [PubMed]
  • Eysenbach G, Powell J, Kuss O, and Eun-Ryoung S. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA. 2002.  May 22; 287(20):2691–2700. [PubMed]
  • Berland GK. et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001.  May 23–30; 285(20):2612–21. [PubMed]
  • Friedman DB. et al. Readability of cancer information on the internet. J Cancer Educ. 2004.  Summer; 19(2):117–22. [PubMed]
  • Jaffery JB, Becker BN. Evaluation of ehealth web sites for patients with chronic kidney disease. Am J Kidney Dis. 2004.  Jul; 44(1):71–76. [PubMed]
  • Glenton C, Paulsen EJ, and Oxman AD. Portals to wonderland: health portals lead to confusing information about the effects of health care. BMC Med Inform Decis Mak. 2005.  Mar 15; 5(1):7. [PMC free article] [PubMed]
  • MedlinePlus: Contact us. [Web document]. [cited 1 Feb 2006]. <http://apps.nlm.nih.gov/medlineplus/contact/index.cfm>.
  • Webtrends. [Web document]. [cited 1 Feb 2006]. (accessed 23 January 2006.).

Articles from Journal of the Medical Library Association : JMLA are provided here courtesy of Medical Library Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...