• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jamiaJAMIA - The Journal of the American Medical Informatics AssociationInstructions for authorsCurrent TOC
J Am Med Inform Assoc. 2008 Jul-Aug; 15(4): 484–495.
PMCID: PMC2442260

Consumer Health Information Seeking as Hypothesis Testing

Alla Keselman, PhD, MA, a , b , * Allen C. Browne, MA, a and David R. Kaufman, PhD c

Abstract

Objective

Despite the proliferation of consumer health sites, lay individuals often experience difficulty finding health information online. The present study attempts to understand users' information seeking difficulties by drawing on a hypothesis testing explanatory framework. It also addresses the role of user competencies and their interaction with internet resources.

Design

Twenty participants were interviewed about their understanding of a hypothetical scenario about a family member suffering from stable angina and then searched MedlinePlus® consumer health information portal for information on the problem presented in the scenario. Participants' understanding of heart disease was analyzed via semantic analysis. Thematic coding was used to describe information seeking trajectories in terms of three key strategies: verification of the primary hypothesis, narrowing search within the general hypothesis area and bottom-up search.

Results

Compared to an expert model, participants' understanding of heart disease involved different key concepts, which were also differently grouped and defined. This understanding provided the framework for search-guiding hypotheses and results interpretation. Incorrect or imprecise domain knowledge led individuals to search for information on irrelevant sites, often seeking out data to confirm their incorrect initial hypotheses. Online search skills enhanced search efficiency, but did not eliminate these difficulties.

Conclusions

Regardless of their web experience and general search skills, lay individuals may experience difficulty with health information searches. These difficulties may be related to formulating and evaluating hypotheses that are rooted in their domain knowledge. Informatics can provide support at the levels of health information portals, individual websites, and consumer education tools.

Introduction

A basic tenet of consumer health informatics is that access to quality electronic health resources empowers the public by increasing knowledge and improving decision-making. The Internet plays a major role in making this information accessible. As of 2006, some 113 million American adults have searched for health information online. 1 The topics ranged from specific diseases or medical problems (64% of users) and prescription or over-the-counter drugs (37%) to Medicare and Medicaid information (13%). Studies point to the positive aspects of growing access to online consumer health materials. 2 Presenting consumer health information via the Internet makes it more accessible, allows information tailoring for specific audience groups, and provides desired anonymity for addressing sensitive topics. 3 In one study, patients with HIV or multiple sclerosis who frequently searched the Internet for information on their conditions had greater knowledge than non-searchers and were also more likely to use this knowledge in their interaction with physicians. 4 In addition, these searchers played a more active role in their care. Internet users overwhelmingly prefer it as a source of information to pamphlets. 5

However, there are many reasons to be concerned that mere access to electronic health information does not necessarily empower consumers and patients. Despite the rapid growth in the number of consumer health sites, lay individuals often have difficulty locating, comprehending, and applying health information. 3,6,7 Even Internet-savvy users may experience difficulty searching for health information. 8 Roadblocks to finding useful information include information overload (famously compared to “drinking from a firehose”) 9,10 and the technical language of the materials. 11 Impetus for improvements in electronic resources is unlikely to come from the lay searchers themselves, as they are frequently unaware of the limitations of their search strategies 12 and often rate themselves as satisfied with their search. 1

Many individuals seeking health information for themselves (rather than for loved ones) perform online information searches before seeing a doctor, “perhaps to see what the diagnosis might be.” 13 Lay people are often inaccurate when they try to self-diagnose. 14 Searching for information in the absence of a specific diagnosis from a doctor is particularly challenging for lay people. In the absence of guidance from a health-care professional, searchers who have little domain knowledge and familiarity with relevant medical vocabulary, are often confronted by an extremely large search space, and may have difficulty separating relevant from irrelevant information. Understanding the nature of the search challenges that users encounter is a prerequisite for developing effective consumer health information resources.

The broad goal of this study is to understand the process of consumer health information seeking in the absence of a diagnosis. The focus is on locating and evaluating rather than applying health information. The study investigates a group of individuals asked to search the MedlinePlus® consumer health information portal for information about a hypothetical diagnostic scenario involving heart disease. The specific objectives of the study are to identify the most common patterns relating their initial theories and search strategies and to characterize the success or failure of these patterns. We are also interested in describing the role of various user competencies and electronic resource features in strategy selection and search outcome. To understand the nature of information seeking barriers, we combine an existing cognitive framework for understanding information seeking with a critical reasoning and hypothesis testing framework. 15,16

Theoretical Framework

The theoretical framework is designed to elucidate the nature of users' health information seeking problems. The framework draws upon two paradigms: 1) a human-computer interaction (HCI) approach to studies of information seeking, and 2) a critical reasoning and hypothesis-testing paradigm that is used in educational research. Human-computer interaction is a discipline for understanding the process by which individuals interact with technology resources in order to achieve a goal or complete a task. A goal-action framework is common in HCI 17 research and has also been extended to work on information seeking (see Sutcliffe and Ennis' cognitive theory of information retrieval). 15 The framework characterizes performance on any task as an iterative process. Each cycle in the process consists of a) setting a goal (e.g., to learn about a heart test that involves radioactive injection), b) performing some actions towards achieving the goal (e.g., conducting “heart disease diagnosis” query, following results links), and c) interpreting and responding to the resulting change in the system (e.g., recognizing the page about nuclear heart scan as appropriate). The response also involves evaluating whether the goals have been met and if necessary, setting the goal for the next iteration of the process. The goals typically have a hierarchical structure, with the completion of the overall goal (e.g., learning about the risks of a diagnostic procedure) dependent upon completion of intermediate level sub-goals (e.g., finding out the procedure's exact name).

Successful completion of the information-seeking cycle leads to retrieving appropriate webpages and extracting required information from their content. A common cause of navigation performance failures are the mismatch between users' competencies (e.g., domain knowledge, familiarity with the online resource, etc.) and the support provided by the system. HCI approaches to information seeking classify user competencies and describe their role in the search process. 15,18 For example, Sutcliffe and Ennis list four types of knowledge that are instrumental in the process: 1) domain, 2) device (knowledge of a particular information-seeking system), 3) information resources (awareness of specific databases underlying the system), and 4) information retrieval knowledge (specific search strategies). Studies of online search behavior support the notion that both domain knowledge and navigational expertise affect search performance. 19 The structure of the resource can somewhat mitigate the role of knowledge by providing supports in places of potential knowledge problems (e.g., query assistance tools like Clinical Queries in PubMed which serve to filter results by category such as therapy).

The HCI approach views users' insufficient knowledge and skills as leading to information retrieval difficulties. However, sub-processes such as information need formulation and results interpretation have not been studied in depth. Although this tradition recognizes that information needs are affected by domain knowledge, the organization of the domain knowledge into explanatory models and their role are not detailed beyond query term selection. In addition, this body of work has not focused on how domain knowledge interacts with critical reasoning in affecting the strategies of selecting and evaluating information. It is our contention that focusing on this relationship between knowledge and reasoning can shed light on some of the difficulties of consumer health information seeking. We also believe that the theoretical framework for understanding this relationship can be found in cognition and education studies describing how people generate knowledge by combining their existing beliefs with information extracted from external sources. This research focuses on learning as hypothesis testing, and describes situations when individuals have an opportunity to seek new information (evidence) and reconcile it with their existing beliefs (theory). 16 The term “lay theory” as used in this paradigm does not imply the accuracy, complexity and completeness characteristic of professional scientific theories. Rather, it reflects a stable and reasonably coherent system of causally connected beliefs (often based on experience), which provides the basis for explanatory models. 20 For example, beliefs that 1) viruses are related to dirt and 2) dirt can penetrate skin over a period of time, can lead to conclusions that HIV can be transmitted via superficial contact and prevented by good hygiene. 21 Jointly, these beliefs may constitute part of an individual's theory of HIV infection. Although the theory is inaccurate and not very complex, the conclusions may be quite resistant to change, due to the coherent and causal nature of the belief structure.

Evidence generation and interpretation by lay people are affected by their pre-existing theories in systematic ways. 16 People often set out to search for the evidence that supports their existing beliefs, a phenomenon referred to as confirmation bias. 22 Individuals are also typically reluctant to revise their viewpoints, often ignoring or reinterpreting contradictory evidence. 23 Cognitive decision-making literature has identified numerous biases that reflect deviation from rational thinking and are characteristic of lay as well as of professional reasoning. 24,25 Many of them are social (e.g., accepting the beliefs of the majority without questioning) or probabilistic in nature (e.g., assigning unjustifiably high weight to highly visible cases). There are also particular biases that may be of relevance to information seeking tasks. 26 Examples include screening out information that is viewed as irrelevant (selective perception) 27 and terminating search for evidence after encountering the first seemingly reasonable alternative (“satisficing,” premature search termination).

Information seeking can be construed as a kind of learning; it often involves looking for information in the context of a preexisting theory (i.e., a set of background assumptions that may guide the search). We therefore suggest that consumer health information seeking (especially in the absence of a diagnosis) may be largely influenced by pre-existing explanatory models of illness and complicated by the same challenges as other kinds of learning that involve hypothesis testing. In particular, we propose a framework that 1) reconceptualizes the iterative cycles of information seeking as attempts to verify or reject a pre-existing theory, and 2) integrates stages of the information-seeking process with hypothesis testing ([triangle]).

Figure 1
Combining Hypothesis-testing Framework with Cognitive Human–Computer Interaction approach.

In our modified framework, background knowledge (theory) and initial hypotheses constitute the beginning state (State 1), which defines the perceived information need and gives rise to the search goal (State 2). Theory and hypotheses influence search action steps by shaping the search goal, (State 3). Retrieved information is then evaluated in light of the existing theory (State 4), and theory revision is made only after the failure to “reconcile” the data with the theory. Individuals may also attempt to reconcile discrepant evidence with pre-existing theory, possibly by employing heuristics that are rooted in cognitive biases. Combining a hypothesis testing perspective with the HCI perspective may provide additional insight into consumers' difficulties in locating and interpreting health information. This hypothesis-testing approach to information-seeking points to two potential pitfalls in the search process. The first occurs at the point where knowledge-based hypothesis (hypothesis-testing framework) influences the search goal (HCI framework), where an incorrect hypothesis may lead to searching irrelevant resources. The second occurs during information evaluation, where prior hypothesis and background knowledge structure affect evidence interpretation. Our framework also links information seeking with comprehension, another core competency for utilizing health information effectively.

The study is conducted in the tradition of qualitative cognitive science research, which is based on the assumption that despite many individual differences, human reasoning and problem solving patterns are characterized by substantial regularities of the human information-processing system (e.g., limitations of working memory). 28 Thus the cognitive approach typically involves very detailed analysis of performance of a relatively small number of participants.

Methods

Participants

A convenience sample of 20 lay individuals (9 males, 11 females) participated in the study. Twelve were white, 6 African-American, one Asian, and one Hispanic. For 10 participants, the highest completed level of education was high school. Of the remaining 10, one held a bachelor's degree; seven held master's degrees; and two held doctoral degrees. As the study was carried out by a government agency (US National Library of Medicine) and only nine participants were recruited from the general public, no clearance was required, as per Paperwork Reduction Act (PRA) of 2005, 45 USC Sec. 3502 (see http://www.archives.gov/federal-register/laws/paperwork-reduction/3502.html). The rest of the participants were recruited among the employees and contractors working in the US National Library of Medicine.

Scenario and Procedure

Participants read a hypothetical scenario describing a relative who experienced symptoms typical of stable angina (Appendix A). The scenario was developed with the help of a medical expert. For simplicity, the terms “stable angina” and “angina” are used interchangeably in this paper to refer to stable angina. Participants discussed possible causes of the symptoms during semi-structured interviews and then searched MedlinePlus® for information on the disease presented in the scenario while thinking aloud. The semi-structured interview and the information seeking were audio-recorded. The search process was recorded with TechSmith Morae® screen capture and video-analysis software. Morae® provides a video of all screen activity and logs a wide range of events and system interactions including mouse clicks, text entries, web-page changes, and windows dialogue events as well as an audio recording of all verbalizations.

Data Analysis

All audio files were professionally transcribed. Sections of the transcripts were checked for accuracy by the first author and found to be satisfactory. The semi-structured interviews about participants' theories about the causes, condition and severity of the situation described in the scenario were analyzed independently from the information-seeking performance trajectories.

Analysis of Semi-Structured Interviews - Participants' Background Knowledge vs. Reference Model

Participants' theories reflecting their understanding of the scenario were coded via semantic analysis. 29 Sections of verbal protocols were segmented into pairs of concepts connected by a relation (e.g., propositions), and then represented as diagrams. Completeness and connectedness of participants' knowledge structures were then compared to a reference model of stable angina, as represented in consumer health materials. A reference model is a kind of gold standard for evaluating participants' knowledge or reasoning. 30 The reference model identified core concepts that explained the condition and specified semantic relationship among concepts. It was derived from the semantic analysis of several consumer health publications, primarily Angina and Coronary Artery Disease from the National Heart, Lung and Blood Institute (NHLBI) that emerged among the top results of a MedlinePLUS “angina” query. 31,32 Face validity of the model was verified in discussion with a clinician who held an MS degree in nursing and a doctorate in biomedical informatics. [triangle] illustrates the process of extracting the reference model's concepts and relationships from the consumer health texts.

Table 1
Table 1 Converting Consumer Health Text into a Reference Model

The reference model is presented in [triangle]. The model links three key concepts implicated in the scenario: a physiological process (atherosclerosis), a resulting disease (coronary artery disease, CAD) and its primary symptom (stable angina chest pain). The model also includes the concept of heart attack, a condition that may occur as a result of coronary artery disease (CAD) progression, and is characterized by actual damage to the heart tissue. Finally, the model includes a general concept of heart disease, which encompasses both CAD and its potential outcome, heart attack. The five symptoms from the hypothetical scenario (chest pain, neck pain, shortness of breath and nausea) are clustered as manifestations of a single, diagnostically significant problem.

Figure 2
Semantic analysis of stable angina model in the National Heart Lung and Blood Institute health text (reference model). Bold indicates concepts, accepted as correct answers in response to the scenario.

As the model is based on materials written for consumers rather than health professionals, it represents some simplification of the problem. For example, it does not describe the pathophysiological mechanisms by which CAD/angina may progress to a heart attack, and views them as related diseases, rather than two forms of the same condition. It does, however, make an important distinction between stable angina chest pain that often emerges as a response to physical exertion, lasts a few minutes and is relieved by rest or medication, and more severe and persistent chest pain that usually accompanies a heart attack. Descriptions of unstable and variant angina were not included in the reference model, as less relevant to the scenario.

Analysis of Information Seeking—Hypotheses, Search Goals, Search Actions, and Information Evaluation

Transcribed protocols of participants' information seeking verbalizations were integrated with Morae® logs of web-page changes and windows events. 33 Hypotheses statements made during the knowledge interviews were added to the transcripts. The resulting protocols were imported into QSR NVivo® software for qualitative thematic analysis. The software allows labeling segments of the data with descriptive coding categories and searching for patterns within and among the categories. Each protocol was then labeled with two kinds of codes: action-related codes and competency codes. Combining Morae® recordings with verbal transcripts allowed us to closely monitor users' behavior by coding each action and coordinating it with the verbal transcript. With the NVivo® analysis of the resulting protocols, we could examine search strategies across periods of time and visualize trends across subjects' behaviors.

Action-related codes pertained to steps of information-seeking cycle, outlined in our theoretical framework ([triangle]). Reflecting the core methodologies of cognitive studies of human performance, our scheme involved decomposing information seeking into a number of iterative goal-action-system response cycles (indicated in Bold). 15 The innovation involved expanding these steps into a system of sub-codes related to hypothesis-testing and evidence evaluation (indicated in [triangle] in italics). Although the top-level codes were predefined, sub-codes emerged from multiple readings of the data. After action coding was completed, each participant's information seeking path was graphically represented as series of states (user's knowledge statements and system's displays) and actions. Subsequently, we derived a set of common information-seeking and reasoning patterns based on individuals' trajectories in completing the task.

Table 2
Table 2 Information Seeking Codes

Finally, in order to characterize the role of user competencies evidenced during the performance, we modified the scheme by Sutcliffe and Ennis, 15 to include the following:

  • Domain Knowledge (e.g., of heart disease)
  • General Search Strategies (e.g., query expansion)
  • Resource Knowledge (of MedlinePlus®)
  • Metaknowledge (of desirable site characteristics)
  • Language (spelling and vocabulary knowledge)

These competency codes were assigned to specific utterances or action steps made during the information seeking process. Competency codes were assigned qualitatively, without value ranking, when evidenced in participants' utterances or behavior. For example, a statement that MedlinePlus® health topic pages had a Diagnosis section was coded as representing Resource Knowledge competency.

Results

In this section, we first present models of participants' understanding of the scenario. Next, we present common information-seeking patterns and relate it to their initial hypotheses. We conclude by characterizing the impact of user competencies and resource features during different search stages.

Participants' Demographics and Characteristics

Half of the participants had no formal education beyond high school, while the other half had at least college education. [triangle] summarizes participants' characteristics by education level, based on an introductory questionnaire. The table suggests that although all participants had significant computer and Internet experience, college-educated participants were more likely to have used MedlinePlus® prior to the study. Females were more likely than males to report frequently using the Internet to obtain health information. In the high-school education subgroup, three of the four participants reporting frequent (“often”) Health Internet use were females. In the college-or-above education subgroup, all four participants reporting frequent Health Internet use were females.

Table 3
Table 3 Demographics and Characteristics

Participants' Models of the Scenario: Background Knowledge and Hypotheses

This section corresponds to the content of the first component of our theoretical framework, background knowledge, theories and hypotheses. On the surface level, participants' understanding of the scenario could be labeled as incorrect or imprecise. None of the participants were able to identify stable angina (or angina) as the condition described in the scenario; none made a reference to coronary artery disease (CAD) as resulting in the symptoms. Their hypotheses about the situation differed in breadth and certainty. Seven participants proposed the Specific Hypothesis that the symptoms described in the scenario most likely signified a heart attack or a potential heart attack. While the “potential heart attack” hypothesis was not technically incorrect (stable angina increases the likelihood of a heart attack in the future), it is imprecise and not optimal in terms of prospective information-seeking trajectories. Eight believed that the character in the scenario suffered from some “heart problem,” while one suggested it was “old age” (Area Hypothesis). Four listed a number of cardiac and non-cardiac (e.g., stroke) illnesses that could explain the symptoms in the scenario (Assorted Hypothesis).

On a deeper, structural level, participants' understanding of the scenario differed from the reference model in three critical aspects: key concepts, symptoms' grouping and symptoms' characteristics. First, key concepts in the participants' models were different from those in the reference model. Two of the three disease key concepts of the reference model, coronary artery disease (CAD) and angina, were not mentioned by the participants. With respect to the third key reference model concept, atherosclerosis, a distinction needs to be made between participants' use of the term and their reference to the relevant concept. While the lexical form of the term was mentioned by only 3 of 20 participants, 18 made some reference to the blockage of blood vessels.

For all but one participant, cardiac concepts were prominent in their explanation of the disease. Eighteen participants mentioned heart attack either as their primary hypothesis or as a candidate hypothesis (classification of participants' hypothesis is presented in the following section). Blockage of blood vessels was seen as a main mechanism for causing cardiac problems; other mechanisms mentioned involved tear to the heart muscles, irregular heart beat and “electrical problem with the heart.” Eight participants suggested that the symptoms could be related to non-cardiac problems, such as stroke, arthritis, asthma and diabetes. These were not always supported by a physiological explanation (e.g., “This could be diabetes, because this disease can do weird things”).

The second distinction, symptoms' grouping, involves the tendency to connect the symptoms to either single or multiple conditions. The reference model presents all symptoms in the scenario as potential indicators of one condition. In the participants' models, nausea and dizziness were sometimes seen as unrelated to a cardiac problem, and indicative of a co-occurring non-cardiac condition (perhaps less worthy of immediate concern). The third distinction between the reference and participants' models involves the importance ascribed to certain symptoms' characteristics. None of the participants noted the significance of the short duration or pain, its relation to exertion and response to rest.

[triangle] presents a model of Participant 17, a woman with a high-school education, which encompasses the three characteristics of lay models described in this section. The concepts mentioned in her explanation did not include those prominent in the reference model (key concepts). She viewed all of the symptoms described in the scenario as potentially indicative of either stroke or heart attack (symptoms' grouping). In her model, both conditions were related to blockage of blood vessels in the brain (with heart attack being the body's response to the resulting stress). She also felt that shortness of breath could be indicative of asthma (symptoms' grouping). Characteristics of the chest pain are not discussed (symptoms' characteristics).

Figure 3
Semantic analysis of domain knowledge, participant 17.

The language used by participants in their discussions also suggests a potential problem related to the lack of medical vocabulary knowledge. While many participants seemed aware that the scenario did not conform to classic symptoms of a heart attack, and used descriptors such as “potential,” they did not seem to possess the vocabulary that would allow them to “legitimize” the phenomenon by labeling it, and later searching for it.

Information-Seeking Processes: Goal Setting, Search Execution and Information Evaluation

This section corresponds to steps 2–4 of our theoretical framework, analyzing the flow of the information-seeking process from goal setting, search action steps and information evaluation. The theoretical framework for this study suggests that information seekers' search goals are influenced by their prior knowledge and hypotheses. The analysis of the knowledge models in the previous section suggests that for many participants, their initial search moves will not directly involve (stable) angina and coronary artery disease (CAD), but will often lie within the domain of cardiac problems. Participants are also likely to look for information supporting their initial beliefs. Differences in the understanding of the relevant concepts and the relations among them may negatively impact information evaluation and navigation choices. However, the strength of the connection between knowledge/understanding and action is likely to vary, and strategies are likely to emerge that are not predicted by research on theory-evidence coordination. As the focus of the study is on characterizing information-seeking trajectories, we chose to partition the data on the basis of the initial search goals and moves rather than hypotheses, and to relate patterns or knowledge use to these trajectories. In all cases, the initial action was consistent with the explicitly expressed goal. We thus categorized the participants into three clusters: Verification-First; Problem Area Search-First; and Bottom-Up. Subsequent switches to another strategy did not affect cluster assignment. [triangle] summarizes key statistics for each cluster. [triangle] [triangle] [triangle] represent prototypical information seeking sequences in each cluster, described in-depth in the following three subsections.

Figure 4
Information-seeking sequences of verification-first participants with specific hypotheses.*
Figure 5
Information-seeking sequences of problem-area narrowing-first participants with area hypotheses.*
Figure 6
Information-seeking sequences of bottom-up-first participants with area hypotheses. *
Table 4
Table 4 Cluster Data Summary

Verification-First Cluster

Eight of the participants—P 1, 4, 6, 9, 11, 15, 17, 20— (40%) started by attempting to verify a specific illness. For five of the eight, the highest completed level of education was high school. Compared to the other clusters, participants in this cluster were more likely to start out with a specific hypothesis namely, a heart attack. All other hypotheses in this cluster also referenced heart attack. All participants navigated to a high-quality heart-attack site soon after starting their search. None used any strategies other than verification (e.g., bottom-up or narrowing). Seven participants arrived at the incorrect conclusion that the situation described in the scenario involved heart attack. They did it by focusing on the similarities between the descriptions of the heart attack on the sites and the scenario (squeezing chest, neck and shoulder pain; shortness of breath, nausea). At the same time, they ignored the differences: symptoms in the scenario emerged upon exertion, lasted 2–3 minutes only, and were alleviated by rest. As described in the previous section, the relative importance ascribed to various symptoms' characteristics is different in participants' models and the text-based reference model. Ignoring symptoms' characteristics that are viewed as non-essential can also be seen as exemplary of the selective perception bias. 27

A notable reasoning pattern in this cluster involved the confirmation bias: starting with the heart attack hypothesis, navigating to a heart attack site and concluding that the information confirmed their hypothesis ([triangle]). Three participants in this group also demonstrated what can be interpreted as premature search termination bias by stopping their search after reviewing only one (in all cases, incorrect) content topic and judging the information to be satisfactory.

The search trajectory of Participant 4 (high school education) illustrates the confirmation bias pattern in this cluster ([triangle]). In responding to the scenario during the semi-structured interview, she expressed Specific heart attack hypothesis by saying, “I guess, it almost sounds like she's having a heart attack. Especially with all her symptoms. Um, the squeezing of the chest. The pain down her arm. Or I should say her shoulder and having the pain after physical activity and also being nauseous are the ones that probably concern me the most.” Afterwards, she explained her information-seeking goal as wanting “to look up heart attack.” She then typed “heart attack” query into MedlinePlus® search window (Verification Strategy), selected National Library of Medicine Heart Attack Portal from the list of results, and proceeded to Diagnosis/Symptoms section, “What Are the Symptoms of a Heart Attack?” site by the Cleveland Clinic Foundation (Search Actions). Once on the page, she scrolled to the section that stated “Symptoms of a heart attack include.” While reading the symptoms' list, she noted that some of the symptoms were present in the scenario (e.g., chest pain and nausea), while others were absent (e.g., sweating and light-headedness). She then concluded that in her opinion, the character in the scenario was suffering a heart attack. Only one participant in this group did not follow the pattern, noting the difference in the duration of the symptoms. He then followed a link from the MedlinePlus® heart attack encyclopedia page to the angina page, and concluded that the scenario described angina. This participant had graduate-level education and had used MedlinePlus® on many occasions in the past.

Figure 7
Participant 4 search trajectory (verification-first, confirmation bias).

Problem Area Narrowing-First Cluster

Five participants—P2, 3, 8, 13 and 19—(25%) started with problem area search. High school was the highest completed level of education for two of the five. Four of these participants had Area hypotheses, one had Assorted hypothesis. These participants started with either a general query such as “heart disease” (3 participants) or by browsing the site index tree (2 participants). One participant eventually switched to a bottom-up “chest pain” query; the rest continued with the narrowing strategy throughout the session. All eventually navigated to sites describing specific diseases. However, three of the five spent time on sites that had little potential for answering their questions (e.g., health news about a specific treatment procedure). Unlike the participants in the verification-first cluster, these participants were more likely to leave without a conclusion than with an incorrect conclusion.

[triangle] represents a prototypical model of problem-area narrowing-first search, representing 4 out of 5 participants in this cluster. One of the two trajectories in this model is exemplified by Participant 3's performance. This participant held a master's degree and was very familiar with MedlinePlus®. When discussing the scenario, she states “I'd be concerned about heart, because she is not getting enough oxygen or something” (Area hypothesis). She then proceeded to type “heart diseases” query into MedlinePlus® search window (Narrowing Strategy), and followed the link to the National Library of Medicine Heart Diseases portal. She then read the links under the Diagnosis and Symptoms subsection and said, “What I am finding here are all tests, nothing about symptoms, so, I am disappointed. But here is one, Heart Attack, Stroke and Cardiac Arrest.” She then followed that link to the American Heart Association site, and read the list and description of heart attack symptoms: chest discomfort, discomfort in the other areas of the upper body, shortness of breath, and others (cold sweat, nausea and lightheadedness). The description of chest discomfort indicated that it “lasts more than a few minutes” or “goes away and comes back.” She then concluded, “These look like they could be symptoms, so I feel like they you know help me feel better about saying this could be something really serious … she could be having very minor heart attacks.” As the pain episode in the scenario only lasts 2–3 minutes, this can be viewed as selective perception bias. However, this reasoning may also be ascribed to the ambiguity in the text and the scenario. It is not clear from the text how long the period of recurrence is in “goes away and come back,” while the scenario states that the character “has been troubled by periodic squeezing pain in her chest” for the past year.

We can also partially account for some of the problems by considering the configuration of the web resources. The results of many users' queries and browses displayed links to relevant information about angina. These links, however, were not prominently displayed in the users' view and some did not realize that the choices were available to them. For example, a “heart disease” query produced a list of subtopics that included the relevant Coronary Heart Disease and Heart Diseases. However, these followed a number of specific irrelevant subtopics (e.g., Heart Valve Diseases). Some relevant results appeared so far down the screen they could not be seen without scrolling. In the topics index tree, Angina was listed under Heart and Circulation, but users did not select it from the alphabetical sequence, perhaps because they did not know the term.

Bottom-Up First Cluster

[triangle] represents prototypical information seeking sequences in this cluster. Seven participants P5, 7, 10, 12, 14, 16, 18 (35%) started their search by attempting a bottom-up search. Five of these participants began without a specific hypothesis. For two of the participants, the bottom-up strategy proved immediately unsuccessful: they attempted to locate a general purpose diagnostic tool not included in MedlinePlus®. These participants switched to hypothesis-driven strategies and arrived at the incorrect heart attack conclusion. The five remaining participants made progress to varying degrees towards the accurate conclusion. One chose a heart attack site from the list of results and concluded that the scenario in fact described a heart attack. Another navigated to a low-relevance site about the mechanism of pain. Three participants went to a potentially useful familydoctor.org site that contained chest pain diagnostic flowcharts. However, one of them selected the flowchart for diagnosing “acute” rather than “chronic” chest pain, and ended up concluding that it was a heart attack. The other two employed a “chronic chest pain” flowchart available on one of the sites, but were unable to follow it to angina.

Like the participants from the narrowing group, these searchers encountered relevant links, but they were scattered throughout variably relevant subtopics, sometimes below the fold (necessitating scrolling). Consistent with the previous research, 5 this was partly the function of imprecise queries. For example, entering the terms “recurrent, 2—3 minute episodes, squeezing chest pain” results in the “Angina” subtopic presented at the top of the results list. However, the users' queries were much less specific (“chest pain,” “chest pain and nausea”), most likely reflecting the perceived relevance of various symptoms and symptoms' characteristics in the scenario. This issue is illustrated by the search performance of Participant 12 (who had a high-school education). He started the session by stating an intention to find out “what it means when somebody has pain and nausea when they are being physically active,” and proceeded to type in “chest pain” as a query. The list of resulting topics was divided into the following categories: Pain, Back Pain, Angina and Abdominal Pain. This participant followed the link to Why Do I Have Pain? children's site by Nemours Foundation. 34 The site described the brain's processing of the sensation of pain. The participant spent a significant amount of time on the site, praising the amount of information it contained, “This is what research is. If you have a computer at home, you got a library at home.” He then clicked back to the list of results, stating that he would have to study all of them carefully, and that he would also have to study the results for nausea. The participant finally clicked on several links under Pain, Back Pain and Angina subheadings, scanning the content rather than reading, and finished the session with a positive comment about his experience, but didn't reach a conclusion.

As in the other clusters, some participants tended to ignore details of textual information to which they did not ascribe significance. For example, Participant 14 read the following description of a heart attack, “… chest pain or discomfort in the center of the chest, a squeezing, heaviness or crushing feeling. Lasts more than a few minutes or goes away and comes back” and concluded, “Wow. That seems like a lot like what she's having.” As in the example in the previous section, this can be interpreted either as selective perception bias or as an appropriate response to the text's ambiguity. Early termination of the search was less common than in Verification-First Cluster, with only two of the participants restricting their review to a single content webpage.

Other Factors: The Role of Web Resources, Education Level and User Competencies

One of our goals was to characterize the role of different kinds of competencies during the task, as well as the role of web resources in mediating the effect of these competencies. The frequency distribution of different kinds of qualitative competency codes suggests that different types of knowledge were instrumental during different search stages. Domain knowledge codes were more likely to appear during goal setting and information evaluation, whereas domain understanding provided the context that determined the direction of the search as well as the interpretation of the results. Resource knowledge, strategies and metaknowledge codes were more likely to appear during navigational action steps. [triangle] illustrates this point by presenting micro-coded protocol excerpts for Participant 17. In the first statement of the coded segment 1, this participant expresses her intention to execute a “heart attack symptoms” query, because in her domain knowledge model chest pain is indicative of a heart attack. This statement concerns goal setting, and domain knowledge defines the intended search trajectory by providing a hypothesis to verify. The second part of the coded segment 1 and the coded segment 2 concern the navigation to a heart attack site. They involve a strategic choice between conducting a query and following a link and the knowledge of functions available in a browser (e.g., clicking the back button). Despite the failure of the initial query due to a spelling error, the participant's strategic repertoire helped her to bypass the problem and find the site she perceived as relevant. In the final (evaluation) segment, the domain knowledge once again becomes central. The participant evaluates the information on the site and draws on her background knowledge, and concludes that the character in the scenario could be suffering from a heart attack.

Table 5
Table 5 Excerpts of Coded Information Seeking Protocol of Participant 17

Level of education appeared to differentially impact the various information seeking competencies. Regardless of their level of education, participants demonstrated comparable levels of understanding of the symptoms described in the scenario (domain knowledge), as evidenced in the Participants' Models of the Scenario section. At the same time, participants with higher levels of education were more likely to be familiar with MedlinePlus® (only one college-educated participant has never heard of this portal, as opposed to eight high-school educated participants), have a repertoire of efficient search strategies (e.g., using keyboard shortcuts to open multiple tabs, exhibited by three college-educated participants and none of the high school participants), and make meta-level comments (e.g., judging the authoritativeness of a source). However, incomplete and inaccurate domain knowledge often led these participants to apply their efficient strategies to the wrong pages. It also resulted in them disregarding certain key aspects of the information.

Lack of medical vocabulary knowledge presented an additional challenge. While most participants mentioned clogging of the arteries when discussing the possible mechanisms of heart disease, they lacked precise labels for their concepts. As a result, they could not use the relevant terms in queries, and did not recognize them when scanning topics or lists of results. Many participants read the term “angina” while perusing an index tree, yet they did not perceive it as relevant. Participants with high school education were more likely to comment on the vocabulary-related difficulties. For example, one such participant started the session by selecting the Heart and Circulation subtopic of MedlinePlus® index. Upon scanning the resulting list of topics, he noted, “Now not knowing the doctor's stuff… I don't see [anything] for the average, every day Joe Public here. Maybe I'll go back up here to a search engine.” When asked to clarify what caused this difficulty, the participant explained, “See, they say ‘For Cardiac Disease see Heart Disease.' Heart disease I know, but cardiac disease, I don't know what that is.”

Two aspects of the MedlinePlus® interface (redesigned since the study was conducted) might have contributed to shaping the search trajectories that emerged in this study. The first was the lack of explicitness in relating lay and professional terms in the index. For example, in the alphabetical list of topics, the Chest Pain title suggested that the reader see Angina. However, the Angina title made no references to chest pain. The second aspect had to do with the order and organization of query results lists, where specific relevant links were presented after general and less relevant links. The search of Participant 12, presented in a previous section, illustrates this point, where the list of responses to the “chest pain” query started with the general Pain category, and were followed by Back Pain category, and only then by Angina (below the fold on many monitors).

Conclusions

Results of this study suggest that when individuals search the web for complex information in the absence of a diagnosis, they may not get to the pages that provide the needed information. Participants with strong specific hypotheses (incorrect in all cases) were most likely to start out by attempting to verify their specific ideas, and to interpret the information they found as supporting their initial beliefs. They were also least likely to relinquish their initial strategy in favor of others. Participants in the other two clusters, on the other hand, were more likely to terminate their search before reaching a conclusion (and thus feel dissatisfied with their results). If individuals approach the task with a specific preconception of what the answer might be, they may leave satisfied and with incorrect beliefs that match their initial hypotheses. If their preconceived notion is not specific, they are more likely to leave without an answer. Many will discuss health information they found online with their health care providers, and may have their misconceptions corrected. However, 35% of individuals seeking online health information for themselves do it independently of a doctor's visit, 1 and supporting their independent search is one of the goals of the consumer health effort.

Lay health theories are often marked by gaps and misconceptions that impact reasoning; yet, the relevant body of theory-evidence coordination research has not played a role in decision-making and information-seeking theories thus far. This study contributes to the field by relating consumer health information-seeking difficulties to theory-evidence coordination. Our theoretical framework informed a methodological approach for studying the mechanism by which individuals' health knowledge affects the trajectory and the outcome of health information seeking. The framework had suggested that the influence of prior knowledge/hypotheses should be traced throughout all stages of information seeking. It also pointed to hypothesis generation and information evaluation as two points where pre-existing beliefs are most likely to lead to sub-optimal choices and biased reasoning strategies.

Not unexpectedly, lay health theories differ from the professional ones not only in their breadth, but also in their focus and organization. 35 For better or for worse, knowledge provides guiding assumptions for setting information goals and evaluating retrieved information. Incomplete or inaccurate knowledge increases the chance that information seekers will be influenced by irrelevant information. It may also affect the precision of search queries, leading to sub-optimal outcomes. Moreover, efficient search skills, internet experience and education may not compensate for the lack of domain knowledge. Our results suggest that at least some health searches may be conceptualized as a sort of hypothesis testing, complicated by difficulties of hypothesis formulation, evidence selection and interpretation. Even in cases where participants initially attempted to suspend their judgment and perform a bottom-up search, they relied on their background knowledge and hypotheses in selecting what they perceived as relevant from the list of results. In analyzing participants' information-seeking patterns, we noted several cognitive biases that could be particularly relevant to evidence gathering and knowledge generation (e.g., confirmation bias, selective perception, and premature termination of search for evidence). Lau and Coiera 26 identify other biases that may be instrumental in health information-seeking, and provide evidence that the effect of biases is not limited to lay people.

Limitations of lay health knowledge do not mean that we should not try to facilitate health consumers' access to online health information. Health educators and designers of consumer health sites need awareness of the situation and ways to implement supports in places where the potential for errors is high and the consequences potentially serious. Support can be provided at three levels: that of information portals such as MedlinePlus®, individual websites, and education tools (e.g., online tutorials). On the level of portals, we can address the problem of imprecise, short and poorly worded queries, implicated in this study, as well as in the works of other researchers. 36 This can be done via providing users with query formulation support tools, suggesting additional or alternative query terms, and therefore making the query more specific. 37 We can also attempt to counter confirmation bias by asking users to identify their information needs within a taxonomy that distinguishes between information-seeking in the presence and absence of a diagnosis. In addition, the portal may offer affordances or guides that facilitate hypothesis testing by presenting information so as to emphasize possible alternatives, as well as to suggest a consultation with a health professional. On the level of individual sites, we can attempt to address the needs of our target users as well as the needs of those who are “lost” by placing prominent pointers to related topics and explaining how they are related. On all levels, we need to rely on consumer-friendly terminology, 38 providing consumer-friendly definitions and explicitly linking them to professional terms (e.g., “angina is a medical term for chest pain”) and defining qualifiers (e.g., “chronic” and “acute”). Intervening at the level of consumer health education may involve teaching consumers to formulate specific queries, to evaluate qualifiers in the information and to not terminate their searches prematurely, for example, after viewing what seems at a glance like plausible evidence that supports their hypothesis. However, providing such consumer education may be as difficult as compelling software users to read the manual.

Some challenges of extrapolating from the results of this study to the experience of various user groups involve the artificiality of the scenario and the sample of participants, partially recruited from the employees of the National Library of Medicine. While convenience sampling and use of scenarios are common in qualitative research, they need to be considered in a discussion of the generalizability of the results. One of the tenets of cognitive science is that although individuals differ greatly in their knowledge and experience, their reasoning and decision-making are characterized by consistent patterns, or regularities. 28 These patterns can sometimes be identified via an in-depth analysis of performance of a small sample of participants. 39 Data convergence into a limited number of patterns points to these patterns' significance. A larger quantitative study could provide a useful follow-up by supplementing our descriptive data with the information about frequency of various patterns and their impact on information-seeking performance across the range of topics and user competencies. If these patterns are linked to suboptimal user performance on computer systems, they also provide valuable information for design improvement. Statistical analysis of users' search trajectories in MedlinePlus® is a potentially interesting approach to investigating patterns' frequency.

This study identified three patterns of information seeking, which were partially explained by the participants' domain knowledge and hypotheses, as well as faulty evidence evaluation strategies. The information-seeking difficulties experienced by the participants were systematic and could be largely explained by the theoretical framework. In addition, some of the problems evidenced in this study are analogous to findings pertaining to lay decision making and scientific reasoning. 40 We suggest that these sort of problems are worthy of consideration in the context of the design of consumer e-health information resources. Informatics support features implemented in the design may improve the accuracy of search and results interpretation. However, like any in-depth qualitative cognitive analysis, our study does not provide information about the distribution of the identified information-seeking patterns in the general population, and does not exclude the possibility of other influences on information seeking. One specific limitation of the sample includes a convenience subsample of individuals with uniformly frequent Internet use (which may have obscured “navigational” difficulties).

The very low success rate in the study may be partially explained by characteristics of the research situation: the hypothetical nature of the scenario, which may have affected the users' motivation; the complexity of the diagnostic task and the similarity between angina and heart attack symptoms. The scenario asked participant to imagine a family member experiencing symptoms of stable angina. This scenario is realistic given that nearly all participants were able to recount stories of family members or friends, suffering from some cardiac symptoms. At the same time, a hypothetical situation does not produce the level of emotional investment and motivation that would be evoked by a real health hazard to oneself or to a loved one. In this regard, validity and generalizability considerations are similar to those presented in the discussion of the study sample. The study provides information about cognitive barriers to effective information-seeking strategies, but does not address the potential mediating impact of emotional and motivational factors. Future studies should focus on diversifying the skill level of the sample, varying the set of tasks and domains in which tasks are presented, and further integrating the analysis of knowledge and competency usage with the analysis of information seeking as hypothesis testing steps. Ultimately, such studies can result in a model of information seeking that will account for variations in information behavior from hypothesis generation to conclusions across levels of competency. This model may not only seed ideas for design, but provide targets for training in terms of fostering the various competencies.

Acknowledgments

The authors thank Tony Tse for discussions throughout the project and assistance with earlier versions of this manuscript, Ilyse Rosenberg and Hyeoneui Kim for help in developing the scenario and providing medical expertise, Rob Logan and Laurel Graham for comments on earlier versions of the manuscript, Tiago Soromenho-Ramos for assistance with the graphics, and Jessica Ancker for her invaluable editorial help. This research was supported by the Intramural Research Program of the NIH, National Library of Medicine and by NIH grant 1R01LM008799-01A1.

Appendix A

Hypothetical Scenario about a Relative Suffering from Stable Angina

You are visiting your mother * who lives alone, in another part of the country. On the second day of your visit, she carries several bags of groceries up two flights of stairs and stops with a pained expression on her face. When you press her to tell you what is wrong, she admits that she is having chest pain. She says that the pain feels as if something were squeezing her chest. She is also nauseous and out of breath. She lies down to rest. The discomfort lasts 2–3 minutes, after which the pain stops.

When you talk to her about this incident, she admits that for the past year, she has been troubled by periodic squeezing pain in her chest. Sometimes she can also feel the pain in her neck and shoulders. The pain usually happens after she does something physically active: climbs several flights of stairs, does some heavy housework, unloads groceries, etc. When this happens, she also often feels nauseous and out of breath. She also feels very tired.

The pain typically lasts a few minutes and goes away after she rests a while.

She tells you not to worry, it is probably nothing, and is probably normal for her age.

Footnotes

* For younger participants, the mother's age was specified as “late sixties”.

References

1. Fox S. Pew Internet Project Report: Online health search 2006http://www.pewinternet.org/report_display.asp?r=190Accessed June 2008.
2. Department of Health and Human Services Expanding the Reach and Impact of Consumer E-HEALTH TOOLShttp://www.health.gov/communication/ehealth/ehealthTools/pdf/ehealthreport.pdfAccessed June 2008.
3. Cline R, Haynes KM. Consumer health information seek-ing on the Internet: the state of art Health Educ Res 2001;16(6):671-692Dec;. [PubMed]
4. Brewer NT. The impact of Internet use on health cognitions and behaviorIn: Haugtvedt CP, Machleit KM, Yalch R, editors. Online consumer psychology: Understanding and influencing consumer behavior in the virtual world. Mahwah, NJ: Lawrence Erlbaum; 2005.
5. James N, Daniels H, Rahman R, McConkey C, Derry J, Young A. A Study of Information Seeking by Cancer Patients and their Carers Clin Oncol (R Coll Radiol) 2007;19:356-362. [PubMed]
6. Hogan TP, Palmer CL. Information preferences and practices among people living with HIV/AIDS: results from a nationwide survey J Med Libr Assoc 2005;93(4):431-439. [PMC free article] [PubMed]
7. Fox S, Fallows D. Pew Internet Report: Online health resources. 2003http://www.pewinternet.org/pdfs/PIP_Health_Report_July_2003.pdf 2005. Accessed June 2008.
8. Pereira J, Bruera E. The Internet as a resource for palliative care and hospice: a review and proposals J Pain Symptom Manage 1998;16(1):59-68Jul;. [PubMed]
9. McLellan F. “Like hunger, like thirst”: patients, journals, and the internet Lancet 1998;352(Suppl 2):SII39-SII43Oct;. [PubMed]
10. Fava GA, Guidi J. Information overload, the patient and the clinician Psychother Psychosom 2007;76(1):1-3. [PubMed]
11. Chapman K, Abraham C, Jenkins V, Fallowfield L. Lay understanding of terms used in cancer consultations Psychooncology 2003;12(6):557-566. [PubMed]
12. Lorence DPGL. The zeitgeist of online health search. Implications for a consumer-centric health system. J Gen Intern Med 2006;21(2):134-139. [PMC free article] [PubMed]
13. Fox S. Pew Internet Report: The Online Health Care Revolution: How the Web helps Americans take better care of themselves. 2000[updated 2000; cited]http://www.pewinternet.org/pdfs/PIP_Health_Report.pdf 2006. Accessed June 2008.
14. Ferris DG, Nyirjesy P, Sobel JD, Soper D, Pavletic A, Litaker MS. Over-the-counter antifungal drug misuse associated with patient-diagnosed vulvovaginal candidiasis Obstet Gynecol 2002;99(3):419-425. [PubMed]
15. Sutcliffe A, Ennis M. Towards a cognitive theory of in-formation retrieval Interacting with Computers 1998;10:321-351.
16. Kuhn D. How do people know? Psychol Sci 2001;12:1-8. [PubMed]
17. Norman DA, Draper S. User Centered System Design: New Perspectives on Human-Computer InteractionHillsdale, NJ: Lawrence Erlbaum Associates; 1986.
18. Marcionini G. Information seeking in electronic environmentsCambridge: Cambridge University Press; 1995.
19. Jansen BJ, Pooch U. Web user studies: a review and framework for future work JASIST 2001;52(3):235-246.
20. Vosniadou S. Conceptual change research: State of the art and future directionsIn: Schnotz W, Vosniadou S, Carretero M, editors. New perspectives on conceptual change. Amsterdam: Pergamon; 1999.
21. Keselman A, Kaufman DR, Patel VL. “You can exercise your way out of HIV” and other stories: The role of biological knowledge in adolescents' evaluation of myths Sci Ed 2004;88(4):548-573.
22. Chapman GB, Elstein AS. Cognitive processes and biases in medical decision makingIn: Chapman GB, Sonnenberg FS, editors. Decision making in health care: theory, psychology and applications. Cambridge: Cambridge University Press; 2000. pp. 183-210.
23. Chinn CA, Brewer WF. The role of anomalous data in knowledge acquisition: A theoretical framework and im-plications for science instruction RER 1993;63:1-49.
24. Kahneman D, Slovic P, Tversky A. Judgment under uncertainty: Heuristics and biasesNew York: Cambridge University Press; 1982.
25. Elstein AS. Heuristics and biases: Selected errors in clinical reasoning Acad Med 1999;74:791-794. [PubMed]
26. Lau AYS, Coiera EW. Do people experience cognitive biases while searching for information? J Am Med Inform Assoc 2007;14(5):599-608. [PMC free article] [PubMed]
27. Burgoon JK, Burgoon M, Miller GR. Learning theory approaches to persuasion Human Comm Res 1981;7(2):161-179.
28. Patel VL, Kaufman DR, Arocha JF. A primer on aspects of cognition for medical informatics J Am Med Inform Assoc 2001;8(4):324-343. [PMC free article] [PubMed]
29. Patel VL, Groen GJ. The general and specific nature of medical expertise: A critical lookIn: Ericsson A, Smith J, editors. Toward a General Theory of Expertise: Prospects and Limits. New York: Cambridge University Press; 1991. pp. 93-125.
30. Patel VL, Kaufman DR. Cognitive science and biomedical informaticsIn: Shortliffe EH, Cimino JJ, editors. Biomedical Informatics; Computer Applications in Health Care and Biomedicine. 3rd ed.. New York, NY: Springer; 2006.
31. NHLBI Anginahttp://www.nhlbi.nih.gov/health/dci/Diseases/Angina/Angina_WhatIs.html 2006. Accessed June 2008.
32. NHLBI Coronary Artery Diseasehttp://www.nhlbi.nih.gov/health/dci/Diseases/Cad/CAD_WhatIs.html 2006. Accessed June 2008.
33. Kaufman DR, Patel VL, Hilliman C, Morin PC, Pevzner J, Weinstock RS, et al. Usability in the real world: assessing medical information technologies in patients' homes J Biomed Inform 2003;36(1–2):45-60. [PubMed]
34. Nemours Foundation Why do I have pain?http://www.kidshealth.org/kid/talk/qa/pain.html 2003. Accessed June 2008.
35. Patel VL, Arocha JF, Kushniruk AW. Patients' and physicians' understanding of health and biomedical concepts: relationship to the design of EMR systems J Biomed Inform 2002;35(1):8-16. [PubMed]
36. Eysenbach G, Kohler C. How do consumers search for and appraise health information on the world wide web?. Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002;324(7337):573-577. [PMC free article] [PubMed]
37. Zeng QT, Crowell J, Plovnick RM, Kim E, Ngo L, Dibble E. Assisting consumer health information retrieval with query recommendations J Am Med Inform Assoc 2006;13(1):80-90Jan-Feb. [PMC free article] [PubMed]
38. Zeng QT, Tse T. Exploring and developing consumer health vocabularies J Am Med Inform Assoc 2006;13(1):24-29Jan-Feb. [PMC free article] [PubMed]
39. Preece J, Rogers Y, Sharp H. Interaction design: Beyond human–computer interactionNew York, NY: Wiley; 2002.
40. Kuhn D. Children and adults as intuitive scientists Psychol Rev 1989;96:674-689. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles