• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of poqLink to Publisher's site
Public Opin Q. 2008; 72(5): 892–913.
Published online Dec 12, 2008. doi:  10.1093/poq/nfn059
PMCID: PMC3022327
NIHMSID: NIHMS107787

Eye-Tracking Data

New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding

Abstract

Survey researchers since Cannell have worried that respondents may take various shortcuts to reduce the effort needed to complete a survey. The evidence for such shortcuts is often indirect. For instance, preferences for earlier versus later response options have been interpreted as evidence that respondents do not read beyond the first few options. This is really only a hypothesis, however, that is not supported by direct evidence regarding the allocation of respondent attention. In the current study, we used a new method to more directly observe what respondents do and do not look at by recording their eye movements while they answered questions in a Web survey. The eye-tracking data indicate that respondents do in fact spend more time looking at the first few options in a list of response options than those at the end of the list; this helps explain their tendency to select the options presented first regardless of their content. In addition, the eye-tracking data reveal that respondents are reluctant to invest effort in reading definitions of survey concepts that are only a mouse click away or paying attention to initially hidden response options. It is clear from the eye-tracking data that some respondents are more prone to these and other cognitive shortcuts than others, providing relatively direct evidence for what had been suspected based on more conventional measures.

Introduction

Cannell, Miller, and Oksenberg (1981) presented what was probably the first “cognitive” model of the survey response process. Their model featured two tracks, or routes, respondents take to arrive at their answers. In one track, respondents first understand the question and then carry out a series of potentially burdensome cognitive processes such as recalling relevant facts and organizing these memories. This track, according to Cannell and his coworkers, is the one that leads to accurate responses. In the other track, respondents bypass much of this cognitive work and instead give answers marked by acquiescence or other response sets. Much of the research reported by Cannell and his collaborators sought to find methods to keep respondents motivated so that they stayed on the first track as they answered survey questions. More recently, Krosnick (1991, 1999) has published work on survey satisficing that extends the idea that respondents may vary in how much cognitive effort they are willing or able to expend in answering survey questions. Highly motivated or highly able respondents may “optimize,” carefully executing each of the components of the response process. Others with less motivation or less cognitive capacity may “satisfice,” taking various shortcuts to complete the questionnaire.

One way that survey satisficing manifests itself is in response order effects (Krosnick and Alwin 1987; Miller and Krosnick 1998). Response order effects (in which the order of the response categories affects the distribution of the answers) can occur in self-administered surveys when respondents process the first response options more deeply than the later options. Krosnick and his colleagues argue that working memory is limited and that respondents are unable to give the later options as much attention as the ones they consider initially. Because respondents are prone to “confirmation” bias, this deeper scrutiny makes respondents more likely to select the first options they consider; that is, response options in surveys are generally designed to sound plausible and respondents are likely to react favorably to them. A related possibility is that some respondents simply process the options sequentially and stop processing completely when they come to an answer that's good enough (Tourangeau 1984); such respondents never even consider what may be better answers that are listed after the one they chose.

Additional evidence of the limited effort survey respondents sometimes invest comes from studies on providing definitional material in Web surveys. Conrad et al. (2006) examined several methods for presenting definitions for key terms in web survey questions and found that few respondents (about one in six) accessed the definitions at all. In addition, the more effort it required to get the definitions, the less likely the respondents were to consult them. Fewer respondents opened definitions when it took two mouse clicks to access them than when it took just one. And those respondents who did obtain definitions might not have attended to the details of the definitions (Tourangeau et al. 2006).

All of these findings, however, are based on relatively indirect data about respondents’ behavior during the survey: their answers, response latencies, and mouse movements. These data are helpful but they cannot answer all the questions we have about mechanisms underlying survey responses. In this study, we use eye-tracking data to answer such previously unresolved questions. Are primacy effects mostly due to shallower processing of response options positioned lower on the list or to skipping them altogether? Will initially hidden response options receive the same amount of attention as options that were visible immediately? Even if definitions were easy to access or always visible on the screen, would respondents read them or would they skip them anyway? As opposed to other methods, eye tracking gives us a relatively direct window into what respondents attend as they process the questions and helps us to test different hypotheses about their survey behavior.

For more than a century, tracking of eye movements has been used in studies of cognitive processes in reading, visual search, scene perception, and face recognition (cf. Rayner 1998). Eye tracking is also often used in studying cognition in real world situations such as watching TV commercials (Woltman-Elpers, Wedel, and Pieters 2003) and reading print advertisements (Pieters, Rosbergen, and Wedel 1999; Wedel and Pieters 2000; Rayner et al. 2001; Liechty, Pieters, and Wedel 2003), making a sandwich (Land, Mennie, and Rusted 1999; Land and Hayhoe 2001), flying a plane (Thomas and Wickens 2004), or driving a car (Ho et al. 2001). Eye tracking is still rarely used in survey methodological research, however, even though the processing of self-administered survey questions involves reading and visual search. For two exceptions, see Redline and Lankford (2001) and Graesser et al. (2006). Technology for tracking eye movements has become increasingly easy to use in recent years. Current eye-tracking equipment no longer requires awkward and invasive apparatuses such as special lenses or helmets. Instead, the combination of near-infrared beams that reflect off of the retina and digital cameras that track head position makes it possible to record eye movements more or less unobtrusively with an adequate precision for many practical applications.

We recorded respondents’ eye movements while they were completing a Web survey. The questionnaire included several experiments that manipulated response order, the format of the response options, and the accessibility of definitions of key concepts. We knew from previous studies (Schuman and Presser 1981; Krosnick and Alwin 1987; Couper, Tourangeau, and Kenyon 2004; Conrad et al. 2006) that these manipulations could produce changes in the answers. Our study examines whether these changes can be explained by the amount of time people spend looking at different parts of the questions. For example, we compared how much time respondents fixated on the top half of a list of options with the time they spent fixated on the bottom half. We also examined how much time they spent looking at definitions of key terms, depending on how much effort was needed to access the definitions. In addition, we used eye tracking to study more general characteristics of respondents’ behavior such as the way they worked through longer and shorter lists of response options, the change in their answers due to the time they spent processing definitions, and the consistencies in the level of vigilance (or the lack of it) they show throughout the questionnaire.

Methods

Eye-tracking equipment

We used the Tobii 1750 eye-tracking system (www.tobii.com) which includes hardware for unobtrusive tracking of eye movements and software for data analysis. The hardware resembles an ordinary computer monitor and uses near-infrared beams and video images to capture the respondent's eye movements; no special helmets, lenses, or other equipment are needed. The procedure is harmless for the respondents. The accuracy of recordings is satisfactory for our purposes and for most practical applications: the margin of error for timing of eye movements is ± 3 milliseconds and for position of eye fixations ± 0.5–1° (a degree of visual angle amounts to approximately 1 cm on the screen at a 50-cm distance). The frame rate is 50 Hz, meaning that a data point is produced every 20 milliseconds. The ability to track eye movements without any cumbersome gear or head restraints, however, brings some limitations. For instance, this level of spatial precision is not enough to determine which words exactly the respondents read—at least unless letters are made to be unusually large. Also, calibration (positioning of respondent's eyes) tended to decrease in accuracy as the questionnaire progressed. Our study took 10 minutes to complete; in studies that are longer than ours, one or more additional calibrations might be necessary at 10- to 15-minute intervals.

Sample

The study was conducted in February and March 2006. We recruited 120 respondents through fliers on the University of Maryland campus and ads in local (College Park and Greenbelt, MD) newspapers. Technical difficulties prevented recording of eye movements for four of the respondents. In addition, in each of the three experiments, 10 respondents (not always the same ones) had recordings that were not of satisfactory quality due to systematic shifts in the tracking of the eye movements relative to the probable locations of the fixations and/or due to unreliable tracking measures (with high unsystematic error). The systematic shifts were easy to spot visually—the apparent positions of the fixations were consistently a fixed distance above (or below) the lines of text on the screen. These recordings were excluded from the subsequent analyses, leaving 108 respondents with good quality recordings in at least one of the three experiments and 106 with good recordings in all three. Of those 108, 50 percent were between 18 and 24 years old, 35 percent between 25 and 34, and 15 percent between 35 and 64 years of age; 51 percent were male; 13 percent had high school education, 52 percent some college, and 35 percent had a college degree. Most (83 percent) used the Internet every day, 60 percent considered themselves advanced or expert users of the Internet, and 77 percent had already participated in at least one prior Web survey.

Procedure

Respondents were seated in front of the Tobii screen. After they signed a consent form that described the eye-tracking procedure and completed a short calibration exercise (in which they followed dots displayed at different parts of the screen with their eyes), the respondents completed the Web questionnaire that included the experiments presented in this paper. The questionnaire took about 10 minutes to complete; the respondents’ eye movements were recorded during that time. After that, the respondents participated in an unrelated experiment for another 20 minutes and were paid $20. The study was approved by the IRBs at both the University of Maryland and University of Michigan, the home institutions of the authors.

Materials and experimental design

The questionnaire included about 20 questions. The first two-thirds included questions belonging to different experiments and the last third asked about various demographic characteristics. There were five experiments, three of which are reported here, while the remaining two (examining pictures as sources of visual context effects, and the effects of the content and font of response options on answers) are reported elsewhere. The complete questionnaire is available from the authors on request. We describe each experiment in more detail in the following section.

Results

In all analyses, we took into account all fixations that lasted at least 100 milliseconds and encompassed 20 pixels (about one word of text) in our questionnaire. As a sensitivity check, we repeated all analyses with a 50-millisecond cut-off, but all of our conclusions remained unchanged (the results are available on request). Average fixation duration across the questions analyzed in this study was 242 milliseconds (SD = 182.0). The average size of a saccade (a fast eye movement between two fixations) was 52 pixels (SD = 92.6) and was similar for all experimental conditions described below. The variance of saccade sizes is large because our survey questionnaire often required jumps of different sizes (i.e., between item texts, different response options, scale labels, buttons, etc.).

Experiment 1: Effects of order of response options

Survey researchers have known for some time that the order of the answer options can affect which answer the respondents select. With response options presented visually (rather than aloud), it is often observed that the options near the top of the list are more likely to be selected than those closer to the bottom (Krosnick 1991, 1999). As we noted earlier, there are at least two possible mechanisms underlying these primacy effects. First, respondents may just select the first acceptable answer they read, not bothering to read the later options at all. On the other hand, respondents may read all the options but spend fewer cognitive resources processing the later options compared to those presented earlier (cf. Krosnick and Alwin 1987; Krosnick 1991). Either way, if eye fixations reflect the attention respondents give to an option, respondents would spend more time looking at the top options. In addition, the first mechanism would predict very little or no time at all spent on looking at the options that come after the one that is chosen.

Experiments that only examine the respondents’ answers cannot discern whether one or the other mechanism (or both) is responsible for primacy effects. In contrast, eye movement data can provide evidence for distinguishing the two mechanisms. In this experiment, we reversed the order of the response options in four questions for a random half of respondents and measured the time they spent looking at the options in the first versus the second half of the list. All questions were adapted from major surveys and were used in previous studies of response order effects (Schuman and Presser 1981; Krosnick and Alwin 1987). The full text of the questions is available in the Appendix. One question asked about desirable qualities of a child and had 12 answer options; others asked about trust in police officers and had 5 options, and attitudes towards crime and morality had 2 options each. All of these questions were positioned in the second half of the questionnaire and each question was shown on a separate screen.

Similar to large-scale surveys, we observed response order effects in our laboratory study—respondents were more likely to choose the options in the first half of the list than the second half for all items. These primacy effects were more pronounced for the items with longer lists of options. For the question about child qualities with 12 response options, 60 percent of the respondents chose a quality from the first half of the list, significantly more than the 50 percent that would be expected had the position had no effect (χ2(1) = 4.20, p =.04). For the question about trust in police officers on a five-point scale, if we ignore the respondents who chose the middle option1 (n = 80), 58 percent chose one of the first two options (χ2(1) = 2.45, p =.12), and 42 percent chose one of the last two—whether those options indicated trust or distrust. For the two questions with only two response options, the upper option was chosen by 54 and 51 percent of respondents for questions about crime and morality, respectively (n.s.).

Did respondents also attend for longer periods of time to the options in the first half of the list? For all four items, we observed similar patterns (see table 1): respondents spent more total time looking at the response options in the first half of the list than at those in the second. It is possible that some of these differences reflect the time needed to click on the selected option so that the added time spent represents response execution rather than the selection of the preferred option. Table 1 therefore shows what happens when we subtracted the time spent fixating on the radio buttons2 from the total time that respondents spent looking at the text of the response options (corrected fixation times and number of fixations). The difference in time spent on the first half of the options is still greater than those in the second half.

Table 1
Total Fixation Time (Raw and Corrected), by Position of Options, Order, and Item

In analyses of variance with repeated measures on the two halves of the list of response options and with the order of options as a between-subjects factor, the within-subject effects of first versus second half were significant for all four items—F(1, 104) = 18.55, F(1, 104) = 24.10, F(1, 104) = 8.42, and F(1, 104) = 10.63, all p =.01, for the questions on children, police, crime, and morality, respectively. There were no interactions between the fixation times on the two halves and order of response options, suggesting that options in the first half of the list were fixated longer regardless of their content. The same conclusions followed from the analysis of number of fixations in each half.

We found support for both of the mechanisms that might explain response order effects. First, as shown in figure 1, the larger the proportion of time respondents spent looking at the top half of the options, the more likely they were to choose an option from that part. According to logistic regression analyses, odds ratios were 2.08, 2.19, 4.62, and 1.60 (all p =.01) for the questions on children, police, crime, and morality, respectively. If we assume that the time spent fixating on an option reflects, at least in part, the time spent on processing that option (Just and Carpenter 1980), this gives support to the idea that primacy effects are partially caused by processing the later options less deeply than the earlier ones. Interestingly, respondents who spent little time fixating second half of response options in one question were not more likely to show the same behavior in another question. The highest correlation between the proportions of time spent fixating top halves of options of different questions was r(107) =.24 (p =.02), for the two items with two options each.

Figure 1
Relationship between the Proportion of Time Spent Looking at the First Half of Response Options, and Likelihood of Choosing an Option from the First Half, for Each Question.

Second, some respondents did not look at the later options at all. For example, with the item listing 12 characteristics of children, 10 percent of the respondents did not fixate on either of the last two options. For the question about trust in police which used an ordered five-point response scale (from “great deal of trust” to “great deal of distrust”), the results were more dramatic. Only 54 percent fixated on the last option of the scale. These results support the idea that the observed primacy effects are a product of two behavioral patterns: reading all options but paying more attention to the first ones on the list and skipping the latter options altogether. These patterns, reminiscent of Krosnick's (1991) description of “weak” and “strong” satisficing, are for the first time directly observed. Traditional methods, using only respondents’ answers, were able only to find supporting data for these assumptions, but tracking respondents’ eye movements gives us the opportunity to look directly at how they processed the survey questions.

Experiment 2: Effects of response format

Many Web surveys use drop-down lists to elicit answers. Couper et al. (2004) compared answers to the same questions in three formats: radio buttons with all the response options visible immediately; a scrollable drop-down list with only half of the options visible at the outset; and a drop-down list with none of the options visible until the respondent clicked on the list (see figure 2). They also compared two orders for the response options and found primacy effects with all three formats: the options from the top half of the list were more likely to be selected than those from the bottom. The effect was far more marked, however, in the drop-down box condition in which only the first half of the options were initially visible. There are two possible explanations for this difference across response formats. First, the respondents might never activate the drop-down list to reveal the options that weren't visible initially—either because they are unwilling to expend the additional effort or because they don't know how to activate the list. Second, even if they do uncover the bottom options, it is possible that they still pay more attention to the options that were visible from the outset. Eye tracking can help us to resolve this issue by revealing not only how many people activated the list but also how long they looked at different parts of the list.

Figure 2
Response Formats Used in the Second Experiment.

We replicated the Couper et al. (2004) experiment described above, this time recording the respondents’ eye movements. We used two questions with 10 substantive response options each. One question asked about the nutrients that were important in choosing a breakfast cereal and the other asked about features that were important when deciding on which automobile to purchase. The exact wording of the questions is available in the Appendix. We reversed the order of the substantive options for half of the respondents; the option “None of the above” was always at the end of the list.

The questions were presented in three formats (see screenshots in figure 2). A third of the respondents got the response options for both questions in radio button format with all options immediately visible; a third got both as drop-down lists with five options visible initially; and the final third got both questions as drop-down lists with no immediately visible options. The respondents got both questions in the same format.

For both questions, the options in the top part of the list were chosen more often than those in the bottom half. Of the respondents who selected one of the substantive answers, 55 percent across the three versions chose one of the top five options in the question about breakfast cereal and 66 percent did so in the question about automobiles (though only the latter percentage is significantly different than 50 percent, (χ2(1) = 10.78, p =.001). This effect was more pronounced, however, when only five options were visible initially (63 and 78 percent for the questions about cereal and automobiles, respectively) than when all of the options (52 and 69 percent) or none of them (52 and 53 percent) were visible at the outset. How are these findings related to fixation times?

As table 2 shows, respondents looked longer at the first half of the list of response options in most conditions. As with Experiment 1, we present both uncorrected fixation times and fixation times corrected for the fixations that occurred during mouse clicks. On average, the respondents spent 61 percent (for the question about cereal) and 57 percent (for the question about automobiles) of their time looking at options in the top half of the list. We carried out paired t-tests (on the corrected data) that examined the difference between the time each respondent fixated on the top and bottom half of the options. The results indicated that the corrected fixation time in the top half was significantly higher for both questions: t(91) = 3.80 (p =.01) and t(100) = 3.65 (p =.01) for cereal and automobiles, respectively. This difference was most pronounced, however, when five options were visible initially (for cereal: 76 percent, t(29) = 5.40, p =.001; for automobiles: 69 percent, t(32) = 4.61, p =.001). Some respondents never activated the drop-down list and thus never saw the bottom options (43 percent for breakfast cereal and 41 percent, for automobiles question, respectively). Even those who did activate the list spent more time looking at the first than at the second five options (64 percent, t(16) = 2.42, p =.03; and 60 percent, t(20) = 3.10, p =.006, respectively).

Table 2
Total Fixation Time (Raw and Corrected) and Number of Fixations, by Position of Options, Question Format, and Item

The advantage for the first half was much smaller in the radio buttons condition in which all options were visible from the outset (for cereal: 59 percent, t(32) = 1.65, p =.11; for automobiles: 54 percent, t(35) = 1.24, p =.22) and disappeared completely when none of the options were visible initially (for cereal: 48 percent, t(28) = −.61; for automobiles: 51 percent, t(32) =.33). The overall effect of question format on the time spent looking at the two halves of the list was significant for both questions (for cereal: F(2, 89) = 16.84; for automobiles: F(2, 98) = 6.51, both p =.01), with post hoc tests indicating that the proportion of time spent on the first five options was significantly longer when only those five options were visible than when all or none of the options were visible initially. When we took into account only those respondents in the version with five initially visible options who activated the drop-down list, the overall difference between the formats was still pronounced for the question about breakfast cereal (F(2, 76) = 3.99, p =.02), though not for the question about automobiles (F(2, 90) =.71). The number of fixations in different formats followed the same pattern as the total fixation time.

As in Experiment 1, the longer the respondents looked at the response options in the top part, the more likely they were to choose one of them. Regression analyses controlling for question format showed that the respondents who spent more than half of the time fixating on the first half of options were much more likely to choose one of those options for both questions (odds ratios were 15.00 and 5.43, both p =.01).

Experiment 3: Reading definitions

Surveys often ask about concepts that are not understood in the same way by everyone or concepts that are unfamiliar to an ordinary respondent. In such cases, a clarification about the intended meaning of questions can improve the accuracy of respondents’ answers, sometimes dramatically (Schober and Conrad 1997; Conrad and Schober 2000; Schober, Conrad, and Fricker 2004). In Web surveys, clarification can be provided in different ways. For example, respondents can sometimes click on a term to open a separate window with the definition. Another possibility is to allow the respondents to obtain definitions by merely positioning (“rolling over”) the mouse pointer on the term; this involves less effort because they do not need to plan and execute a mouse click. The rollover strategy has proved to be very effective in a previous Web experiment (Conrad et al. 2006, Experiment 2): four times as many respondents obtained definitions by means of rollovers (37 percent) as did by clicking (8 percent).

If the amount of effort is such an important issue, would simply making the definitions always visible lead to a further increase in the percentage of respondents who read them? Or would the respondents simply ignore them as they do with other unrequested online content such as banners and Web page headers? This question cannot be answered without determining whether respondents actually look at the definitions. We conducted an experiment using the rollover interface from the second experiment from Conrad et al. (2006), added a condition in which definitions were always visible, and recorded respondents’ eye movements while they were estimating their consumption of eight different nutrients, including fat, dietary supplements, and vegetables. See the Appendix for the full text of the questions. The questions were shown in two grids of four questions each. A definition was available for each nutrient and was either always visible (for a random half of the respondents) or had to be activated by positioning the mouse pointer above the key term (for the other half). Figure 3 shows the two experimental conditions.

Figure 3
Presentation of Definitions in the Third Experiment.

The definitions were not always intuitive. Sometimes they included surprising elements, such as that, “Fat supplies essential fatty acids which reduce chances for heart attacks, cancer, asthma, depression, accelerated aging, obesity, diabetes, arthritis, and Alzheimer's disease”; that potato chips and French fries are vegetables; or that some dietary supplements improve sexual performance. The average number of words in the definitions was 29, ranging from 20 to 32.

In the “rollover” condition, each definition was opened and looked at by an average of 10 percent of the respondents compared to an average of 78 percent of the respondents in the “always-on” condition. Of the respondents in the “rollover” condition, 28 percent activated at least one definition and only one respondent opened all eight definitions. In the “always-on” condition, all respondents fixated on at least one definition and 45 percent looked at all eight definitions. Although fewer respondents opened the definitions in the “rollover” condition, those who did spent significantly more time reading them than the respondents who looked at the definitions in the “always-on” condition. When respondents obtained definitions in the “rollover” condition, the mean reading time across the eight items was 4.2 seconds compared to 2.5 seconds in the “always-on” condition (in paired t-test, t(14) = 2.35, p =.03).

The definitions in both conditions affected estimates of consumption for some of the nutrients. For example, the longer the respondents fixated on the definition of dietary supplements which touted them as helping to protect cells against aging, improve sexual performance, and reduce stress, the more they felt they were not consuming enough of them (r(104) = −.27, p =.01). The more time they spent fixated on the definition of grain products, which included drinks such as beer, the more they felt they were consuming too much of them (r(104) =.23, p =.02). Small changes also occurred with longer fixations on the definitions for fat (r(104) = −.11), vegetables (r(104) =.14), and calcium (r(104) = −.13). Regression analyses including the interaction of reading time and the format of the definitions showed that these results held for both the rollover and the always-on condition.3 The more respondents read the definitions, the more the definitions seemed to affect their answers.

General observations about respondents’ answering styles

The respondents tended to read response options from top to bottom and only about half of them revisited options they’d read before (from 43 percent for the question about trust in police officers to 57 percent for the question about child qualities). Some respondents merely skimmed through the response options, especially when the lists were longer. For example, only 75 percent of all respondents fixated for at least 100 milliseconds on each of the 12 options for the question on desirable qualities for children. Even in the two questions with only two response options (about attitudes related to crime and morality), only 80 and 77 percent fixated on both options. Finally, when the response options were ordered, forming a graded scale, such as in the question that asked about level of trust in police officers, only 26 percent of the respondents fixated on each one of them; the remaining respondents appeared to have inferred what the rest of the options were from reading just a few. This is reminiscent of the eye-tracking results reported by Graesser et al. (2006). They found that respondents frequently answered the question before reading the entire question stem, especially when the question syntax was complex and among respondents who had scored poorly on an intelligence test.

Some respondents seemed to adopt a particular style for working through questions. The average correlation between the number of options that were fixated on in the four questions mentioned above was r(105) =.57, p =.001. This means that some respondents were more likely than others to skim through the options. In fact, only 23 percent of the respondents fixated on every one of the options in all four of those questions; the remaining 77 percent skipped at least one option. Those who fixated on all options were also more conscientious on other questions. For example, they opened more definitions in the “rollover” condition (1.9 versus 0.4 definitions, t(52) = −2.88, p =.01) and when they opened them, they spent more time reading the definitions (20.8 versus 4.9 seconds on average, t(13) = −1.84, p =.09). They also spent more time reading the general instructions on the very first page of the questionnaire (11.7 versus 7.7 seconds, t(105) = −2.69, p =.01). These results suggest that some respondents were more prone than others to “satisficing” behavior (cf. Krosnick and Alwin 1987), which was reflected in different ways throughout the survey. We did not find any significant demographic differences between the “conscientious” respondents and the other respondents nor did we observe any difference in their previous experience with Web surveys.

Discussion

Eye tracking proved to be a useful method for investigating the mechanisms underlying response order effects and for understanding the impact of the format of the items and of definitional material. We were able to answer three previously unresolved questions. First, what is the mechanism behind primacy effects? Our results show that these effects occur because respondents often spend more time processing the top than the bottom part of the list of response options and because some respondents don't bother to read the bottom response options at all. These findings are consistent with theoretical arguments put forward by Krosnick and Alwin (1987), Krosnick (1991), and Tourangeau (1984). Figure 4 shows two examples of gaze plots for the question on child qualities: the respondent in the top panel fixated on (and presumably read) all options, while the respondent in the bottom panel of the figure did not read the last few options.

Figure 4
Gaze Plots of Two Different Respondents to the Question about Child Qualities.

Second, will initially hidden response options get the same amount of attention as the immediately visible ones once they are uncovered? Our experiments suggest that this is not the case. Even when the initially hidden options are revealed, they tend to get less fixation time than the options that were visible from the beginning. This may contribute to a bias toward choosing the initially visible options more often. These results suggest that using formats with only a few options visible should be avoided.

The third question deals with how to present definitions of potentially difficult survey concepts. Should they be always visible, potentially cluttering up the screen, or should they be accessible only on request? Our results suggest that making the definitions always visible on the screen might be the best way to increase their chances of being read to any extent. Formats that require even the smallest amount of respondent effort such as rolling the mouse over a term to obtain its definition will decrease the chance that respondents will access and read the definitions. This is in accord with the results of Gray and Fu (2004) whose respondents, while trying to program a VCR, found even an eye movement to be too much effort and instead relied on what they remembered. In our experiment, only about one-third of the respondents in the rollover condition activated at least one of the eight definitions and most activated only one. On the other hand, once they activated a definition, they spend more time reading it than the respondents who “got it for free” in the always-on condition. In addition, the more time the respondents spent reading the definitions, the more the definitions affected their answers. Perhaps the optimal approach is to motivate respondents to request definitions but still require them to take some initiative, even at a low level of effort (such as the rollover).

Our results confirm the suspicion of survey researchers that some respondents are more conscientious than others (Cannell, Miller, and Oksenberg 1981). The same respondents who took care to read all the response options also spent more time reading instructions and definitions in other parts of the questionnaire. We did not find any differences between more and less conscientious respondents in their sociodemographic characteristics or other relevant variables. The level of conscientiousness can, however, lead to apparent substantive differences between those two groups in combination with certain questionnaire properties (for instance, listing the response options in the same order for everyone). Survey researchers should therefore make every effort to motivate the respondents to be attentive to all aspects of questions—for instance, by making the questionnaire short and interesting. In addition, as some respondents will probably take cognitive shortcuts in spite of all our efforts, it is wise to design the questionnaire to offset the effects of these burden-reducing strategies—for example, by rotating the order of response options or by making all materials and response options immediately visible.

Summary and Conclusions

We recorded respondents’ eye movements while they were participating in experiments that varied the order and format of the response options and the accessibility of definitions for terms in survey questions. When all response options were visible initially, respondents looked at the options near the top of the list longer than they looked at those at the bottom. In addition, they were more likely to choose the options at the top. This phenomenon was even more pronounced when only some options were visible initially and the others had to be uncovered with a mouse click. Definitions were more likely to be read at all when they were always present on the screen compared to when even a slight effort such as positioning the mouse pointer over a defined term was required to reveal the definition. Definitions were likely to be read in more depth, however, when respondents requested them via a rollover. Respondents were mostly reading from top to bottom but with a noticeable amount of backtracking to earlier options. Nevertheless, many respondents skimmed through at least some of the response options and such behavior was consistent across questions and reflected in other indicators of the level of respondent conscientiousness. Our results suggest that seemingly minor differences in visual design may produce significant changes in respondents’ answering behavior and offer strong support to the idea that some respondents are merely satisficing as they work their way through questionnaires.

Eye tracking has great potential to advance our knowledge of the processes underlying survey response. Questions such as whether primacy effects result from shallower processing of later options or from skipping them altogether, whether respondents read all options in drop-down lists, and whether they read definitions accompanying survey questions cannot be definitively answered without data on respondents’ eye movements. The eye-tracking data add important evidence in support of existing hypotheses and allow for a more nuanced understanding of survey response effects.

Appendix

Text of questions used in the three experiments.

Experiment 1. Response order effects.
[Child qualities]
Which three qualities listed below would you say are the most desirable for a child to have?
  That he has good manners
  That he tried hard to succeed
  That he is honest
  That he is neat and clean
  That he has good sense and sound judgment
  That he has self-control
  That he acts like a boy or she acts like a girl
  That he gets along well with other children
  That he obeys his parents well
  That he is responsible
  That he is considerate of others
  That he is interested in how and why things happen
[Crime]
Some say individuals are more to blame than social conditions for crime and lawlessness in this country. Others say the contrary—social conditions are more to blame than individuals for crime and lawlessness in this country. Which one of these two statements comes closest to your opinion on this issue?
 Individuals are more to blame
 Social conditions are more to blame
[Police]
Next, we would like you to think about the amount of trust you have that the police officers in your area will always do what is right. Would you say you have
  A great deal of trust
  A moderate amount of trust
  Equal amounts of trust and distrust
  A moderate amount of distrust
  A great deal of distrust
[Morality]
In your opinion, should government (federal, state, or local) have some responsibility for preventing the breakdown of morality, or should private organizations and individuals be entirely responsible for preventing the breakdown of morality?
 Government is responsible
 Private organizations and individuals are responsible
Experiment 2. Question format effects
[Breakfast cereal]
Which of the following nutrients is MOST important to you when selecting breakfast cereal? (select one)
  Protein
  Carbohydrates
  Sugar
  Fat
  Fiber
  Vitamin A
  Vitamin C
  Calcium
  Iron
  Vitamin E
  None of the above
[Automobiles]
Which of the following features do you feel is MOST important in making a decision on which automobile to purchase? (select one)
  Engine size
  Transmission type (manual or automatic)
  Color
  Antilock brakes
  Make and model
  4-wheel drive
  Automatic steering
  Electronic windows and locks
  Gas mileage
  Price
  None of the above
Experiment 3. Definitions
How much of the following items do you typically consume?
[1st grid]
 Fat
 Dietary supplements
 Grain products
 Poultry
[2nd grid]
 Vegetables
 Dairy products
 Cholesterol
 Calcium
[Response options for all nutrients]
 Much less than I should
 Somewhat less than I should
 As much as I should
 Somewhat more than I should
 Much more than I should

Footnotes

1We removed the middle option selections from this analysis because by definition, this option is neither first half nor second half.

2This time was ~212 milliseconds in our sample, which corresponds to a 200-millisecond estimate from the human–computer interaction studies; cf. Card, Moran, and Newell (1983); Kieras (1993).

3We corrected the fixation time in the “always-on” condition by deducting the fixations that occurred when respondents were moving their eyes up and down between the radio buttons within the grid and the labels of the response options at the top of the grid. To estimate the duration of such fixations, we used the data from the “rollover” definition condition, in which definitions were not visible unless activated by the respondents. This correction does not change the overall conclusions.

References

  • Cannell Charles F., Miller Peter V., Oksenberg Lois. Research on Interviewing Techniques. In: Leinhardt Samuel., editor. Sociological Methodology, 1981. San Francisco, CA: Jossey-Bass; 1981. pp. 389–437.
  • Card Stuart, K., Moran Thomas P., Newell Allen. The Psychology of Human-Computer Interaction. Hillsdale, NJ: Lawrence Erlbaum Associates; 1983.
  • Conrad Frederick G., Schober Michael F. Clarifying Question Meaning in a Household Telephone Survey. Public Opinion Quarterly. 2000;64:1–28. [PubMed]
  • Conrad Frederick G., Couper Mick P., Tourangeau Roger, Peytchev Andrey. Use and Non-use of Clarification Features in Web Surveys. Journal of Official Statistics. 2006;22:245–69.
  • Couper Mick P., Tourangeau Roger, Conrad Frederick G., Crawford Scott. What They See Is What We Get: Response Options for Web Surveys. Social Science Computing Review. 2004;2:111–27.
  • Couper Mick P., Tourangeau Roger, Kenyon Kristen. Picture This! Exploring Visual Effects in Web Surveys. Public Opinion Quarterly. 2004;68:255–66.
  • Graesser Arthur C., Cai Zhiqiang, Louwerse Max M., Daniel Frances. Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility. Public Opinion Quarterly. 2006;70:1–20.
  • Gray Wayne D., Fu Wai-Tat. Soft Constraints in Interactive Behavior: The Case of Ignoring Perfect Knowledge In-The-World for Imperfect Knowledge In-The-Head. Cognitive Science. 2004;28:359–82.
  • Ho Geoffrey, Scialfa Charles T., Caird Jeff K., Graw Trevor. Visual Search for Traffic Signs: The Effects of Clutter, Luminance, and Aging. Human Factors. 2001;43:194–207. [PubMed]
  • Just Marcel A., Carpenter Patricia A. A Theory of Reading: From Eye Fixations to Comprehension. Psychological Review. 1980;87:329–54. [PubMed]
  • Kieras David. Using the Keystroke-Level Model to Estimate Execution Times. 1993. University of Michigan, available at ftp://www.eecs.umich.edu/people/kieras/GOMS/KLM.pdf .
  • Krosnick Jon A. Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys. Applied Cognitive Psychology. 1991;5:213–36.
  • Krosnick Jon A. Survey Research. Annual Review of Psychology. 1999;50:537–67. [PubMed]
  • Krosnick Jon A., Alwin Duane. An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement. Public Opinion Quarterly. 1987;51:201–20.
  • Land Michael F., Hayhoe Mary. In What Ways Do Eye Movements Contribute to Everyday Activities. Vision Research. 2001;41:3559–65. [PubMed]
  • Land Michael F., Mennie Neil, Rusted Jenny. The Roles of Vision and Eye Movements in the Control of Activities of Daily Living. Perception. 1999;28:1311–28. [PubMed]
  • Liechty John, Pieters Rik, Wedel Michel. Global and Local Covert Visual Attention: Evidence from a Bayesian Hidden Markov Model. Psychometrika. 2003;68:519–41.
  • Miller Joanne M., Krosnick Jon A. The Impact of Candidate Name Order on Election Outcomes. Public Opinion Quarterly. 1998;62:291–330.
  • Pieters Rik, Rosbergen Edward, Wedel Michel. Visual Attention to Repeated Print Advertising: A Test of Scanpath Theory. Journal of Marketing Research. 1999;36:424–38.
  • Rayner Keith. Eye Movements in Reading and Information Processing: 20 Years of Research. Psychological Bulletin. 1998;124:372–422. [PubMed]
  • Rayner Keith, Rotello Caren M., Stewart Andrew J., Keir Jessica, Duffy Susan A. Integrating Text and Pictorial Information: Eye Movements When Looking at Print Advertisements. Journal of Experimental Psychology: Applied. 2001;7:219–26. [PubMed]
  • Redline Cleo D., Lankford Christopher P. Eye-Movement Analysis: A New Tool for Evaluating the Design of Visually Administered Instruments (Paper and Web); 2001. Proceedings of the Survey Research Methods Section of the American Statistical Association, available at www.amstat.org/Sections/Srms/Proceedings/y2001/Proceed/00248.pdf .
  • Schober Michael F., Conrad Frederick G. Does Conversational Interviewing Reduce Survey Measurement Error? Public Opinion Quarterly. 1997;61:576–603.
  • Schober Michael F., Conrad Frederick G., Fricker Scott S. Misunderstanding Standardized Language in Research Interviews. Applied Cognitive Psychology. 2004;17:22–41.
  • Schuman Howard, Presser Stanley. Questions and Answers in Attitude Surveys. San Diego, CA: Academic Press; 1981.
  • Thomas Lisa C., Wickens Christopher D. Eye-Tracking and Individual Differences in Off-Normal Event Detection When Flying With a Synthetic Vision System Display. 2004. Paper read at Human Factors and Ergonomics Society, 48th Annual Meeting, Santa Monica, CA, USA.
  • Tourangeau Roger. Cognitive Science and Survey Methods. In: Jabine Thomas B., Straf Miron L., Tanur Judith M., Tourangeau Roger., editors. Cognitive Aspects of Survey Design: Building a Bridge Between Disciplines. Washington, DC: National Academy Press; 1984. pp. 73–100.
  • Tourangeau Roger, Conrad Fred G., Arens Zachary, Fricker Scott, Lee Sunghee, Smith Elisha. Everyday Concepts and Classification Errors: Judgments of Disability and Residence. Journal of Official Statistics. 2006;22:385–418.
  • Wedel Michel, Pieters Rik. Eye Fixations on Advertisements and Memory for Brands: A Model and Findings. Marketing Science. 2000;19:297–312.
  • Woltman-Elpers Josephine L. C. M., Wedel Michel, Pieters Rik G. M. Why Do Consumers Stop Viewing Television Commericals? Two Experiments on the Influence of Moment-to-Moment Entertainment and Information Value. Journal of Marketing Research. 2003;40:437–53.

Articles from Public Opinion Quarterly are provided here courtesy of Oxford University Press
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Links

  • MedGen
    MedGen
    Related information in MedGen
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...