Purpose: To understand how methodological factors influence prevalence estimates of health-risk behaviors obtained from surveys, we examined the effect of varying question wording and honesty appeals while holding other aspects of the surveys constant.
Methods: A convenience sample of students (n = 4140) in grades 9 through 12 was randomly assigned to complete one of six versions of a paper-and-pencil questionnaire in classrooms. Each questionnaire version represented a different combination of honesty appeal (standard vs. strong) and questionnaire type. The questionnaire types varied in wording and in the number of questions assessing particular types of behaviors. The questionnaires were based on those used in three national surveys--the Youth Risk Behavior Survey, Monitoring the Future, and the National Household Survey on Drug Abuse. Logistic regression analyses examined how responses to each survey question assessing behavior were associated with questionnaire type, honesty appeal, and the interaction of those two variables.
Results: Among 32 behaviors with different question wording across questionnaire types, 12 showed a significant effect of questionnaire type. Among 45 behaviors with identical question wording across questionnaire types, five showed a significant main effect of questionnaire type. Among all 77 behaviors, one showed a significant main effect for honesty appeal and two showed a significant interaction between honesty appeal and questionnaire type.
Conclusions: When population, setting, questionnaire context, mode of administration, and data-editing protocols are held constant, differences in question wording can create statistically significant differences in some prevalence estimates. Varying honesty appeals does not have an effect on prevalence estimates.