Send to

Choose Destination
Med Decis Making. 2019 Aug;39(6):693-703. doi: 10.1177/0272989X19856617. Epub 2019 Aug 28.

Exclusion Criteria as Measurements I: Identifying Invalid Responses.

Author information

Department of Engineering & Public Policy, Carnegie Mellon University, Pittsburgh, PA, USA.
The Institute for Politics and Strategy, Carnegie Mellon University, Pittsburgh, PA, USA.
Department of Social and Decision Sciences, Carnegie Mellon University, Pittsburgh, PA, USA.
Division of General Internal Medicine, University of Pittsburgh, Pittsburgh, PA, USA.
Department of Health Policy and Management, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA, USA.


Background. In a systematic review, Engel et al. found large variation in the exclusion criteria used to remove responses held not to represent genuine preferences in health state valuation studies. We offer an empirical approach to characterizing the similarities and differences among such criteria. Setting. Our analyses use data from an online survey that elicited preferences for health states defined by domains from the Patient-Reported Outcomes Measurement Information System (PROMIS®), with a U.S. nationally representative sample (N = 1164). Methods. We use multidimensional scaling to investigate how 10 commonly used exclusion criteria classify participants and their responses. Results. We find that the effects of exclusion criteria do not always match the reasons advanced for applying them. For example, excluding very high and very low values has been justified as removing aberrant responses. However, people who give very high and very low values prove to be systematically different in ways suggesting that such responses may reflect different processes. Conclusions. Exclusion criteria intended to remove low-quality responses from health state valuation studies may actually remove deliberate but unusual ones. A companion article examines the effects of the exclusion criteria on societal utility estimates.


exclusion criteria; health state valuation; preference-based measures; study design

[Available on 2020-08-28]

Supplemental Content

Full text links

Icon for Atypon
Loading ...
Support Center