Abstract

Health care payers and policy makers need information about the cost and effectiveness of medical treatments. While randomized controlled trials historically are the primary source of medical information, they are expensive and labor-intensive, and often have limited utility for answering questions about "real-world" patient populations. These problems have led to an increasing reliance on claims database research in making policy decisions about treatment options. However, both researchers and decision makers should recognize the limitations and unique features of claims databases. Recommendations for avoiding or minimizing threats to internal validity, construct validity, and external validity are: (1) use of a study design that includes comparisons; (2) ensuring that the study design and conclusions are consistent with the database; (3) a priori conceptual modeling of the research question; (4) use of appropriate constructs; (5) explicit examination of alternative explanations for study findings; (6) sensitivity analyses of key assumptions; (7) awareness of the distinction between statistical and practical significance of findings; (8) generalization only when appropriate; and (9) reporting of relevant information. Given that any study design or data source has limitations, we hope that this paper will encourage a philosophy of methodological pluralism in outcomes research. Awareness and accurate reporting of validity issues will strengthen and extend the information resources currently available to decision makers.