Send to

Choose Destination
Pharmacoepidemiol Drug Saf. 2019 Nov 17. doi: 10.1002/pds.4892. [Epub ahead of print]

Quantifying how small variations in design elements affect risk in an incident cohort study in claims.

Author information

US Food and Drug Administration, Center for Drug Evaluations and Research, Silver Spring, Maryland.
Division of Biostatistics and Study Methodology, The George Washington University, Children's National Research Institute, Silver Spring, Maryland.
Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts.



Epidemiological study reporting is improving but is not transparent enough for easy evaluation or replication. One barrier is insufficient details about design elements in published studies.


Using a previously conducted drug safety evaluation in claims as a test case, we investigated the impact of small changes in five key design elements on risk estimation. These elements are index day of incident exposure's determination of look-back or follow-up periods, exposure duration algorithms, heparin exposure exclusion, propensity score model variables, and Cox proportional hazard model stratification. We covaried these elements using a fractional factorial design, resulting in 24 risk estimates for one outcome. We repeated eight of these combinations for two additional outcomes. We measured design effects on cohort sizes, follow-up time, and risk estimates.


Small changes in specifications of index day and exposure algorithm affected the risk estimation process the most. They affected cohort size on average by 8 to 10%, follow-up time by up to 31%, and magnitude of log hazard ratios by up to 0.22. Other elements affected cohort before matching or risk estimate's precision but not its magnitude. Any change in design substantially altered the matched control-group subjects in 1:1 matching.


Exposure-related design elements require attention from investigators initiating, evaluating, or wishing to replicate a study or from analysts standardizing definitions. The methods we developed, using factorial design and mapping design effect on causal estimation process, are applicable to planning of sensitivity analyses in similar studies.


claims data; design elements; exposure algorithm; fractional factorial design; index date; risk estimation process; sensitivity analyses


Supplemental Content

Full text links

Icon for Wiley
Loading ...
Support Center