• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Stat Med. Author manuscript; available in PMC Jan 25, 2010.
Published in final edited form as:
PMCID: PMC2811095
NIHMSID: NIHMS168143

Estimation of average treatment effect with incompletely observed longitudinal data: Application to a smoking cessation study

Abstract

We study the problem of estimation and inference on the average treatment effect in a smoking cessation trial where an outcome and some auxiliary information were measured longitudinally, and both were subject to missing values. Dynamic generalized linear mixed effects models linking the outcome, the auxiliary information, and the covariates are proposed. The maximum likelihood approach is applied to the estimation and inference on the model parameters. The average treatment effect is estimated by the G-computation approach, and the sensitivity of the treatment effect estimate to the nonignorable missing data mechanisms is investigated through the local sensitivity analysis approach. The proposed approach can handle missing data that form arbitrary missing patterns over time. We applied the proposed method to the analysis of the smoking cessation trial.

Keywords: Causal effect, Potential outcomes, Robust estimator, Surrogate outcome

1 Introduction

The smoking cessation trial to be analyzed is a part of “IT'S TIME” study, a 2-year longitudinal study consisting of clinic-based interventions delivered to woman smokers of child-bearing age to help them quit smoking. In this study, 1706 eligible smokers were accrued in 12 Chicago area clinics. Subjects were randomized either to the treated arm receiving an educational program helping them to quit smoking or to the controlled arm where no such a program was offered. Follow-up telephone interviews were conducted at 2, 6, 12, and 18 months into the study. The primary outcome of interest in the study is QUIT, a binary variable denoting a woman's report on whether she was abstinent (had not smoked for at least 7 days) at a specific wave. The primary objective of the statistical analysis of the smoking cessation data is to estimate the average treatment effects of the intervention relative to that of the control for a given subpopulation. One challenging problem in estimating the treatment effect is the incompletely observed data. Only about 20% of the subjects completed all follow-up measures on the primary outcome. See Table 1 for more details.

Table 1
Unadjusted point-prevalence of QUIT at each wave by treatment condition

When no missing data are involved, the average treatment effect can be easily obtained. With missing data, estimating the average treatment effect is no longer straightforward. When data are missing at random (MAR) [1], two different approaches may be followed. The first estimates the missing data probability from the data and then uses the weighted estimating equation approach based on the marginal model to obtain the estimate. The other approach uses the maximum likelihood to estimate a joint model on the outcome and potential confounders and then marginalize to obtain the estimate. The former approach is simple but can be very inefficient when the proportion of complete cases is low. Methods for gaining efficiency can be very difficult to implement with non-monotonic missing data patterns [2], [3]. The second approach is more efficient when the joint model is correctly specified. When data are not missing at random (NMAR), the second approach can be taken by modeling the missing data mechanism and the outcome and confounders jointly and then marginalizing the joint model for the full data to obtain the average treatment effect. In this article, we follow the idea of the second approach to estimating the average treatment effect.

Modeling longitudinal data has been extensively discussed in the literature, perhaps, starting from [4] for the continuous outcomes. The discrete outcomes are more challenging to model. Generalized linear mixed models [5],[6], generalized estimating equations [7],[8], and the Markov transitional models [9],[10] are usually considered the three main models of choice. Further development combines features of the three types of models [11],[12],[13]. We propose a mixture of the random effects model and the Markov transitional model, which we call the dynamic generalized linear mixed model for categorical outcomes. For continuous outcomes, the dynamic generalized linear mixed model is known to be equivalent to the linear mixed effects model with autocorrelated errors [14]. For discrete outcomes, no such correspondence exists. In comparison to the dynamic latent trait models [13], such dynamic models are more transparent and stable, and can be used to directly model the relationship among multiple outcomes measured longitudinally.

In the smoking cessation study, in addition to the primary outcome:QUIT, a self-reported auxiliary variable was also measured when the subject had not quit at a given wave. This auxiliary variable measures a subject's readiness to quit at the given time. This variable was called STAGE. A subject classified herself into one of the 5 ordinal stages with respect to her readiness to quit, with 1 corresponding to the least likely to quit and 5 to the most likely to quit. It is speculated that the current stage of readiness to quit smoking of a subject may be informative in predicting the quitting status of that subject in the future. From this point of view, STAGE may be loosely regarded as a surrogate for the future outcome QUIT [15],[16], [17]. Since the auxiliary variable may provide useful information on the mechanism leading to the event [18]: quit, it is of practical importance to include the auxiliary information in the statistical model. Furthermore, including the auxiliary covariate in the model may help to correct bias and gain efficiency in the parameter estimation and make a MAR assumption more plausible. It is therefore of substantial interest to jointly model both the primary outcome and the auxiliary information when data are subject to missing values. In the smoking cessation study, both QUIT and STAGE were subject to missing values and the missingness occurred synchronized. We model them jointly using the dynamic generalized linear mixed models and evaluate the usefulness of the auxiliary information in helping estimate the average treatment effects.

The incompletely observed data are a challenging issue to handle in analyzing the smoking cessation data because the missing data formed complex non-monotonic missing patterns. Handling missing outcomes in longitudinal studies has been studied by many authors. Among them are [19], [20], [21], [22] for general models; [2], [3] for the generalized estimating equations, [23] for linear mixed models. More recently, [24],[25], [26],[27] discussed handling missing outcomes in various longitudinal data models. The missing data mechanism models proposed in the literature usually cannot accommodate complex missing data patterns that are nonmonotone and nonsaturated (not all potential missing patterns occur). When data are missing at random, the modeling approach we take is robust against misspecification of the MAR missing data mechanism. When data are not missing at random, rather than carrying out a global sensitivity analysis of the missing data mechanism models on the treatment effect estimate, we take the simpler approach by performing a local sensitivity analysis [28],[29],[30],[31],[32]. The general form of the logistic regression models we proposed for modeling the missing data mechanism accommodates the nonmonotone nonsaturated missing data patterns. The proposal reduces to a series of logistic regression models usually used for modeling the missing data mechanism when the missing data form monotonic patterns.

The remainder of this article is organized in the following way. In section 2, we discuss a general framework of the dynamic mixed-effects generalized linear models. The framework is then specified for the smoking cessation data. Arbitrary missing data patterns are then modeled by the general logistic models at individual time points. The maximum likelihood approach is proposed for estimating and making inference on the model parameters under the MAR assumption. The approach for estimating the average treatment effect for a subpopulation is discussed in section 3. Sensitivity of the average treatment effect estimate to the possible nonignorable missing data mechanism models is studied through the local sensitivity analysis procedure in section 4. Section 5 presents a detailed analysis of the smoking cessation data using the framework discussed in sections 2, 3, 4. Section 6 concludes the article with a discussion on the merits and limitations of the proposed approach along with possible extensions of the proposed approach in future works.

2 Statistical method for the data analysis

2.1 The modeling framework for statistical analysis

Note that the main objective of the statistical analysis of the smoking cessation trial is to determine if the intervention group has a higher rate of quitting compared with that of the control group. More specifically, compared with smokers in the control group, it is of interest to know whether those in the intervention group would be more likely to have actually quit smoking over time. The statistical problem in analyzing the data can be formulated in the following way. Let Yit, (t = 1, 2, …, T) denote the outcome of subject i measured at time t. In the smoking cessation study, Yit is the quitting status measured at time t for subject i. Let Xi denote the collection of the treatment assignment, the baseline covariates, and possibly time-varying covariates that are not affected by the treatment. We use time-dependent notation to denote all of them until section 3 where we need to distinguish the time-varying and the time-independent covariates when we discuss the average causal effect estimator. Let Xi = (Xi1, (...), XiT) denote all the covariates. In addition to the outcome and covariates, a time-dependent auxiliary variable, denoted by Vi = (Vi1, (...), ViT), was measured in the study. In contrast to the time-varying covariates, the auxiliary variables are those that may be affected by the treatments under study. In the smoking cessation study, Vi recorded the variable STAGE over time. When data are completely observed, the auxiliary variable appears not very useful in the estimation of the treatment effect. When the primary outcome is subject to missing values, modeling the auxiliary variable in the analysis may help correct bias and gain efficiency in the estimation of the treatment effect.

There are many ways to decompose the joint distribution of (Y, V) given X. One natural and useful way to decompose the joint distribution is by the following sequence of conditional distributions in the order of time.

P(Yi,ViX)=t=1TP(VitYit,Vi(t1),Yi(t1),,Vi1,Yi1,Xi)P(YitVi(t1),Yi(t1),,Vi1,Yi1,Xi),

where V0 is assumed to be a constant if no such measure is recorded. The long-dependence on the recorded history in the conditional distributions can be a problem both in interpreting and in fitting the model. On the other hand, a short-dependence is easier to interpret in practice. This leads us to consider models with a short chain of dependence plus random effects. The idea behind the modeling strategy is that it is anticipated that the random effects capture the average long-term dependence while the short chain dependence captures the short-term dependence in addition to the average long-term dependence. Such a model appears as,

P(Yi,Vi,ai,biXi)=[t=1TP(VitYit,Yi(t1),Xit,ai)P(YitYi(t1),Yi(t1),Xit,bi)]P(ai,bi)

We assume further that P(Vit|Yit, Vi(t−1),Xit,ai) is known up to an unknown parameter ψ, P(Yit|Vi(t−1), Yi(t−1), Xit, bi) is known up to an unknown parameter β, and P(ai, bi) is known up to an unknown Σ, which is a function of parameter θ.

For simplicity of presentation, we assume that covariates are always observed. Further, either both (Y,V) are observed or both are missing. Let Ri = (Ri1, …, Rit)′ be the missing data indicator, whose tth component, Rit, equals 1 if (Yit, Vit) is observed for individual i and is 0 if (Yit, Vit) is missing. When data are missing at random, it is well known that the maximum likelihood method can yield valid inferences concerning the parameters β, ψ, and θ. In general, the missingness may depend on the missing values resulting in nonignorable missing data. In this case, a parametric model on the missing data probabilities may allow the identification of the model parameter. Under this situation, the maximum likelihood approach can still be used to obtain the parameter estimates and inference. However, if the model for the missing data probabilities is sufficiently large, the model may become unidentifiable. Therefore, nonignorable models for the missing data probabilities tend to serve as a framework for sensitivity analysis rather than as a way to identify the model parameter. We have the same purpose in mind here in specifying such a model. We specify the model for the missing data probabilities sequentially and assume parametric models as

P(RiYi,Vi,Xi)=t=1TP(RitRi(t1),,Ri1,Yi,Vi,Xi)=t=1TP{RitRi(t1),,Ri1,(Yit,Vit,Xit),,(Yi1,Vi1,Xi1),α},

where α is the unknown parameter.

With the foregoing model specification, the marginal likelihood for the observed data {Ri, Ri(Yi, Vi), Xi}, i = 1, (...), n, can be written as

i=1nP(RiYi,Vi,Xi,α),P(Yi,Vi,ai,biXi,β,ψ,θ)daidbidRi(Yi,Vi),
(1)

where Ri(Yi, Vi) and Ri(Yi, Vi) denote respectively the observed and missing parts of (Yi, Vi) determined by Ri. For any categorical component of (Yi, Vi), the integration with respect to that component should also be understood as summation over the range. Assume that the parameters are identifiable either from the model restriction or from the fixation of some parameter values in the model for the missing data probabilities. Estimation and inference on the model parameters can be carried out by the maximum likelihood approach. The maximum likelihood estimator can be obtained by the EM algorithm. More details are given in the next subsection when we deal with the specific application.

2.2 Model specification for the application

For the smoking cessation data, Yit is a binary variable taking value 1 if the subject is considered to have quit smoking at time t, 0 otherwise. Vit, the readiness to quit, takes value 6 if Yit = 1 and takes values 1 to 5 indicating the level of readiness. It can be seen that Vit predicts Yit perfectly for t = 1, (...), T and we can express Vit as

Vit=6Yit+(1Yit)it0,

where Vit0 takes categories 1 to 5 when Yit = 0. We have that Yit = 1{Vit=6}. Although modeling the distribution of {Vit, t = 1, (...), T} alone is equivalent to modeling {Yit,Vit0) jointly for the estimation of the treatment effect, the fact that the event Vit = 6 carries so much weight in this clinical trial makes it very much worthwhile to treat it separately. As a result of this consideration, we model {Vit, t = 1, (...), T}, the mathematical equivalent of {Yit,Vit0,t=1,,T) jointly rather than directly. The models for {Yit,Vit0,t=1,,T) are

P(VitYit,Vi(t1),Yi(t1),,Vi1,Yi1,Xi,ai,bi)={P(Vit0Vi(t1),,Vi1,Xi,ai,bi)}1Yit.

P(YitVi(t1),Yi(t1),,Vi1,Yi1,Xi,ai,bi)=P(YitVi(t1),,Vi1,Xi,ai,bi).

The short-chain dependence mixed effects model that was proposed in the last subsection becomes

P(Vit0Vi(t1),,Vi1,Xi,ai,bi)=P(Vit0Vi(t1),Xit,ai,ψ),

and

P(YitVi(t1),,Vi1,Xi,ai,bi)=P(YitVi(t1),Xit,bi,β).

Note that Vit0 takes one of the K(= 5) discrete values. We consider in this paper the baseline logit model for P(Vit0Vi(t1),Xit,ai) which specifies

logP(Vit0=kVi(t1),Xit,ai)P(Vit0=KVi(t1),Xit,ai)=ψ0k+ai+ψ1kVi(t1)+ψ2kXit,

for k = 1, (...), K − 1. Note, however, that other models such as the proportional odds ratio model specified as

logP(Vit0>kVi(t1),Xit,ai)P(Vit0kVi(t1),Xit,ai)=ψ0k+ai+ψ1Vi(t1)+ψ2Xit,

for k = 1, (...), K − 1, may also be used. The model for the distribution of Yit given Xit and bi is specified by the logistic regression as

logP(Yit=1Vi(t1),Xit,bi)P(Yit=0Vi(t1),Xit,bi)=β0+bi+β1Vi(t1)+β2Xit.

The random effects distribution for (ai, bi) is specified as N(0, Σ(θ)).

Specification of models for the missing data mechanism can be accomplished by using the general logistic regression models as follow. Let πitr = P{Rit = 1|Ri(t−1), (...), Ri1, (Yit, Vit, Xit), (...), (Yi1, Vi1, Xi1)}, where r denote the missing data pattern formed by (Ri1, (...), Ri(t−1)). Logistic model

logπitr1πitr=αt0r+αt1r(Yi1,Vi1,Xi1)++αttr(Yit,Vit,Xit)

may be fit when patterns (Ri1 (...), Ri(t−1), Rit = 1) and (Ri1, (...), Ri(t−1), Rit = 0) are both observed, where αtjr = (αtjry, αtjrv, αtjrx), j = 1, (...), t, are respectively coefficients for each component of (Yij, Vij, Xij). When either (Ri1, (...), Ri(t−1), Rit = 1) or (Ri1, (...), Ri(t−1), Rit = 0) is unobserved, parameter αtr = (αttr, (...), αt1r, αt0r) may become inestimable when fitting the logistic regression model even under missing completely at random assumption. Those probabilities are estimated by the degenerated probabilities 1 or 0 in this paper. This can be seen as a direct extension of the modeling strategy routinely used for monotonic missing data to nonmonotonic missing data. The probability for the observable pattern (Ri1, (...), Rit) for subject i is

t=1TπitrRit(1πitr)1Rit,

where r is conformable with the observed pattern (Ri1, (...) RiT).

With the foregoing model specification, we can now assemble the likelihood in a more concrete way as

L(α,β,ψ,θ)=i=1n[t=1TπitrRit(αtr){1πitr(αtr)}1RitP(YitVi(t1),Xit,ai,β)][{P(Vit0Vi(t1),Xit,bi,ψ)}1YitP(ai,biθ)daidbidRi(Yi,Vi)].

where α is the collection of αtr for t = 1, (...), T and for all the missing patterns where πtr is not degenerated. Note that, for any of the models specified and used so far, by virtually the same approach, more complex models can be specified and used in place of the relatively simple ones whenever the simple models do not fit the data well. For example, more than one-step dependence on history may be added to the model. We suppress these developments for simplicity of presentation.

2.3 Parameter estimation and inference

We use the maximum likelihood approach to estimate and make inferences on the parameters. The EM algorithm is adopted for maximizing the likelihood function by augmenting the observed data {Ri,Ri(Yi,Vi),Xi} naturally to (Ri,Yi,Vi,Xi, ai, bi). The likelihood for the augmented data is

LF(α,β,ψ,θ)=i=1n[t=1TπitrRit(αtr){1πitr(αtr)}1RitP(YitVi(t1),Xit,bi,β)][{P(Vit0Vi(t1),Xit,ai,ψ)}1YitP{ai,biΣ(θ)}].

Let liF=logLiF and li = log Li which are respectively the contribution of the ith subject to the augmented data likelihood and to the observed data likelihood. Let lFΣi=1nliF and l=Σi=1nli. The objective function to be maximized in the EM algorithm is

Q(α,β,ψ,θα,β,ψ,θ)=Σi=1nE[liF(α,β,ψ,θ){Ri,Ri(Yi,Vi),Xi},α,β,ψ,θ].

The conditional distribution used for the calculation of the contribution to the objective function from the ith subject is

P{Ri,Yi,Vi,Xi,ai,biRi,Ri(Yi,Vi),Xi,α,β,ψ,θ}=LiF(α,β,ψ,θ)Li(α,β,ψ,θ).

To compute the expectations under this distribution, we need to resolve the issue of computing integrations. We applied Gauss quadrature approximation to the integrations. Other numeric approaches or the Monte Carlo simulation approach may be applied. Such methods usually require intensive computation.

The EM algorithm applied to the example does not yield a closed-form solution to the maximization. Iterative approaches, such as the Newton-Raphson method, need to be applied. Such algorithms require computing both the first and the second derivatives of the augmented data log-likelihood. Score functions from the augmented data log-likelihood for the parameter αtr, β, ψ, and θ are given in the Appendix.

Before applying the computational method to the data, we need to sort out the missing data patterns observed with degenerated probabilities. For longitudinal data with T time points, the total number of potential missing patterns is 2T. However, we seldom observed all the missing data patterns in a single data set. This was the case for the smoking cessation data. For the smoking cessation data, there are potentially 24 = 16 missing data patterns, of which only 14 were observed in this data set, as is seen from Table 1. To determine the missing patterns with degenerated probabilities, we first sort the missing data indicators in a dictionary order as is done in Table 2. Determination of the degenerated probabilities that are not required to model can then be done forwardly in time. For any given missing data pattern up to time t − 1, r = (r1, (...), rt−1), if both Rt = 0 and 1 occur in the data at time t, πtr is not degenerated. Otherwise, it is degenerated. Whether πtr = 0 or 1 for the degenerated case can be easily determined by whether Rt = 0 or 1. Finally, patterns associated with 0 probability of a degenerated pattern is also degenerated. When we apply this approach of determination to the observed missing data patterns in Table 2, it can be seen that the degenerated probabilities are π3(00) = 0, π4(100) = 0, and all the patterns associated with the zero probabilities. Such patterns are (0,0,*,*), and (1,0,0,*), where * can be filled by either 0 or 1. Altogether, there are 21 + 20 = 3 patterns whose probabilities are either 0 or 1 (degenerated). In other words, we need 24 − 3 = 13 logistic models for the missing data probabilities. Table 3 showed the regression coefficient estimates from the fit of 13 logistic regression models with different intercepts and common coefficients. The missingness appears to be significantly affected by the education level and the number of years of smoking. The result suggests that those who did not complete high school education and/or have long smoking history are more likely to stay in the trial.

Table 2
Patterns of Missing Outcome
Table 3
Estimation of MAR missing data mechanism model

3 Estimating the average treatment effect

We use G-computation approach [33], [34], [35], to estimate the average treatment effect for a subpopulation defined by the baseline covariates and possibly by the time-dependent covariate processes that are not affected by the treatment. In the smoking cessation study, the covariates include the treatment assignment and the baseline covariates, such as race and educational levels. They are all time-independent covariates. To simplify the notation, from now on, we use Z to denote the treatment assignment and X to denote the baseline covariates. There are two treatment conditions: one is treated (Z = 1) and the other is control (Z = 0). Let Yt(0) and Yt(1) respectively denote the potential outcomes [36], [37], [38], if control or treatment is applied to a unit. Similarly, Vt(0) and Vt(1) denote respectively the potential auxiliary variable values if control or treatment is applied to the unit. We are interested in estimating the average treatment effect in the form of

ACE(X)=E{YT(1)YT(0)X}

for a subpopulation defined by the covariates that are not affected by treatment. Note that the average causal effect is defined at time T primarily for notational simplicity. It is easy to see that the definition can be extended to any time point. If there were no missing data, the data observed from the trial would be ((Yit(Zi),Vit(Zi),t=1,,T;Xi,Zi)) for i = 1, …, n. Suppose that treatment assignment is independent of the potential outcomes. More precisely,

P(ZYt(0),Yt(1),Vt(0),Vt(1),t=1,,T;a(0),a(1),b(0),b(1),X)=P(ZX).
(2)

It follows from the no-unmeasured-confounder assumption (2) that

P(Yt(u)Vt1(u),Yt1(u),,V1(u),Y1(u),a(u),X)=P(Yt(Z)Vt1(Z),Yt1(Z),,V1(Z),Y1(Z),a(Z),X,Z=u),P(Vt1(u)Yt1(u),Vt2(u),,V1(u),Y1(u),b(u),X)=P(Vt1(Z)Yt1(Z),Vt2(Z),,V1(Z),Y1(Z),b(Z),X,Z=u),

for t = 1, …, T, and

P(a(u),b(u)X)=P(a(Z),b(Z)Z=u),

because the random effects are independent of the covariates. This leads to

E{h(YT(u))X}=h(YT(u))t=1T{P(Yt(u)Vt1(u),Yt1(u),,V1(u),Y1(u),a(u),X)}{P(Vt1(u)Yt1(u),Vt2(u),,V1(u),Y1(u),b(u),X)}dYt(u)dVt1(u)dV1(u)dY1(u)P(a(u),b(u)X)da(u)db(u),=h(YT(Z))t=1T{P(Yt(Z)Vt1(Z),Yt1(Z),,V1(Z),Y1(Z),a(Z),X,Z=u)}{P(Vt1(Z)Yt1(Z),Vt2(Z),,V1(Z),Y1(Z),b(Z),X,Z=u)}P(a(Z),b(Z)Z=u)dYt(Z)dVt1(Z)dV1(Z)dY1(Z)da(Z)db(Z),
(3)

where u = 0, 1 and h is an integrable function. For the smoking cessation data, suppose that all the models specified in the previous section are true for the observed data when Yit and Vit are replaced by Yit(Zi) and Vit(Zi).

E{h(YT(u))X}=h(YT(Z))t=1T[P(Yt(Z)Vt1(Z),a(Z),X,Z=u,β)][{P(Vt1(Z)0Vt1(Z),b(Z),X,Z=u,ψ)}1Yt1(Z)]P(a(Z),b(Z)Z=u,θ)dYt(Z)dVt1(Z)dV1(Z)dY1(Z)da(Z)db(Z),
(4)

Now suppose that the assumptions on the missing data mechanism are correct and parameter (β, ψ, θ) can be estimated from the likelihood consistently. Denote the estimate by ([beta], [psi], [theta w/ hat]). If we use the distribution estimated from the maximum likelihood in computing the expectation in (4) with h(y) = y, denoted by E^{YT(u)X} for u = 0, 1, the maximum likelihood estimator of the average causal effect can be obtained as

AC^E(X)=E^(YT(1)X)E^(YT(0)X).

The variance of the estimator can be obtained by the asymptotic approximation using the δ-method. See Appendix for more details. Finally, the average causal effect, E(YT(1)YT(0)), for the population where the sample was drawn can be estimated by

AC^E=1nΣi=1n{E^(YT(1)Xi)E^(YT(0)Xi)}.

The variance of the estimator can be estimated by

1nΣi=1nVAi+1n1Σi=1n{AC^E(Xi)AC^E}2,

where VAi = VA(Xi) and VA(x) is the estimated variance of the limit of n{AC^E(x)ACE(x)} in distribution.

4 Sensitivity analysis with nonignorable missing data

In the sensitivity analysis of the estimates to nonignorable missing data mechanism models, we concentrate on the analysis of the influence of nonignorable missing data mechanisms on the estimate of the average causal effect of the treatment versus the control. Although the framework proposed in the previous section can be used to obtain a global sensitivity analysis of the average treatment effect to the nonignorable missing data mechanisms, the computation involved in such an analysis can be very intensive. We choose to perform a local sensitivity analysis which approximates the global sensitivity analysis. Note that the average causal effect can be expressed as

m(β,ψ,θ)=E(YT(1)X,β,ψ,θ)E(YT(0)X,β,ψ,θ).

To simplify notation, we use η = (β, ψ, θ) in this section. In the local sensitivity analysis, we first estimate η using the likelihood approach by ignoring the nonignorable missing data mechanism model. The estimated η is thus biased and is an implicit function of the parameter α. As a result, m([eta w/ hat]) is an implicit function of α through η, and using m([eta w/ hat]) as the estimate of m(η) is subject to bias. The idea of the local sensitivity analysis is to try to correct the bias. We consider the set of missing data mechanisms that are modeled in the previous section. From now on, we use (γ, α) to denote all the parameters in the missing data mechanism model with α denoting the part of the parameters that define a NMAR mechanism if αα*, where α* is often 0. γ denotes the rest of the parameters in the missing data mechanism model.

For a fixed α, let the estimate of (γ, η) be denoted by {[gamma with circumflex](α), [eta w/ hat](α)}. Suppose that the functions of m on η and η on α are continuously differentiable. To compute the local sensitivity approximation to m(η), we need first to obtain the local sensitivity approximation to η. Note that the local sensitivity approximation to η is

η^(α)η^(α)+(αα){η^(α)α}.

It then follows that

m{η^(α)}m{η^(α)}+(αα){η^(α)α}mη{η^(α)}.

Note that when α = α*, the missing data mechanism is ignorable for any γ. From Appendix B, we see that

η^(α)α=Σi=1n2lαη{Ri,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}[Σi=1n2lη2{Ri,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}]1.

The second term on the right-hand side is the inverse of the observed information matrix for the incomplete data, which can be obtained by fitting an ignorable model to the observed data. Since

2lαη{Ri,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}=E{2lFαηRi,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}+Cov{lFα,lFηRi,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)},

and 2lFαη=0 when α and η are variation independent parameters, the first term on the right-hand side becomes the conditional covariance between the score for the nonignorable part of the missing data mechanism model and the score from the full data model given the observed data.

From Appendix A, we see that

mη{η^(α)}=Cov[lFη{α,γ^(α),η^(α)},YT(1)YT(0)X,η^(α)]=Cov[lFη{α,γ^(α),η^(α)},YT(Z)X,Z=1,η^(α)]Cov[lFη{α,γ^(α),η^(α)},YT(Z)X,Z=0,η^(α)].

Note that (α, γ) and η are in general variation-independent. As a result, lFη does not involve the missing data mechanism model. The sensitivity of m(η) with respect to α can then be computed by the linear extrapolation.

When the local sensitivity of the average causal effect, E(YT(1)YT(0)), to the nonignorable missing data models is of interest, it follows that

1nΣi=1nmi{η^(α)}1nΣi=1nmi{η^(α)}+(αα){η^(α)α}1nΣi=1nmiη{η^(α)}.

where mi(β,ψ,θ)=E(YT(1)Xi,β,ψ,θ)E(YT(0)Xi,β,ψ,θ). The computation of the local sensitivity direction can be carried out by the foregoing approach.

5 Analysis of the smoking cessation data

Table 1 lists the rates of abstinence at each wave respectively for the control group and the intervention group. It shows that the abstinence rates have improved for both groups over the 18-month period. The intervention group appears to have a higher rate of abstinence than that of the control group. Note, however, that the rates were computed based on the actual number of subjects under observation. In view of the large number of subjects with missing outcomes, this improvement may be due to the selection bias. It is possible that the rate of abstinence at a wave could be increased over the previous wave even when no new subject became abstinent if subjects who were not going to quit smoking were more likely to drop out of the study. Table 2 displays the missing data patterns for the outcome: QUIT, and the auxiliary variable: STAGE. Note that QUIT and STAGE are either both missing or both observed. As a result, only one indicator of the missing values at a given time point was used. It can be seen from the table that the missing values form nonmonotone patterns and the total number of missing patterns (14) is less than the number of all potential patterns (16). In the following analysis, we aim at addressing the issue of missing outcomes in estimating the average treatment effect.

Before focusing on the issue of missing outcomes, we note that incomplete covariate information also occurred in some subjects. Among those 1706 women eligible at baseline, 14 did not report their race, 6 did not provide information on their educational level. Variables on years since started smoking and the number of daily cigarettes smoked had 81 and 22 missing values, respectively. The fraction of missing covariates is relatively small in comparison to that of missing outcomes. Rather than modeling the covariate distribution and maximizing the joint likelihood, which can be computationally much more challenging to be carried out, we took two simplified approaches: One is to exclude subjects with incomplete covariate information from the analysis; the other is to impute the incomplete covariate information by regression models fit to the available cases. Since the analytical results for the deleted data are close to those for the imputed data, we only present the analysis of the imputed data set in this article.

Table 1 shows that the rates of abstinence change over time for both the treatment group and the control group. Based on this observation, we model the logit of the abstinent rate (QUIT) as a function of time and treatment condition after accounting for the effect of pre-treatment conditions. Specifically, two types of models were fit. The first one is a simple model that does not include the auxiliary variable into the model. This type of models can be schematically expressed as

logP{QUIT=1at timet}P{QUIT=0at timet}=intercept(t)+treatment(t)+pre-treatment covariates+lag1QUIT+random effect.
(5)

The other type includes the auxiliary variable in the models. This type of models can be schematically expressed as two joint models with the logistic model,

logP{QUIT=1at timet}P{QUIT=0at timet}=intercept(t)+treatment(t)+pre-treatment covariates+lag1QUIT+lag1STAGE+random effect,
(6)

for QUIT, and the baseline logit model,

logP{STAGE=kat timet}P{STAGE=5at timet}=intercept(k)+treatment(k)+pre-treatment covariates(k)+lag1QUIT&STAGE(k)+random effect,
(7)

for the STAGE, where k = 1, …, 5.

We fitted one model of the first type and two models of the second type to the observed data under MAR assumption. Model I had the form (5) and did not included V. Models II and III included V and had the joint model form of (6) and (7). All three models included the same baseline covariates. The variables lag-1 QUIT and lag-1 STAGE were combined as one variable and were treated as an ordinal variable in Model II. That is, STAGE=6 when QUIT=1. Model III treated 1{STAGE=6} differently from those of stages 1 to 5, which were again treated as an ordinal variable.

Results on parameter estimation for the outcome from Models I, II, and III are listed in Table 4. Results on parameter estimates for the auxiliary variable from Models III are listed in Table 5. Note that we did not list parameter estimates for the auxiliary variables from Model II because they are very close to those of Model III. Results in Table 4 suggest that the treatment has significant effects on the QUIT status at months 2 and 6, but not significant effects on the QUIT status at months 12 and 18 when both pre-treatment covariates and the auxiliary variable STAGE at the previous time are adjusted. It can also be seen that the current QUIT status strongly depends on the STAGE at the previous time point even after adjusted for the pretreatment covariates. Subjects with more advanced stage at the previous time point tends to have a higher probability of quitting smoke in the current time point, which is consistent with intuition. In addition, the strong positive association between the current quitting status and the quitting status at the previous time suggests that a quitter at the previous time point is much more likely to remain quitting than to relapse to smoking in the current time point. Results in Table 5 also show that the current STAGE strongly depends on STAGE at the previous time point. Although the coefficient estimate for the lag-1 STAGE is negative, nevertheless, the result should be interpreted as such that subjects were more likely to advance their stage of readiness to quit over time because the base category used in the baseline logit model was STAGE=5 rather than STAGE=1.

Table 4
Parameter estimates for outcome models under MAR assumption
Table 5
Parameter estimates for auxiliary covariate in model III under MAR assumption

Since STAGE is a post-treatment variable, it is important to remember that the treatment effect may be diluted when STAGE is included in the adjustment. This issue was addressed by computing the average treatment effect using the G-computation approach. The results are given on the left panel of Table 6 along with the naive estimate of treatment effect, which is the difference of point-prevalence rates for treated and control groups. Those results indicate that, on average, treatment has significant effects on QUIT at months 2 and 6, but has insignificant effects on QUIT at months 12 and 18. The conclusion drawn from the average treatment effect estimates is consistent with those from the conditional model estimates and the naive rate estimates.

Table 6
Estimated average treatment effects and their sensitivity to nonignorable missing data models

The treatment effect estimates are consistent based on different methods under MAR. It is of substantial interest to know how the conclusion changes if the MAR assumption is violated. Assume that missing values on Y and V were generated from the logistic models

logπitr1πitr=intercept(r)+Baseline covariates+α1YYit+α2YYi(t1)+α1VVit+α2VVi(t1),

where t denotes the time and r denotes the missing pattern. Under this model, the local sensitivity analysis was performed. The results are displayed on the right panel of Table 6. The numbers under each variable indicate the amount of change in the average treatment effect under the corresponding model at the given time point when the coefficient of that variable in the missing data mechanism model is changed by one unit. The results allow us to assess the impact of the nonignorable missing data on the average treatment effect estimation for any given combination of the coefficients in the missing data mechanism model. For example, if the actual missing data mechanism has (α1Y, α2Y, α1V, α2V) = (0, 0.5, 0.1, 0), approximately, the average treatment effects at 2, 6, 12, and 18 months based on Model II are 5.28%, 4.10%, 0.68%, and −1.03% respectively. The results in Table 6 suggest that the average treatment effect at month 6 is much more sensitivity to the dependence of the missing data mechanism on the STAGE variable than at month 18 under both models II and III. The results also suggest that, when missingness depends on the current or previous QUIT or STAGE, the treatment observed under MAR can be substantially altered if the dependence is relatively strong. Overall, if Quitter or subjects in the advance stage of readiness to quit are more likely to dropout, the average treatment effect estimate will be much smaller than estimated under the MAR assumption.

The Fortran code for performing the analysis can be obtained from the authors by email to ude.ciu@nehcyh.

6 Discussion

We presented a framework for the analysis of the longitudinal data with missing outcome and auxiliary variables. The proposed framework was used in the analysis of the smoking cessation data. Although the example we treated in this paper is special in that both the outcome and the auxiliary variables are observed or missing simultaneously, nevertheless, the framework is easily extended to handle the case when they are not observed or missing simultaneously. The auxiliary variable can also be viewed as a time-dependent covariate. In this way, the proposed method can be regarded as a joint modeling approach to the missing data problem with both the longitudinal outcomes and longitudinal covariates subject to missing values. Since the missing data form non-monotonic patterns, modeling the time-dependent covariate is necessary in the application of the likelihood approach. This framework can also handle nonmonotone and unsaturated missing data. One possible argument against our treatment of the unsaturated missing data patterns is that we implicitly assume that the missing data patterns not observed cannot happen in the future. But we view a pattern that is not observed is so unlikely to happen that we estimate the probability of its happening zero rather than an estimated nonzero probability (not necessarily small as one may think it is) based on a potentially saturated model. In addition, a saturated logistic model can behave erratically when fitting to the data with observed unsaturated missing patterns. Our modeling approach naturally reduces to the consecutive logistic models usually used for modeling missing data probability when the missing data form monotonic patterns.

We applied the G-computation approach to obtain the average treatment effect estimate, which is of primary interest in trials of the similar type. We proposed a sensitivity analysis of the average treatment effect to the missing data mechanism models through the local sensitivity analysis approach. Since one may not know whether the observed data deviated from the MAR assumption, and if it does, how strong the deviation is, the results only provided us a way to quantify the potential impact on the estimated average causal effect of the deviation from the MAR assumption rather than a decisive conclusion, which may not be achieved without additional information from subject-matter experts. Note that the asymptotic unbiasedness of the proposed estimators in this article relies on the correct model assumptions, which need to be carefully evaluated. Alternative approaches that provide some protection against the bias of the parameter estimator, such as the doubly robust approach, may be applied under MAR assumption. However, when the MAR assumption on the missing data mechanism does not hold, the local sensitivity analysis can provide useful information while keeping the computation simple.

When the average causal effect of a subpopulation defined by the baseline covariates is of interest, the proposed approach can be modified to give an estimator of the effect. However, if the subpopulation is defined by only a subset of the covariates in the model, a model for the covariates not used in defining the subpopulation given the covariates used in defining the subpopulation is required to be specified. We leave those issues for further studies in the future.

ACKNOWLEDGMENT

We thank reviewers for helpful comments on the earlier versions of the paper, which led to substantial improvements on the presentation. The research is partially supported by NIH/NCI grant R01 CA106355.

Appendix A: Derivatives for the log-likelihood

The first derivatives are for the augmented data are

lFαtr=Σi=1nRitπitr(1πitr)logπitrαtr,lFβ=Σi=1nΣt=1TβlogP(YitVi(t1),Xit,bi,β),lFψ=Σi=1nΣt=1T(1Yit)ψlogP(Vit0Vi(t1),Xit,ai,ψ),lFθ=Σi=1nθlogP{ai,biΣ(θ)}.

The second derivatives can be obtained similarly. The first derivatives for the observed data are the conditional expectations of the foregoing derivatives given the observed data. The variance estimates for the maximum likelihood estimator can be obtained by the inverse of the observed data information matrix, which can be computed by Louis' (1984) formula as

2l(α,β,ψ,θ)2(α,β,ψ,θ)=Σi=1nE{2liF(α,β,ψ,θ)2(α,β,ψ,θ){Ri,Ri(Yi,Vi),Xi},α,β,ψ,θ}+Σi=1nVar{liF(α,β,ψ,θ)(α,β,ψ,θ){Ri,Ri(Yi,Vi),Xi},α,β,ψ,θ}.

Appendix B: Variances of the causal effect estimators

We first derive a variance formula for AC^E(X)=E^(YT(1)X)E^(YT(0)X). Use δ-method,

AC^E(X)=ACE(X)+(η^η0)Tη{E(YT(1)X)E(YT(0)X)}+op(η^η0),

where [eta w/ hat]η = ([beta]β, [psi]ψ, [theta w/ hat]θ). This implies that n{AC^E(X)ACE(X)} is asymptotically normal with variance

VA=η{E(YT(1)X)E(YT(0)X)}Ω[η{E(YT(1)X)E(YT(0)X)}],

where Ω is the asymptotic variance for {n(β^β0),n(ψ^ψ0),n(θ^θ0)}. Note that,

ηE(YT(u)X)=Cov{YT(u),lFηX},

for u=0,1, which can be obtained by G-computation. It follows that

η{E(YT(1)X)E(YT(0)X)}=Cov{YT(1)YT(0),lFηX}.

The variance of the estimator of ACE(X) can be obtained based on the following expansion.

AC^E(X)=ACE(X)+(η^η0)Tη{E(YT(1)X)E(YT(0)X)E(YT(0)X)}+op(η^η0).

Therefore, the variance has a similar sandwich form as

VR=η{E(YT(1)X)E(YT(0)X)E(YT(0)X)}Ω[η{E(YT(1)X)E(YT(0)X)E(YT(0)X)}],

where

η{E(YT(1)YT(0)X)E(YT(0)X)}=Cov(YT(1)YT(0),lFηX)E(YT(0)X)E(YT(1)YT(0)X)E(YT(0)X)Cov(YT(0),lFηX)E(YT(0)X).

The estimated variances can be obtained by replacing Ω by its estimate obtained from the inverse of the observed information matrix and computing the covariance using the estimated models for the full data.

We can obtain the variance of AĈE similarly in the following way. Note that

n{AC^EACE}=n(η^η0)T1nΣi=1nη{E(YT(1)YT(0)Xi)}+1nΣi=1n{ACE(Xi)ACE}+op(nη^η0).

Note that

n(η^η0)=1nΣi=1ns{Ri,Ri(Yi,Vi,Xi)}+op(1),

where E{s(Y, V, X)|X} = 0. It is now easy to see that the variance estimate is a consistent estimate of the asymptotic variance of n(AC^EACE).

Appendix C: Derivatives in local sensitivity analysis

Note that the estimate of (γ, η) satisfies

(γ,η)l{α,γ^(α),η^(α)}=0,

for any α. By differentiating with respect to α on both sides of the equation, it follows that

{γ^(α),η^(α)}α=2l{α,γ^(α),η^(α)}α(γ,η){2l{α,γ^(α),η^(α)}2(γ,η)}1.

Note that the observed data likelihood score can be written as conditional expectations of the full data likelihood scores, which separate the score for γ from the score for η as

l(γ,η){α,γ^(α),η^(α)}=(E[l1Fγ{α,γ^(α)}R,R(Y,V),X,α,γ^(α),η^(α)]),(E[l2Fη{η^(α)}R,R(Y,V),X,α,γ^(α),η^(α)]),

where l1F and l2F are respectively the likelihood for missing data mechanism model and the likelihood for the full data when (R, V, Y, X) is observed. It can be seen that

2lαγ{α,γ^(α),η^(α)}=E{2l1FαγR,R(Y,V),X,α,γ^(α),η^(α)}+γ^(α)αE{2l1Fγ2R,R(Y,V),X,α,γ^(α),η^(α)}+Cov{l1Fγ,l1FαR,R(Y,V),X,α,γ^(α),η^(α)}+γ^(α)αCov{l1Fγ,l1FγR,R(Y,V),X,α,γ^(α),η^(α)}+η^(α)αCov{l1Fγ,l2FηR,R(Y,V),X,α,γ^(α),η^(α)}.

Similarly, it follows that

2lαη{α,γ^(α),η^(α)}=η^(α)αE{2l2Fγ2R,R(Y,V),X,α,γ^(α),η^(α)}+Cov{l2Fη,l1FαR,R(Y,V),X,α,γ^(α),η^(α)}+γ^(α)αCov{l2Fη,l1FγR,R(Y,V),X,α,γ^(α),η^(α)}+η^(α)αVar{l2FηR,R(Y,V),X,α,γ^(α),η^(α)}.

When α* is chosen such that the missing data mechanism is ignorable, it can be seen that

l1FγE{l1FγR,R(Y,V),X,α,γ^(α),η^(α)}=0.

This implies that

γ^(α)α=Σi=1nE{2l1FαγRi,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}[Σi=1nE{2l1Fγ2Ri,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}]1=Σi=1n2l1iFαγ{α,γ^(α),η^(α)}[Σi=1n2l1iFγ2{α,γ^{α),η^(α)]1

and that

η^(α)α=Σi=1nCov{l2Fη,l1FαRi,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}[Σi=1nE{2l2Fη2Ri,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}][Σi=1nVar{l2FηRi,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}]1=Σi=1n2lηα{Ri,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}[Σi=1n2lη2{Ri,Ri(Yi,Vi),Xi,α,γ^(α),η^(α)}}1.

References

1. Little RJA, Rubin DB. Statistical Analysis with Missing Values. 2nd edition Wiley; New York: 2002.
2. Robins JM, Rotnitzky A, Zhao LP. Analysis of semiparametric regression for repeated outcomes in the presence of missing data. Journal of American Statistical Association. 1995;90:106–121.
3. Rotnitzky A, Robins JM, Scharfstein DO. Semiparametric regression for repeated outcomes with nonignorable nonresponse. Journal of American Statistical Association. 1998;93:1321–1339.
4. Laird NM, Ware JH. Random effects models for longitudinal data. Biometrics. 1982;38:963–974. [PubMed]
5. Stiratelli R, Laird N, Ware JH. Random-Effects Models for Serial Observations with Binary Response. Biometrics. 1984;40:961–971. [PubMed]
6. Breslow NE, Clayton DG. Approximate inference in generalized linear mixed models. The Journal of American Statistical Association. 1993;88:9–25.
7. Liang KY, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73:11–22.
8. Liang KY, Zeger SL, Qaqish B. Multivariate regression analysis for categorical data (with discussion) Journal of the Royal Statistical Society, Ser. B. 1992;54:3–40.
9. Diggle PJ, Heagerty PJ, Liang KY, Zeger SL. Analysis of Longitudinal Data. 2nd edition Clarendon Press; Oxford: 2002.
10. Albert PS. A transitional model for longitudinal binary data subject to nonignorable missing data. Biometrics. 2000;56:602–608. [PubMed]
11. Daniels MJ, Pourahmadi M. Bayesian analysis of covariance metrices and dynamic models for longitudinal data. Biometrika. 2002;89:553–566.
12. Heagerty PJ. Marginalized transition models and likelihood inference for longitudinal categorical data. Biometrics. 2002;58:342–351. [PubMed]
13. Dunson DB. Dynamic latent trait models for multidimensional longitudinal data. Journal of the American Statistical Association. 2003;98:555–563.
14. Pourahmadi M, Daniels M. Dynamic conditionally linear mixed models. Biometrics. 2002;58:225–231. [PMC free article] [PubMed]
15. Prentice RL. Surrogate endpoints in clinical trials: Definition and operational criteria. Statistics in Medicine. 1989;8:431–440. [PubMed]
16. Buyse M, Molenburghs G. Criteria for the validation of surrogate endpoints in randomized experiments. Biometrics. 1998;54:1014–1029. [PubMed]
17. Frangakis CE, Rubin DB. Principal stratification in causal inference. Biometrics. 2002;58:21–29. [PMC free article] [PubMed]
18. Aalen OO, Frigessi A. What can statistics contribute to a causal understanding? Scandinavian Journal of Statistics. 2007;34:155–168.
19. Diggle P, Kenward MG. Informative drop-out in longitudinal data analysis (with discussion) Applied Statistics. 1994;43:49–93.
20. Little RJA. Modeling the drop-Out mechanism in repeated-measures studies. Journal of the American Statistical Association. 1995;90:1112–1121.
21. Robins JM. Non-response models for the analysis of non-monotone non-ignorable missing data. Statistics in Medecine. 1997;16:21–37. [PubMed]
22. Robins JM. Non-response models for the analysis of non-monotone ignorable missing data. Statistics in Medecine. 1997;16:39–56. [PubMed]
23. Ibrahim JG, Chen MH, Lipsitz SR. Missing responses in generalised linear mixed models when the missing data mechanism is nonignorable. 2001;88:551–564.
24. Cole BF, Bonetti M, Zaslavsky AM, Gelber RD. A multistate Markov chain model for longitudinal, categorical quality-of-life data subject to nonignorable missingness. Statistics in Medicine. 2006;24:2317–2334. [PubMed]
25. Parzen M, Lipsitz SR, Fitzmaurice GM, Ibrahim JG, Troxel A. Pseudo-likelihood methods for longitudinal binary data with non-ignorable missing responses and covariates. Statistics in Medicine. 2006;25:2784–2796. [PubMed]
26. Kurland BF, Heagerty PJ. Marginalized transition models for longitudinal binary data with ignorable and non-ignorable drop-out. Statistics in Medicine. 2004;23:2673–2695. [PubMed]
27. Stubbendick AL, Ibrahim JG. Likelihood-based inference with nonignorable missing responses and covariates in models for discrete longitudinal data. Statistica Sinica. 2006;16:1143–1167.
28. Copas JB, Eguchi S. Local sensitivity approximations for selection bias. Journal of the Royal Statistical Society. 2001;67:459–513.
29. Copas JB, Eguchi S. Local model uncertainty and incomplete-data bias (with discussion) Journal of the Royal Statistical Society. 2005;59:55–95.
30. Troxel AB, Ma G, Heitjan DF. An index of sensitivity to nonignorability. Statistica Sinica. 2004;14:1221–1237.
31. Ma GG, Troxel AB, Heitjan DF. An index of local sensitivity to nonignorable drop-out in longitudinal modelling. Statistics in Medicine. 2005;24:2129–2150. [PubMed]
32. Verbeke G, Molenberghs G, Thijs H, Lesaffre E, Kenward MG. Sensitivity analysis for non-random dropout: a local influence approach. Biometrics. 2001;57:7–14. [PubMed]
33. Robins JM. A new approach to causal inference in mortality studies with sustained exposure periods—application to control of the healthy worker survivor effect. Mathematical Modelling. 1986;7:1393–1512.
34. Robins JM. Addendum to “A new approach to causal inference in mortality studies with sustained exposure periods — application to control of the healthy worker survivor effect” Computers and Mathematics with Applications. 1987;14:923–945.
35. Robins JM. Causal inference from complex longitudinal data. In: Berkane M, editor. Lecture notes in Statistics: Latent Variable Modeling and Applications to Causality. Vol. 120. 1997. pp. 67–117.
36. Rubin DB. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology. 1974;66:688–701.
37. Rubin DB. Neyman (1923) and causal inference in experiments and observational studies. Statistical Science. 1990;5:1151–1172.
38. Rubin DB. Causal inference using potential outcomes: Design, modeling and decisions. The Journal of American Statistical Association. 2005;100:322–331.
39. Holland PW. Statistics and causal inference (with discussion) The Journal of American Statistical Association. 1986;81:945–970.
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...