- We are sorry, but NCBI web applications do not support your browser and may not function properly. More information

- Journal List
- NIHPA Author Manuscripts
- PMC2967047

# Correlated *z*-values and the accuracy of large-scale statistical estimates

^{*}

^{*}Department of Statistics, Stanford University

## Abstract

We consider large-scale studies in which there are hundreds or thousands of correlated cases to investigate, each represented by its own normal variate, typically a *z*-value. A familiar example is provided by a microarray experiment comparing healthy with sick subjects' expression levels for thousands of genes. This paper concerns the accuracy of summary statistics for the collection of normal variates, such as their empirical cdf or a false discovery rate statistic. It seems like we must estimate an *N* by *N* correlation matrix, *N* the number of cases, but our main result shows that this is not necessary: good accuracy approximations can be based on the root mean square correlation over all *N* · (*N* − 1)/2 pairs, a quantity often easily estimated. A second result shows that *z*-values closely follow normal distributions even under non-null conditions, supporting application of the main theorem. Practical application of the theory is illustrated for a large leukemia microarray study.

**Keywords:**rms correlation, non-null

*z*-values, correlation penalty, Mehler's identity, empirical process, acceleration

## 1 Introduction

Modern scientific studies routinely produce data on thousands of related situations. A familiar example is a microarray experiment in which thousands of genes are being investigated for possible disease involvement. Each gene might produce a *z*-value, say *z _{i}*, for the

*i*th gene, by definition a test statistic theoretically having a standard normal distribution

under the null hypothesis *H*_{0} of no disease involvement. A great deal of the current literature was developed under the assumption of independence among the *z _{i}*'s. This can be grossly unrealistic in practice, as discussed in Owen (2005) and Efron (2007a), among others. This paper concerns the accuracy of summary statistics of the

*z*'s, for example, their empirical cdf (cumulative distribution function), under conditions of substantial correlation.

_{i}Figure 1 concerns a leukemia microarray study by Golub et al. (1999) that we will use for motivation and illustration. Two forms of leukemia are being examined for possible genetic differences: acute lymphoblastic leukemia (ALL) and acute myeloid leukemia (AML). In the version of the data discussed here there are *n*_{1} = 47 ALL patients and *n*_{2} = 25 AML patients, with expression levels on the same *N* = 7128 genes measured on each patient.

*z*-values for

*N*= 7128 genes, leukemia study, Golub et al. (1999).

*Dashed curve*(

*x*), smooth fit to histogram;

*solid curve*“empirical null”, normal density fit from central 50% of histogram, is much wider than

**...**

A two-sample *t*-statistic *t _{i}* comparing AML with ALL expression levels was computed for each gene and converted to a

*z*-value,

where Φ and *F*_{70} are the cumulative distribution functions for a standard normal and a Student-*t* distribution with 70 degrees of freedom. Figure 1 shows a histogram of the *z _{i}*'s, which turns out to be much wider than (1.1) suggests: its central spread is estimated to be

_{0}= 1.68 rather than 1, as discussed in Section 3.

Here is an example of the results to be derived in Sections 2 through 4. Let (*x*) be the right-sided cdf (“survival curve”) of the *z*-values,

Then a good approximation for the variance of (*x*) is

The first term in (1.4) is the usual binomial variance, while the second term is a *correlation penalty* accounting for dependence between the *z _{i}*'s. The quantities occurring in the correlation penalty are

_{0}, the estimate of central spread (1.68 above);- , an estimate of the root-mean-square of the correlations between the
*N*(*N*− 1)/2 pairs of*z*'s (equaling about .11 for the leukemia data, as calculated from the simple formula in Section 3);_{i} ^{(1)}(*x*), the first derivative of a smooth fit to the*z*-value histogram (estimated by a Poisson spline regression in Figure 1).

The row marked $\widehat{\mathrm{sd}}$ in Table 1 is the square root of formula (1.4) applied to the leukemia data. (4) = .025 is seen to have $\widehat{\mathrm{sd}}=.0040$, more than double ${\widehat{\mathrm{sd}}}_{0}=.0018$, the binomial standard deviation obtained by ignoring the second term in (1.4). The permutation standard deviation, obtained from repeated permutations of the 72 patients, is only .0001 at *x* = 4. Permutation methods, which preserve within-microarray correlations, have been advocated for large-scale hypothesis testing (see Westfall and Young, 1993; Dudoit, Shaffer and Boldrick, 2003, Sect. 2.6), but they are inappropriate for the accuracy considerations of this paper.

*x*) (1.3); $\widehat{\mathrm{sd}}$ square root of formula (1.4); ${\widehat{\mathrm{sd}}}_{0}$ square root of first term in (1.4); ${\widehat{\mathrm{sd}}}_{\mathrm{perm}}$ permutation standard deviation. Accuracy of False Discovery Rate estimate $\widehat{\mathrm{Fdr}}\left(x\right)$ discussed

**...**

Formula (1.4), and more ambitious versions that include covariances across different values of *x*, are derived in Section 2 and Section 3: an exact expression is derived first, followed by a series of simplifying approximations and techniques for their estimation. The basic results are extended to provide more general accuracy estimates in Section 4: comparing, for example, the variability of local versus tail-area false discovery rates.

All of our results depend on the assumption that the *z _{i}*'s are normal with possibly different means and variances,

There is no requirement that they be “*z*-values” in the hypothesis-testing sense of (1.1), and in fact this paper is more concerned with estimation than testing. However, *z*-values are ubiquitous in large-scale applications, and not only in the two-sample setting of the leukemia study. Section 5 concerns the *non-null distribution of z-values*. A theorem is derived justifying (1.5) as a good approximation, allowing results like (1.4) to be applied to the leukemia study *z*-values. Section 6 and Section 7 close with remarks and a brief summary.

The statistics microarray literature has shown considerable interest in the effects of large-scale correlation, some good references being Dudoit, van der Laan and Pollard (2004), Owen (2005), Qiu, Klebanov and Yakovlev (2005b), Qiu, Brooks, Klebanov and Yakovlev (2005a) and Desai, Deller and McCormick (2009). Efron (2007a) used a *z*-value setting to examine the effects of correlation on false discovery rate analysis; that paper's Section 2 theorem is a null hypothesis version of the general result developed here. A useful extension along different lines appears in Schwartzman and Lin (2009).

Clarke and Hall's (2009) asymptotic calculations support the use of the independence standard deviation ${\widehat{\mathrm{sd}}}_{0}$ in Table 1, even in the face of correlation. The situations they consider are low-correlation by the standard here, with the root-mean-square value of (1.4) approaching zero (from their assumption (3.2)). Since is often easy to estimate, formulas such as (1.4) provide a quantitative check on the use of ${\widehat{\mathrm{sd}}}_{0}$.

## 2 The distribution of correlated normal variates

Given *N* correlated normal variates *z*_{1}, *z*_{2}, … , *z _{N}*, with possibly different means and standard deviations, let (

*x*) denote their right-sided empirical cdf

^{1}

This section presents tractable formulas for the mean and covariance of the process {(*x*), −∞ < *x* < ∞}, and a simpler approximation that we will see is nicely suited for applications.

Rather than work directly with cdfs, it will be easier, and in a sense more basic, to first derive results for a discretized version of the empirical *density* of the *z _{i}* values. We partition the range

*Ƶ*into

*K*bins

*Ƶ*,

_{k} each bin being of width Δ. Let *x _{k}* indicate the midpoint of

*Ƶ*, and

_{k}*y*the number of

_{k}*z*'s in

_{i}*Ƶ*,

_{k} We will derive expressions for the mean and covariance of the vector ** y** = (

*y*

_{1},

*y*

_{2}, … ,

*y*)′. In effect,

_{K}**is the order statistic of**

*y**z*= (

*z*

_{1},

*z*

_{2}, … ,

*z*)′, becoming exactly that as the bin width Δ → 0. (In which case the

_{N}*y*values go to 1 or 0, with the non-zero bin

_{k}*x*values indicating the locations of the ordered

_{k}*z*'s, assuming no ties.) Familiar statistical applications, of the type described in Section 4, depend on

_{i}*z*only through

**.**

*y*Suppose that the *z _{i}*'s are divided into a finite number of classes, with members of the

*c*th class

*having mean*

_{c}*μ*and standard deviation

_{c}*σ*,

_{c} Let *N _{c}* be the number of members of

*, with*

_{c}*p*the proportion

_{c} so Σ* _{c} N_{c}* =

*N*and Σ

*= 1. The use of model (2.4) for*

_{c}p_{c}*z*-values is supported by the results of Section 5.

If ** x** is the

*K*-vector of bin midpoints, let

*x*= (

_{kc}*x*−

_{k}*μ*)/

_{c}*σ*and

_{c} Likewise, for any real-valued function *h*(*x*) we define * h_{c}* to be the

*K*-vector of function values

also denoted by *h*(*x _{c}*) in what follows.

It is easy to calculate the expectation of the count vector ** y** under the multi-class model (2.4)-(2.5). Let

*π*equal the probability that

_{kc}*z*from class

_{i}*falls into the*

_{c}*k*th bin,

Here (*x*) = exp(−*x*^{2}/2)/√2π, the standard normal density. The approximation *π _{kc}* Δ(

*x*)/

_{kc}*σ*from (2.4) becomes arbitrarily accurate for Δ sufficiently small, and we will take it as exact in what follows. Then

_{c}The *K* × *K* covariance matrix of the count vector ** y** depends on the

*N*×

*N*correlation matrix of

*z*, but in a reasonably simple way discussed next. Two important definitions are needed to state the first result: there are

*M*=

*N*(

*N*− 1)/2 correlations

*ρ*

_{ii′}between pairs (

*z*,

_{i}*z*

_{i′}) of members of

*z*, and we denote by “

*g*(

*ρ*)” the distribution putting weight 1

*/M*on each

*ρ*

_{ii′}. Also, for

_{ρ}(

*u, v*) the bivariate normal density having zero means, unit standard deviations, and correlation

*ρ*, we define

and

(the integral notation being shorthand for summing over *M* discrete points).

### Lemma 1

*Under the multi-class model* (2.4)-(2.5), *the covariance of the count vector *** y** (2.3)

*has two components,*

*where*

*and*

Here diag(* π_{c}*) is the

*K*×

*K*diagonal matrix having diagonal elements

*π*, similarly diag(

_{kc}*), while*

**π**_{d}*is the*

**λ**_{cd}*K*×

*K*matrix with

*kl*th element λ(

*x*); the summations are over all classes.

_{kc}, x_{ld}*Note.* Equation (2.14) assumes that the correlation distribution *g*(*ρ*) is the same across all classes * _{c}*. The proof of Lemma 1, which is similar to that for the simpler situation of Efron (2007a), appears in Remark C of Section 6.

The **cov**_{0} term in (2.12)-(2.13) is the sum of the multinomial covariance matrices that would apply if the *z _{i}*'s were mutually independent with fixed numbers drawn from each class;

**cov**

_{1}is a penalty for correlation, almost always increasing

**cov**(

**). The**

*y**N*

^{2}factor in (2.14) makes the correlation penalty more severe as

*N*increases, assuming

*g*(

*ρ*) stays the same.

Expression (2.14) for the correlation penalty can be considerably simplified. *Mehler's identity* for λ_{ρ}(*u*, *v*) (2.10) is

where *h _{j}* is the

*j*th Hermite polynomial. (See Lancaster, 1958 for an enlightening discussion of (2.15), also known as the “tetrachoric series”, and its connections to the singular value decomposition, canonical correlation, Pearson's coefficient of contingency, and correspondence analysis.) Denoting the

*j*th moment of the correlation distribution

*g*(

*ρ*) by

*α*,

_{j}(2.11) becomes

so **λ**_{cd} in (2.14) can be written in outer product notation as

Making use of (2.8), taken as exact,

where ${\phi}_{c}^{\left(j\right)}$ indicates the *j*th derivative of (*u*) evaluated at each component of * x_{c}* (using

^{(j)}(

*u*) = (−1)

^{j}(

*u*)

*h*(

_{j}*u*)).

Rearranging (2.14) then gives a simplified formula.

### Lemma 2.

*Defining*

(2.14) *for the correlation penalty becomes*

A convenient approximation to **cov**_{1} is based on three reductions of (2.21):

- The second term in (2.21) is neglible for large
*N*. - Common standardization methods for large-scale data sets often make
*α*_{1}, the expectation of*g*(*ρ*), exactly or nearly zero, as illustrated in Section 3 for the leukemia data; see Section 3 of Efron (2009).

The root mean square (rms) correlation

featured in Efron (2007a) (where it is called the *total correlation*), is a single-number summary of *z _{i}*'s entire correlation structure. Carrying out the three reductions above produces a greatly simplified form of (2.21),

with ^{(2)} in (2.20) depending on the second derivative of the normal density, ^{(2)}(*u*) = (*u*) · (*u*^{2} − 1).

Figure 2 compares the exact formulas (2.12)-(2.14) for **cov**(** y**) with the simplified formula based on the rms approximation (2.23); for a numerical example having

*N*= 6000,

*α*= .10, and two classes (2.4)-(2.5), initially with

but then recentered as in the leukemia example; see Remark D of Section 6 for more detail. The plotted curves show the standard deviations $\mathrm{sd}\left\{{y}_{k}\right\}={\mathrm{cov}}_{\mathit{kk}}{\left(\mathit{y}\right)}^{\frac{1}{2}}$ from (2.12), the corresponding rms approximation (2.23), and also ${\mathrm{sd}}_{0}\left\{{y}_{k}\right\}={\left({\mathrm{cov}}_{0\mathit{kk}}\right)}^{\frac{1}{2}}$ from (2.13). We can see there is a substantial correlation penalty over most of the range of *z*, and also that the rms approximation is quite satisfactory here.

Returning to right-sided cdfs (2.1), let ** B** be the

*K*×

*K*matrix

so

is a *K*-vector with *k*th component the proportion of *z _{i}*'s in bins indexed ≥

*k*,

(** B** would be transposed if we were dealing with left-sided cdfs.) The expectation of

**is both obvious and easy to obtain from (2.9),**

where Φ^{+}(*u*) = 1 − Φ(*u*). Now that we are working with tail areas rather than densities we can let Δ → 0, making (2.28) exact.

** has covariance matrix **** Bcov**(

**)**

*y***/**

*B*′*N*

^{2}. The same kind of calculations as in (2.28) applied to Lemma 1 gives the following theorem.

### Theorem 1

*Under the multiclass model* (2.4)-(2.5),

*where ***Cov**_{0 }*has klth entry*

*and*

Here *p _{c}* is from (2.5),

*x*and

_{kc}*x*from (2.6),

_{lc}*α*is as in (2.16) and

_{j} (Notice the distinction between * and (2.20), and between ***Cov** and **cov** etc., Lemma 1.)

The three-step reduction leading to (2.31) also can be applied to **Cov**_{1}: for *α* as in (2.22),

with ^{(1)} depending on the first derivative of the normal density, ^{(1)}(*u*) = −(*u*)*u*. Section 3 shows that (2.33) is especially convenient for applications.

Figure 3 is the version of Figure 2 applying to **: the heavy curve tracks sd(*** _{k}*) from (2.29), the dotted curve is from Rms approximation (2.33), and the dashed curve shows the standard deviations from

**Cov**

_{0}(2.30), ignoring the correlation penalty. Once again the simple approximation formula performs well, particularly for extreme values of

*z*, which are likely to be the ones of interest in applications. The correlation penalty is more severe here than in Figure 2, especially in the tails.

The **Cov**_{0} formula (2.30) is essentially the covariance function for a Brownian bridge. Results related to Theorem 1 can be found in the empirical process literature; see equation (2.2) of Csörgő and Mielniczuk (1996) for example, which applies to the “one-class” case when all the *z _{i}*'s are (0, 1). Desai et al. (2009) extend the covariance calculations in Efron (2007a) to include skewness corrections.

## 3 Estimation of the correlation parameters

Application of Section 2's theory requires us to estimate several parameters: the rms correlation *α* (2.22), and the class components (*p _{c}*,

*μ*,

_{c}*σ*) in (2.4)-(2.5) (though we will see that the latter task can be avoided under some assumptions). This section illustrates the estimation process in terms of the leukemia study of Section 1.

_{c}**, the data matrix for the study, has**

*X**N*= 7128 rows, one for each gene, and

*n*= 72 columns, one for each patient; the

*n*

_{1}= 47 ALL patients precede the

*n*

_{2}= 25 AML patients. Entry

*x*of

_{ij}**is the expression level for gene**

*X**i*on patient

*j*. The columns of

**were individually standardized to have mean 0 and variance 1; see Remark E.**

*X*The *i*th row of ** X** gives

*t*, the two-sample

_{i}*t*-statistic comparing expression levels on gene

*i*for AML versus ALL patients. These are converted to

*z*-values

*z*= Φ

_{i}^{−1}(

*F*

_{70}(

*t*)) (1.2), whose histogram appears in Figure 1. As noted before, the histogram is much wider near its center than a theoretical (0, 1) null distribution: analysis using the

_{i}`locfdr`program described in Efron (2007b, 2008) estimated that proportion

*p*

_{0}= .93 of the genes were “null” (i.e., identically distributed for ALL and AML), and that

*z*-values for the null genes followed a (.09, 1.68

^{2}) distribution.

We wish to estimate the rms correlation *α* (2.22). Let *X*_{0} indicate an *N* × *n*_{0} subset of ** X** pertaining to a single population of subjects, for example the 47 ALL patients. There are

*N*· (

*N*− 1)/2 sample correlations

_{ii′}between rows

*i*and

*i*′ of

*X*_{0}. Computing all of these, or a sufficiently large random sample, yields the empirical mean and variance (

*m*,

*v*) of the distribution,

(*m*, *v*) = (.002,.190^{2}) for the ALL patients. As discussed in Section 3 of Efron (2009), standardizing the columns of *X*_{0} to have mean 0 forces *m* 0, and we will assume *m* = 0 in what follows. (This is equivalent to taking *α*_{1} = 0 as we did following (2.21).)

The obvious choice $\stackrel{-}{\alpha}={v}^{\frac{1}{2}}$ tends to greatly overestimate *α* : each _{ii′} is nearly unbiased for its true correlation *ρ*_{ii′}, a normal-theory approximation for mean and variance being

(Johnson and Kotz, 1970), but the considerable variances in (3.2) can greatly broaden the empirical distribution of the 's. Two corrected estimates of *α* are developed in Efron (2009). The simpler correction formula is

based on an identity between the row and column correlations of *X*_{0}. The second approach uses an empirical Bayes analysis of the variance term in (3.3) to justify a more elaborate formula,

The first three columns of Table 2 compare with for *X*_{0} based on the ALL patients, the AML patients, and both. The final column reports mean ± standard deviation for and in 100 simulations of model (2.24): *N* = 6000, *n*_{1} = *n*_{2} = 40 patients in each class, true *α* = .10; see Remark D. The two estimates are effectively linear functions of each other for typical values of *v*; , the simpler choice, is preferred by the author.

It seems that we need to estimate the class components (*p _{c}*,

*μ*, σ

_{c}_{c}) in (2.4)-(2.5) in order to apply the theory of Section 2, but under certain assumptions this can be finessed, as discussed next.

The marginal density *f*(*z*) under model (2.4)-(2.5) is

so, letting ** f** =

*f*(

*x*) (the density evaluated at the

*K*-vector of bin midpoints), we have Δ ·

**= Σ**

*f**as in (2.8). Formula (2.13) can be expressed as*

_{c}p_{c}**π**_{c} Here we are assuming, as in (2.5), that the class sample sizes *N _{c}* are fixed. A more realistic assumption might be that the numbers

*N*

_{1},

*N*

_{2}, … ,

*N*are a multinomial sample of size

_{C}*N*, sampled with probabilities

*p*

_{1},

*p*

_{2}, … ,

*p*, in which case (3.6) becomes the usual multinomial covariance matrix

_{C} A smooth curve ** fit to the histogram heights**^{2} as in Figure 1 then yields ${\widehat{\mathbf{cov}}}_{0}$ by substitution into (3.7), *without requiring knowledge of the class structure* (2.4)-(2.5). In the same way, we can estimate the **Cov**_{0} for ** in (2.30) by the standard multinomial formula **

Under some circumstances, a similar tactic can be applied to estimate the correlation penalties **cov**_{1} and **Cov**_{1}, (2.23) and (2.33). The first and second derivatives *f*^{(1)}(*z*) and *f*^{(2)}(*z*) of (3.5) are

Suppose we make the *homogeneity assumption* that all *σ _{c}* values are the same, say

*σ*=

_{c}*σ*

_{0}. Comparison with definitions (2.32) and (2.20) then gives

with *f*^{(j)} = (*f*^{(j)}(*x _{k}*))′. This leads to the convenient covariance penalty formulas,

A smooth estimate (*z*) of *f*(*z*) can be differentiated to give estimated values of **Cov**_{1} and **cov**_{1}, for example,

for the correlation penalty standard deviation of (*x _{k}*) (2.27). (This provides the second term in formula (1.4).) The heavy curve in Figure 4 shows (3.12) for the leukemia data, using

_{0}= 1.68, = .114, and (

*z*) from Figure 1.

Suppose we are unwilling to make the homogeneity assumption. A straightforward approach to estimating **Cov**_{1} or **cov**_{1} requires assessments of the parameters (*p _{c}*,

*μ*,

_{c}*σ*) in (2.4)-(2.5). These can be based on the “non-null counts” (Efron, 2007b), the small bars plotted negatively in Figure 1; see Remark B. The figure suggests three classes, left, center and right, with parameter values as estimated in Table 3.

_{c}The dashed curve in Figure 4 shows sd_{1}(* _{k}*) estimated directly from (2.32)-(2.33) using the values in Table 3. It is similar to the homogeneity estimate (3.12) except in the extreme tails.

Formula (1.4) for the standard deviation of (*x*) was tested in a simulation experiment. The specifications were the same as in Figure 3, with *N* = 6000, *α* = .10, and two classes of *z*-values (2.24). One hundred ** X** matrices were generated as in the simulation for Table 2, each yielding a vector of 6000 correlated

*z*-values, followed by

_{0}, and

^{(1)}(

*x*) for use in (1.4); see Remark D for further details. Finally, $\widehat{\mathrm{sd}}$, the square root of (1.4), was calculated along with ${\widehat{\mathrm{sd}}}_{0}$, the square root of just the first term.

The solid curve in Figure 5 shows the average of the $\widehat{\mathrm{sd}}$ values for *x* between −4 and 4.5, with solid bars indicating standard deviations of the 100 $\widehat{\mathrm{sd}}\u2019\mathrm{s}$. There is a good match of the average with the exact sd curve from Figure 3. The error bars indicate moderate variability across the replications. The average for ${\widehat{\mathrm{sd}}}_{0}$, dashed curve, agrees with the corresponding curve in Figure 3 and shows that correlation cannot be ignored in this situation.

## 4 Applications

Correlation usually degrades statistical accuracy, an important question for the data analyst being the severity of its effects on the estimates and tests at hand. The purpose of Section 2 and Section 3 was to develop practical methods for honestly assessing the accuracy of inferences made in the presence of large-scale correlation. This section presents a few examples of the methodology in action.

We have already seen one example: in Table 1 the accuracy of (*x*), the right-sided empirical cdf for the leukemia data, computed from the usual binomial formula that assumes independence among the *z*-values,

was compared with $\widehat{\mathrm{sd}}$ from formula (1.4) in which the correlation penalty term was included: $\widehat{\mathrm{sd}}$ more than doubled ${\widehat{\mathrm{sd}}}_{0}$ over most of the range.

Suppose we assume, as in Efron (2008), that each of the *N* cases (the *N* genes in the leukemia study) is either *null* or *non-null* with prior probability *p*_{0} or *p*_{1} = 1 − *p*_{0}, and with the corresponding *z*-values having density either *f*_{0}(*z*) or *f*_{1}(*z*),

Let *F*_{0} and *F*_{1} be the right-sided cdfs of *f*_{0} and *f*_{1}, and *F* the mixture cdf

The probability of a case being null given that *z* exceeds *x* is

according to Bayes theorem, “Fdr” standing for false discovery rate.

If *p*_{0} and *F*_{0} are known then Fdr has the obvious estimate

(2.1). Benjamini and Hochberg's celebrated 1995 algorithm uses $\widehat{\mathrm{Fdr}}\left(x\right)$ for simultaneous hypothesis testing, but it can also be thought of as an empirical Bayes estimator of the Bayesian probability Fdr(*x*). The bottom row of Table 1 shows $\widehat{\mathrm{Fdr}}\left(x\right)$ for the leukemia data, taking *p*_{0} = .93 and *F*_{0} ~ *N*(.09, 1.68^{2}) as in Figure 1. (Later we will do a more ambitious calculation taking into account the estimation of *p*_{0} and *F*_{0}.)

The coefficient of variation for $\widehat{\mathrm{Fdr}}\left(x\right)$ approximately equals that for (*x*) (when *p*_{0}*F*_{0}(*x*) is known in (4.5)). At *x* = 5 we have $\widehat{\mathrm{Fdr}}\left(5\right)=.15$, with coefficient of variation about .19. An $\widehat{\mathrm{Fdr}}$ of .15 might be considered small enough to trigger significance in the Benjamini–Hochberg algorithm, but in any case it seems clear that the probabiltiy of being null is quite low for the 71 genes having *z _{i}* above 5. Even taking account of correlation effects, we have a rough upper confidence limit of .21 (i.e., .15 · (1 + 2 · .19)) for Fdr(5).

Next we consider accuracy estimates for a general class of statistics *Q*(** y**), where

*Q*is a

*q*-dimensional function of the count vector

**(2.3). As in Section 5 of Efron (2007b), we assume that a small change**

*y**d*in the count vector (considered as varying continuously) produces change

**y***dQ*in

*Q*according to

If $\widehat{\mathbf{cov}}\left(\mathit{y}\right)$ is a covariance estimate for ** y**, obtained perhaps as in (2.12), (3.8), or (3.11), then the usual delta-method estimate for cov(

*Q*) is

In a theoretical context, where **cov**(** y**) is known, we might instead use

now with the derivative matrix ** D** evaluated at the expectation of

**.**

*y*Model (4.2) yields the *local false discovery rate*

*f*(*x*) being the mixture density

fdr(*x*) is inferentially more appropriate than Fdr(*x*) from a Bayesian point of view, but it is not as immediately available since it involves estimating the *density **f*(*x*). However, because *z*-value densities are mixtures of near-normals as shown in Section 5, it is usually straightforward to carry out the estimation.

`Locfdr`, the algorithm discussed in Efron (2007a, 2008), estimates *f*(*x*) by means of Poisson regression of the counts *y _{k}* as a spline function of the

*x*, the bin midpoints in (2.2)-(2.3). The structure matrix

_{k}**for the Poisson regression is**

*M**K*×

*d*, where

*K*is the number of bins and

*d*is degrees of freedom (e.g., the number of free parameters of the spline fit; see Remark G for details). Let

**be the vector of fitted values (**

*x*), and

_{k}**the vector with components**

*= log((*

_{k}*x*)). Then, as discussed in (Efron, 2007b, Sect. 5), (4.6) takes the form

_{k}For any function *v*(*x*) define the vector

as with ** and **** above. If **

then

implying, if *p*_{0} and *f*_{0} are known, that

with ** D** =

**(**

*M***′ diag(**

*M**N*Δ

**)**

*f***)**

*M*^{−1}

**′ (4.8).**

*M*The solid curves in Figure 6 plot standard deviations for $\mathrm{log}\left(\widehat{\mathrm{fdr}}\left(x\right)\right)$, obtained as square root of the diagonal elements of $\mathbf{cov}\left(\widehat{\mathbf{lfdr}}\right)$ (4.15), for model (2.24) with *N* = 6000 and rms correlation *α* equal 0, .1, or .2; see Remark D. The horizontal axes are plotted in terms of the upper percentiles of *F*(*x*), the right end of each plot corresponding to the far right tail of the *z*-value distribution. For *α* = 0, $\mathrm{sd}\left(\mathrm{log}\phantom{\rule{thinmathspace}{0ex}}\widehat{\mathrm{fdr}}\left(x\right)\right)$ increases from .03 to .08 as we move from the fifth to the first percentile of *F*. The coefficient of variation (CV) of $\widehat{\mathrm{fdr}}\left(x\right)$ nearly equals $\mathrm{sd}\left(\mathrm{log}\phantom{\rule{thinmathspace}{0ex}}\widehat{\mathrm{fdr}}\left(x\right)\right)$, so $\widehat{\mathrm{fdr}}\left(x\right)$ is quite accurately estimated for *α* = 0, but substantially less so for *α* = .2. Reducing *N* to 1500 doubles the standard deviation estimates for *α* = 0, but has less effect in the correlated situations: for *α* = .1 for example, the increase is only 20% at percentile .025. Simulations confirmed the correctness of these results.

*Solid curves*show standard deviation of $\mathrm{log}\left(\widehat{\mathrm{fdr}}\left(x\right)\right)$ as a function of

*x*at the upper percentiles of the

*z*-value distribution for model (2.24),

*N*= 6000 and

*α*= 0, .1, .2.

*Dotted curves*(green) same for $\mathrm{log}\left(\widehat{\mathrm{Fdr}}\left(x\right)\right)$ (4.5), nonparametric Fdr estimator.

**...**

Intuitively it seems that fdr should be harder to estimate than Fdr, but that is not what Figure 6 shows. Let * _{k}* = log((

*x*)), with corresponding vector

_{k}**. Then**

**in (4.6) has**

with ** B** as in (2.25), giving an estimate of

**cov**(

**) from (4.7) or (4.8). The same argument as (4.13)-(4.15) shows that this also estimates $\mathbf{cov}\left(\mathrm{log}\phantom{\rule{thinmathspace}{0ex}}\widehat{\mathbf{Fdr}}\right)$, the log of vector (4.5), assuming**

*p*

_{0}

*F*

_{0}(

*x*) is known. The dotted curves in Figure 6 show standard deviations for $\mathrm{log}\left(\widehat{\mathrm{Fdr}}\left(x\right)\right)$. If anything, Figure 6 suggests that $\widehat{\mathrm{fdr}}$ is

*less*variable than $\widehat{\mathrm{Fdr}}$, particularly at the smaller percentiles.

Here we are comparing the nonparametric estimator $\widehat{\mathrm{Fdr}}\left(x\right)$ (4.5) with the parametric estimator $\widehat{\mathrm{fdr}}\left(x\right)$. The Poisson spline estimate (*x*) that gave $\widehat{\mathrm{fdr}}\left(x\right)$ can be summed to give parametric estimates of *F*(*x*) and Fdr(*x*), say $\stackrel{~}{\mathrm{Fdr}}\left(x\right)$. Straightforward calculations show that the derivative matrix ** for $\stackrel{~}{\mathrm{Fdr}}\left(x\right)$ is **

with ** B** from (2.25) and

*equaling in (4.11). Standard deviations for $\mathrm{log}\left(\stackrel{~}{\mathrm{Fdr}}\right)$, shown by the dashed curves in Figure 6, indicate about the same accuracy for $\stackrel{~}{\mathrm{Fdr}}\left(x\right)$ as for $\widehat{\mathrm{Fdr}}$.*

_{f}All of these calculations assumed that *p*_{0} and *f*_{0}(*z*) (or *F*_{0}(*z*)) in (4.2) were known. This is unrealistic in situations like the leukemia study, where there is clear evidence that a textbook *N*(0, 1) theoretical null distribution is too narrow. Estimating an “empirical null” distribution, such as *N*(.09, 1.68^{2}) in Figure 1, is both necessary and feasible (see Efron, 2008) but can greatly increase variability, as discussed next.

Formula (4.14) becomes

when *p*_{0} and *f*_{0} are themselves estimated. The corresponding derivative matrix $\widehat{\mathit{D}}=d\widehat{\mathbf{lfdr}}\u2215d\mathit{y}$ in (4.6) appears as equation (5.8) in Efron (2007b), this formula applying to the *central matching* method for estimating *p*_{0}*f*_{0}(*z*). The second row of Table 4 shows $\left\{\mathrm{log}\phantom{\rule{thinmathspace}{0ex}}\widehat{\mathrm{fdr}}\left(x\right)\right\}$ obtained from ** D**cov(

**)**

*y***′ for the same situation as in the middle panel of Figure 6. Comparison with the theoretical null standard deviations (from the solid curve in the middle panel) shows that estimating the null distribution greatly increases variability.**

*D*Here are some points to note:

- Accuracy is worse for $\mathrm{log}\left(\widehat{\mathrm{Fdr}}\right)$ then for $\mathrm{log}\left(\widehat{\mathrm{fdr}}\right)$ in the top line of Table 4.
- Accuracy is somewhat better when
*p*_{0}*f*_{0}(*z*) is estimated by the MLE option in`locfdr`(Lemma 2 of Efron, 2007b). - The big empirical null standard deviations in Table 4 are at least partially misleading: some of the variability in $1+O\left({n}^{-\frac{1}{2}}\right)$ is “signal” rather than “noise”, tracking conditional changes in the appropriate value of fdr(
*x*). See Figure 2 of Efron (2007a) and the discussion in that paper.

Remark H of Section 6 describes a parametric bootstrap resampling scheme that avoids the Taylor series computations of (4.7), but which has not yet been carefully investigated.

## 5 The non-null distribution of *z*-values

The results of the previous sections depend on the variates *z _{i}* having normal distributions (1.5). By definition, a

*z*-value is a statistic having a

*N*(0, 1) distribution under a null hypothesis

*H*

_{0}of interest (1.1): but will it still be normal for non-null conditions? This section shows that under repeated sampling the non-null distribution of

*z*will typically have mean

*O*(1), standard deviation $1+O\left({n}^{-\frac{1}{2}}\right)$, and non-normality

*O*(

_{p}*n*

^{−1}) (as measured by the magnitude of skewness and kurtosis). In other words, normality degrades more slowly than unit standard deviation as we move away from the null hypothesis.

Figure 7 illustrates the phenomenon for the case of non-central *t* distributions,

the notation indicating a non-central *t* variable with *ν* degrees of freedom and non-centrality parameter *δ* (*not δ*^{2}), as described in Chapter 31 of Johnson and Kotz (1970). Here, as in (1.2), *F _{ν}* is the cdf of a

*central t*distribution. The standard deviation of

_{ν}*z*decreases as |

*δ*| increases; for

*δ*= 5, ν = 20,

*z*has (mean, sd) equal (4.01, 0.71). The useful and perhaps surprising observation is that normality holds up quite well even far from the null case

*δ*= 0. We tacitly used this fact to justify application of our theoretical results to the leukemia study.

To begin the theoretical development, suppose that *y*_{1}, *y*_{2}, … , *y*_{n} are independent and identically distributed (iid) observations sampled from *F _{θ}*, a member of a one-parameter family of distributions,

having its moment parameters {mean, standard deviation, skewness, kurtosis}, denoted

defined differentiably in *θ*. The results that follow are heuristic in the sense that they only demonstrate second-order Cornish–Fisher expansion properties, with no attempt to provide strict error bounds.

Under the null hypothesis *H*_{0} : *θ* = 0, which we can write as

the standardized variate

satisfies

Normality can be improved to second order by means of a Cornish–Fisher transformation,

which reduces the skewness in (5.6) from $O\left({n}^{-\frac{1}{2}}\right)$ to *O*(*n*^{−1}),

See Chapter 1 of Johnson and Kotz (1970) or, for much greater detail, Section 2.2 of Hall (1992). We can interpret (5.8) as saying that *Z*_{0} is a *second-order z*-*value*,

e.g., *a test statistic giving standard normal p*-*values accurate to O*(*n*^{−1}).

Suppose now that *H*_{0} is false, and instead *H*_{1} is true, with *y*_{1}, *y*_{2}, … , *y*_{n} iid according to

rather than (5.4). Setting

makes *Z*_{1} second-order normal under *H*_{1},

We wish to calculate the distribution of *Z*_{0} (5.7) under *H*_{1}. Define

Some simple algebra yields the following relationship between *Z*_{0} and *Z*_{1}.

### Lemma 3

*Under definitions* (5.7), (5.11) *and* (5.13),

*where*

The asymptotic relationships claimed at the start of this section are easily derived from Lemma 3. We consider a sequence of alternatives *θ _{n}* approaching the null hypothesis value

*θ*

_{0}at rate ${n}^{-\frac{1}{2}}$,

The parameter *d* = *√n*(*μ _{θn}* −

*μ*

_{0})/

*σ*

_{0}defined in (5.13) is then of order

*O*(1), as is

while standard Taylor series calculations give

the dot indicating differentiation with respect to *θ*.

### Theorem 2

*Under model* (5.2), (5.16), *and the assumptions of Lemma 3,*

*with M and S as given in* (5.17)-(5.18). *Moreover,*

*Proof*. The proof of Theorem 2 uses Lemma 3, with *θ _{n}* playing the role of

*H*1 in (5.14). Both 1 −

*c*

^{2}and (

*γ*

_{1}/

*γ*

_{0})

*S*−

*c*

^{2}are of order $O\left({n}^{-\frac{1}{2}}\right)$; the former from (5.18) and the latter using

*γ*

_{1}/

*γ*

_{0}= 1 + (

_{0}/

*γ*

_{0})(

*θ*−

_{n}*θ*) +

_{0}*O*(

*n*

^{−1}). Since ${Y}_{1}^{2}-1$ is

*O*(1), this makes the bracketed term in (5.14) ${O}_{p}\left({n}^{-\frac{1}{2}}\right)$; multiplying by

_{p}*g*

_{0}=

*γ*

_{0}/(6 √

*n*) reduces it to

*O*(

_{p}*n*

^{−1}), and (5.19) follows from (5.12). Differentiating

*M*and

*S*in (5.17)-(5.18) with respect to

*d*verifies (5.20). ■

Theorem 2 supports our claim that, under non-null alternatives, the null hypothesis normality of *Z*_{0} degrades more slowly than its unit standard deviation, the comparison being *O _{p}*(

*n*

^{−1}) versus $O\left({n}^{-\frac{1}{2}}\right)$.

One-parameter exponential families are an important special case of (5.2). With *θ* the natural parameter of and *y* its sufficient statistic, i.e., with densities proportional to exp{*θ _{y}*}

*g*

_{0}(

*y*), (5.20) reduces to

The parameter *γ*_{0}=(6√*n*) is called the *acceleration* in Efron (1987), interpreted as “the rate of change of standard deviation with respect to expectation on the normalized scale,” which agrees with its role in (5.21).

As an example, suppose ${y}_{1},{y}_{2},\dots ,{y}_{n}\stackrel{\mathrm{iid}}{\sim}\theta {\Gamma}_{1}$, Γ* _{n}* indicating a standard gamma distribution with

*n*degrees of freedom, so

*gθ*(

*y*) = (1/

*θ*)exp(

*y*/

*θ*) for

*y*≤ 0. (Equivalently, ~

*θ*Γ

*n*/

*n*.) This is an exponential family having skewness

*γ*

_{0}= 2 for any choice of

*θ*

_{0}. An exact

*z*-value for testing

*H*

_{0}:

*θ*=

*θ*

_{o}is

where *G _{n}* is the cdf of Γ

_{n}. Table 5 shows the mean, standard deviation, skewness and kurtosis of

*Z*

_{0}for

*n*= 10,

*θ*

_{0}= 1, evaluated for several choices of the alternative

*θ*

_{1}. The standard deviation of

*Z*

_{0}increases steadily with

*θ*

_{1}; here

*γ*

_{0}/(6√

*n*) = .1054, matching to better than three decimal places the observed numerical derivative

*dS/dM*. Skewness and kurtosis are both very small; in the equivalent of Figure 7, there is no visible discrepancy at all between the density curves for

*Z*

_{0}and their matching normal equivalents.

So far we have considered *z*-values obtained from an average of iid observations, but the results of Theorem 2 hold in greater generality. Section 5 of Efron (1987) considers one-parameter families where , an estimator of *θ* , has MLE-like asymptotic properties in terms of its bias, standard deviation, skewness and kurtosis,

Letting play the role of and *μ _{θ}* =

*θ*+

*β*/

_{θ}*n*in definitions (5.5)-(5.12), Lemma 3 and Theorem 2 remain true, assuming only the validity of the Cornish–Fisher transformations (5.9)-(5.12). Ignoring the bias

*β*, i.e., taking

_{θ}*Y*

_{0}= √

*n*( −

*θ*

_{0})/

*σ*

_{0}at (5.5), adds an $O\left({n}^{-\frac{1}{2}}\right)$ term to

*M*in (5.17).

Moving beyond one-parameter families, suppose is a *p*-parameter exponential family, having densities proportional to $\mathrm{exp}\{{\eta}_{1}{x}_{1}+{\eta}_{2}^{\prime}{x}_{2}\}{g}_{0}({x}_{1},{x}_{2})$, where *η*_{1} and *x*_{1} are real-valued while *η*_{2} and *x*_{2} are (*p* − 1)-dimensional vectors, but where we are only interested in *η*_{1}, not the nuisance vector *η*_{2}. The *conditional* distribution of *x*_{1} given *x*_{2} is then a one-parameter exponential family with natural parameter *η*_{1}, which puts us back in the context of Theorem 2. Remark H of Section 6 suggests a further extension where the parameter of interest “ *θ*” can be a general real-valued function of *η*, not just a coordinate such as *η*_{1}.

The non-central *t* family does not meet the conditions of Lemma 3 or Theorem 2: (5.1) is symmetric in *δ* around zero, causing *γ*_{0} in (5.14) to equal zero and likewise the derivative in (5.20). Nevertheless, as Figure 7 shows, it does exhibit impressive non-null normality. Table 6 displays the moment parameters of *z* = Φ^{−1}(*F _{ν}*(

*t*)) (1.3), for

*t*~

*t*(

_{ν}*δ*),

*ν*= 20 and

*δ*= 0, 1, 2, 3, 4, 5. The non-null normality isn't quite as good as in the gamma example of Table 5, but is still quite satisfactory for its application in Section 4.

Microarray studies can be more elaborate than two-sample comparisons. Suppose that in addition to the *N* × *n* expression matrix ** X** we have measured a primary response variable

*y*and covariates

_{j}*w*

_{j1},

*w*

_{j2}, … ,

*w*on each of the

_{jp}*n*subjects. Given the observed expression levels

*x*

_{i1},

*x*

_{i2}, … ,

*x*for gene

_{in}*i*, we could calculate

*t*, the usual

_{i}*t*-value for

*y*as a function of

_{j}*x*, in a linear model that includes the

_{ij}*p*covariates. Then

is a *z*-value (1.1) under the usual Gaussian assumption, showing behavior like that in Table 6 for non-null genes.

## 6 Remarks

Some remarks, proofs, and details relating to the previous sections are presented here.

### A. Poisson regression

The curve (*z*) in Figure 1 is a Poisson regression fit to the counts *y _{k}*, as a natural spline function of the bin centers

*x*. Here the

_{k}*x*ranged from −7.8 to 7.8 in steps of Δ = .2, while the spline had five degrees of freedom, so

_{k}**in (4.11) was 79 × 6 (including the intercept column). See Section 5 of Efron (2007b).**

*M*### B. Table 3

Section 3 of Efron (2008) defines the *non-null counts* ${y}_{k}^{\left(1\right)}=(1-\widehat{\mathrm{fdr}}\left({x}_{k}\right))\cdot {y}_{k}$. Since, under model (4.2), $1-\widehat{\mathrm{fdr}}\left({x}_{k}\right)$ estimates the proportion of non-null *z*-values in bin *k*, ${y}_{k}^{\left(1\right)}$ estimates the number of non-nulls. The ${y}_{k}^{\left(1\right)}$ values are plotted below the x axis in Figure 1, determining the “left” and “right” distribution parameters in Table 3. “Center” was determined by the empirical null fit from `locfdr`, using the MLE method described in Section 4 of Efron (2007b). This method tends to underestimate the non-null counts near *z* = 0, and also the *σ _{c}* values for the left and right classes, but increasing then to 1.68 had little effect on the dashed curve in Figure 4.

### C. Proof of Lemma 1

Let *I _{k}*(

*i*) denote the indicator function of the event

*z*

_{i}*Ƶ*(2.2) so that the number of

_{k}*z*'s from class

_{i}

_{c }*Ƶ*is

_{k} the boldface subscript indicating summation over the members of * _{c}*. We first compute

*E*{

*y*} for bins

_{kc}y_{ld}*k*and

*l*,

*k*≠

*l*, and classes

*c*and

*d*,

following notation (2.5)-(2.10), with *χ _{ij}* the indicator function of event

*i*=

*j*(which can only occur if

*c*=

*d*). This reduces to

under the assumption that the same correlation distribution *g*(*ρ*) applies across all class combinations. Since *y _{k}* = Σ

_{c }

*y*(2.3), we obtain

_{kc}the non-bold subscripts indicating summation over classes.

Subtracting

from (6.4) results, after some rearrangement, in

Using *π _{kc}* = Δ · (

*x*)/

_{kc}*σ*as in (2.8), expression (6.6) is seen to equal the

_{c}*kl*th element of

**cov**(

**) in Lemma 1, when**

*y**k*≠

*l*.

The case *k* = *l* proceeds in the same way, the only difference being that *N*Δ*p _{c}χ_{cd}*(

*x*)/

_{kc}*σ*must be added to formula (6.3). This adds

_{c}*N*ΔΣ

_{c}

*p*(

_{c}*x*)/

_{kc}*σ*to (6.6), again in agreement with

_{c}**cov**(

**) in Lemma 1.**

*y*The assumption that *g*(*ρ*) is the same across all classes can be weakened for the rms approximations (2.23) and (2.33), where we only need the second moments *α*_{2} to be the same. In fact, the class structure can disappear entirely for rms formulas, as seen in (3.12).

###
*D. Model*
(2.24)

Specifications (2.24) were recentered to give overall expectation 0 in (2.4), (2.5):

these being the parameter values used in Figures Figures2,2, ,3,3, ,55 and and6.6. Recentering overall expectations to zero is common in practice, a consequence of the data matrix ** X** having its column-wise means subtracted off.

The 6000 × 80 data matrices ** X** used in the simulations for Table 2 and Figure 5 had entries

*x*~ (

_{ij}*δ*, 1) independent across columns

_{ij}*j*:

*δ*= 0 for

_{ij}*j*≤ 40, while for columns

*j*> 40,

*z*-values based on the difference of means between the last and first 40 “patients” then satisfy (2.4), (6.7). The correlation distribution *g*(*ρ*) was supported on two points, 20% *ρ* = .20 and 80% *ρ* = −.05, giving *α* = .10.

### E. Leukemia data standardization

The original entries *x _{ij}* of the leukemia data matrix were genetic expression levels obtained using Affymetrix oligonucleotide microarrays. For the analyses here, each column of

**was replaced by its normal score values**

*X**= Φ*

_{ij}^{−1}((

*r*−.5)/7128), where

_{ij}*r*was the rank of

_{ij}*x*in its column. Transformations such as this reduce the disturbing effects of sensitivity differences between microarrays; see Bolstad, Irizarry, Astrand and Speed (2003).

_{ij}### F. A parametric bootstrap method

Section 3 of Efron (2007a) discusses a hierarchical Poisson simulation scheme that can be adapted to the more general context of this paper. Following the notation in (3.11), we first simulate a vector ** u**,

and then take ** y** ~ Poisson(

**), that is ${y}_{k}\stackrel{\mathrm{ind}}{\sim}\text{Poisson}\left({u}_{k}\right)$. The simulated**

*u***vectors can then be used to assess the variability of any function**

*y**Q*(

**), obviating the need for the derivative matrix**

*y***. This amounts to a parametric bootstrap approach to accuracy estimation. It produced similar answers to (4.11) when applied to the leukemia data but seemed prone to biases in other applications. (Nonparametric bootstrapping, resampling columns of the data matrix**

**, can produce erratic results for the kind of large-scale accuracy problems considered in Section 4.)**

*X*### G. z-value densities

Suppose test statistic *t* has possible densities {*f _{θ}*(

*t*), θ Θ}, with corresponding cdfs

*F*(

_{θ}*t*), and we wish to test

*H*

_{0}:

*θ*=

*θ*

_{0}. The

*z*-value statistic

*z*= Φ

^{−1}{

*F*

_{θ0}(

*t*)} then has densities

The density curves in Figure 7 were obtained from (6.10), with *f _{θ}*(

*t*) the noncentral

*t*(

_{ν}*θ*) density,

*ν*= 20 and

*θ*= 0, 1, 2, 3, 4, 5.

### H. Extensions of Theorem 2

In some circumstances, Theorem 2 can be extended to multi-parameter families = {*F _{η}*} where we wish to test

*θ*=

*θ*

_{0}for

*θ*a real-valued function of

*η*. This is straightforward to verify in the context of Efron (1985), which includes for example Fieller's problem, and is conjectured to be true in general exponential families.

## 7 Summary

The paper considers studies where a large number *N* of cases are under investigation, *N* perhaps in the hundreds or thousands, each represented by its own *z*-value *z _{i}*, and where there is the possibility of substantial correlation among the

*z*'s. Our main result is a simple approximation formula for the accuracy of summary statistics such as the empirical cdf of the

_{i}*z*-values or an estimated false discovery rate. The argument proceeds in five steps:

- Exact formulas for the accuracy of correlated
*z*-value cdfs are derived under normal distribution assumptions (Section 2). - Practical estimates for the approximation formulas are derived and demonstrated through simulations and application to a microarray study (Section 3 and Section 4).
- Delta-method arguments are used to extend the cdf results to more general summary statistics (Section 3 and Section 4).
- Under reasonable assumptions, it is shown that
*z*scores tend to have nearly normal distributions, even in non-null situations (Section 5), justifying application of the theory to studies in which the individual variates are*z*-values.

Our main conclusion is that by dealing with normal variates, a practical assessment of large-scale correlation effects on statistical estimates is possible.

## Acknowledgments

This work was supported in part by NIH grant 8R01 EB002784 and NSF grant DMS0505673.

## Footnotes

## References

- Bolstad B, Irizarry R, Astrand M, Speed T. A comparison of normalization methods for high density oligonucleotide array data based on variance and bias. Bioinformatics. 2003;19:185–193. [PubMed]
- Clarke S, Hall P. Robustness of multiple testing procedures against dependence. Ann. Statist. 2009;37:332–358.
- Csörgő S, Mielniczuk J. The empirical process of a short-range dependent stationary sequence under Gaussian subordination. Probab. Theory Related Fields. 1996;104:15–25.
- Desai K, Deller J, McCormick J. The distribution of number of false discoveries for highly correlated null hypotheses. Ann. Appl. Statist. 2009 Submitted, under review.
- Dudoit S, Laan M. J. van der, Pollard KS. Multiple testing. I. Single-step procedures for control of general type I error rates. Stat. Appl. Genet. Mol. Biol. 2004;3:71. Art. 13. electronic. [PubMed]
- Dudoit S, Shaffer JP, Boldrick JC. Multiple hypothesis testing in microarray experiments. Statist. Sci. 2003;18:71–103.
- Efron B. Bootstrap confidence intervals for a class of parametric problems. Biometrika. 1985;72:45–58.
- Efron B. Better bootstrap confidence intervals. J. Amer. Statist. Assoc. 1987;82:171–200. with comments and a rejoinder by the author.
- Efron B. Correlation and large-scale simultaneous significance testing. J. Amer. Statist. Assoc. 2007a;102:93–103.
- Efron B. Size, power and false discovery rates. Ann. Statist. 2007b;35:1351–1377.
- Efron B. Microarrays, empirical Bayes and the two-groups model. Statist. Sci. 2008;23:1–22.
- Efron B. Are a set of microarrays independent of each other? Ann. Appl. Statist. 2009 To appear. [PMC free article] [PubMed]
- Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES. Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring. Science. 1999;286:531–537. [PubMed]
- Hall P. The bootstrap and Edgeworth expansion. Springer-Verlag; New York: 1992. (Springer Series in Statistics).
- Johnson NL, Kotz S. Distributions in statistics. Continuous univariate distributions. 1. Houghton Mifflin Co; Boston, Mass: 1970.
- Lancaster HO. The structure of bivariate distributions. Ann. Math. Statist. 1958;29:719–736.
- Owen AB. Variance of the number of false discoveries. J. R. Stat. Soc. Ser. B Stat. Methodol. 2005;67:411–426.
- Qiu X, Brooks A, Klebanov L, Yakovlev A. The effects of normalization on the correlation structure of microarray data. BMC Bioinformatics. 2005a;6:120. [PMC free article] [PubMed]
- Qiu X, Klebanov L, Yakovlev A. Correlation between gene expression levels and limitations of the empirical Bayes methodology for finding differentially expressed genes. Stat. Appl. Genet. Mol. Biol. 2005b;4:32. Art. 34. electronic. [PubMed]
- Schwartzman A, Lin X. The effect of correlation in false discovery rate estimation. Harvard University; 2009. (Biostatistics Working Paper Series number 106).
- Westfall P, Young S. Resampling-based Multiple Testing: Examples and Methods for p-Value Adjustment. Wiley-Interscience; New York, NY: 1993.

## Formats:

- Article |
- PubReader |
- ePub (beta) |
- PDF (1.5M)

- Selection of differentially expressed genes in microarray data analysis.[Pharmacogenomics J. 2007]
*Chen JJ, Wang SJ, Tsai CA, Lin CJ.**Pharmacogenomics J. 2007 Jun; 7(3):212-20. Epub 2006 Aug 29.* - Empirical Bayes screening of many p-values with applications to microarray studies.[Bioinformatics. 2005]
*Datta S, Datta S.**Bioinformatics. 2005 May 1; 21(9):1987-94. Epub 2005 Feb 2.* - Improving the statistical detection of regulated genes from microarray data using intensity-based variance estimation.[BMC Genomics. 2004]
*Comander J, Natarajan S, Gimbrone MA Jr, García-Cardeña G.**BMC Genomics. 2004 Feb 27; 5(1):17. Epub 2004 Feb 27.* - The significance of non-significance.[QJM. 1998]
*Berry EM, Coustère-Yakir C, Grover NB.**QJM. 1998 Sep; 91(9):647-53.* - Estimating renal function in children: a new GFR-model based on serum cystatin C and body cell mass.[Dan Med J. 2012]
*Andersen TB.**Dan Med J. 2012 Jul; 59(7):B4486.*

- Estimating False Discovery Proportion Under Arbitrary Covariance Dependence[Journal of the American Statistical Associa...]
*Fan J, Han X, Gu W.**Journal of the American Statistical Association. 2012; 107(499)1019-1035* - Large Covariance Estimation by Thresholding Principal Orthogonal Complements[Journal of the Royal Statistical Society. S...]
*Fan J, Liao Y, Mincheva M.**Journal of the Royal Statistical Society. Series B, Statistical methodology. 2013 Sep 1; 75(4)10.1111/rssb.12016* - Computationally efficient permutation-based confidence interval estimation for tail-area FDR[Frontiers in Genetics. ]
*Millstein J, Volfson D.**Frontiers in Genetics. 4179* - Statistical Analysis of Big Data on Pharmacogenomics[Advanced drug delivery reviews. 2013]
*Fan J, Liu H.**Advanced drug delivery reviews. 2013 Jun 30; 65(7)987-1000* - Optimal False Discovery Rate Control for Dependent Data[Statistics and its interface. 2011]
*Xie J, Cai TT, Maris J, Li H.**Statistics and its interface. 2011; 4(4)417-430*

- PubMedPubMedPubMed citations for these articles
- TaxonomyTaxonomyRelated taxonomy entry
- Taxonomy TreeTaxonomy Tree