NSDUH Mental Health Estimates before 2008

Starting in 2001, NSDUH mental health estimates used a cut point methodology based on the K6 score (Kessler et al.; 2003; Office of Applied Studies, 2002). Kessler and colleagues, using a sample of 155 respondents from the Boston area, performed logistic regression modeling to select viable predictors for SMI and concluded that a model based on the K6 alone performed best. Respondents with a total score of 13 or greater on the K6 were classified as having SMI. The cut point between 12 and 13 had been chosen to offset false positives (respondents without SMI but classified as having SMI) and false negatives (respondents with SMI but classified as not having SMI). This cut point methodology was applied to NSDUH national samples from 2001 to 2004 and was reported as SMI prevalence.

Additional changes to mental health questions were made in the 2004 NSDUH including removing the WHODAS that had been added in 2001 and adding a module on major depressive episode (MDE). The modified questionnaire was first implemented in a random half sample of adults in 2004, and then in the full adult sample in 2005. Because of concerns about context effects of the new questionnaire impacting the reporting of K6 items, the 2005–2007 NSDUH national findings changed the descriptive terminology for the K6 cut point measure from SMI to serious psychological distress (SPD).

After analyzing the results of a split-sample experiment conducted within the 2004 NSDUH, SAMHSA concluded that K6 alone was not adequate for the model-based estimation of SMI (Office of Applied Studies, 2005, Appendix B.4.4). An acceptable model would need to include an impairment measure. Consequently, a larger, nationally representative clinical interview sample was needed.

The 2008 Mental Health Surveillance Study (MHSS)

2008 Clinical Follow-up: A technical advisory group (TAG) was convened in 2006 to solicit recommendations for national mental health surveillance strategies. As a result, SAMHSA implemented the MHSS clinical data collection in 2008. As part of the MHSS, specially trained clinicians conducted follow-up interviews with a sample of NSDUH respondents by telephone to arrive at a clinical diagnosis of specific, major mental disorders, and ultimately SMI and AMI (any mental disorder). The NSDUH main interview adult sample was partitioned into two half samples to test alternative impairment measures: the abbreviated WHODAS and SDS (Sheehan Disability Scale). MHSS clinical subsamples were selected from both half-samples and yielded approximately 750 respondents for each impairment measure. An initial objective of the 2008 clinical sample was to select one of the two tested impairment measures for inclusion in future years’ surveys. Because the investigation led to selection of a model with a WHODAS scale rather than an SDS scale, only models with WHODAS predictors are discussed in the remainder of this report, and the MHSS data is limited to the WHODAS subsample.

The NSDUH data collection methodology allows respondents to select an English version or a Spanish version of the main questionnaire. Respondents completing the main NSDUH interview in Spanish were excluded from sample eligibility for the MHSS clinical sample in 2008 and all succeeding years.

The weights used in the 2008 modeling process were developed using fairly simple methods because there were so few cases available for a more complex approach. The weights were the product of four factors: (1) the respondent’s NSDUH weight, (2) the inverse of the probability that the respondent was selected for the clinical sample, (3) a nonresponse adjustment, and (4) a poststratification adjustment that forced the population estimates computed from the clinical sample for some demographic groups to match population totals derived from Census Bureau data. Separate weights were also developed for each half sample of both the main NSDUH survey and the clinical sample. A weighted logistic regression model was fit on clinical diagnostic data collected from the 2008 WHODAS subsample (n = 750). The dependent variable for the model was a diagnosis of SMI (1 = yes, 0 = no; based on a clinical diagnostic interview) and the predictor variables were the psychological distress (K6) score and function impairment measure (abbreviated WHODAS) based on items collected in the NSDUH. The model was used to produce a predicted probability of having SMI for each clinical interview respondent. A cut point was established among the fitted probabilities of having SMI such that if adults with probabilities at or above the cut point were predicted to have SMI and the rest were not, the weighted number of false positives would come as close as possible to equaling the weighted number of false negatives. If these weighted numbers were exactly equal, then the estimated proportion of adults predicted to have SMI would be the same as the estimated proportion actually diagnosed to have SMI.

Since the predictor variables in the model were variables collected on the NSDUH main interview, a probability of having SMI could be predicted for every NSDUH adult respondent using the estimated model parameters. Then using the cut point from the clinical sample, NSDUH respondents were classified as having or not having SMI. The SMI classification variables for NSDUH respondents then were used to compute prevalence estimates of SMI for adults.

The probabilities of having SMI from the 2008 MHSS regression model also were used to make estimates of AMI prevalence for NSDUH respondents. A second cut point was determined so that if adults with probabilities at or above the cut point were predicted to have AMI and the rest were not, the weighted total of false AMI positives and false AMI negatives in the clinical sample would come as close as possible to being equal.

The 2008 Model: The following is the 2008 model for the estimated probability an adult has SMI ($\widehat{\pi }$) :

where the X_k and X_w refer to K6 and WHODAS terms, respectively, and are defined as follows:

• X_k = Alternative Past Year K6 Score: Past year K6 score less than 8 recoded as 0; past year K6 score 8 to 24 recoded as 1 to 17.
• X_w = Alternative WHODAS Score: WHODAS item scores less than 2 recoded as 0; WHODAS item scores 2 to 3 recoded as 1, then summed for a score ranging from 0 to 8.

The 2008 formula for the predicted probability of SMI (SMIPP) can then be expressed using the model parameter estimates above as follows:

If SMIPP was greater than or equal to 0.26971946 (SMI cut point), then the respondent was predicted as having past year SMI. If SMIPP was greater than or equal to 0.02400 (AMI cut point), then the respondent was predicted as having past year AMI. For more information on the 2008 model and 2008 clinical interview collection procedures, see Aldworth and colleagues (2010) or Liao and colleagues (2012).

Mental Illness Estimates from 2009 to 2011: Although SAMHSA continued to conduct clinical interviews after 2008, estimates of SMI and AMI from the 2009, 2010, and 2011 NSDUHs have been based on the 2008 model and parameter estimates developed from the 2008 clinical sample. This approach provided comparability for assessing SMI and AMI trends. Producing a new model each year based on the small annual clinical samples (only 500 interviews in 2009 and 2010) would have resulted in large changes in the model parameters and corresponding prevalence rates due purely to sampling error, making it nearly impossible to detect real trends in SMI and AMI over time. Furthermore, an evaluation of the 2008 model, using 2009 clinical data, found that the model parameter estimates were similar even when refit with the additional 500-case 2009 clinical sample.

The 2012 Analysis

With financial support from the National Institute of Mental Health (NIMH), SAMHSA expanded the clinical interview sample in 2011 and 2012. By the end of 2012, approximately 5,500 clinical interviews had been collected.

SAMHSA conducted two methodological studies using the MHSS clinical data accumulated since 2008. The first developed improved clinical sample weighting procedures; the second assessed a variety of alternative SMI prediction models and estimation approaches.

The New Clinical Sample Weights: An improved weighting scheme was developed that addressed the potential coverage bias from not including adults who responded to the main NSDUH interview in Spanish and the nonresponse biases specific to the clinical interview sample.

There were three conceptual improvements that better accounted for coverage and nonresponse bias in the weight adjustment process. An initial coverage adjustment forced the sum of the adjusted NSDUH weights among adult Hispanics (which were set to zero for Hispanics not completing the survey in English) to equal the sum of the original NSDUH weights among all adult Hispanics. The sum of the adjusted and original NSDUH weights were also equalized among Hispanic within categories found to be correlated with completing (or not completing) the survey in English, such as age group, education level, and years in the United States.

The second conceptual improvement of the new weighting scheme was to break down nonresponse adjustment into two stages: (1) initial nonresponse, caused by failing to agree to a clinical follow-up, and (2) final nonresponse, caused by not completing the clinical follow-up after having agreed to it earlier. About 15 percent of the selected sample (unweighted) did not agree to a clinical interview. An additional 20 percent agreed to but did not participate in the clinical interview. Table 1 shows a few mental health-related differences between respondents and the two types of nonrespondents to the clinical interview with 2008–2011 NSDUH clinical interview data.

Table 1

Estimated Prevalence Rates among Respondents, Initial Refusals, and Final Refusals for Selected Mental Health and Demographic Variables: 2008–2011 NSDUH Clinical Interview Sample.

In the new weighting scheme, the weights first were adjusted for initial nonresponse using variables found to be correlated with initial refusal. Then, the weights were adjusted for final nonresponse using variables found to be correlated with final nonresponse. Paralleling the coverage adjustment, the sum of the adjusted weights among respondents after each stage of nonresponse adjustment was forced to equal the sum of the weight before adjustment among both respondents and nonrespondents.

The third conceptual improvement in the weight adjustment process was a poststratification that forced equality between the weighed means among clinical and adult main NSDUH survey respondents for variables correlated with having SMI. These included sex/age group, race/ethnicity, alternative K6 score, and alternative WHODAS score.

The variance of an estimate that combines data from samples drawn independently across multiple years is minimized when the relative scaling factor attached to each year is proportional to the inverse of the variance of the year’s individual estimate. Because the clinical sample weights were to be used for computing several direct estimates as well as for modeling, there was not a single set of ideal scaling factors. The scaling factors adopted were based on the estimated variances of annual estimates for SMI, AMI, and past year MDE. The scaling factors were 0.12 for 2008, 0.04 for 2009, 0.14 for 2010, 0.35 for 2011, and 0.35 for 2012.

As a result of these adjustments, the final annual weights for a respondent to the clinical interview was the product of seven factors: the respondent’s main NSDUH weight, a coverage adjustment to account for respondents to the main study who completed that survey in Spanish, the inverse of the probability the respondent was selected for the clinical subsample, an initial nonresponse adjustment, a final nonresponse adjustment, a poststratification adjustment, and the scaling factor. These first six factors were computed independently for each year.

The final weights were used to analyze the combined MHSS clinical samples from 2008 to 2012 and examine models and estimation strategies.

Choosing Variables for the Model: Three criteria were used to determine which variables should be included in the model. These were model parsimony, reduction in the overall misclassification rate of mental illness among all adults, and the reduction of significant biases in the estimates of mental illness within subpopulations for which mental illness prevalences are estimated (for a listing of these subpopulations, see Table 4). Only a limited number of predictor variables could reasonably be included in a logistic model for SMI because the clinical data being analyzed derived from a complex survey having only 100 effective degrees of freedom (100 variance strata with two variance replicates each). A useful rule of thumb for analyzing this survey data is that there should be no more than 10 (i.e., ≈ √100) predictor variables in a statistical model.

Table 4

Comparisons of Direct Estimates for SMI with Cut Point Estimates for SMI Based on the 2008 Model and the 2012 Model, by Subpopulation.

Direct indicators of mental illness, such as depression and suicidal thoughts, were investigated for possible inclusion in the model if adding them reduced the overall misclassification rate (the sum of the false positive and false negative rates).

Demographic variables used to define subpopulation for which mental illness prevalences are estimated—such as sex, race, and employment—were not included in the initial model investigation. Adding a demographic variable would tend to fix the relationship between SMI and the subpopulation defined by the particular demographic variable over time. For example, if being employed resulted in an estimated 2 percent decrease in the odds of having SMI when all the other things were being equal, then treating that 2 percent decrease as fixed over time would impede the measurement of any changes in the relationship between SMI and employment. As a result, demographic variables were only considered for the model if their inclusion produced significant decreases in bias for estimates of mental illness with subpopulations for which mental illness prevalences are estimated.

One indirect measure of mental illness was used to define a subpopulation for which mental illness prevalences are estimated: receipt of mental health service. It was also not considered for the model. Because having received services for mental health is highly correlated with SMI, adding service receipt as a predictor variable in a model for SMI would produce a cut point that overpredicts SMI prevalence among adults having received mental health services. That is, resulting estimates of SMI by service use would be biased within that subpopulation.

The revised SMI models were evaluated using two criteria: the overall error rate and subpopulation level bias. The (overall) error rate is the sum of the estimated fraction of false positives and false negatives in the adult population. Models with lower error rates produce more accurate predictions of SMI and AMI than models with higher error rates.

Different ways of recoding and collapsing the K6 and WHODAS variables were evaluated but none led to a meaningful reduction in the error rate when compared to the way in which K6 and WHODAS scores were recoded for use in the 2008 model. By adding two additional NSDUH variables—serious thoughts of suicide in the past year and the experience of MDE in the past year to the model—a meaningful decrease in the error rate was observed (Table 2).

Table 2

Summary Statistics for Cut Point Estimates Computed with Different Models.

An estimator for SMI (or AMI) based on a model has the potential of being systematically biased overall or for subpopulations. The cut point for SMI in 2008 was determined so that the estimated proportion of false positives (adults predicted to have SMI but who did not have SMI diagnosis) and false negatives (adults predicted not to have SMI but who did have SMI diagnosis) in the clinical sample were as close to equal as possible. This property removed the possibility of systematic bias in the estimated proportion of adults in 2008 having SMI using a cut point estimator based on the 2008 model.¹

Unfortunately, using the 2008 SMI cut point among all adults did not ensure the near equality of estimated false positives and false negatives among subpopulations for which mental illness estimates are produced. As a result, the cut point estimator for certain subpopulations for which mental illness prevalences are estimated could be biased.

In order to determine whether or not estimates of SMI by subpopulations were biased, a bias measure was developed. A reasonable bias measure for a subpopulation level estimate is the difference between the weighted proportions across the clinical sample within the subpopulation of respondents predicted to have SMI and those actually diagnosed to have SMI (this is equal to the difference between the false positive rate and the false negative rate in the subpopulation). Under the null hypothesis that there is no bias in the subpopulation, this bias measure would not be significantly different from zero.²

The 2008 model was refit with data from 2008 to 2012 using the revised (and scaled) weights. The bias measure and its test statistic were then applied to age group subpopulations (Table 2). Results of the bias tests indicated that SMI prevalence estimates were significantly biased for certain age groups. This is illustrated in Table 3. The prevalence estimate computed directly from the MHSS clinical sample was slightly larger than the standard cut point estimate for all adults (3.93% as opposed to 3.89%). The standard cut point estimate for the age group of 18 to 25 year olds, however, was much higher (5.77%) than the direct estimate (3.77%) for this age group. Moreover, the measure of the bias of this estimate was significantly different from zero at the .01 level (p-value = .009).

Table 3

The Cut Point Estimator for SMI by Age Group under the Refit 2008 Model.

This finding suggested that adding an age predictor to the model for SMI might reduce the bias in estimates of SMI among these age groups, although care would be necessary to avoid creating new domain-level biases by the addition (as can happen when a cut point methodology is used). Various recoded age variables were added to the model, and the resulting bias and error rates for both SMI and AMI were examined. Based on this investigation, an adjusted age variable was added to the 2012 model: AGE1830, defined as either 12 or the difference between the respondent’s age and 18, whichever was smaller. The variable increased as the respondent aged from 18 to 30 but then leveled off at 12.

Our final investigation concerned the possibility of adding the year of data collection to the 2012 model for SMI either as a single continuous variable or as five categorical variables (2008, 2009, …, 2012). Neither addition was statistically significant at the .1 level, which supported treating the 5 years of data as a single time-invariant data set from which SMI in future years could be made.

Improvements Due to Revisions in the 2012 Model: Table 2 displays some summary statistics for SMI cut point estimates computed with 2008 through 2012 data but with (1) the refitted 2008 model (i.e., the 2008 model was refitted on the combined 2008–2012 clinical sample) and (2) the 2012 model. It also displays summary statistics for AMI cut point estimates derived from those models. Results indicate that the error rates are lower for the 2012 model compared to the 2008 model. The 2012 model incorporates the NSDUH respondent’s age (using the variable AGE1830) and indicators of past year suicide thoughts, and depression, along with the variables that were specified in the 2008 model (i.e., the variants of the K6 and WHODAS scores).

Specifically, the following is the 2012 model for the estimated probability an adult has SMI ($\widehat{\pi }$):

where the X_k, X_w, X_s, X_m, and X_a terms are defined as follows:

• X_k = Alternative Past Year K6 Score: Past year K6 score less than 8 recoded as 0; past year K6 score 8 to 24 recoded as 1 to 17.
• X_w = Alternative WHODAS Score: WHODAS item scores less than 2 recoded as 0; WHODAS item scores 2 to 3 recoded as 1, then summed for a score ranging from 0 to 8.
• X_s = Serious Thoughts of Suicide in the Past Year: Coded as 1 if yes; coded as 0 otherwise.
• X_m = Past Year MDE on the NSDUH: Coded as 1 if the criteria for past year MDE were met; coded as 0 otherwise.
• X_a = AGE1830: Coded as age minus 18 if aged 18 to 30; coded as 12 otherwise.

The 2012 formula for the predicted probability of SMI (SMIPP) can then be expressed using the model parameter estimates as follows:

If SMIPP was greater than or equal to 0.260573529 (SMI cut point), then the respondent was predicted as having past year SMI. If SMIPP was great than or equal to 0.0192519810 (AMI cut point), then the respondent was predicted as having past year AMI.

Tables 4 and 5 display the differences in prevalence estimates for the time period 2008–2012 using estimates generated from the 2012 model and from the original 2008 model. Here, the NSDUH weights are not scaled (i.e., data from each year contributes equally to the analysis), when computing the cut point estimates (the clinical sample weighted remain scaled for the direct estimates). It can be observed that adding the age variable to the 2012 model appears not only to have reduced the bias in the age-based subpopulations but in other subpopulations for which mental illness prevalences are estimated as well.

Table 5

Comparisons of Direct Estimates for AMI with Cut Point Estimates for AMI Based on the 2008 Model and the 2012 Model, by Subpopulations.

One subpopulation, the South region, had a significantly biased AMI estimate when applying the 2012 model. That is not wholly unexpected under the null hypothesis of no subpopulation-level biases, given the number of tests conducted (64). However, both the direct estimate for the South, computed from the MHSS clinical sample only, and the cut point estimate, based on the 2012 model, were below the national average.

Resulting Action

As a result of this research, SAMHSA took the following actions:

• A revised model, the 2012 model—including the additional predictors of past year suicidal thoughts, past year MDE, and an adjusted age—was adopted for 2012 NSDUH estimates of SMI and AMI.
• For comparability in estimating recent trends, the 2008 to 2011 SMI and AMI estimates were updated using the 2012 model and estimation procedures.