Measuring Lexical Diversity in Narrative Discourse of People With Aphasia

Abstract

The cardinal deficit in aphasia is anomia, which is difficulty retrieving a word during discourse or in structured tasks (Goodglass & Wingfield, 1997). For this reason, addressing word-finding deficits has attracted considerable attention in aphasiology. One of the micro-linguistic content analyses available for assessing lexical semantics in people with aphasia (PWA) is lexical diversity (LD). LD can be defined as “the range of vocabulary deployed in a text by a speaker that reflects his/her capacity to access and retrieve target words from a relatively intact knowledge base (i.e., lexicon) for the construction of higher linguistic units” (Fergadiotis & Wright, 2011, p. 1,415).

LD has been used in several applications in aphasiology. For example, it has been used to differentiate PWA from neurologically intact adults (Holmes & Singh, 1996), to measure selected aspects of semispontaneous discourse in PWA in order to capture clinically relevant aspects of noun and verb production (Lind, Kristoffersen, Moen, & Simonsen, 2009), and to examine whether and to what extent LD differs across adults with fluent and nonfluent aphasia (Wright, Silverman, & Newhoff, 2003). LD has also been used as an external criterion to investigate the concurrent validity of the story retell procedure (Doyle et al., 1998), which was designed to elicit language samples in PWA (McNeil et al., 2007). Besides clinical applications, LD has been used as a key variable for theory testing and development. Gordon (2008) studied the productive vocabulary of individuals with fluent and nonfluent aphasia in the context of the “division of labor” hypothesis (Gordon & Dell, 2003). Crepaldi et al. (2011) used LD to assess the predictions of neuropsychological models that may give rise to the characteristic differential noun–verb impairment in aphasia. Finally, LD has been used to investigate the efficacy and generalization to discourse of semantic feature analysis (Rider, Wright, Marshall, & Page, 2008).

LD has also been used in other areas of speech-language pathology, such as to track language development in children with cochlear implants (Ertmer, Strong, & Sadagopan, 2002), to study and diagnose specific language impairment (Owen & Leonard, 2002; Thordardottir & Namazi, 2007), and to differentiate bilingual children with and without specific language impairments (Kapantzoglou, 2012; Klee, Gavin & Stokes, 2007). Despite its widespread use in speech-language pathology, both in the field of aphasiology and other fields, identifying a robust measure to capture LD has been very challenging (Malvern, Richards, Chipere, & Durán, 2004). In this study, we collected and examined validity evidence for four computational tools that have been proposed for measuring the LD of a language sample, and we evaluate their appropriateness and applicability to aphasic discourse.

One of the most commonly used approaches to measure LD is to use the ratio of unique lexical items divided by the total number of words in a sample (type-token ratio, TTR; Chotlos, 1944; Templin, 1957) after standardizing the length of the sample.¹ TTR is inherently flawed because it varies as a function of sample length. As the sample length increases, it is less probable that a speaker will produce new words because the number of lexical items that can be activated at any given time is considered finite (Heap, 1978).

When using TTR to gauge LD, shorter samples often appear to be richer, rendering comparisons across speakers who produce language samples of different lengths problematic. Researchers have attempted to address this issue by proposing a standardized sample size, but this approach has not produced satisfactory results. A major limitation is that in order for results to be comparable across studies, researchers have to agree on the number of tokens required to estimate the TTR. In aphasiology, some researchers have proposed 300 tokens as a standard length (Brookshire & Nicholas, 1994; Prins & Bastiaanse, 2004). However, consensus on this issue is primarily low because it is not always feasible to obtain a predetermined number of tokens. Individuals with aphasia often do not produce long samples; subsequently, researchers truncate samples based on the shortest sample in the study (e.g., Gordon, 2008).

Recently, a new generation of tools for measuring LD has emerged from the field of computational linguistics. These tools have been designed to produce length-invariant estimates of LD without discarding any data. With a few exceptions, these measures have been used in limited applications in the field of speech-language pathology.

The Measure of Textual Lexical Diversity (MTLD; McCarthy, 2005) employs a sequential analysis of a sample to estimate an LD score. Conceptually, MTLD reflects the average number of words in a row for which a certain TTR is maintained. To generate a score, MTLD calculates the TTR for increasingly longer parts of the sample. Every time the TTR drops below a predetermined value, a count (called the factor count) increases by 1, and the TTR evaluations are reset. The algorithm resumes from where it had stopped, and the same process is repeated until the last token of the language sample has been added and the TTR has been estimated. Then, the total number of words in the text is divided by the total factor count. Subsequently, the whole text in the language sample is reversed and another score of MTLD is estimated. The forward and the reversed MTLD scores are averaged to provide the final MTLD estimate.

The Moving Average Type Token Ratio (MATTR; Covington, 2007; Covington & McFall, 2010) measures LD by calculating TTRs for successive nonoverlapping segments of a sample. The algorithm selects a window length of x tokens, and the TTR for tokens 1 to x is estimated. Then, the TTR is estimated for tokens 2 to (x +1), then 3 to (x + 2), and so on for the entire sample. The final score is the average of the estimated TTRs.

The D (Malvern & Richards, 1997; McKee, Malvern, & Richards, 2000) generates LD scores that conceptually reflect how fast TTR decreases in a sample. If a language sample consists of types that are being used repeatedly, TTR would decrease faster as a function of the sample size. The D performs a series of random text samplings to plot an empirical TTR versus number-of-tokens curve for a sample. Thirty-five tokens are randomly drawn from the sample without replacement, and the TTR is estimated. This process is repeated 100 times, and the average TTR for 35 tokens is estimated and plotted. The same routine is then repeated for subsamples of 36 to 50 tokens. The average TTR for each subsample of increasing token size is subsequently plotted to form the empirical curve. Then, the least squares approach is used to obtain an estimate of D that produces a theoretical curve that maximizes the fit to the empirical TTR curve. Lower D values result in steeper theoretical curves that fit the empirical curves of samples with poorer LD. The whole process is repeated three times, and the final D value is the average of the three runs.

Recently, McCarthy and Jarvis (2007) argued that D might be related to probabilities of word occurrence that can be modeled using the hypergeometric distribution (HD). The HD is a discrete probability distribution that expresses the probability of k successes after drawing n items from a finite population of size N containing m successes without replacement. For example, if a container contains m white marbles and N − m black marbles (total number of marbles = N), and drawing a white marble is defined as a success, the HD gives the probability of drawing k white marbles after n draws without replacement.

McCarthy and Jarvis (2007) used the HD to create a new measure of LD called the HD-D. The assumption underlying HD-D is that if a sample consists of many tokens of a specific word, then there is a high probability of drawing a sample that will contain at least one token of that word. McCarthy and Jarvis reported strong linear correlations between HD-D and D scores in two studies (McCarthy & Jarvis, 2007, r = .97; McCarthy & Jarvis, 2010, average r = .91 across several types of discourse evaluated in the study). Based on these findings, McCarthy and Jarvis argued that D is an approximation of HD-D expressed in a different metric. Further, they attributed the less than perfect correlations between the two measures to the main difference in the nature of the two measures—the fact that D is based on random sampling and curve fitting, which introduces error in the estimation process, as opposed to HD-D, which is directly estimated based on probabilities of word occurrence in a language sample.

A feature of HD-D is that it does not require a minimum of 50 tokens to be estimated. By default, D is required to estimate the average TTR for 50-token subsamples in order to establish the empirical curve that is modeled. If there are <50 tokens in the sample, the program terminates without providing a score for the specific sample. This is problematic for researchers who work with PWA (and other clinical populations), who often produce limited verbal output. The reason is twofold. First, language samples with <50 tokens that are discarded may lead to a loss of valuable information. Typically, we reach more robust conclusions about a client’s language skills the more data are available. Second, from a missing data theory perspective, only when data are missing at random or completely at random (both terms introduced by Rubin in 1976) is the missing mechanism ignorable. Conversely, if the data are missing not at random, and this fact is ignored and the data are analyzed, statistical parameter estimates may include substantial bias that may lead to invalid inferences (Enders, 2010; Graham, 2009; Little & Rubin, 2002; Rubin, 1976).

To illustrate this point, Figure 1 shows two probability density functions for a standardized variable X (e.g., height). The first curve corresponds to a simulated complete data set of 1,000 observations, and M₁ is its mean. When data are missing not at random, the probability that data are missing depends on the unseen scores themselves. The second curve corresponds to such a truncated data set for which data are missing not at random and missingness is related to an observation’s value: Values <−1 are not observed systematically. This scenario could occur for instance if our sample consisted of U.S. fighter pilots who have a minimum standing height requirement of 64 in. In this case, M₂ would be a biased estimator of the mean height of the general population. Similarly, if LD scores are not missing in a haphazard fashion when estimated with D, but instead are missing in a systematic way, parameter estimates may also be biased. Further, even though Figure 1 demonstrates the effect of systematic missingness on a mean, the same logic applies to the estimation of any statistical model parameter (e.g., variances, covariances, factor loadings, regression coefficients). Finally, notice that if the data points were missing at random (e.g., some from the lower end of the distribution and some from the higher end of the distribution with equal probability), the estimated mean would not have been biased despite the missing values.

Open in a separate window

Figure 1

Two probability density functions for (a) a complete data set and (b) the same data set for which data are missing not at random from the lower end of the distribution. M₂ is a biased estimate of the population mean.

Method and Results

Relationship of D and HD-D and the Impact of Missing Data

Statistical approach: Curve fitting and analysis of variance (ANOVA)

To determine whether D and HD-D can be used interchangeably to measure LD, we analyzed these scores using the R statistical language (R Development Core Team, 2011) and SPSS 20. The nonlinear least square fit function (Bates & Watts, 1988) into the “stats” package of R was used to fit a linear and an exponential curve. Model fit was based on visual and algebraic information. To determine if the minimum requirement of 50 tokens to estimate D will lead to biased estimates, we conducted a one-way ANOVA in SPSS 20.

Preliminary analysis

Language samples from 101 PWA were included in the study. Data were prepared for statistical analysis following Kline (2010) and Tabachnick and Fidell (2007). Descriptive statistics for the number of types, tokens, and TTRs, as well as the estimated LD indices, are provided in Table 2. After being imported into SPSS, the data were screened for missing values. All variables had complete data except for D, for which ~27% of the data were missing due to an insufficient number of tokens in the samples. Distributions were visually inspected and assessed in terms of the normality assumption; skewness and kurtosis statistics were estimated. Several distributions were noted to be skewed and with various degrees of kurtosis. For this reason, the maximum likelihood ratio (MLR) estimator was used, which estimates parameters using maximum likelihood with standard errors and a χ² test statistic that are robust to nonnormality.

Table 2

Descriptive statistics of the major study variables.

Variable	N	M	SD	Range	Skewness	Kurtosis
Types	101	42.44	21.15	11–108	−0.80	−0.48
Tokens	101	83.08	53.15	17–273	−1.17	−2.85
TTR	101	00.55	00.11	.27–.80	−0.01	−0.09
D	74	31.55	14.88	7.20–90.28	−1.17	−2.85
HD-D	101	−7.78	04.46	−23.83–.76	−0.94	−1.22
MTLD	101	25.11	09.81	10.45–65.89	−1.32	−2.96
MATTR	101	0.76	00.08	.49–.92	−0.56	−0.77

Open in a separate window

Note. TTR = type-token ratio; D (McKee, Malvern, & Richards, 2000); HD-D = Hypergeometric Distribution D (McCarthy & Jarvis, 2007); MTLD = Measure of Textual Lexical Diversity (McCarthy, 2005); MATTR = Moving-Average Type-Token Ratio (Covington, 2007).

Relationship between D and HD-D: Results

To investigate the relationship between D and HD-D, SPSS 20 and R were used to fit a linear and an exponential curve to the data. First, a linear regression analysis was performed to predict D from HD-D for the participants with complete data. The linear regression equation was significant, R² = .85, F(1, 72) = 362.69, p < .001. Then, an exponential model was fit to the same data. The exponential equation was also significant, R² = .99, F(1, 72) = 15972.94, p < .001. However, the difference in variance explained using the exponential model was substantial (ΔR² = .14), which suggests that the exponential model better captured the relationship between D and HD-D than the linear model. The two models were further compared using the residual sum of squares, the residual standard error, and the standard errors of the model parameters. Based on the algebraic information, the exponential model demonstrated considerably better fit (see Table 3). Finally, the data were plotted along with the linear and exponential curves suggested by the models, and fit was evaluated visually (Figure 2). Overall, the exponential model demonstrated better fit compared to the linear model. Substantively, the excellent fit of the exponential model to the bivariate data (graphically and in terms of the R²) provides strong evidence that HD-D and D are essentially isomorphic; that is, using the exponential function, one can estimate D scores based on observed HD-D with excellent accuracy.

Open in a separate window

Figure 2

Bivariate scatter plot of the scores generated by vocD and HD-D with fitted linear and exponential curves. Solid dots represent scores that were imputed using the exponential model.

Table 3

Model summaries and parameter estimates.

Model	R	R2	F	RSS	Residual SE	a (SE)	b (SE)
Linear	0.92*	0.85	392.69	2504	5.9	56.87 (1.45)	3.82 (0.19)
Exponential	0.99*	0.99	15972.94	129.1	1.34	75.08 (0.59)	0.14 (<0.01)

Open in a separate window

Note. RSS = residual sum of squares; SE = standard error.

^*p < .01.

voc-D and missing data: Results

The curve estimation analysis was followed up by an ANOVA in SPSS to explore whether the missing D data were missing not at random. Using the exponential function, D scores were computed based on HD-D for the language samples with <50 tokens, for which D in CLAN was not able to produce estimates (solid data points in Figure 2). Henceforth, voc-D generated scores and imputed D scores based on HD-D will be referred to as D_VOC and D_IMP, respectively, and the combined variable D_VOC + D_IMP will be referred to as D_COM. Also, a binary missingness variable was created that denoted whether D scores were voc-D generated or were missing and estimated from HD-D. To assess the assumption of homogeneity of variance, Levene’s test of equality of error variances was performed, and it was not statistically significant, F(1, 99) = .62, p = .43. A between-subjects one-way ANOVA was conducted with D_COM as the dependent variable and missingness (voc-D values present or not) as the independent variable. There was a significant effect of missingness, F(1, 99) = 14.45, p = <.001, η² = .13, Cohen’s d = .88.

The box plot in Figure 3 shows graphic summaries for D_VOC, D_IMP, and D_COM. Results suggested that the probability of D scores missing was related to the values of D; that is, the missing D scores were more likely to be lower than the scores that were estimated with voc-D. Figure 3 shows the impact of this finding in the current data set. If the number of tokens in a sample did not limit calculating D, the estimate of the mean would have been approximately equal to 28.24 (D_COM). However, the estimation process was more likely to fail for language samples that had lower LD. As a result, values from the lower end of the distribution were eliminated systematically, and the mean estimate of D in our sample using the voc-D program alone was biased upward (D_VOC = 31.55).

Open in a separate window

Figure 3

Boxplots of D scores generated with vocD, imputed with HD-D, and combined. The width of the plots is proportional to the square root of the samples sizes. The black line in each box indicates the median for each set of scores. If the notches of two plots do not overlap, this is “strong evidence” that the two medians differ (Chambers, Cleveland, Kleiner, & Tukey, 1983).

Investigating Construct Validity

After addressing the first two questions, we wanted to determine whether all of the techniques (i.e., MTLD, MATTR, D, HD-D) measure the same construct and whether there was evidence consistent with the presence of systematic residual covariance that would indicate length effects. Because the data generated by voc-D were not missing at random (and would therefore bias the model parameter estimates), we imputed D values for the missing data points from HD-D scores and the exponential function used earlier.

Statistical approach: Confirmatory factor analysis (CFA)

The LD of a sample was conceptualized as an unobserved latent variable, and its relationship with four observed variables (MTLD, MATTR, D, and TTR) was modeled using CFA in Mplus 6. Two models that reflected competing hypotheses were specified a priori; they were evaluated in terms of global fit and localized areas of strain and were compared directly using a χ² difference test. Following Bollen (1989), the magnitude of the standardized loadings and error covariances from the best fitting model were used to compare the relative influence of the factor on the manifest variables and to answer the substantive questions of the paper.

The first model assumed a single latent common factor ζ that was interpreted as the mathematical instantiation of LD in a sample. The loadings, λ_MTLD, λ_MATTR, λ_D, and λ_TTR indicated how strongly the latent variable influenced the observed variables, or, alternatively, how strongly a score generated by a given technique reflected the LD of a sample. Finally, the model stipulated that once the effect of the common factor was taken into account, there was no systematic covariance among the residual terms of the observed indicators. Given that TTR is influenced by sample length, the lack of residual term covariances reflected the hypothesis that MTLD, MATTR, and D did not share TTR’s susceptibility to length effects. In other words, TTR was used not in spite of but because of its known length dependency to help us uncover similar behavior in other indices. The second model was identical to the first except that the error covariance between the TTR and D error terms was allowed to be freely estimated. This specification was consistent with the findings of Owen and Leonard (2002) and Fergadiotis (2011), who suggested that D might be influenced by length. Therefore, allowing the error covariance to be estimated modeled the specific hypothesis that D, similar to TTR, is influenced by a construct unrelated to LD, most possibly, length.

Investigating construct validity: Results

The models were estimated in Mplus 6.1 using the MLR estimator. Four fit indices were taken into account to examine global model fit. Fit indices included the Satorra-Bentler scaled χ² statistic (Satorra & Bentler, 1994) to take into account the nonnormality of the data, the comparative fit index (CFI; Bentler, 1990), the root-mean square error of approximation (RMSEA; Steiger & Lind, 1980), and the standard root mean residual (SRMR; Hu & Bentler, 1998). A good fitting model was expected to have a nonsignificant χ² at the .05 level; a CFI value >.95; an RMSEA value <.08, with the upper bound of the 90% confidence interval <.10; and an SRMR value <.08 (Brown, 2006; Hu & Bentler, 1999; Kline, 2010; Steiger, 2007). To assess for local strains in the solution, modification indices and normalized residuals were considered. Two hypotheses using nested model comparisons were tested followed by a substantive evaluation of model parameters. To perform the nested model comparisons, the scaled difference χ² test statistic (Satorra & Bentler, 2001) was used.

First, a unidimensional CFA model with four indicators (MTLD, MATTR, D_COM, TTR) was fit to the data (see Model A in Figure 4). The covariances among the residual terms of the indicators were fixed to 0. To identify the model, the variance of the latent variable was set equal to 1. The model converged to a solution with no out-of-range parameter values for which the fit indices provided mixed evidence of adequate model fit: χ²(2, N = 101) = 17.74, p < .001; CFI = .93; RMSEA = .30 (90% confidence bands = .19 and .42); and SRMR = .04. The largest normalized residual was associated with the covariance of TTR and D_COM (−1.23). Also, based on the modification indices, allowing the residual variances of TTR and D_COM to covary would improve the model fit significantly (approximate Δχ² = 14.44; expected parameter change = −.45).

Open in a separate window

Figure 4

Two alternative confirmatory factor analysis models of the lexical diversity scores based on language samples from 101 people with aphasia. Completely standardized robust maximum likelihood parameter estimates. For each observed variable, the variance accounted for by the common factor, R² = (1 − residual variance) = λ². For all parameter estimates, p < .001, except for MTLD’s residual variance (p = .004).

In the next step, a model with a single factor defined by the four indicators (MTLD, MATTR, D_COM, TTR) was fit to the same data (see Model B in Figure 4). The model was identical to Model A with one exception: The TTR and D_COM residual terms were allowed to covary. The model converged to a solution for which all fit indices suggested excellent fit to the data, χ²(1, N = 101) =.38, p = .54; CFI = 1.00; RMSEA = .00 (90% confidence bands = .00 and .22); and SRMR = .007. Similarly, no local model strain was noted (highest normalized residual = −.2; no modification indices with values >3.84). Also, none of the standardized parameter estimates took on out-of-range values, and all of the estimates were statistically significant. The two models were further compared using the scaled difference χ² test statistic to explore whether fixing the TTR and D_COM covariance to 0 had a statistically significant impact on global fit. Based on the results, the null hypothesis that Model A did not fit significantly worse than Model B was rejected, Δχ²(1) = 23.02, p < .001. Figure 4 shows the parameter estimates for both models (Mplus output is available upon request). Based on the results, Model B demonstrated excellent global fit and a lack of localized misfit and statistically outperformed Model A.

Discussion

The main purpose of this paper was to collect validity evidence regarding techniques for measuring LD for the study of aphasic discourse. The specific aims were to investigate the relationship between D and HD-D; explore whether using D may lead to biased estimates when studying aphasic discourse; determine whether MTLD, MATTR, and D measure the same latent variable and to what extent; and examine whether there is evidence of length effects for the aforementioned LD indices. In a series of analyses, HD-D and D were found to correlate highly using an exponential function. Also, statistically significant mean differences were uncovered between scores that were estimated using vocD and the scores for which the vocD algorithm failed to produce scores due to inadequate number of tokens in the samples. Finally, a unidimensional CFA model of MTLD, MATTR, D, and TTR that allowed for the residual terms of TTR and D to correlate freely exhibited excellent fit to the data. To the best of our knowledge, this was the first time that missing data theory was employed to study the performance of the D index. Further, it was the first time that CFA was used to study the validity of the MTLD, MATTR, and D score interpretations.

Acknowledgments

Footnotes

References