Chapter  52:  Criteria for Determining Disability in Speech-Language Disorders: Evidence Report/Technology Assessment Number 52

Search Process and Inclusion Criteria

The task of synthesizing the available evidence on all speech and language evaluation instruments was clearly too large an undertaking to complete within the scope of this project. Thus, EPC staff had to select and prioritize instruments in such a way as to address the critical informational needs of the SSA while also limiting the scope to fall within the contractual boundaries of the project. To do this, we assembled a panel of 10 national experts, our Technical Expert Advisory Group (TEAG). They, along with Agency for Healthcare Research and Quality (AHRQ) and SSA staff, identified 19 instruments for literature review and evidence analysis-three each for adult language, adult speech, child speech, and voice, and eight for child language disorders. One speech instrument can be used with both adults and children and thus was counted twice. We later excluded one instrument because it was not a single instrument but instead was an approach to conducting more comprehensive clinical analysis of phonological patterns for which standard "diagnostic test characteristics" would be hard to determine.

The RTI-UNC EPC review team conducted detailed searches of the relevant English-language literature from 1966 (or the initiation of the specific electronic database) to October 2000 using the MEDLINE^®, CINAHL, PsycLIT^®, ERIC, Health and Psychosocial Instruments (HAPI), and Cochrane Collaboration databases. We initially excluded all gray literature. After reviewing abstracts for eligibility, however, we recognized that, for many instruments, data on reliability and validity could be found only in the instrument manuals. Thus, we expanded efforts to include instrument manuals in the review. We also examined reference lists of all included articles and instrument manuals to identify additional studies.

The EPC team applied a series of inclusion and exclusion criteria to the literature searches. Essentially, we included all English-language research on the selected instruments in children and adults (ages 18 through 62) in which the study evaluated the instrument's reliability, validity, or ability to predict future communicative impairment and/or functioning (i.e., predictive validity). Articles reporting the efficacy or effectiveness of speech or language therapy that did not provide information relevant to the key questions were excluded. Because of the need to address issues facing the SSA in establishing disability criteria in the United States, we excluded articles providing normative data from populations other than the United States.

The EPC team selected studies for inclusion from among 1,238 citations using a process of duplicate but independent review of titles, abstracts, and, where necessary, full papers. Discussion leading to consensus was used to resolve disagreements. The number of citations reviewed ranged from three, for the Dysarthria Examination Battery (DEB) and Voice Handicap Index (VHI), to 256, for the Test of Language Development (TOLD).

The team abstracted data, using single abstraction with subsequent review by clinical and methodological experts, from 92 articles whose abstracts met inclusion criteria. Two reviewers with expertise in quantitative psychology and experience in the validation and standardization of educational tests abstracted the data. During the data abstraction phase, we eliminated 53 articles because they did not meet inclusion criteria or did not address the version of the instrument selected by TEAG members.

The EPC study director and clinical experts completed a quality rating for each article and manual. The quality rating scales evaluated research design and conduct, measurement of reliability and validity, development of instrument norms, justifications for conclusions, and external validity concerns. Six additional items evaluated aspects of instrument development or revision for the instrument manuals.

The team compiled the data into a series of five evidence tables for each instrument. The first of these tables provides information on the study design and conduct and the quality scores assigned by the methodologist and the expert clinicians. The subsequent four tables describe the reliability, validity, predictive validity for future communicative functioning, and available normative data found in the reviewed articles and manuals.

Subsequently, the team graded the evidence summarized in the tables, assessing whether the evidence met thresholds for acceptable reliability, validity, and availability of normative data. Where relevant, we used classic criteria for clinical decisionmaking about individuals, not groups of subjects. The criteria employed were:

• Reliability -- the criterion for reliability is "strictly" met if the following three conditions are all met:

  • Internal consistency reliability, measured using either Cronbach's coefficient alpha or Kuder-Richardson statistics (K-R 20), is greater than or equal to 0.90;

  • Test-retest/intra-rater reliability is greater than or equal to 0.90 if measured using a correlation coefficient, or greater than or equal to 0.80 if measured using Cohen's Kappa; and

  • Inter-rater reliability is greater than or equal to 0.90 if measured using a correlation coefficient, or greater than 0.80 if measured using Cohen's Kappa.

• Validity -- the criterion for validity is met if the following conditions are all met:

  • Instrument developers examine relationships between subtests, composite scores, and total scores, establishing hypothesis a priori for these relationships and for patterns of scores for individuals belonging to various groups of import;

  • These relationships are all statistically significant at p < 0.05; and

  • In the case of correlation coefficients, the magnitude of the relationship is at least 0.30, thus providing evidence of a moderate correlation.

• Normative Data -- the criterion for normative data is met if the following conditions are all met:

  • Data are available for the population targeted by the instrument;

  • An adequate sample size is used (i.e., at least 100 per group); and

  • Evidence is provided on how well the sample represents the population.

Some might reasonably argue that we set the criterion for internal consistency reliability too high given the complexity of speech and language functioning and disorders. Additionally the variability in daily performance that arises from these different speech and language disorders suggests that our criterion for test-retest reliability or intra-rater reliability was also set too high. Thus, we defined a "relaxed" criterion, which differs from the strict criterion in that internal consistency reliability may be as low as 0.80 and/or test-retest/intra-rater reliability may be as low as 0.80 (correlation) or 0.70 (Cohen's Kappa). The relaxed criterion is at a level suitable for having confidence in group, rather than individual comparisons.

After grading the psychometric properties of the individual instruments, we graded the strength of the overall body of evidence for groups of instruments identified by age group and disorder. We graded instrument manuals and peer-reviewed literature separately employing the following definitions for both.

• Acceptable: research or analyses were well conducted, had representative samples of reasonable size, and met our psychometric evaluation criteria discussed earlier.

• Unacceptable: studies were poorly conducted, used small or nonrepresentative samples, or had results that did not meet or only partially met the psychometric criteria.

Reliability, Validity, and Availability of Normative Data

The EPC team evaluated the strength of evidence describing the reliability, validity, and availability of normative data separately for instruments assessing adult language, child language, adult speech, child speech, and voice disorders.

Epidemiology and Costs of Speech and Language Disorders

According to the National Institute on Deafness and Other Communication Disorders (NIDCD), approximately 42 million people (1 in 6) in the United States have some type of communication disorder.² Of these, 28 million have communication disorders associated with hearing loss, and 14 million have disorders of speech, voice, and/or language not associated with hearing loss.

The personal and societal costs of these disorders are high. On a personal level, such disorders may affect nearly every aspect of daily life. Estimates of annual societal costs in the United States range from $30 billion³ to $154 billion⁴ in lost productivity, special education, and medical care.

Speech Disorders

A speech disorder is a disorder affecting the articulation of speech sounds, the fluency with which speech is produced, or the quality of the voice. Articulation disorders (also called phonological disorders) include motor speech disorders and functional articulation disorders. Motor speech disorders result from damage to the central or peripheral nervous system. Damage may occur as the result of strokes, traumatic brain injury, or neurogenic diseases including Parkinson's disease, Huntington's disease, and amyotrophic lateral sclerosis; among children, the problems can arise from any of a range of prenatal, perinatal, and postnatal conditions, particularly those resulting in cerebral palsy.

Functional articulation disorders are those that either have no known cause or result from causes other than known neurological insults or physical abnormalities. In the majority of cases in children, articulation disorders fall into this category.⁵ They may stem from problems with the motoric component of speech production or from an internal representation of the phonological rule system of the target language that is immature or disordered. Among preschool and school-age children, articulation disorders are the most prevalent communication disorders, affecting approximately 10 percent of the population, and they are of sufficient severity to require treatment in 8 percent of the population. Among children with articulation disorders, 50 percent to 70 percent exhibit academic difficulties throughout the primary and secondary grades, reflecting at least in part the demonstrated relationship between early phonological disorders and later reading, writing, spelling, and mathematical achievement.

Long-term consequences can persist throughout the lifespan. Studies of adults who were diagnosed and treated for articulation disorders as children have revealed continuing difficulties in processing linguistic information, even though they seldom continued to show overt difficulties with speech sound production. These individuals are less likely to attend college and more likely to hold jobs that involve unskilled labor than their peers without a history of phonological disorder.⁵

Fluency disorders, also referred to as stuttering, involve an interruption in the flow of speaking manifested as an atypical rate, rhythm, repetitions in sounds, syllables, words, and phrases, or some combination of these. Secondary symptoms can include excessive tension, struggle behaviors, and odd behavioral mannerisms.³ Approximately 1 percent of the population (more than 3 million Americans) exhibits a fluency disorder that has persisted beyond 6 years of age.⁶ Children who stutter have a poorer educational adjustment and lower achievement than their peers who do not stutter. The disorder likely is vocationally handicapping as well, given the negative stereotypes of people who stutter and the fact that employers believe that stuttering decreases employability.⁶

Voice disorders are characterized by abnormal pitch, loudness, resonance, quality, or duration of voice, or by an inability to use one's voice, or some combinations of these factors. These disorders result from abnormal laryngeal, respiratory, or vocal tract functioning. They may be caused by habits of vocal misuse and hyperfunction (e.g., repeated clearing of one's throat, or prolonged talking over background noise) that produce physical changes in the vocal folds, by medical conditions (e.g., trauma, neurological disorders, allergies, or cancer), by psychological disorders (e.g., stress or personality disorders), or by a combination of these factors.⁷

Between 3 percent and 9 percent of the population of the United States has a voice disorder. Of the total working population in the United States, approximately 25 percent have jobs that critically require voice use.³ The majority of individuals diagnosed with voice disorders report that their voice problems have negatively affected past, current, and future job performance. Individuals in certain vocations and avocations, including teachers, singers, actors, cheerleaders, and aerobic exercise instructors, are particularly susceptible to voice disorders.⁷

Language Disorders

A language disorder is the impaired comprehension and/or use of spoken, written, and/or other symbol systems used for communication. Between 6 and 8 million individuals in the United States have some form of language impairment.³ Approximately 1 million of these are adults with aphasia, an acquired impairment of language comprehension and/or expression caused by brain damage, usually secondary to strokes. A large proportion of the 2 million adults with progressive dementing diseases (e.g., Alzheimer's disease, Parkinson's disease) have significant language impairments. In addition, language impairments persist among adults who failed to develop normal language skills because of developmental or acquired disorders in childhood (e.g., specific language impairment, autistic disorder, hearing impairment). Approximately 8 percent to 12 percent of preschool children have some form of language impairment.

Specific language impairment (SLI) is defined as a significant deficit in language functioning that is not accompanied by any deficits in hearing, intelligence, or motor functioning that would explain the language deficit. Developmental language disorders tend to concentrate within families,^8,9 and genetic factors have been implicated.¹⁰ The overall prevalence of SLI among kindergarten students is estimated at approximately 7 percent.¹¹

Persistence of language impairment across time is more likely among children initially diagnosed with both receptive and expressive language impairments (65 percent to 100 percent with persisting disorders, as reported across studies) than among children initially diagnosed with expressive language impairments only (0 percent to 54 percent with persisting problems, as reported across studies). Children identified with language disorders as preschoolers are at great risk for learning disabilities at school age, and the vast majority of children identified at school age as learning disabled have concomitant language disorders. One study reported a prevalence rate of 90.5 percent for language disorders among 242 children between 8 and 12 years of age who had learning disabilities.¹²

The rate of comorbidity of psychiatric and communication disorders in children is high, particularly for children diagnosed with attention-deficit/hyperactivity disorder (ADHD).^13-15 The comorbidity of these two types of disorders is frequently unsuspected. Cohen and colleagues found that of 399 children referred for psychiatric outpatient treatment, 25 percent had language impairments that were previously unsuspected, almost equal to the number with previously identified language impairments.¹⁶ The children with previously unsuspected language impairments had the most severe externalizing behavior problems (problems with adverse effects on property or other people), compared to children with identified language disorders or those without language disorders.

Technical Issues Related to Speech and Language Disorders

First, speech and language disorders are apparent across the lifespan, but in many cases the manifestations of the disorders vary at different ages or stages of development. These changes require clinicians and others to apply different evaluation procedures appropriate to the assessment of the disorders at different developmental and chronological stages.

Second, speech and language disorders are diverse in terms of both affected functions and etiology. With respect to affected functions, for instance, broad categories include disorders of speech sound production, voice, fluency, and language, as described above. Within the broad category of language, interrelated subcomponents including semantics (language meaning and vocabulary), syntax (phrase and sentence structure), and pragmatics (appropriate use of language in context) may be differentially affected.

Finally, language comprehension and production abilities will not necessarily be commensurate in a single individual. In addition, communication disorders will differentially affect abilities in different communication and language modalities, such as gesture or sign language, spoken language, and written language.

For etiology, clinicians distinguish between developmental and acquired disorders of speech and language. Developmental communication disorders are those apparent from early on in a child's life, such that speech or language (or both) do not develop as expected. Although these developmental disorders first appear in childhood, impaired communication affects many individuals with these disorders throughout their lives.

Acquired disorders are those that affect an individual (either child or adult) who already has an intact speech and language system or who was progressing normally in the development of such a system. These disorders usually result from neurological damage following such events as strokes or closed head injury, although they also can arise from other types of disease or events (e.g., laryngeal cancer, accidents affecting oral motor structures).

Other factors contribute to the complexity of evaluating speech and language disorders as well. First, speech and language impairments often coexist with and are related to other problems, such as hearing loss, mental retardation, ADHD, autism, cerebral palsy, Parkinson's disease, dementia, hemiplegia, and many others. These coexisting conditions often require that clinicians adapt standard speech and language evaluation procedures. In some cases, speech and language evaluation instruments have been developed specifically for individuals with particular coexisting disorders; a case in point is the Rhode Island Test of Language Structure developed for children with hearing impairments by Engen and Engen.¹⁹

Another major factor, which is increasingly important with the changing demographics in the United States, is that individuals being evaluated for speech and language disorders may speak a nonstandard dialect of English, or they may not be from an English-speaking cultural environment. For example, 24.4 percent of children in kindergarten to grade 3 had limited English proficiency in 1990.²⁰ An estimated 6.2 million culturally and linguistically diverse Americans representing different minority groups in the United States have a communication disorder. To determine the presence, nature, and severity of a speech and language disorder, these individuals must undergo an evaluation that is appropriate for both their language and their culture. However, the task of identifying valid and reliable speech and language assessment instruments and procedures for people from different linguistic and cultural environments is a challenging one for clinicians.

Measurement Issues

Assessment and diagnosis of individuals with suspected speech and language disorders is a process that involves posing a series of clinical questions and choosing the appropriate procedures and instruments to help answer those questions. Rarely, if ever, will a single assessment procedure or instrument be sufficient to ascertain a diagnosis, severity level, prognosis, and treatment recommendations for a speech or language disorder.

Posing clinical questions and choosing assessment procedures and instruments require clinical expertise and often will call for multidisciplinary assessments to clarify the nature of the speech or language disorder, its impact on functioning, and the long-term prognosis for the patient. Findings on standardized measures must be interpreted in light of other important clinical information. For example, two patients who have suffered from traumatic brain injuries may show similar performance profiles on a language measure, but their long-range prognoses could be very different depending on the variable such as the elapsed time interval since injury, the presence or absence of a seizure disorder, or the possible impact of medication on language functioning. Ultimately at issue is the reliability and validity of the entire clinical process in yielding accurate results and sound conclusions pertaining to the possible speech or language disorder.

During the assessment process, clinicians can use a wide variety of procedures and instruments. In almost every case, the assessment will involve an interview and a written questionnaire to obtain the clinical history of the patient. The interview may follow a structured format, or it may have a dynamic format that depends on the patient's responses. Standardized written questionnaires are available to aid in the evaluation of some disorders. In some cases, the patient completes the questionnaires; in others, an informant who is knowledgeable about the patient completes it.

Many standardized instruments are available to assess disorders of speech sound production and of language. In these instruments, the individual is asked to respond to a series of questions or tasks designed to elicit particular speech or language targets or to test perception or comprehension of particular targets. The examiner scores responses against standardized criteria for accuracy or appropriateness.

Another widely used procedure is observation of the patient's communication behaviors and the physiological or neurophysiological structures that underlie those behaviors. The observation may be recorded in some way (e.g., audio- or video-recording) for later analysis. The observed behavior can be analyzed in numerous ways, depending on the nature of the suspected disorder. For instance, if a child is suspected of having a disorder of speech sound production, then a sample of the child's conversational speech might be analyzed for percentage of correct consonants or evaluated for the percentage of speech that is intelligible to an unfamiliar listener. If an individual is suspected of having a fluency disorder, his or her spontaneous speech may be rated using an observational scale such as the Iowa Scale for Rating Severity of Stuttering.²¹ For someone who seems to have a problem with language production, the sample might be analyzed for variables such as mean length of utterance, diversity of vocabulary, or use of a variety of linguistic structures.

In some cases observing the structures or functions of interest is difficult, if not impossible, without the use of special instrumentation. For example, a computer-based system might be used to determine fundamental frequency, shimmer, and jitter for the voice of an individual who is being evaluated for a possible voice disorder. For the same individual, videostroboscopy might be used to observe the structure and function of the vocal folds. For an individual with a language disorder, structural or functional brain imaging (or both) may be a valuable tool for clarifying etiology, severity, and prognosis.

Measurement strategies and instruments will vary depending on the levels at which the disability is being evaluated. The recent terminology drafted by the World Health Organization (WHO) uses "impairment" to refer to problems in body function or structure, including psychological or linguistic functioning.²² "Activity limitations" refer to difficulties an individual may have in executing activities, and "participation restrictions" refer to problems an individual may experience in involvement in life situations. Traditionally, the focus of measurement has been the level of impairment of speech or language functions.

Contemporary practice, however, emphasizes assessment at the level of activity limitations and participation restrictions associated with speech and language impairments. Some standardized instruments have been developed to measure disability at these levels. For example, the Assessment of Intelligibility of Dysarthric Speech by Yorkston and Beukelman²³ and the Functional Assessment of Communication Skills for Adults²⁴ are measures that largely address speech or language disabilities at the level of activity limitations, whereas the Voice Handicap Index^25,26 measures aspects of impairment, activity limitations, and participation restrictions.

The range of disorders, ages, and cultural or linguistic factors described above has prompted the development of a vast array of assessment instruments for speech, voice, and language. One source reviewing commercially available assessment instruments includes more than 140 tools in its most recent edition.²⁷ Given the multiplicity of tools and the complexity of factors involved, the lack of a clear understanding of which speech and language evaluation procedures yield reliable and valid results is not surprising. Assessing individuals from English-speaking cultural environments who have no coexisting disorders is difficult; assessing individuals who have coexisting disorders or who come from non-English-speaking cultural environments is even more challenging.

Nevertheless, the quality of the speech and language evaluation procedures is of paramount importance for sound clinical decisionmaking and robust policies related to services for individuals with speech and language disorders. This report will clarify the existing knowledge base for a representative selection of speech, language, and voice assessment instruments used with children and adults, and it also will point the directions for future research.

Key Questions

Final key questions are as follows:

1. What evaluation procedures for child and adult speech (voice, articulation/intelligibility, fluency) and language disorders have been demonstrated to have the salient characteristics of a good diagnostic tool (e.g., reliability, validity, appropriate normative data, responsiveness) for individuals who are/have:

   • English-speaking, have normal hearing, with or without normal cognition?

   • Non-English-speaking, have normal hearing, with or without normal cognition?

   • Mentally retarded?

   • Learning disorders?

   • Hearing impaired?

2. Are there evaluation procedures that have been demonstrated to have predictive validity for the individual's communicative impairment, performance, or both?

Causal Pathway

Figure 1. Causal Pathway for Determining Disability (more...)

   Figure 1. Causal Pathway for Determining Disability in Speech and Language Disorders

Figure 1

Our evidence report begins with the speech-language evaluation and stops short of the SSA determination of disability per se (which may involve consideration of other impairments coexisting with the speech and/or language impairment). We focus on speech and language evaluation procedures and prediction of outcomes based on these procedures. In particular, we have evaluated the following:

• Reliability (internal consistency, test-retest, inter-rater, and intra-rater);

• Validity (content, construct, concurrent [convergent and divergent/discriminant], and predictive for future communicative functioning);

• Clinical test parameters (sensitivity, specificity, positive predictive value, negative predictive value), where available;

• Responsiveness; and

• Availability of normative data for the evaluation instruments.

In the causal pathway, an individual with suspected speech or language disorder is evaluated using selected evaluation procedures. Clinicians today use many different evaluation procedures to evaluate speech and language disorders in adults and children. Decisions regarding which procedure(s) to employ are now left to the discretion of the health care professional(s) conducting the evaluation.

The choice of evaluation instrument(s) will be determined by five sets of factors:

1. Whether the individual is a child or an adult;

2. The type of disorder (speech sound production, voice, stuttering, or language -- including consideration of semantics, syntax, pragmatics, comprehension, production, and language modality);

3. The presence of other impairments (e.g., cognitive impairments, hearing loss, physical disability); and

4. Whether the individual is from a non-English-speaking cultural environment;

5. Contextual factors, including examiner training and expertise, and setting for evaluation and/or treatment.

Characteristics of these instruments are critical to a successful disability determination process. Evaluation instruments must be reliable and valid for the purposes for which they are employed. If the clinical decisions based on their use are to be sound and objective, then evaluation instruments must be reliable and valid for the purposes for which they are employed. Such purposes include making a diagnosis, determining degree of impairment, and/or having predictive validity for short-term and long-term impairment and/or functional communicative outcomes.

Clinicians and technical experts synthesize information gained from the various procedures and instruments employed in the speech and language evaluation. In this process, they also may incorporate information from other evaluations not directed at the speech and language disorder per se. On the basis of all this information, they will arrive at conclusions regarding diagnosis, severity level, prognosis, and treatment recommendations.

At the final step in the causal pathway, conclusions from the speech-language evaluation are considered in light of the SSA rules for determination of disability (cited in Chapter 1). For all individuals applying for disability benefits under Title II, and for adults applying under Title XVI, disability is defined as the inability to engage in any gainful activity by reason of any medically determinable physical or mental impairment(s) that can be expected to result in death or that has lasted or can be expected to last for a continuous period of not less than 12 months. Children under the age of 18 applying under Title XVI are considered disabled if they have a medically determinable physical or mental impairment that causes marked and severe functional limitations that can be expected to cause death or that have lasted or can be expected to last for a continuous period of not less than 12 months. An individual may be considered disabled on the basis of severe speech-language impairment alone, or the individual's speech-language impairment may be considered in combination with other impairments in determining disability status.

Inclusion and Exclusion Criteria

Table 3. Criteria for Determining Disability in Speech (more...)

Table 3. Criteria for Determining Disability in Speech and Language Disorders: Study Inclusion/Exclusion Criteria

Category	Criteria
Study Population	Humans, Children, Adolescents, and Adults (ages 18 to 62)
Evaluation Tools	Adult Language: Boston Diagnostic Aphasia Examination, 2nd Edition; Porch Index of Communicative Ability; Western Aphasia Battery Child Language: Clinical Evaluation of Language Fundamentals, 3rd Edition (CELF-3); CELF-3 Spanish Edition; CELF-3 Preschool; Test of Language Development-Primary, 3rd Edition; Test of Language Development-Intermediate, 3rd Edition; Preschool Language Scale, 3rd Edition (English); Preschool Language Scale, 3rd Edition (Spanish); Test of Pragmatic Language Adult Speech: Assessment of Intelligibility; Dysarthria Examination Battery; Stuttering Severity Instrument for Children and Adults, 3rd Edition Child Speech: Goldman-Fristoe Test of Articulation, 2nd Edition; Stuttering Severity Instrument for Children and Adults, 3rd Edition; Phonological Process Analysis Voice: GRBAS (Grade, Rough, Breathy, Aesthenic, Strain) Scale; Multidimensional Voice Profile; Voice Handicap Index
Study Setting	Inpatient and outpatient settings, communities, schools
Outcomes Measured	Reliability (internal consistency, test-retest, inter- and intra-rater) Validity (content, construct, concurrent [convergent and divergent/discriminant] Predictive validity (for future communicative functioning), and normative data for particular populations (children or adults ages 18-62) Cognitive impairments (normal, borderline normal, mentally retarded, learning disabled) Hearing impairments, language, or cultural issues (English- versus non-English speaking) Exclude studies evaluating the efficacy or effectiveness of interventions/treatment of speech and/or language disorders
Time Period	Depends on when the evaluation tool(s) were developed and validated
Geographic Site of Study	Exclude based on language of publication. Normative data must be from the United States
Publication Languages	English only
Admissible Evidence (Study Design and Other Criteria)	Randomized controlled trial (RCT) -- double, single-blinded, and cross-over Non-RCT -- nonequivalent control group designs, cohort studies (prospective and retrospective), case-control studies, psychometric evaluations Other designs (meta-analysis, meta-regression, cross-design synthesis, review article for reference list searches) Sample size: >20 subjects per analysis group

We included studies of individuals older than 62 years if the majority of the study sample was age 62 and younger. We excluded studies that: (1) concerned solely elderly adults (i.e., individuals > 62 years of age); (2) were published in languages other than English; and (3) did not report information related to the key questions (except to the extent that they are used to provide background information).

We excluded articles reporting the efficacy or effectiveness of speech or language therapy that did not provide information relevant to the key questions. We also excluded articles providing normative data from populations other than the United States because of our need to address issues facing the SSA in establishing disability criteria in the United States.

Search Terms and Databases

Table D3. Search Terms Employed in the Literature Review (more...)

Table D3. Search Terms Employed in the Literature Review

MeSH Search Terms and Key Words

Exploded: Study Designs: study design, study characteristics, randomized controlled trial [publication type], single-blind method, double-blind method, random allocation, cross-over, case-control studies, retrospective cohort, longitudinal studies, outcomes Disorders: Language development disorders, language disorders, speech disorders, child language disorders, adult language disorders, voice disorders, articulation disorders, dysarthria, stuttering, aphasia, apraxias, developmental apraxia Tests:a Phonological Process Analysis, Clinical Evaluation of Language Fundamentals, Preschool Language Scale, Goldman-Fristoe, Western Aphasia Battery, Test of Pragmatic Language, Test of Language Development, Stuttering Severity Index, Assessment of Intelligibility for Dysarthric Speech, Dysarthria Examination Battery, Boston Diagnostic Aphasia Examination, Voice Handicap Index, Multidimensional Voice Profile, GRBAS, Porch Index of Communicative Ability Primary Outcomes: Predictive value of tests, sensitivity and specificity, reproducibility of results, reliability Other: Quality of life, activities of daily living, functional status, outcomes and process assessment (health care), outcomes assessment (health care), costs and cost analysis, cost-benefit analysis, epidemiologic study characteristics

a

Keyword searches were used for test names to achieve the broadest search possible. As a result, we often found nonrelevant citations because the search engine looks for words in proximity.

We employed a multifaceted approach to identify relevant studies, including use of standard electronic (literature) databases, reference lists of relevant articles, and Cochrane Collaboration resources. We searched electronic data sources, including the MEDLINE, CINAHL, PsycLIT^®, ERIC, Health and Psychosocial Instruments (HAPI), and the Cochrane Collaboration databases. The PsycLIT^® database, produced by the American Psychological Association, covers the professional and academic literature in psychology and related disciplines including medicine, psychiatry, nursing, sociology, education, pharmacology, physiology, linguistics, and other areas. ERIC, produced by the Educational Resource Information Center, indexes and abstracts journal and report literature (1966 to the present) in education and related disciplines. HAPI provides information on measurement instruments (i.e., questionnaires, interview schedules, checklists, index measures, coding schemes/manuals, rating scales, projective techniques, vignettes/scenarios, tests) in the health fields, psychosocial sciences, organizational behavior, and library and information science.

We also reviewed the reference lists of all included articles and instrument manuals to identify additional studies. Finally, we conducted searches of the Cochrane Collaboration Database, a family of electronic databases providing reference information on randomized and other controlled clinical trials and systematic reviews conducted by various Cochrane Collaborative Review Groups. These studies were reviewed for quality, and in some cases we obtained additional information from the original authors or through hand-searches of the literature.

Data Extraction Forms

The study director worked with a psychometrician to develop the two data extraction forms needed to review the peer-reviewed literature and the instrument manuals. Because the selected instruments are for both adults and children and encompass language, speech, and voice disorders, we could not develop a single extraction form that would gather all necessary data. Rather than create a form for each combination of population and disorder (i.e., one each for adult speech, child speech, adult language, child language, voice) for a total of five forms or one for each instrument (20 forms), we opted to create two modular extraction forms that we could use with all instruments and for each population and disorder combination. The modular approach allowed abstractors to use only the portions that were relevant for a specific article or manual.

The forms collect information on study design, population studied, examiner qualifications and training, results (validity, reliability, predictive validity, and normative data), and limitations. The forms also provide essential instructions to abstractors and a description of designs used in the reliability testing and validation of instruments, the appropriate statistics, and benchmarks for evaluating whether an instrument had good reliability and validity. We tested our extraction forms to be sure that the instructions were clear and that abstractors would find the forms easy to use; we made changes as needed. We present the final forms in Figures D2 and D3 in Appendix D.

Abstractors and Training

Two quantitative psychology doctoral students conducted all data abstraction because we were unable to find abstractors who had appropriate methodological skills and speech and language clinical expertise. Both abstractors had course work in quantitative methods and in both classical test and modern test theory (as well as item response theory), and both had experience in the validation and standardization of educational tests. Each article or instrument was reviewed by one abstractor only. We did not blind the abstractor to journal, authors, or institutions.

Both abstractors received training from the study director and a psychometrician. This training included a question-by-question review of the data extraction form and practice abstraction of a manual and an article. The study director and psychometrician reviewed the extraction forms, correcting any errors in the forms and making certain that the abstractors understood the task. The study director monitored progress of the abstraction process and provided feedback to the abstractors, as did the psychometrician.

Quality Control Process

Although evidence-based medicine "best practices" require that two abstractors independently review each article, we were unable to conduct dual abstraction because we could not find enough reviewers with appropriate skills. Instead, the study director, a health services researcher with expertise in quantitative methods and systematic review, reviewed all data extraction forms and evidence tables. EPC clinical experts re-reviewed evidence tables in their areas of expertise to ensure that all information was reproduced accurately. Because the abstractors and EPC staff did not complete data extraction forms independently, we could not check the inter-rater reliability of the data abstraction process, as previous reports have done.

Rating the Quality of Individual Articles and Manuals

Quality rating items previously used by the RTI-UNC EPC are tailored to randomized and nonrandomized clinical trials, designs seldom used in psychometric evaluation; furthermore, these items do not address reliability and validity testing. Thus, we sought guidance from the educational and psychological testing literature for criteria and standards to use in our quality evaluation. The American Educational Research Association (AERA), American Psychological Association, and the National Council on Measurement in Education regularly publish standards for test construction, evaluation, and documentation. The most recent version, 1999 Standards for Educational and Psychological Testing, hereafter known as the 1999 Standards, forms the basis for our quality rating scales.²⁸ Our use of the 1999 Standards is not precedent-setting. In 1984, McCauley and Swisher had adapted an earlier edition to evaluate the psychometric properties of preschool language and articulation instruments.²⁹

Figures D4 and D5 in Appendix D present the quality rating items and scoring. The individual article quality rating forms compare 35 criteria (for a maximum score of 35 points) in seven main categories. They are research design and conduct, the measurement of reliability, validity, and development of instrument norms, justifications for conclusions, and external validity concerns. The manual rating form includes six additional criteria addressing aspects of instrument development or revision; the total score for this form is 41 points. We rescaled the quality score, reporting it as a percentage. Because some items on the forms did not apply to a particular article or manual, we subtracted the points for that item from both the numerator and the denominator when calculating the percentage score.

The EPC study director and clinical experts completed a quality form for each manual and article. The clinical experts evaluated instruments in their areas of expertise; the study director evaluated all manuals and articles from a methodological standpoint. We report the quality rating scores separately for the clinical and methodological experts rather than averaging the scores across the quality reviewers. We did not use quality scores for inclusion/exclusion decisions.

Evaluating the Psychometric Properties of the Instruments

Even when we examined the psychometric literature, we found it particularly difficult to identify thresholds for reliability and validity. Although the 1999 Standards ²⁸ has reliability, validity, and normative data criteria, it does not venture to set thresholds for them.²⁸ The Scientific Advisory Committee (SAC) of the Medical Outcomes Trust cited some standards for reliability coefficients,³⁰ which generally match those found in seminal educational and psychological testing texts,^31-33 but the SAC did not provide them for validity. Anastasi³¹ and Cronbach³³ provide limited information for setting validity thresholds; we use what they presented in creating our criteria and thresholds. For normative data, we use the 1999 Standards ²⁸ and the criteria employed by McCauley and Swisher.²⁹

The criteria selected are those usually used to make individual comparisons, or in this case, decisions about individuals. It is important to minimize the possibility that a patient is misdiagnosed -- that is, to make certain that patients with true impairments are not missed and that "normal" individuals not be labeled as having an impairment. This is particularly important in the context of disability determination, where incorrect decisions can result in individuals being wrongly kept off or struck from SSA disability eligibility rolls or conversely in healthy individuals inappropriately being covered within the disability program.

From these various sources, we established the following criteria:

• Reliability -- the criterion for reliability is "strictly" met if the following three conditions are all met:

  • Internal consistency reliability, measured using either Cronbach's coefficient alpha or Kuder-Richardson statistics (K-R 20), is greater than or equal to 0.90;

  • Test-retest/intra-rater reliability is greater than or equal to 0.90 if measured using a correlation coefficient, or greater than or equal to 0.80 if measured using Cohen's Kappa; and

  • Inter-rater reliability is greater than or equal to 0.90 if measured using a correlation coefficient, or greater than 0.80 if measured using Cohen's Kappa.

Some might reasonably argue that the criterion for internal consistency reliability is set too high given the complexity of speech and language functioning and disorders. Additionally the resultant variability in daily performance suggests that our criterion for test-retest reliability or intra-rater reliability also may be too high. Thus, we defined a "relaxed" criterion, which differs from the strict criterion in that internal consistency reliability may be as low as 0.80 and/or test-retest/intra-rater reliability may be as low as 0.80 (correlations) or 0.70 (Cohen's Kappa). The relaxed criterion is at a level suitable for having confidence in group, rather than individual, comparisons.

• Validity -- the criterion for validity is met if the following conditions are all met:

  • Instrument developers examine relationships between subtests, composite scores, and total scores, establishing hypotheses a priori for these relationships and for patterns of scores for individuals belonging to various groups of import;

  • These relationships all are statistically significant at p < 0.05; and

  • In the case of correlation coefficients, the magnitude of the relationship is at least 0.30, thus providing evidence of a moderate correlation.

• Normative Data -- the criterion for normative data is met if the following conditions are all met:

  • Data are available for the population targeted by the instrument;

  • An adequate sample size is used (i.e., at least 100 per group); and

  • Evidence is provided on how well the sample represents the population.

Reliability

Reliability of measurement refers to the extent to which the observed variation in scores is due to variation in the "true" score (i.e., the "true" value of an underlying construct such as expressive language or articulation) rather than random error. Thus, the scores on an instrument, such as the TOLD-P:3 should not change unless underlying expressive or receptive language ability change. That an instrument is reliable is a necessary condition for use; an instrument that does not produce consistent results has little utility. When instrument developers measure reliability, they generally consider several types: internal consistency (or inter-item consistency), test-retest, intra-rater, and inter-rater (or inter-observer) reliability.

Validity

The validity of an instrument refers to "how well it measures what it purports to measure"(p. 83).³² When a test developer validates or examines the validity of an instrument, she or he broadly examines the relationships between an individual's performance on the instrument and particular observable facts about the behaviors studied. In the case of a patient's expressive language abilities, the instrument items that are designed to measure expressive language should indeed measure the individual's ability. The procedures for evaluating validity have been given a variety of names over the years. We employ the names given in the 1999 Standards -- content-related, construct-related, and criterion-related validation.²⁸

As we described earlier for reliability, validity should not be couched in terms of whether an instrument is "valid or not." Rather, the correct way to think about it is whether the instrument is valid for a particular purpose, population, and situation.

Key Question No. 1

Goodglass and Kaplan standardized the BDAE-2 using 242 male patients treated at the Boston Veterans Administration (VA) Medical Center between 1976 and 1982.⁴³ No additional information was provided about patient demographics. They made no claim that the standardization sample is representative of individuals with aphasia in the United States; thus, we ought to take care not to generalize the results beyond this population.

Key Question No. 1

Porch standardized the PICA using 150 adults with a diagnosis of brain injury or referral for investigation of a communication disability (Evidence Tables 5-7).⁴⁶ He described the age, race, educational attainment, and time since onset of aphasia. No information was provided with which to judge whether this sample was representative of aphasic adults in the United States.

Key Question No. 2

Lendrem and Lincoln, using data from 52 mildly to moderately aphasic adults, found that PICA verbal, gestural, and graphic components accounted for 69 percent of the total variance in 34-week scores, suggesting that early PICA scores (at 4 weeks) could be used to predict impairment at 6 months.⁴⁸ In a related study, Lincoln and McGuirk compared two methods (e.g., slope of improvement and a statistical t-test) for predicting impairment.⁴⁹ Their sample differed from that of Lendrum and Lincoln in that 68 of the 124 adults received speech-language treatment. Neither prediction method fared well. The proportions of patients with accurate predictions (i.e., predicted within plus or minus 10 percent of actual score) were low for both treated and untreated patients, suggesting that PICA scores poorly predict recovery. Both studies employed relatively small samples and provided little information to allow readers to assess whether the results could be replicated with a different group of aphasic adults.

Key Question No. 1

Shewan and Kertesz standardized the original WAB in Ontario, Canada, using a sample of 150 aphasic adults and 59 control subjects.⁵¹ The subjects included individuals with communication disorders resulting from brain injury and other causes. They provided limited demographic information (age only); therefore, we were unable to judge whether these individuals were representative of aphasic and/or "normal" adults in the United States. Kertesz did not provide formal standardization of the revised WAB; rather, he presented only a comparison between the revised and original instruments.⁵⁰

Key Question No. 2

Key Question No. 1

Semel et al. standardized the CELF-3 using 2,450 English-speaking children, adolescents, and young adults in 47 states.⁵⁷ None of the sample was receiving language therapy or had a diagnosed or identified language disorder. The sample was representative of US children with respect to race, sex, residence (urban or rural), family income, educational attainment of parents, and geographic region

Key Question No. 2

Key Question No. 1

Semel et al. standardized the CELF-3Sp using 1,050 Spanish-speaking children, adolescents, and young adults from 19 states (Evidence Tables 18-21).⁶⁸ Individuals were included only if they were fluent in Spanish. Like the CELF-3 standardization sample, none had hearing deficits or diagnosed language disorders. The investigators documented that the sample was representative of the US Hispanic population with respect to educational attainment of parents and geographic region.

Key Question No. 1

Wiig et al. standardized the CELF-P using 800 English-speaking children ages 3-0 through 6-11 years (i.e., children ages 3 years, 0 months to 6 years, 11 months), none of whom was receiving language therapy or had a diagnosed or identified language disorder (Evidence Tables 22 through 25).⁶¹ They documented that the sample was representative of US children with respect to race, sex, mother's educational attainment, and geographic region

Key Question No. 1

Newcomer and Hammill standardized the TOLD-P:3 using 1,000 English-speaking children, ages 4-0 through 8-11 years, in 28 states (Evidence Tables 26-30).⁶⁹ Children with learning disabilities, speech-language disorders and delays, mental retardation, and other handicaps were purposively included to be able to provide reliability and validity information for these groups. They documented that the sample was representative with respect to race, sex, residence (urban or rural), family income, educational attainment of parents, and geographic region.

Key Question No. 2

We found no literature describing the predictive validity of the TOLD-P:3. Lewis et al. assessed the predictive validity of preschool TOLD-P:2 scores for school-age reading, language, and spelling skills in children identified with moderate to severe speech sound disorders as preschoolers.⁷² Preschool TOLD-P:2 scores predicted language scores, as measured by the CELF-R, and spelling scores.

They also evaluated the ability of the TOLD-P:2 Word Discrimination subtest and Semantic and Syntactic composite scores to predicting school-age impairments in language, reading, and spelling. The Word Discrimination subtest predicted spelling disorders; the Semantic composite predicted reading; and Syntax composite predicted all three types of disorders.

Key Question No. 1

Hammill and Newcomer standardized the TOLD-I:3 with 779 English-speaking children, ages 8-0 through 12-11 years, in 23 states (Evidence Tables 31-34).⁷¹ Children with learning disabilities, speech-language disorders, mental retardation, hearing impairments, and other handicaps were purposively included to be able to provide reliability and validity information for these groups. Additionally, they documented that the sample was representative with respect to race, sex, residence (urban or rural), family income, educational attainment of parents, and geographic region.

Key Question No. 1

Zimmerman et al. standardized the PLS-3 using 1,200 English-speaking children, ages 2 weeks through 6-11 years, in 40 states and the District of Columbia (Evidence Tables 35-38).⁷⁴ They excluded children previously identified as language disordered or receiving language remediation services following the diagnosis of a language disorder and those who had difficulties at birth, including those who did not go home from the hospital with the mother, who had a hospital stay of more than 1 week, or a significant birth or genetic defect. They documented that the standardization sample closely matched the US population with respect to geographic region, sex, mother's educational attainment, and race or ethnic group.

Key Question No. 1

Zimmerman et al. standardized the PLS-3Sp using 181 Spanish-speaking children ages 0-0 to 6-11 years from six states and Puerto Rico (Evidence Tables 39-42).⁸⁴ Although the children spoke a variety of dialects, the authors indicated that the sample was not representative of Hispanic children in the United States.

Key Question No. 1

Phelps-Terasaki and Phelps-Gunn standardized the TOPL using a sample of 1,096 children from 24 states and one Canadian province (Evidence Tables 44-46).⁸⁵ They documented that it was representative with respect to gender, residence (urban or rural), race, geographic region, and ethnicity of US children ages 5 through 13.

Key Question No. 1

Yorkston and Beukelman standardized the AIDS using a small sample (n = 12) of dysarthric speakers (Evidence Tables 47-49); they did not provide demographic information about the subjects.²³ We identified no evidence on internal consistency, construct validity, concurrent validity or predictive validity (Key Question No. 2).

Key Question No. 1

Drummond standardized the DEB using 20 men and women who were either "mildly" dysarthric or normal (nondysarthric and not brain-damaged) and treated in a university medical center clinic (Evidence Tables 51-53).⁸⁸ Additionally, she used data from 34 or 38 individuals described in three other studies to assess reliability and validity (determining the actual number of individuals with the information given in the other three studies is difficult). No information was provided with which to assess the generalizability of results to other dysarthric individuals.

Key Question No. 1

Riley standardized the SSI-3 using 271 children and adults (ages 2 to 17+) from California (Evidence Tables 54-57).⁸⁹ Subjects under age 8 had no prior treatment for stuttering whereas most of those older than 8 had received treatment. He made no attempt to assess whether the sample was representative of US children and adults who stutter. Thus, the results he reported may not be generalizable to individuals with characteristics different from the standardization sample.

Key Question No. 1

Goldman and Fristoe standardized the GFTA-2 using 2,350 children and adolescents ages 2 through 21 years (Evidence Tables 58-62).⁹² This sample closely matches the US population with respect to sex, race or ethnic group, geographic region, and mother's education. Additionally, they included representative proportions of students enrolled in special education under different categories of eligibility.

Key Question No. 2

Key Question No. 1

We were unable to obtain a copy of the original GRBAS Scale and associated documentation. Hirano provided the first English-language description of the scale but provided no information on either reliability or validity.⁹⁸

Key Question No. 1

Key Question No. 1

Benninger et al and Jacobson et al. standardized the VHI using a population of patients attending a voice clinic in a major hospital.^{25, 26} Although they provided detailed demographic and clinical information about their populations, they made no claim as to whether their populations were representative of US voice patients (Evidence Tables 70-72).

Adult Language Disorder Instruments

User manuals varied widely in how well they met our usability criteria. The PICA manual was particularly user-friendly, meeting all eight criteria. Both reviewers agreed that the WAB manual met the least number of criteria of all instruments examined, not just those for adult language. This result is not surprising given the brevity of this manual (seven pages inclusive of references). The BDAE-2 manual ratings were intermediate; both reviewers agreed that information on examiner requirements and training (Criteria 3 and 4) was insufficient for the expected user. The two reviewers disagreed as to whether the derived aphasia classifications and their interpretations were well explained.

Child Language Disorder Instruments

Our reviewers rated the child language instrument manuals as quite user-friendly. In general, these manuals met the majority of the criteria (ranging from 6.6 to 8). Our raters scored the two Preschool Language Scale-3 manuals in similar ways, disagreeing on whether they described examiner qualifications and training (Criteria 3 and 4) in sufficient detail. In a similar vein, the raters disagreed on the examiner qualification criterion for the CELF-P.

Adult Speech Disorder Instruments

The AIDS manual met all eight criteria as evaluated by both reviewers. The SSI-3 met three of the criteria completely (both reviewers agreed) and two partially (i.e., one reviewer rated that the criterion had been met, the other did not). The reviewer who indicated that the SSI-3 manual did not meet Criterion 1 (description of procedures allows duplication of administration) and Criterion 2 (description allows duplication of scoring procedures) noted that procedures in the manual allowed the examiner to interrupt the stutterer to simulate normal conversation; her concern was that this instruction might be interpreted differently by each examiner and during each administration. Thus, the scores obtained might vary. Both reviewers indicated that the SSI-3 manual lacked information on examiner qualifications and training and on environmental and equipment requirements (Criteria 3, 4, and 5) for instrument administration. The DEB fared only slightly better than the SSI-3. Our reviewers disagreed in their assessments for half of the criteria. Both agreed that the administration procedures and scoring instructions could be easily followed and that examiner training had been adequately described. However, one reviewer was very concerned about the fact that many of the measures are subjective and might lead "different examiners to assign ratings differently based purely on their subjection impressions."

Child Speech Disorder Instruments

The GFTA-2 manual met all eight criteria as evaluated by both reviewers. The SSI-3 met three criteria completely (both reviewers agreed) and two partially (i.e., one reviewer rated that the criterion had been met, the other did not). As above, the rater who indicated that the SSI-3 manual did not meet Criteria 1 and 2 was concerned about examiner interruptions. Both raters indicated that the SSI-3 manual lacked information on examiner qualifications and training (Criteria 3 and 4) and on environmental and equipment requirements for instrument administration (Criterion 5).

Voice Disorder Instruments

Our usability reviewers scored only the MDVP instrument manual for the eight criteria. We were unable to rate the usability for the GRBAS Scale because we could not obtain an English-language manual. No manual was available for the VHI; all data on development and evaluation of reliability and validity were reported in the peer-reviewed literature.

The MDVP manual met three criteria (1, 4, and 5) as assessed by both reviewers. Our reviewers disagreed about Criteria 2 and 3. The reviewer who indicated that Criterion 2 was not met noted that although the manual does not report standardized scores, many different authorities claim various standard scores for the parameters tested by the MDVP system. The other reviewer made no comments to shed light on her rating of Criterion 3.

Evaluation Criteria and Summary Results

Tabular Summaries

Table 24 summarizes the available reliability and validity data and normative information for the 18 instruments that we reviewed and indicates whether the psychometric evaluation criteria are met. We document whether each instrument (listed in alphabetical order within disorder) met either the strict or the relaxed criterion for reliability; the next two sets of columns indicate with an X whether the instrument met construct and concurrent validity criteria and whether normative data are available and demonstrably representative of the US population. Finally, the right-most columns show (with an X) whether the instrument can comfortably be said to be applicable to several special populations. (In all cases, a blank means the instrument does not meet the relevant evaluation variable.)

Table 25. Strength of Available Evidence for Instruments, (more...)

Table 25. Strength of Available Evidence for Instruments, by Key Questiona

Instrument	Key Question No. 1	Key Question No. 2
Adult Language
BDAE-2	Unacceptable
PICA	Unacceptable	Unacceptable
WAB	Unacceptable (original only)
Child Language
CELF-3	Unacceptable	Acceptable (earlier version)
CELF-P	Unacceptable
CELF-3Sp	Acceptable
PLS-3	Acceptable (except for children 0-8 months old)
PLS-3Sp	Unacceptable
TOLD-P:3	Acceptable	Acceptable (earlier version)
TOLD-I:3	Acceptable
TOPL	Unacceptable
Adult Speech
AIDS	Unacceptable
DEB	Unacceptable
SSI-3	Unacceptable
Child Speech
GFTA-2	Unacceptable	Acceptable (earlier version)
SSI-3	Unacceptable
Voice
GRBAS	Unacceptable
MDVP	Acceptable
VHI	Acceptable

a

Blank cells indicate that we found no data on predictive validity and thus could not evaluate Key Question No. 2.

Table 25 summarizes for each instrument the strength of available evidence for the two key questions. We document whether the available evidence is acceptable or unacceptable. In all cases, a blank means that no evidence exists to address the key question. The following paragraphs discuss how the instruments measure up against our psychometric evaluation and evidence strength criteria.

Adult Language Disorder Instruments

The two PICA studies provide limited but contradictory evidence of its ability to predict future impairment at 6 months. According to Lendrem and Lincoln,⁴⁸ the PICA can do so, but according to a later study by Lincoln and McGuirk,⁴⁹ it cannot. In both studies, the samples were small and derived from investigations conducted for another purpose; the investigators presented no information on whether the results could be generalized to typical adult aphasics. Further, the studies employed different methods (which were generally poorly documented). Given this picture, we cannot conclude that evidence supports predictive validity for this category of instruments.

Child Language Disorder Instruments

We identified limited predictive validity evidence related to the revised version of the CELF (CELF-R) and the second edition of the TOLD-P (TOLD-P:2); none was found for the versions of these instruments selected for this evidence review. Kotsopoulos et al. reported data on the ability of the CELF-R instrument to predict reading, spelling, and math performance in children with severe behavioral and psychiatric disorders who were attending a day treatment program.⁵⁹ Although the initial CELF-R performance significantly predicted the children's gains in grade-level scores for reading and math (but not spelling) across the academic year, we caution, for several reasons, that users should not conclude that CELF-R scores predict future impairment. In particular, the relatively small sample comprised children with psychiatric and behavioral disorders, so one cannot evaluate the generalizability of the results to populations beyond those having characteristics similar to the sample.

In 2000, Lewis et al. evaluated the predictive validity of the TOLD-P:2 for school-age language, reading, and spelling skills among 87 children who as preschoolers had been identified with moderate to severe speech sound disorders.⁷² School-age (elementary school) follow-up data were available for 52 of the original sample. Preschool TOLD-P:2 scores were a significant predictor of school-age language and reading skills but not spelling skills. This research group also examined the discriminative power of one subtest and several composites to predict school-age impairments in language, reading, and spelling. For the TOLD-P:2 Semantic composite score, sensitivity was 0.69 and specificity was 0.76 for discriminating between children with and without reading disorders at school age, but this version of the instrument did not discriminate between children with and without language or spelling disorders at school age. The TOLD-P:2 Syntax composite score was able to discriminate between children with and without all three types of disorders at school age. Although Lewis and colleagues conducted their analyses carefully, information remains insufficient to evaluate the generalizability of their results. Thus, we urge caution in concluding that this study supports predictive validity.

Child Speech Disorder Instruments

Although we found no literature describing the predictive validity of the GFTA-2 (or the SSI-3), Lewis et al. also provided some evidence for the original version of the GFTA.⁷² Preschool scores on the original GFTA significantly predicted school-age reading but not language or spelling skills. Although the sample was relatively small, the research group carefully documented that differential attrition did not occur, but they did not provide sufficient information with which to evaluate the generalizability of their results. Thus, we would urge caution in concluding that this study supports the predictive validity of the GFTA (or GFTA-2).

Summary

Limitations in the studies, including sample size and generalizability, as well as insufficient documentation of methods, weaken our ability to conclude that any of the instruments examined in this evidence report have adequate evidence of predictive validity for either communicative impairment or functioning. Of the four instruments for which we identified evidence (of any sort) of predictive validity, only the TOLD-P:3 had demonstrated "acceptable" evidence of reliability and validity (Key Question No. 1). Unfortunately, evidence to support predictive validity derived from the previous version of the instrument, the TOLD-P:2. Thus, complete and acceptable evidence on both key questions is not available for any instrument that we reviewed. Only for the TOLD-P:3 does the mixed evidence begin to reach this level of acceptability.

Key Questions and Instruments

Our results and conclusions in this evidence report are, of course, heavily contingent on the structure and focus of the key questions. The original questions (posed as part of the government's request for proposal) were those judged within the SSA to be of high priority to them in assisting their later development of criteria for determining disability in individuals with speech and/or language disorders. We tailored and revised the questions to some extent in early discussions with SSA and AHRQ, to make them compatible with the nature of a systematic evidence review (one without formal recommendations) and manageable within the time and budget constraints of this work. In the end we could devise no completely satisfactory resolution to the SSA's need for a broad canvass of the field; the agency needed a review that was not restricted by patient group or disorder type, and we needed to avoid an unmanageable number of citations that would have been generated by reviewing all relevant assessment instruments.

Given the priority of including both adults and children and three major disorders (speech, language, and voice) while keeping the review within reasonable limits, we engaged in a systematic instrument selection process. Our aim was to optimize scientific and clinical appreciation of issues in diagnosing and caring for patients with these conditions and the likelihood of finding information in an evidence-based approach, while taking account of SSA's time frame and our overall budget. Within these constraints, a national panel of experts selected the 18 instruments through a formal, iterative process at a meeting early in the project. They based their selections on (a) judgments as to the extent to which instruments are already known and used in this field and assess a range of ages and types of speech and language performance and (b) knowledge or beliefs that peer-reviewed information might well exist for the instruments in question.

Thus, these instruments do not cover the universe of instruments in this field or for a population from 0 to 62 years of age; neither do they address all relevant aspects of speech or language disorders in adults and children. Furthermore, they were selected by one group of experts at a particular time. Although we have no special reason to question the selections, we are cognizant of the fact that other expert panels (or this one at some other point in time) might identify a different set of instruments to accord high priority for review.

We emphasize these points because we wish to caution readers not to assume that these particular instruments were necessarily regarded, a priori, as the "best" or "most comprehensive" for evaluating speech language disorders in children or adults. Nonetheless, we are confident that they do represent an appropriate, reasonable, timely, broad-based selection of instruments by which SSA could gain a rigorous view of the state of the science in detecting and predicting the likely outcomes from speech and language disorders.

Generic Deficiencies in this Body of Literature

As is clear from Chapter 3 and the summary above, the peer-reviewed literature rarely yielded data on the reliability and validity for the majority of selected instruments. In comparison with many other clinical fields for which the published evidence base is extensive (and of high quality), even for screening and diagnostic tests, the peer-reviewed knowledge base about ways to identify speech and language disorders and their potential for causing disability is very small. The peer-reviewed literature is the hallmark of systematic evidence reviews, so this lack of information posed a considerable limitation for our work.

Consequently, we elected to expand our efforts to review and abstract data from instrument manuals. Instrument manuals provided reliability and validity data in varying degrees of comprehensiveness. Some very thorough manuals provided reliability and validity information for children by sex, age group, race or ethnicity, presence of speech or language disorder (or both), and similar factors. In some cases, manuals presented data on all aspects of reliability, providing information on internal consistency, test-retest or intra-rater reliability, and inter-rater reliability, and on construct and concurrent or criterion validity. Others provided information on only some aspects of either reliability or validity.

Given this heterogeneity even in non-peer-reviewed publications and to supplement the variable data in instrument manuals, we reviewed, if available, reliability and validity studies in the peer-reviewed literature for the version immediately before the selected version. For example, for child language disorders, we examined peer-reviewed literature for the revised version of the CELF,⁶² the original version of the GFTA,⁹⁴ and the Stuttering Severity Instrument for Children and Adults, Revised.⁸⁹ This tactic has its own limitations. First, in some cases the publications are very old (20 to 30 years in the past). Second, content may well have changed in important ways across the various versions of the instruments. Thus, although such early data may be indicative of reliability and validity of later versions, we cannot at this stage confirm that these data truly and adequately apply to today's versions of the instruments.

Finally, we note study design, conduct, and documentation difficulties in this literature. In a substantial number of reviewed studies, authors provided little information on the sample studied (e.g., demographic characteristics, information on type and severity of speech, language, or voice impairment) or on how they had recruited subjects from the larger population. Disorder-specific samples were often small, and sometimes investigators combined individuals with different types of disorders for analysis purposes without prior analyses to demonstrate that the groups did not differ from each other in any material way. Often, the research teams poorly documented their statistical methods; occasionally, they did not specify statistical approaches at all. For the most part, they did not correct p-values when making multiple statistical comparisons. Statistics not in common use today were sometimes employed, but the publications lacked reference to statistical texts or original articles; hence, we could not determine whether the choice of statistics was appropriate for the past state of knowledge and computing resources.

Limitations Specific to the Reviewed Instruments

Inherent Quality of Speech and Language Instruments

Publication and Documentation

In the spirit of evidence-based practice, we note that publishing the information called for above in peer-reviewed journals is also critical. As noted in Chapters 2 and 4 of this report, we relaxed what might be regarded as the commonly followed standards of systematic reviews by including substantial amounts of gray literature in the form of users' manuals and instruction guides. That is, we learned in doing this evidence report that data on the reliability and validity of speech and language assessment instruments are rarely found in the peer-reviewed literature; such information is largely confined to the instrument manuals, which do not undergo a peer-review process and cannot really be considered acceptable venues for documenting these properties of the instruments. Thus, we suggest that journal editors in fields concerned with speech and language disorders encourage the submission of reports on instrument reliability and validity, identify peer reviewers who are qualified to evaluate the rigor of these types of reports, and then publish such data in their journals.

Normative Data and Samples

Instruments developed to measure speech and language disorders in adults have used samples of individuals with disorders as the normative sample. Unquestionably, finding the numbers of individuals with disorders needed for the development of psychometrically sound assessment instruments is challenging indeed. Nonetheless, developers report almost no evidence regarding the representativeness of their samples used in standardizing these instruments. This issue needs attention in future research.

A related issue is the importance of developing normative data for subgroups and avoiding inappropriate aggregation of subgroups defined by different disease processes, comorbidities, or prognoses. For example, in the standardization of adult language instruments, individuals with traumatic brain injury and those with cerebrovascular accidents have been grouped together. Patterns of cognitive-linguistic functioning following these two types of neurological insults are different, however, and clinicians' ability to interpret assessment results is impeded when normative databases combine data on these two subgroups.

An issue with several measures of child speech and language development reviewed for this report pertains to the systematic exclusion of children with disabilities from the normative sample, resulting in a lack of representativeness of the samples and possible inflation of the numbers of children who would be identified with speech or language disorders using these measures. In short, the representativeness of samples used in instrument development and standardization is an area for attention in future research across the age span and various disorders comprising the focus of this report.

Costs, Benefits, and Harms

Assessing the feasibility of use of these instruments in clinical settings was beyond the purview of this evidence report. Consequently, we did not formally evaluate the cost or the burden on patients, clinicians, or administrators of actually applying these instruments. (Chapter 3 and its instrument-specific tables describes the basic steps for using the instruments and comments on usability.) Information on costs and burden to patients and to those in health care delivery settings might well be valuable, however, in helping SSA or clinicians in selecting among otherwise seemingly similar instruments.

In addition, in extreme cases of impairment, using the types of instruments reviewed in this report may be an unnecessary cost, with disability status being readily determined on the basis of gross clinical criteria that do not require fine-grained assessment or quantitative comparison to normative samples. Thus, areas for future research are (a) to compare the relative sensitivity and specificity of different approaches to disability determination for different types and degrees of speech and language impairment and (b) to determine when the relative costs and benefits justify the addition of standardized instruments to the assessment process rather than relying solely on clinical judgments.

Also not dealt with in this report or literature on this topic is the matter of the adverse effects or harms of diagnostic testing or disability evaluation. Although clinicians, SSA experts, and others may not judge harms stemming from the use of these instruments to be large, nonetheless the possibility exists that patients may be misdiagnosed or categorized wrongly. Persons with true disease may be missed and, thus, wrongly kept off or struck from SSA disability eligibility rolls; persons identified as disabled who truly are not may suffer emotionally or financially from such mislabeling. In both cases, SSA faces the twin problems of inappropriate persons covered, or not covered, within the disability program. We urge that researchers take a broader perspective to the investigation of speech and language instruments, so as to shed some light on the likelihoods that adults or children may be mislabeled (in both positive and negative ways) and on the consequences of such labeling.

Normative Data and Subpopulations

With the increasing cultural, linguistic, and racial diversity of the US population, the applicability of assessment instruments to individuals who are members of different subpopulations is of crucial importance to clinical diagnosis and the process of disability determination. In addition to demographic subpopulations, the applicability of speech and language assessment instruments for reliable and valid assessment of individuals with different disorders is important, because speech and language impairments may contribute to the disability of people identified with other disorders, such as severe physical impairment, mental retardation, learning disorders, and hearing impairment. Inclusion of representative numbers of subpopulation members in normative samples during instrument standardization is important, but it is not sufficient to ensure the applicability of instruments to various subgroups.

Improving the evidence base requires analyses examining reliability and validity of instruments for subpopulations, not just for the total normative sample. The only instruments we reviewed that have provided good quality evidence in this respect were the Test of Language Development (TOLD)-Primary, 3^rd Edition and the Test of Language Development-Intermediate, 3^rd Edition. Subgroups for which TOLD information can be found include children who are speech-language disordered, learning disabled, mentally retarded, and hard of hearing. All three adult language instruments were standardized on individuals who have language and cognitive impairments associated with strokes and traumatic brain injury, but as noted earlier, the data from these two groups were combined and analyses for subgroups were not presented.

Despite this start on instrumentation for various subgroups, clinicians and policymakers need to recognize that dialect, language, or cultural differences, or functional differences due to certain types of impairment, may well preclude reliable and valid assessment with existing instruments. Despite the existence of a large number of speech and language assessment instruments, we still lack appropriate instruments for reliably and validly assessing speech and language in many of these diverse subgroups. Thus, future research funding and priorities should be directed at addressing these serious deficiencies. Funding sources should encourage research teams that represent collaborations among professionals with expertise in speech and language disorders, cultural experts for the demographic subpopulations of interest, professionals with expertise in disorders that often co-occur with speech and language impairment, and psychometric experts.

Evaluating Existing and Emerging Therapies and Treatments

Some work in the evidence-based practice field (for instance, that done on behalf of the current US Preventive Services Task Force¹²⁰) tends to examine clinical questions about diagnostic (or screening tests) within a context of effective therapies. We did not take that approach in this evidence report because of the already expansive scope of the assignment; hence, questions concerning the efficacy or effectiveness of various treatment of speech and language disorders have here gone unexamined.

We note, however, that a rich agenda of research remains to be pursued concerning appropriate ways to manage speech, language, or voice disorders in both adults and children. A necessary part of such investigations involves tracking patients' progress over time, and obviously the types of instruments reviewed here could play a part in such outcomes assessments.

We caution, however, that the deficiencies in many of these popular and well-known instruments must be addressed before they can be used with confidence in treatment trials or studies. The basic measurement issues were discussed above, but in addition, methodologic work needs to be done on the responsiveness of these instruments (that is, on their sensitivity to change and on the calculation of appropriate effect sizes that reflect change over time for individuals and groups). One strategy for those engaging in or supporting research on the management of patients with speech and language disorders is to build solid methdologic research directly into treatment and rehabilitation studies, thereby strengthening both the given studies and the measurement field as a whole.

Research on treatment efficacy or effectiveness is also relevant to the concerns of the SSA in disability determination. One factor that must be considered in developing a prognosis for a patient's functioning at the end of a 12-month (or longer) interval is the expected response of the patient to intervention. Thus, building the evidence base on treatment outcomes in speech and language disorders will contribute to policy development and more informed clinical decision-making in this area.

Impairment and Disability

Most of the instruments we reviewed were designed to provide a measure of the type or degree of impairment (or both) that an individual experiences. The literature has a dearth of information on the relationship between the type or degree of impairment and the functioning of the person in any usual life activities, including those of concern to the SSA in its disability determinations. This "real world" functioning question suggests a rich research agenda that would not only assist the SSA in decisions about disability but also contribute to the "ecological validity" of all speech and language assessments. We need both more instruments providing direct measurement of activity limitations and participation restrictions and more research demonstrating the relationship between speech and language impairment and activity limitations or participation restrictions.

This is not a completely unexplored area in speech and language instrumentation. For example, the Functional Assessment of Communication Skills for Adults was developed to evaluate the ability to communicate in daily life activities despite existing speech, language, cognitive or hearing impairment.²⁴ Our national panel of experts chose instruments based in part on the likelihood of finding an evidence base in the peer-reviewed literature. Thus, the bias toward instruments measuring impairment in the current report reflects the fact that expert measures of speech and language functioning are very limited. The interests of current policymakers and clinical practitioners may be better served by investing resources into refining such measures and developing an evidence base for them rather than or in addition to focusing on measurement of impairment.

Prediction of Future Functioning

We do call for further research on the second key question considered in this report, that is, the ability of speech and language assessment instruments to predict future functioning or performance. As discussed in Chapters 3 and 4, we found very limited evidence in the literature related to this topic. Predicting future functioning is a key criterion for disability determination, so the SSA will need research results that document the ability of these measures to provide robust, longer-term predictions. Said another way, evidence must be relevant to predicting how an individual will function at least one year from the time of the initial disability determination.

Research on this question needs to be large scale. Moreover, it should not be limited to the predictive value of instruments in assessing specific intervention programs or in predicting future performance of a restricted subgroup. Rather, in terms of broad nation concerns about disability, research should consider prediction of future test performance and future adaptive performance in everyday life among key subpopulations reflecting age, language, co-existing conditions, and other factors.

Key Clinical Questions

During the meeting the discussion of the key clinical questions concerned (1) whether language impairment included oral, aural, and written impairments; (2) which populations to include or exclude; and (3) differences between impairment and function and whether it is appropriate to use tools for impairment to evaluate future functioning or performance. The first issue was addressed by leaving the terminology in the key clinical questions broad (i.e., speech and language disorders) rather than by limiting to spoken language (current SSA criteria examine spoken or verbal rather than written language). The second issue was not resolved; rather, the RTI-UNC EPC evaluated tools looking for evidence for all of the populations listed. The final issue of using measures of impairment to predict future functioning was addressed by rewording the second key clinical question to ensure that measures of impairment are used to predict future impairment and that functioning/performance tools are used to predict future functioning/performance.

Causal Pathway

During the meeting, participants suggested minor revisions to the causal pathway. Specifically, we included voice impairments as a separate category of impairments and added responsiveness (i.e., the ability of the test to detect changes in impairment and to differentiate between levels of severity) and appropriate normative data to the relevant test characteristics of the evaluation tools. We made the former change because voice production is a separate issue with different evaluation tools and complexities. The latter change was made to describe more comprehensively the characteristics of evaluation instruments.

Supplemental Analysis -- Usability Analysis

When deciding which instrument to use, a clinician must evaluate whether the manual provides sufficient information on how to administer and score the instrument. As part of our analyses, we evaluated the usability of the instrument manuals. Two second-year speech and language pathology graduate students in the Division of Speech and Hearing Sciences at the University of North Carolina at Chapel Hill independently evaluated each manual for comprehensiveness and ease of use. Each has completed a minimum of 80 hours of supervised training in speech and language disorder assessment divided equally between adults and children. Thus, the evaluations of these raters represent what we might be expect if an experienced speech and language pathologist used an unfamiliar instrument for the first time.

Kappa values for the individual criterion ranged from 0.34 to 1.00, suggesting slight moderate to almost perfect agreement.³⁹ Inter-rater agreement ranged from 76.5 percent (13/17) to 100% (17/17). We re-reviewed the criteria for which we observed the lowest level of agreement. In most cases, the disagreement amounted not to whether instrument met the criterion but how much information the manual provided to the reviewer. One of the reviewers seemed to require more detail on use to be comfortable with the instrument and, thus, to rate the criterion as having been met.