Relationship between Self-Assessed and Tested Non-English Language Proficiency among Primary Care Providers
Abstract
Background
Individuals with limited English proficiency (LEP) experience poor patient-clinician communication. Most studies of language concordance have not measured clinician non-English language proficiency.
Objectives
To evaluate the accuracy of self-assessment of non-English language proficiency by clinicians compared to an oral proficiency interview.
Subjects
Primary care providers (PCPs) in California and Massachusetts.
Measures
PCPs first completed a self-assessment of non-English language proficiency using a version of the Interagency Language Roundtable (ILR) Scale, followed by the Clinician Cultural and Linguistic Assessment (CCLA), a validated oral proficiency interview. We used non-parametric approaches to analyze CCLA scores at each ILR scale level and the correlation between CCLA and ILR scale scores.
Results
Sixteen PCPs in California and 51 in Massachusetts participated (n=67). Participants spoke Spanish (79%), followed by Cantonese, Mandarin, French, Portuguese, and Vietnamese. The respondents self-assessed as having “Excellent” proficiency 9% of the time, “Very Good” 24%, “Good” 46%, “Fair” 18% and “Poor” proficiency 3% of the time. The average CCLA score was 76/100. There was a positive correlation between self-reported ILR scale and CCLA score (rho=0.49, p<0.001). The variance in CCLA scores was wider in the middle categories than in the low or high ILR categories (p=0.003).
Conclusions
Self-assessment of non-English language proficiency using the ILR correlates to tested language proficiency, particularly on the low and high ends of the scale. Participants who self-assess in the middle of the scale may require additional testing. Further research needs to be done to identify the characteristics of PCP whose self-assessments are inaccurate and, thus, require proficiency testing.
Introduction
Clear communication between clinicians and patients is essential.1 Patient-clinician communication impacts patient satisfaction, adherence to recommendations, and health outcomes,2 and is influenced by language differences.3 Individuals with limited English proficiency (LEP) often experience poor patient-clinician communication.4, 5
Patient-clinician language concordance is associated with better patient satisfaction,6, 7 medication adherence,8 understanding of diagnoses and treatment,9 patient centeredness,10 and health education.7, 11 Language concordance leads to lower emergency room use, likelihood of missing medications, and cost.8, 12, 13 While the few language concordance and cancer screening studies have found lower screening rates for LEP patients with language concordant clinicians,14-16 it is unclear if the “language concordant” clinicians in these studies were truly fluent or had adequate non-English language skills.
There are two main structured methods used to assess clinician language proficiency
Standardized oral proficiency interviews can formally assess a person's general speaking ability.17 Self assessment tools include the Interagency Language Roundtable (ILR) scale, the Industry Collaboration Effort (ICE) scale, and others designed by individual investigators.10, 13, 18-20 Self-assessment tools take less time and cost less than oral proficiency interviews. However, the accuracy of self-reporting tools has not been sufficiently validated. In one study, medical student Spanish proficiency self-assessments correlated with standardized oral fluency test performance.21 In Kaiser Permanente data, 86% of physicians who self-reported non-English language fluency passed an oral proficiency test.22 Other research has shown that self-assessment is problematic. One large study of physicians, nurses, and clerical staff, self-assessed as adequate for LEP encounters, showed that 20% had inadequate non-English language proficiency with an oral proficiency test.23 Most of these studies only included clinicians with strong self-assessed language proficiency, excluding non-fluent clinicians.
Clinician non-English language proficiency standards are needed to ensure quality communication with LEP patients. While some healthcare organizations have instituted bilingual staff language proficiency testing using interpreting skills tests,23 few have begun testing clinicians.22, 24 More studies are necessary to provide healthcare organizations with practical information on assessing clinician non-English language proficiency and inform guidelines.
To address this gap, we performed a study to evaluate the accuracy of a structured non-English language proficiency self-assessment against a validated clinician oral proficiency interview. Based on previous research,25-27 we hypothesized that clinician oral proficiency interview results on the low and high ends of the self-assessment scale would be more accurate than those in the middle.
Methods
Setting
The study was conducted in two settings, the Palo Alto Medical Foundation (PAMF) and Massachusetts Community Health Centers (MA CHC). PAMF, a multi-specialty organization in the San Francisco Bay Area, had approximately 350 Primary Care Providers (PCPs) at the time of the study (2010). PAMF catchment areas have over 27% LEP adults.25, 28 Approximately 25% of PAMF PCPs self-assess fluency in a non-English language; 46% report some proficiency, most commonly in Spanish, Mandarin, and Cantonese. The MA CHCs top five represented study sites care for approximately 125,000 patients; over one-third of adult patients prefer medical care in a non-English language. The most common languages spoken at the MA CHCs are Spanish, Portuguese, and Vietnamese. Both settings have large LEP patient populations but represent different practice types, geographic regions and patient socioeconomic backgrounds. PCPs were physicians, physician assistants (PAs), nurse practitioners (NPs), and nurse midwives (CNMs).
Recruitment
Data was collected from 16 PCPs at PAMF and 51 in the MA CHCs (n=67). Because Cantonese, Mandarin, French, Portuguese, Spanish, and Vietnamese are the most commonly spoken patient languages at the sites, we recruited clinicians with varying self-reported levels of proficiency in these languages. PCPs were recruited via email to complete a survey which asked them to self-report their language proficiencies using the ILR, followed by testing with the Clinician Cultural and Linguistic Assessment (CCLA). Participants were given a $50 gift card or charity donation. The PAMF IRB approved the study.
Non-English Language Proficiency Assessment
We used the ILR Scale for self-assessment.18 Other organizations, such as the American Council on the Teaching of Foreign Languages (ACTFL), have adapted the ILR scale for their own proficiency guidelines but it has not been widely adopted within health-care. The scale consists of 5 levels, each with descriptive explanations, adapted by the authors for medical situations (Table 1).25 Completion takes less than 5 minutes.
Table 1
Adapted ILR Scale for Physicians 18
| Excellent | Speaks proficiently, equivalent to that of an educated speaker, and is skilled at incorporating appropriate medical terminology and concepts into communication. Has complete fluency in the language such that speech in all levels is fully accepted by educated native speakers in all its features, including breadth of vocabulary and idioms, colloquialisms, and pertinent cultural references. |
| Very Good | Able to use the language fluently and accurately on all levels related to work needs in a healthcare setting. Can understand and participate in any conversation within the range of his/her experience with a high degree of fluency and precision of vocabulary. Unaffected by rate of speech. Language ability only rarely hinders him/her in performing any task requiring language; yet, the individual would seldom be perceived as a native. |
| Good | Able to speak the language with sufficient accuracy and vocabulary to have effective formal and informal conversations on most familiar topics. Although cultural references, proverbs and the implications of nuances and idiom may not be fully understood, the individual can easily repair the conversation. May have some difficulty communicating necessary health concepts. |
| Fair | Meets basic conversational needs. Able to understand and respond to simple questions. Can handle casual conversation about work, school, and family. Has difficulty with vocabulary and grammar. The individual can get the gist of most everyday conversations but has difficulty communicating about healthcare concepts. |
| Poor | Satisfies elementary needs and minimum courtesy requirements. Able to understand and respond to 2-3 word entry level questions. May require slow speech and repetition to understand. Unable to understand or communicate most healthcare concepts. |
After ILR completion, clinicians were invited to take the CCLA, the only validated oral proficiency interview designed to assess clinicians’ ability to communicate directly with LEP patients in their preferred language. Other oral proficiency interviews focus either on non-medical settings or on a person's ability to function as an interpreter.29 The CCLA has been validated in 17 languages, including those we tested. Clinicians were given the test's description, practice questions, and an outline of the scoring procedure. The CCLA, available 24 hours per day, is administered via telephone, costs $100 per test, and takes about 30-40 minutes, comparable to other available tests.
Testing was paid for by The California Endowment, which funded the study. Prompts and instructions during the test are pre-recorded by native speakers to ensure reliability. Assessments are scored separately by two professional raters.24 The CCLA passing score of 80% was established by test development experts.30
Analysis
The Wilcoxon-Mann-Whitney test was used to assess equality in CCLA test score by language; the Spearman test assessed correlation between CCLA scores and the ILR scale for overall sample and by language. The Kruskal-Wallis squared rank test assessed equality of CCLA score variance across ILR categories because means and variances in test scores in each ILR group were correlated. We used non-parametric approaches because the sample size in some groups was too small and the test scores in the cells were not normally distributed.31 Language proficiency overestimation was defined as having a CCLA score <80 with an ILR scale score of “Very Good” or “Excellent.” Underestimation was defined as having a CCLA score ≥ 80 with an ILR scale score less than “Very Good.” The analyses were performed using Stata version 11.2 (StataCorp, College Station, TX).
Results
Sixty-seven PCPs participated. Spanish was the most common language (n=53; 79%) followed by Chinese (Mandarin and Cantonese combined, n=9; 13%), Portuguese (n=3), Vietnamese (n=1) and French (n=1) (Table 2). Most PCPs (88%) were physicians. The majority (78%) were female. The respondents rated their ILR proficiency levels as “Good” (46%), “Very Good” (24%), “Fair” (18%), “Excellent” (9%), and “Poor” (3%). The mean CCLA test score was 76 out of 100 (SD=75.9; range 10-92).
Table 2
Summary Statistics of Sample Characteristics
| Freq (%) or Mean [SD] | |
|---|---|
| Language | |
| Spanish | 53 (79.1%) |
| Chinese | 9 (13.4%) |
| Portuguese | 3 (4.5%) |
| Vietnamese | 1 (1.5%) |
| French | 1 (1.5%) |
| Female | 52 (77.6%) |
| Provider title | |
| MD or DO | 59 (88.1%) |
| PA, NP or CNM | 16 (23.9%) |
| Test score | 75.9 [15.7] |
| ILR | |
| Excellent | 6 (9.0%) |
| Very Good | 16 (23.9%) |
| Good | 31 (46.3%) |
| Fair | 12 (17.9%) |
| Poor | 2 (3.0%) |
As illustrated in Figure 1, there was a positive correlation between self-reported ILR proficiency and CCLA score (rho=0.49, p<0.001). Twenty three clinicians (34.3%) underestimated their skill (i.e., passed the CCLA test but self-reported below “Very Good”), and 5 clinicians (7.5%) overestimated (i.e., failed the CCLA test but self-reported as “Very Good” or “Excellent”). Respondents who self-reported “Excellent” on the ILR scored 87 on average on the CCLA, and all of them passed the CCLA test.
There were only 2 points difference in the mean CCLA score between those who reported “Very Good” (80.8) and those who reported “Good” (78.5) on the ILR scale. Variance in CCLA scores was narrow for high ILR categories: “Excellent” (SD: 3.5; range: 83-92), “Very Good” (SD: 7.0; range: 59-91), and was wider for middle and low ILR categories: “Good” (SD: 12.6; range: 37-92), “Fair” (SD: 16.4; range: 30-88), and “Poor” (SD: 33.2; range: 10-57) (p<0.01).
The correlation between ILR scale and CCLA scores was significant for those tested in Spanish (n=53, rho=0.45, p<0.001) or other languages combined (n=5, rho=0.95, p<0.05) while there was no significant correlation for those tested in Chinese (n=9, rho=0.42, p=0.25). On average, Spanish respondents scored higher (77.9) than Chinese (60.8) (p<0.05).
Conclusions
We found a positive correlation between self-assessed non-English language proficiency and test scores on a validated measure of non-English language proficiency. This is in part due to anchoring at the lower and higher ends of the ILR scale. Clinicians who self-assessed their non-English language proficiency as “Fair” or “Poor” and those who reported “Excellent” appear to be more accurate than those reporting they were in the middle of the proficiency scale. We also found a difference in average scores by language, with Spanish CCLA scores higher than Cantonese or Mandarin. This may reflect a need for different CCLA passing scores by language or our participants’ lower Chinese proficiency. Our Chinese sample size was too small for further analyses. Further research will be done to identify whether the ILR scale works well for Chinese languages.
Most prior self-assessed language proficiency validation studies have only included clinicians who self-assess on the high end. In one study, 75% of medical students whose self-assessed Spanish abilities were at “intermediate,” “advanced” or “native speaker” levels accurately assessed their own Spanish proficiency levels.21 Only two of these students were native speakers – the rest were Spanish language learners. As the authors note, these participants likely represented a select group interested in pursuing a Spanish course. Kaiser Permanente found that only 86% of physicians who self-reported as “fluent” passed the CCLA, but the study was limited to physicians at this high self-reported level.22 One study assessed the relationship between physician self-rated Spanish-language ability and their Spanish-speaking patients’ reports of interpersonal care processes. Patients of physicians who rated themselves as “fluent” scored their physicians significantly higher on their ability to elicit and respond to the patients’ problems and concerns.10 However, the physician self-assessment scale was not validated. It is troubling that other studies on language concordance and quality of care have not reported their clinician language proficiency measures.8, 12, 14-16 No prior studies have attempted to assess clinicians at all levels of the non-English language proficiency spectrum.
Structured, validated tools are used less commonly outside of research. Written proficiency tests may not successfully assess oral communication ability. Bilingual staff assessment interviews are also employed, but generally without a validated tool. Many structured, non-validated oral proficiency interviews exist. According to a review of testing options (prior to publication of the CCLA, used in our study),24 approximately 35 oral proficiency tests were available, but most evaluated interpreting skills, not direct patient-clinician interactions.29 Most were based on the ACTFL guidelines, which led the creation of national standards for foreign language learning. Tests ranged from $50-325 and took between 10 minutes and 4 hours. Although no studies have examined how frequently these approaches are applied in practice, organizations are likely using informal assessments without understanding their limitations.
Healthcare organizations could use the adapted ILR to screen clinicians who wish to use their non-English language skills with patients. Policies about the use of non-English language skills among physicians at the low and high ends of self-assessed language proficiency should be clear.26 Those at the lower end should always use professional interpreters. Those at the higher end may be able to use their own non-English language skills without proficiency testing. Both groups should document their use of interpreters or their own skills in the patients’ records. Setting policy for clinicians in the middle range is less clear. Our findings suggest that self-assessment and oral proficiency testing for such clinicians do not correlate well. Further research is needed to assess the impact of using middle range non-English language skills with LEP patients on outcomes. It is also not clear why some clinicians are inaccurate in their non-English language self-assessments using the ILR.
Our study has limitations. It was small and focused only on language proficiency. We are currently conducting research to better understand how clinician demographics, non-English language acquisition, and interpreter use vary by non-English language proficiency. The ILR scale, while validated, is not usually self-administered although it has been used this way in non-medical settings.18 The ILR was adapted by the authors to address clinician-patient interactions.25 However, there are no existing, validated self-assessment tools for clinician non-English language proficiency that could be substituted.
Language concordance improves healthcare quality. It is essential that healthcare organizations and providers know how to accurately measure clinicians’ non-English language proficiency. There is a need for further research to determine what level of clinical interaction is acceptable for clinicians who score in the middle of the ILR. Future research will need to determine best practices for partially bilingual clinicians.
Acknowledgments
Funding: National Cancer Institute R21 (1 R21 CA168489-01), The California Endowment Grant # 20082043
Contributor Information
Lisa Diamond, Immigrant Health and Cancer Disparities Service, Department of Psychiatry and Behavioral Sciences and Department of Medicine, Memorial Sloan-Kettering Cancer Center, 1275 York Ave., New York, NY 10021 Department of Public Health, Weill Cornell Medical College, 402 E. 67th St., New York, NY 10065.
Sukyung, Palo Alto Medical Foundation Research Institute, 795 El Camino Real, Ames Building, Palo Alto, CA 94301-2302.
Warren Ferguson, Department of Family Medicine and Community Health, University of Massachusetts Medical School, 55 Lake Avenue North, Worcester, MA 01655.
Javier Gonzalez, Immigrant Health and Cancer Disparities Service, Department of Psychiatry and Behavioral Sciences, Memorial Sloan-Kettering Cancer Center, 1275 York Ave., New York, NY 10021.
Elizabeth A. Jacobs, Department of Medicine & Health Innovation Program, University of Wisconsin- Madison, 800 University Bay Drive, Suite 210, MC 9445, Madison, WI 53705.
Francesca Gany, Immigrant Health and Cancer Disparities Service, Department of Psychiatry and Behavioral Sciences and Department of Medicine, Memorial Sloan-Kettering Cancer Center, 1275 York Ave., New York, NY 10021 Department of Public Health and Medicine, Weill Cornell Medical College, 402 E. 67th St., New York, NY 10065.