Adapted from Cooper LA, Hill MN, Powe NR. J Gen Intern Med 2002;17:477–486 with copyright permission (pending) from the Society of General Internal Medicine.
 |
|
|
The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-Based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.
To bring the broadest range of experts into the development of evidence reports and health technology assessments, AHRQ encourages the EPCs to form partnerships and enter into collaborations with other medical and research organizations. The EPCs work with these partner organizations to ensure that the evidence reports and technology assessments they produce will become building blocks for health care quality improvement projects throughout the Nation. The reports undergo peer review prior to their release.
AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers as well as the health care system as a whole by providing important information to help improve health care quality.
We welcome written comments on this evidence report. They may be sent to: Director, Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, 540 Gaither Road, Rockville, MD 20850.
Carolyn M. Clancy, M.D.
Director
Agency for Healthcare Research and Quality
Jean Slutsky, P.A., M.S.P.H.
Acting Director, Center for Outcomes and Evidence
The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services of a particular drug, device, test, treatment, or other clinical service.
We thank Christine Napolitano for data entry and management, as well as for assistance with preparation of the report.
Context. The healthcare system in the United States does not provide the same quality of care for ethnic minority populations that it does for the majority white population. Despite awareness of inequities in healthcare quality, little is known about strategies with the potential to improve the quality of healthcare for minority populations.
Objectives. We performed a systematic review of evidence concerning the effectiveness of interventions designed to improve the quality of healthcare in racial or ethnic minorities. Our report focused on evaluations of interventions targeted at healthcare providers or organizations, as provider and organizational factors contribute substantially to disparities and inequities in access to and quality of healthcare.
Data Sources. Electronic searches of MEDLINE®, the Cochrane Collaboration's CENTRAL Register of Controlled Trials, EMBASE, and three specialty databases were performed. Hand searching of key journals and references lists was also performed. Electronic searching was completed in February 2003 and hand searching was completed to June 15, 2003.
Study Selection. Articles included in this evidence synthesis were English-language reports of evaluations of interventions that addressed one of the specific research questions.
Data Extraction. Pairs of reviewers assessed the study quality and abstracted data for each eligible article. Data were entered into a relational database.
Data Synthesis. Ninety-one articles were identified. Twenty-seven articles evaluated strategies targeted at healthcare providers or organizations to improve minority healthcare quality. The majority of these studies targeted physicians and most addressed aspects of prevention. There is excellent evidence that tracking/reminder systems can improve quality of care, and fair evidence that multifaceted interventions, provider education interventions, and interventions that bypass the physician to offer screening services to racial/ethnic minority patients can improve quality of care. Sixty-four articles evaluated cultural competence training as a strategy to improve the quality of healthcare in minority populations. Curricula addressed specific or general concepts of culture and were primarily group discussions and lectures. The lack of consistency in intervention methods and measured outcomes limited the evidence synthesis. There is, however, excellent evidence for improvement in provider knowledge, good evidence for improvement in provider attitudes and skills, and good evidence for improvement in patient satisfaction.
Conclusions. There is some evidence that interventions to improve quality of healthcare for minorities, including cultural competence training, are effective. More research is needed on quality improvement interventions specifically designed to reduce disparities. For example, interventions should target conditions and healthcare processes for which disparities have been documented. Also needed is more research on cultural competence training that uses rigorous study designs, well-described interventions and measurable objectives that are linked to process and outcome variables. Valid, reliable, and objective measurement of cultural competence is needed. As the literature grows, this information needs continued systematic review, updated on a regular basis and disseminated to clinicians, other healthcare decision-makers, educators, and the medical and health services research community.
In recent years, it has become clear that the healthcare system in the United States does not provide the same quality of care for minority populations that it does for the majority white population. Racial and ethnic disparities in access to and quality of healthcare have been extensively documented.1 The Institute of Medicine (IOM) report “Unequal Treatment” confirmed that racial and ethnic disparities in healthcare are not entirely explained by differences in access, clinical appropriateness, or patient preferences.2 There is also increasing evidence that provider behaviors and practice patterns contribute to disparities in care.3 Moreover, researchers assert that variations in healthcare organizational processes compromise quality and that healthcare disparities signal a potentially ripe area for quality improvement.4
Despite awareness of inequities in healthcare quality, little is known about strategies with the potential to improve the quality of healthcare for ethnic minority populations. For those interested in quality improvement, there is a need for an evaluation and synthesis of the strategies proved to be effective in bettering the quality of healthcare for minorities. Moreover, it is unknown whether strategies specifically designed to reduce disparities in healthcare between racial/ethnic minorities and whites have been implemented successfully. One empirical question is whether interventions incorporating quality improvement strategies with documented efficacy for the general population are sufficient to improve quality of care for minorities and reduce disparities, or whether interventions must be specifically targeted to ethnic minority populations in order to improve quality and achieve equity. It has been suggested that cultural competence on the part of healthcare providers and organizations may be one such targeted strategy,5 but with the exception of one recent systematic review of healthcare system interventions6 the effectiveness of cultural competence initiatives targeting healthcare providers has not been systematically evaluated.
The purpose of this report is to systematically review the evidence to determine the effectiveness of interventions designed to improve the quality of healthcare and/or to reduce disparities for racial/ethnic minorities. Our report focuses on evaluations of interventions aimed at healthcare providers or organizations, as recent work suggests provider and organizational factors contribute substantially to the inequities. We began broadly by examining any type of strategy aimed at improving the quality of care in a racial/ethnic minority population of patients, and then we looked specifically at strategies designed to advance the cultural competence of healthcare providers or organizations. This evidence report was requested by the National Quality Forum (NQF) to address recommended priority actions that were outlined in its report “Improving Healthcare Quality for Minority Patients.”7
The National Quality Forum (NQF) requested an evidence report on strategies for improving minority healthcare quality. In September 2002, the Agency for Healthcare Research and Quality (AHRQ) awarded a contract to the Johns Hopkins University Evidence-based Practice Center (EPC) to prepare an evidence report on this topic. We established a team and work plan to develop a report that would identify and synthesize the best available evidence on strategies shown to improve minority healthcare quality. The project consisted of recruiting technical experts, formulating and refining the specific questions, performing a comprehensive literature search, reviewing the content and quality of the literature, constructing the evidence tables, synthesizing the evidence, and submitting the report for peer review.
We recruited technical experts to provide input during the project, four of whom were experts from the Johns Hopkins University and had expertise in public health, quality improvement, physician-patient communication, and nursing. We recruited five external technical experts who had a special interest in improving minority healthcare quality and represented different perspectives including academic medical centers, professional societies and foundations (see Appendix D). We requested specific feedback from the partner (NQF) and from the internal and external technical experts for key decisions, such as selection and refinement of the questions.
We also sought comprehensive feedback on the draft evidence report from the partner, technical experts, and other peer reviewers. Similar to the technical experts, the other peer reviewers were recruited from a variety of organizations and included those based in universities, professional societies and foundations. Experts and peer reviewers were identified by team members in consultation with internal experts and AHRQ. (See Appendix D for a list of experts and peer reviewers.)
The original questions were refined through team discussions, input from internal experts, and review and feedback from the external technical experts. Listed below are the questions addressed in this report.
What strategies targeted at healthcare providers or organizations have been shown to improve minority healthcare quality?
Which of these strategies have been shown to be effective in reducing disparities in health or in healthcare between minority and white populations?
What are the costs of these strategies?
What strategies have been shown to improve the cultural competence of healthcare providers or organizations?
What are the costs of these strategies?
Components of these questions were further defined for use in our review. Minority was defined as all non-Caucasian or non-white racial and ethnic categories, including, but not limited to, African American, Hispanic, American Indian/Alaska Native, and Asian/Pacific Islander. All clinicians were considered healthcare providers. This category included dentists, dental assistants, nurses, nurse assistants, physicians, physician assistants, pharmacists, mental health workers, community healthcare workers, social workers, and others such as alternative healers. For the purposes of this review, our research questions were meant to include any health professional or healthcare organization that provides health services to patients.
Adapted from Cooper LA, Hill MN, Powe NR. J Gen Intern Med 2002;17:477–486 with copyright permission (pending) from the Society of General Internal Medicine.
) and its use in the report.
In 1993, the Institute of Medicine's (IOM) Committee on Monitoring Access to Personal Health Services set out to resolve many conceptual problems in the definitions of equitable access to health care. The Committee developed a model that provided a useful starting point for the conceptual framework that is used in this report.2 Indicators in this model are grouped according to barriers (personal, structural, and financial) that cause underuse of services and mediators (such as appropriateness or efficacy of treatment received, quality of provider skills, or patient adherence) that affect health outcomes and equity of services.
Cooper and colleagues modified the Institute of Medicine's access model to provide more specific directions for designing and implementing effective interventions to eliminate healthcare disparities.8 They expanded the scope of personal and structural barriers, specified utilization measures to include the type of setting, provider, and procedure, incorporated provider communication skills and cultural competence as measures of the quality of providers (a mediator in the original IOM model), and included patient views of care or patient-centeredness (a component of healthcare quality from Crossing the Quality Chasm) as important outcome measures.9
Specifically, they included additional personal barriers/facilitators documented in recent research on disparities to differentiate between the quality of healthcare received by patient race/ethnicity or to determine differences on the use of health services or in health outcomes for whites and ethnic minorities. These variables include family structure, patient preferences and expectations of treatment, patient involvement in medical decision-making, personal health behaviors, beliefs about health and disease, and health literacy.2
Cooper and colleagues also included structural barriers/facilitators within the system in their refined version of the IOM model. For example, in addition to the availability of care, how care is organized, and transportation, they included level of difficulty in getting any appointments at all with primary care physicians and specialists and the timeliness of appointments.10 A rationale for including these structural barriers or facilitators to health service utilization is provided by recent work showing that minority patients seen in primary care settings report more difficulty getting an appointment and waiting longer during appointments, even after adjustment for sociodemographic and health status characteristics.11
The IOM's access model included a category for mediators. A mediator is a variable (intermediate, contingent, intervening, causal) that occurs in a causal pathway from an independent to a dependent variable. It causes variation in the dependent variable (outcomes), and it is also caused to vary by the independent variable (barriers and facilitators). Health service use and quality of care variables are mediators between barriers/facilitators and health outcomes. Because studies of healthcare disparities document that ethnic minority patients are often cared for by physicians with poorer indicators of technical quality (such as lower procedure volume rates and higher risk-adjusted mortality rates) and that interpersonal care, including patientprovider communication, differs by patient ethnicity and by ethnic concordance in the patientprovider relationship,12 Cooper and colleagues expanded the quality of providers (a mediator between barriers and outcomes of care) to include technical skills,interpersonal/communication skills, medical knowledge, and cultural and linguistic competence.
Appropriateness of care, one of the categories of mediators, was conceptualized as the degree to which the care delivered to patients is consistent with current standards of care (for example, beta-blocker use for acute myocardial infarction, or guideline-concordant care for major depression). Efficacy of treatment, in contrast, was conceptualized as the degree to which a specific intervention, procedure, regimen, service, or treatment produces beneficial results under ideal circumstances.13 For example, patient knowledge about injury prevention might be considered a measure of the efficacy of a provider intervention targeting patient education and counseling skills regarding injury prevention. Patient adherence to recommended treatment (e.g., medication refills, health behavior modification, appointment-keeping) is another healthcare process measure. We included all mediators from Cooper's model in a broad category of healthcare system processes. Finally, in addition to health status and equity of services, Cooper and colleagues included patient views about healthcare, including their attitudes toward and experiences with care and satisfaction, since these have emerged as important outcomes that may differ by race, ethnicity, social class, and language.14
Interventions to improve quality of care and to eliminate racial and ethnic disparities in healthcare might address a number of personal, structural, or financial barriers/facilitators and healthcare system processes from our conceptual model. Ideally, the intervention should target factors known to contribute to disparities in healthcare quality. For example, an intervention to eliminate racial disparities in cardiovascular procedure use might focus on patient preferences, patient-provider communication, and provider knowledge of treatment guidelines. An intervention to eliminate racial disparities in mental health care might target patient attitudes, such as stigma or fear of medications, primary care provider skills in recognition of mental health problems, or structural barriers such as the availability of case managers to improve coordination of care between primary care and mental health treatment settings.
Our conceptualization of cultural competence deserves further attention, since Question 2 in this report specifically addresses the state of the evidence regarding interventions targeting cultural and linguistic competence. No single definition of cultural competence is universally accepted. However, several definitions currently in use share the requirement that healthcare professionals adjust and recognize their own culture in order to understand the culture of the patient.15 Lack of cultural and linguistic competence can be conceptualized in terms of organizational, structural, and clinical (interpersonal) barriers to care.5 The Office of Minority Health defines cultural competence as the ability of healthcare providers and healthcare organizations to understand and respond effectively to the cultural and linguistic needs of patients.16 At the patient-provider level, cultural competence may be defined as the ability of individuals to establish effective interpersonal and working relationships that overcome cultural differences.12 The Liaison Committee on Medical Education includes the need for medical students to recognize and address personal biases in their interactions with patients among their objectives for cultural competence training.17 Medical educators have defined eight content areas (general cultural concepts, racism and stereotyping, physician-patient relationships, language, specific cultural content, access issues, socioeconomic status, and gender roles and sexuality) that are taught within a commonly accepted rubric of cross-cultural education curricula.18 We conceptualized cultural competence interventions as those targeting the relevant provider knowledge, attitudes, and skills (healthcare system mediators in our conceptual model).
In addressing our research questions, we acknowledge the potential for a conceptual overlap in interventions targeting quality of care broadly and those targeting cultural competence specifically. There may also be an overlap in interventions that are targeting an organization broadly and those that are targeting providers specifically. One example of this overlap would be an intervention that incorporates interpreter services. While one might consider interpreter services to be an organizational quality improvement strategy that targets structural barriers to care, this type of intervention also affects healthcare system mediators at the provider level, including patient-provider communication and provider cultural competence.
shows the elements of the model addressed by the studies included in the systematic review. We circled the major categories of healthcare system processes that were targeted by the interventions included in the articles that were eligible for our systematic review.The process of searching the literature included identifying reference sources, formulating a search strategy for each source, and executing and documenting each search.
Our search plan included electronic and hand searching. Several electronic databases were searched. In February 2003, we searched MEDLINE®, the Cochrane CENTRAL Register of Controlled Trials (Issue 1, 2003), EMBASE, and the following three specialty databases: the specialized register of Effective Practice and Organization of Care Cochrane Review Group (EPOC) which contains studies that report objective measures of professional performance, patient outcomes or resource utilization identified through extensive electronic and hand searching; the Research and Development Resource Base in Continuing Medical Education (RDRB/CME) a Web accessed database of materials concerning program evaluation, physician performance, change, and healthcare outcomes identified through electronic and hand searching; and the Cumulative Index of Nursing and Allied Health Literature (CINAHL®).
Hand searching for possibly relevant citations took several forms. First, priority journals were identified through an analysis of the frequency of citations per journal in the database of search results as well as through discussions among the EPC team. We identified 12 journals to be hand searched (Appendix A). To ensure identification of recent publications, we scanned the table of contents of each of the 12 journals for relevant citations from January or February 2003 to June 15, 2003 based on the coverage of these journals in MEDLINE®. On the basis of its coverage, the journal Ethnicity and Disease was searched from the fall 2002 issue forward.
For the second form of hand searching, we scanned the reference lists of key reviews and reference articles. We used ProCite, a reference management software, to create a database of reference material identified through an electronic search for relevant guidelines and reviews, through discussions with experts, and through the article review process. The principal investigator reviewed a list of the titles and abstracts from this database to identify key reviews. We then examined the reference lists from these key reviews to identify any additional articles for consideration.
Finally, we examined the reference lists of eligible articles to identify any potentially relevant articles. This was completed by the second reviewer as part of the article review process (see description of article review process below).
Search strategies, specific to each database, were designed to maximize sensitivity. Initially, we developed a core strategy for MEDLINE®, accessed via PubMed, based on an analysis of the Medical Subject Headings (MeSH) and text words of key articles identified a priori. Because of the exclusion criterion related to study design, the component of the strategy specific to Question 1 was combined with the first phase of a previously validated strategy for the identification of controlled trials 19. No limits were based on type of healthcare provider or specific minority group. The PubMed strategy was the basis for the strategies developed for the other electronic databases (Appendix A).
Whenever possible, the results of the searches were downloaded and imported into ProCite. We used the duplication check in ProCite to include in the Minority Health Citations Database only articles that were not previously retrieved. This database was used to store citations and to track the search results and sources. We also used this database to track the results of the abstract review process and the retrieval of full-text copies of articles.
Specific inclusion and exclusion criteria were applied at each of the three levels of review. Criteria became more stringent as the process moved from searching, to reviewing abstracts, to reviewing full-text articles. After identifying a citation, two team members independently reviewed the title and abstract, and articles were included or excluded from the article review according to the criteria described below.
During the abstract review process, emphasis was placed on identifying all articles that might have original data pertinent to the questions. As previously described, the technical experts were consulted during the development of inclusion and exclusion criteria. In evaluating titles and abstracts, the following criteria were used to exclude articles from further consideration:
published prior to 1980
not in English
did not include human data
contained no original data
a meeting abstract only (no full article for review)
not relevant to minority health
no intervention
not targeted to healthcare providers or organizations
no evaluation of an intervention
article did not apply to any of the study questions
The following additional exclusion criteria were applied to articles addressing Question 1 or strategies to improve minority healthcare quality:
not a randomized controlled trial or a concurrent (non-historical) controlled trial
not conducted in the United States
The rationale for these was to focus on studies that were more likely to provide valid evidence on the effectiveness of interventions and that could be applied to the healthcare system in the United States. Strategies employed in other countries may only apply to the healthcare systems in those countries and may not be amenable to translation to the healthcare system in the United States. This restriction was not placed on articles addressing Question 2 since it was felt that educational methods and other strategies to improve cultural competence were likely to be applicable to providers in the United States. We did not apply any study design limits on articles addressing Question 2 because preliminary search results indicated that very few of these studies would meet the more stringent criteria.
Finally, for Question 1 the exclusion criterion of “not relevant to minority health” was further specified to focus our review on interventions applicable to quality in minority healthcare. A study was excluded if less than 50 percent of the patients was from a single minority group or multiple minority groups, or if no subgroup analysis based on racial or ethnic group was completed.
Titles and abstracts of all articles retrieved by the literature search were printed on an abstract form and distributed to two reviewers (Appendix B). The reviewers screened the abstracts for eligibility and classified them by the research question addressed. When reviewers agreed there was insufficient information to decide eligibility, the full article was retrieved for review.
The results of the abstract review process were entered into the Minority Health Citations Database. Deleted citations were tagged with the reason for exclusion. Citations were returned to the reviewers for adjudication if they disagreed on eligibility.
The purpose of the article review was to confirm the relevance of each article to the research questions, to determine methodological characteristics pertaining to study quality, and to collect evidence pertinent to the research questions.
Forms were developed to confirm eligibility for full article review, assess study characteristics, and extract the relevant data for the study questions. The forms were developed through an iterative process that included the review of forms used for previous EPC projects, discussions among team members and experts, and pilot testing. This process was challenging because of the heterogenous literature. We developed separate forms to abstract data for each question. We used one form to assess the quality of each study. The forms were color coded to aid reviewers (Appendix B).
The study quality assessment form had three sections, and reviewers completed the form for each study. The first section included the exclusion criteria so that reviewers could confirm the eligibility of the article before proceeding with the full article review. The second section listed the research questions, thus allowing reviewers to tag articles by the question addressed. The final section contained questions designed to provide an assessment of study quality. These questions were designed to assess methodological strengths and weaknesses in several domains: 1) representativeness of targeted healthcare providers and, if appropriate, targeted patients; 2) potential bias and confounding; 3) description of the intervention; 4) outcomes of the intervention; and 5) analytic approach, statistical quality, and interpretation. In terms of generalizability, studies were given credit for adequately describing their populations, but no judgment was made about whether those populations were representative of the broader population of minority patients or their providers. Each item was scored for each study with a value ranging from 0 to 2. We calculated percentage scores for each domain by adding the total value of the responses and dividing by the total number of possible points for that domain and for that article (excluding items that were not applicable to certain study design types). We used the scores to categorize quality assessment for presentation on the evidence tables. For each domain, scores of 80 percent or higher were given a full circle, scores of 50 to 79 percent were given a half-filled circle, and scores of less than 50 percent were given an empty circle.
We used a separate form for each question to abstract information such as study design, intervention, and outcome assessment. For articles addressing Question 1, an additional group description form was completed for each group (or “arm”) in the study. Articles addressing Question 1 were categorized as addressing specific clinical areas by using the IOM list of priority areas 20. We further classified these articles by the IOM framework of consumer perspectives of healthcare needs that included the categories of staying healthy, getting better, living with illness, and coping with the end of life.9
A serial article review process was employed. In this process, the quality assessment and abstraction forms were completed by the primary reviewer. The second reviewer, after reading the article, checked each item on the forms for completeness and accuracy. The second reviewer also scanned the reference lists of eligible articles to identify potentially relevant articles. The reviewer pairs were formed to include personnel with domain-specific and/or methodological expertise.
All information from the article review process was entered in a relational database (Minority Health Evidence Database). The database was used to maintain and clean the data, as well as to create evidence and summary tables.
| Overall Assessment | Quality | Quantity | Consistency of Findings | ||
|---|---|---|---|---|---|
| Study Designs | Objective Assessment a | ||||
| A | Excellent | at least one RCTb | ≥ 75% studies | ≥ 4 studies | consistent |
| B | Good | at least one controlled trial | ≥ 50% studies | ≥ 3 studies | reasonably consistent |
| C | Fair | no controlled trials | < 50% studies | ≥ 2 studies | inconsistent |
| D | Poor | no controlled trials | < 50% studies | < 2 studies | too few studies to determine |
Objective assessment as rated on the Quality Assessment Form, Question 15
RCT= Randomized controlled trial
In terms of quantity, there had to be at least four studies to meet criteria for Grade A, three to meet the criteria for Grade B, two to meet the criteria for Grade C, or fewer than two studies to meet the criteria for Grade D. In terms of consistency, the results of the studies had to be consistent to meet the criteria for Grade A, reasonably consistent to meet the criteria for Grade B, and inconsistent to meet the criteria for Grade C. Where there where too few studies to judge the consistency the article was assigned Grade D. The grading of the evidence was discussed at a team meeting and consensus was reached on each criterion. The evidence received a final “grade” that reflected the lowest rank on each of the four criteria (two for quality and one each for quantity and consistency).
Throughout the project, feedback was sought from the technical experts through formal and ad hoc requests for guidance. A draft of the completed report was sent to the technical experts, as well as to the partner (NQF), AHRQ and other peer reviewers. Substantive comments were catalogued and entered into a database. Revisions were made to the evidence report as warranted, and a summary of the comments and their disposition was submitted to AHRQ with the final report.
Results from the search and the abstract review process were maintained in a database developed in ProCite. A summary of the results of the search and review processes is provided in Figure 2
Of the 4,389 citations retrieved by the search methods, 3,710 were uniquely identified, that is, not previously included in the Minority Health Citations Database. We reviewed 3,703 of these citations at the abstract review level. We could not determine eligibility for seven articles that we were unable to retrieve.21–27
Of the 3,703 citations reviewed, we identified 288 (8 percent) as eligible for full article review. Reviewers did not need to agree on what exclusion criterion applied at the abstract level. The most frequent reason for exclusion was that the article was not relevant to minority health (used by one or both reviewers to delete 1,873 citations) and that the article did not describe an intervention (1,655 citations). Reviewers agreed on the reason for 1,876 of the 3,415 citations deleted (55 percent). For these articles, the same criteria were the most frequently applied: not relevant to minority health (806 citations, 43 percent) and no intervention (670 citations, 36 percent).
From the abstract review process, 288 citations were identified for inclusion in the article review phase. At the article review level, 197 articles (68 percent) were excluded. The most frequent reasons for exclusion were no evaluation of an intervention (23 percent), not relevant to minority health (22 percent), and not targeted to healthcare provider or organization (16 percent). A listing of the excluded studies, with each article labeled with the reason for exclusion, is included in this report.
Of the 91 included articles, Question 1 was addressed by 27 articles and Question 2 was addressed by 64 articles. One article was identified as addressing strategies to address disparities (Question 1a) and one article was identified as addressing the costs of strategies shown to improve healthcare quality in minority populations (Question 1b). Of the articles addressing Question 2, six articles addressed the costs of the strategies to improve cultural competence (Question 2a). Three articles were identified that included interventions targeted to both providers and organizations, and each of these addressed Question 1.
The identified literature addressing strategies to improve healthcare quality in minority populations was heterogeneous. The articles were published in a variety of nursing and medical publications. There were 144 different journals represented by the 288 articles eligible for review. The Journal of Nursing Education (16 articles), Academic Medicine (13 articles), and Journal of Transcultural Nursing (12 articles) had the highest proportion of eligible articles.
As shown in Figure 3
, starting around 1992, the number of publications addressing healthcare quality in minority populations greatly increased. Thirty-three percent (96 articles) of the eligible articles were published after 2000.Of the 27 studies eligible for review, only three studies were published before 1990,28–30 20 were published between 1990 and 1999,31–50 and four were published after 200051–54 (see Figure 4
). All studies were randomized controlled trials (n=20) or concurrent controlled trials (n=7) (see Evidence Table 1). Despite this, the methodology employed by the studies varied widely, making synthesis and presentation of the evidence difficult. Some studies examined the same providers or patients across intervention groups whereas other studies employed a crossover design. The method of allocation also varied; the intervention assignment was made at the patient level (n=7), at the patient and provider level (n=6), at the provider level (n=6), at the provider and clinic/center level (n=2), and at the clinic/center level (n=6).
The majority of articles was in the area of prevention: general prevention (which included studies on cancer screening, immunization, etc.) (n=3), cancer screening only (n=10), tobacco cessation (n=2), cholesterol lowering (n=1), and prevention in children/adolescents (n=3). There were three studies published in the area of mental health (either depression or alcohol abuse) and one each in the area of chronic renal disease, asthma, acute respiratory tract infections, emergency medical systems, and advance directives (see Figure 5
). From the perspective of the framework proposed by the Institute of Medicine,9 there were 18 studies in the area of staying healthy, four in the area of getting better, one in the area of living with illness, one that dealt with coping at the end of life, and three that were in more than one category. In terms of the dimensions of quality,9 the majority of studies addressed effectiveness (n=24), although there were some that addressed timeliness (n=10), safety (n=2), and patient-centeredness (n=5).
Almost all studies were targeted at physicians: either at physicians only (n=17) or at physicians and mid-level providers (nurses, nurse practitioners, and/or physician assistants) (n=8). Two studies were not targeted at physicians: one was directed solely at nurses and medical assistants and the other was aimed at emergency medical personnel. The specialty of the targeted physicians was most often internal medicine (n=7), but there were also general primary care (n=3), pediatrics (n=3), family medicine (n=2), adolescent medicine (n=1), and one or more of the above (n=9). The interventions targeted practicing professionals (n=15), professionals in training (n=6), or both (n=6). Most interventions occurred in the outpatient setting, either a hospital outpatient center (n=14), a community health center (n=4), a group practice (n=2), or in one or more of the above (n=6). One intervention took place in the community.
The racial/ethnic background of the patients in these 27 articles is shown in Figure 6
. Most studies had more than 50 percent African American patients (n=19),28,
29,
31–36,
38,
40,
42,
44–47,
49–51,
54 only two had patient populations that were specified as more than 50 percent Hispanic.37,
39 and none had more than 50 percent Asian/Pacific Islander patients or more than 50 percent American Indian/Alaska Native patients. The remaining six studies had mixed groups (but no more than 50 percent in any one racial/ethnic category).30,
41,
43,
48,
52,
53 The mean age of patients was younger than 20 years in one study, between 20 and 39 years in two studies, between 40 and 59 years in seven studies, and over 60 years in five studies. The mean age was not specified in 12 studies.
Adapted from Cooper LA, Hill MN, Powe NR. J Gen Intern Med 2002;17:477–486 with copyright permission (pending) from the Society of General Internal Medicine.
| Main Intervention Method b | Health Process Outcomes | Patient Outcomes | ||||||
|---|---|---|---|---|---|---|---|---|
| Utilization | Quality of Providers | Appropriateness of Care | Efficacy of Treatment | Patient Adherence | Health Status | Patient Satisfaction | ||
| PREVENTION, ADULT | ||||||||
| Adult General Prevention | ||||||||
| Gemson, 1995 | Education | NA/++/++/++/NA/NA/NA/++/++ | ||||||
| McDonald, 1984 | T/R | ++/0 | ++/++ | 0 | ||||
| Turner, 1989 | T/R | NA/NA/NA/NA/ NA/NA | ||||||
| Adult Cancer Screening | ||||||||
| Burack, 1994 | T/R | ++/++/NA | ||||||
| Burack, 1996 | T/R | 0 | + | |||||
| Burack, 1997 | T/R | ++ | ||||||
| Burack, 1998 | T/R | 0 | ++ | |||||
| Burack, 2003 | T/R | ++/0 | ++/0 | |||||
| Chambers, 1989 | T/R | ++ | ||||||
| Dietrich, 1998 | multi | ++ | 0/0/0/0/0/0/0 | |||||
| Mandelblatt, 1993 | bypass MD | ++/++ | ||||||
| Manfredi, 1998 | multi | ++/++/0/++ | ||||||
| McCarthy, 1997 | bypass MD | NA | ||||||
| Adult Tobacco Cessation | ||||||||
| Ahluwalia, 1999 | T/R | ++/++/+/++ | ||||||
| Allen, 1998 | multi | + | 0/0/0/0 | |||||
| Adult Cholesterol | ||||||||
| Keyserling, 1997 | multi | 0 | 0/0 | ++ | ++/+/+ | |||
| PREVENTION, CHILDREN | ||||||||
| Health Behavior Screening | ||||||||
| Schubiner, 1994 | STQ | +/+/++/+/+ | 0 | |||||
| Injury Prevention | ||||||||
| Gielen, 2001 | Education | + | 0 | 0 | ++ | |||
| Well Baby Care | ||||||||
| Hornberger, 1996 | Translation | 0 | +/+/+/+/+/+ | + | ||||
| MENTAL HEALTH | ||||||||
| Alcohol | ||||||||
| Burge, 1997 | multi | 0/++/0/0/0/ 0/0/0 | ||||||
| Depression | ||||||||
| Miranda, 2003 | multi | ++ | ++/0 | |||||
| Callahan, 1994 | multi | ++/++/0/0 | 0 | |||||
| OTHER CLINICAL AREAS | ||||||||
| Acute Respiratory Infections | ||||||||
| Harris, 2003 | multi | 0/0 | ||||||
| Asthma | ||||||||
| Evans, 1997 | multi | NA/NA | ++/++/NA | |||||
| Chronic Renal Disease | ||||||||
| Harris, 1998 | NC | ++/0/0/+ | 0 | 0/0/0 | ||||
| Emergency Systems | ||||||||
| Kellerman, 1993 | defib | 0/0/0 | ||||||
| End of Life | ||||||||
| Dexter, 1998 | T/R | ++/+ | +/+/+ | |||||
The outcome assessments in this table are based on between-group comparisons. “++” indicates significant improvement, “+” indicates improvement, “0” indicates no improvement, and “NA” indicates between-group comparison not available. If a cell is left blank, then the outcome type was not measured for that study. Outcomes separated by “/” indicate more than one outcome type.
“T/R” indicates tracking and reminder, “multi” indicates multifaceted, “bypass MD” indicates bypass doctor with nurse, “STQ” indicates safe times questionnaire, “NC” indicates nephrologist consultation, and “defib” indicates defibrillators on emergency medical vehicles.
and Table 3 and detailed in Evidence Table 3) were related to healthcare process: use of services (7 studies, 13 outcomes), appropriateness of care (18 studies, 43 outcomes), quality of providers (9 studies, 30 outcomes), patient adherence (4 studies, 9 outcomes), and efficacy of treatment (1 study, 1 outcome). Patient health status (7 studies, 21 outcomes), and patient satisfaction (3 studies, 3 outcomes) were also measured. Most authors reported that their intervention resulted in overall improvement (n=13) or partial/mixed improvement (n=10), but a few reported that their intervention resulted in no improvement (n=4).
The studies were fairly well described in terms of representativeness (20 of 27 clearly described healthcare providers and setting) and intervention description (24 of 27 described the intervention sufficiently to ensure replication). In terms of bias and confounding, although there were 20 randomized controlled trials, the randomization was considered adequate (in that investigators could not predict assignment) in only 11 studies. Also, although there were seven concurrent controlled trials, there was one study in which the comparison group was considered inadequate (dissimilar).
The studies were not as well designed in terms of their outcome assessment and analysis. Although all studies used objective methods to evaluate outcomes, only nine of 27 studies had masked outcome assessment, and 13 of 27 studies performed a pre- and a post-intervention evaluation. Approximately half (15 of 27) reported the numbers for and reasons for non-inclusion in the study analysis, and almost all (21 of 27) performed a complete statistical analysis (including the magnitude of difference between groups, an index of variability, and a test statistic).
| Intervention Typea | Clinical Area (# studies) | Total # Studies | Overall Outcome Assessment across Studies | Rating of Evidence |
|---|---|---|---|---|
| Tracking/ Reminders | Adult, Prevention (9) | 10 | All studies reported favorable outcomes. | A |
| End of Life (1) | ||||
| Multifaceted | Adult, Prevention (4) | 9 | Although almost all studies reported some positive impact on one or more outcome type, results within outcome type were inconsistent across studies. | C |
| Depression (2) | ||||
| Alchohol Abuse (1) | ||||
| Asthma (1) | ||||
| Upper Respiratory Tract Infections (1) | ||||
| Bypass MD | Adult, Prevention (2) | 2 | Both studies reported favorable outcomes in the provision of preventive services to patients. | C |
| Provider Education | Adult, Prevention (1) | 2 | Both studies reported favorable impact on provider counseling behaviors. | C |
| Child, Prevention (1) | ||||
| Use of STQb | Child, Prevention (1) | 1 | One study reported favorable impact on provider counseling behavior. | D |
| Use of RSTc | Child, Prevention (1) | 1 | One study reported favorable outcomes on accuracy of translation and provider/patient satisfaction. | D |
| Use of Specialty Consult | Chronic Renal Disease (1) | 1 | One study did not demonstrate improvement. | D |
| Use of Defibrillators | Emergency Medicine (1) | 1 | One study did not demonstrate improvement. | D |
STQ = safe times questionnaire
RST = remote simultaneous translation
General prevention. Three studies evaluated the impact of quality improvement in the area of general prevention; two primarily used a tracking/reminder system,28, 29 and one primarily used provider education and “prevention prescription forms”.36 All studies demonstrated improvements in healthcare processes, such as likelihood of physicians applying preventive care to eligible patients28, 29 and a variety of specific physician counseling behaviors.36 Only one study28 attempted to measure patient outcomes, and it failed to show improvements in any physiologic measures such as blood pressure, serum glucose, and serum potassium.
Cancer screening. Ten studies evaluated the impact of quality improvement in the area of cancer screening,30, 31, 34, 38, 40, 42, 44, 47, 48, 54 and each of these used a unique combination of provider and patient methods in the intervention and control arms of the study. Most (n=8) used some form of a tracking/reminder system, and two studies used medical assistants or nurse practitioners to offer screening directly to patients.31, 42 All of these studies measured the impact of the interventions on the healthcare process, and all studies found some sort of improvement in cancer screening rates (appropriateness of care) or provider counseling behavior (quality of providers) for some of their outcomes. None of the studies evaluated the impact of quality improvement on patient outcomes.
Tobacco cessation. Two studies evaluated the impact of quality improvement in the area of tobacco cessation; one utilized a tracking/reminder system only,49 and one used a provider education and reward system.46 Both interventions improved provider counseling behavior, but the one study that evaluated patient adherence found no effect of the intervention on patient quit rates.46
Cholesteral. Only one study evaluated the impact of quality improvement in the area of hypercholesterolemia.43 The intervention in the study involved provider education, an intensive tracking/reminder system, and patient education. Although the study did not demonstrate any improvement in appropriateness of care, there was a significant improvement in patient adherence to diet therapy and in cholesterol levels for patients in the intervention group.43
Injury prevention. One study evaluated the impact of quality improvement in the area of injury prevention in children.51 This study compared the effects of a minimal provider education program with an enhanced provider education program and found that patients of the enhanced intervention providers received more provider injury prevention counseling and were more satisfied with the injury prevention information provided to them than patients of minimal intervention providers.51 However, there were no differences in parents' knowledge of injury prevention or in their injury prevention practices between the two groups.51
Well-baby care. One study evaluated the relative impact of two different interpretative systems for non-English-speaking mothers during well-baby care visits.37 The two interpretative systems were remote simultaneous (in which the interpreter translates simultaneously with the speaker but is not in the exam room) and proximal consecutive (in which the interpreter is in the exam room, waits for each person to complete a thought, and then translates the completed thought). The study found that both patients and physicians preferred the remote simultaneous translation and that it was associated with fewer misinterpretations and more complete information.37
Health behaviour screening. Only one study evaluated the impact of quality improvement in the area of health-risk-behavior screening in adolescents.35 The study compared the effects of a minimal provider education program with provision of a patient-completed standardized health-behavior-screening instrument to providers. The study found that providers who received the patient-completed screening instrument spent more time in health behavior counseling versus assessment and had higher agreement with psychiatrist ratings in assessment of patients at risk.35 There were no differences in patient satisfaction for either group of patients.
Depression. Two studies evaluated the impact of quality improvement in the area of depression: one that used an intensive program of provider education, depression protocols, and increased provider visits with depressed patients compared with a minimal provider education program33 and one that combined a provider education program with an intensive patient intervention for comparison with a minimal provider education program.53 Both studies found an improvement in healthcare processes (appropriateness of depression care) for depressed minority patients.33, 53 In terms of health outcomes, the results were mixed; one study found that depression in both the intervention and control groups improved,33 and the other study found that depression improved more for the intervention patients than for the control patients.53
Alcohol. One study addressed quality improvement in the area of alcohol abuse.39 This study evaluated the relative and combined impact of a physician education and patient psychoeducation quality improvement program compared with no intervention, and it found addiction severity decreased over time for all groups (including the no intervention group) and physiologic measures of health worsened for all groups.39
Asthma. One study evaluated the impact of a minimal quality improvement intervention compared with an intensive quality improvement intervention (including provider education, provision or review of practice guidelines, and consultation with expert-physicians) in the care of asthma for children in public health clinics.41 The study demonstrated improvements in continuity of care and appropriateness of asthma care for patients in intervention clinics compared with controls.
Chronic renal disease. One study evaluated the impact of a single nephrology consultation for patients with chronic renal insufficiency compared with usual care by a primary care physician.45 The study found that intervention patients had more visits to ophthalmologists, but had no improvements in health outcomes.
Acute respiratory tract infections. One study evaluated the impact of a provider education/minimal patient education intervention with a provider education/intensive patient education intervention, both aimed at decreasing antibiotic prescription rates for acute upper respiratory tract infections.52 The study found that both interventions were effective in decreasing unnecessary antibiotic prescriptions without differences between groups
Emergency systems. One study evaluated the effect of providing automated external defibrillators on emergency medical equipment (and training firefighters in their use) compared with standard emergency care (which involved cardiopulmonary resuscitation without defibrillation).32 That study found no differences in health outcomes between the two groups of patients.
Completion of advance directives. One study used a physician reminder system to encourage physicians to discuss advance directives with patients.50 The study found that patients of physicians who had been reminded were more likely to be counseled and to complete advance directives.50
Tracking/reminder systems: Ten studies used tracking and/or reminder systems to improve quality of care. Of these, two were in adult general prevention,28, 29 six were in adult cancer screening,29, 34, 38, 40, 44, 54 one in tobacco cessation,49 and one was in end-of-life care (completion of advance directives).50 All ten studies demonstrated positive outcomes, primarily in the appropriateness of care (such as provision of preventive care, tobacco cessation counseling, or advance directive counseling) category. Overall, there is excellent evidence supporting the use of tracking/reminder systems aimed at providers of racial/ethnic minority patients (Evidence Grade A).
Multifaceted interventions: Nine studies used an intervention that we characterize as multifaceted, meaning that there two or more (usually more) main intervention methods.33, 39, 43, 46, 47, 53, 41, 48, 52 Two of these interventions were in adult cancer screening,47, 48 one in tobacco cessation,46 one in cholesterol reduction,43 three in mental health,33, 39, 53 one in acute upper respiratory tract infections,52 and one in asthma.41 Outcomes of these studies are mixed, with most studies showing improvements in one or two (but not all) outcomes measured. Overall, there is fair evidence supporting the use of multifaceted interventions aimed at providers of racial/ethnic minority patients (Evidence Grade C).
Bypass the physician: Two studies (both in adult cancer screening) bypassed the physician and had either a nurse or a nurse practitioner offer screening directly to patients, and both studies demonstrated improvements in the provision of preventive services to patients.31, 42 Overall, there is fair evidence supporting the use of bypassing the providers of racial/ethnic minority patients to offer standardized services directly to patients (Evidence Grade C).
Provider education: Two studies primarily used provider education as the main intervention strategy, one in the area of adult general prevention36 and one in prevention of injuries in children.51 Both studies found improvements in provider counseling behaviors,36, 51 but one measured and did not find any positive effect of the intervention on parental knowledge of injury prevention (the only outcome categorized as efficacy of treatment) or parental adherence to provider advice 51. Overall, there is fair evidence supporting the use of provider education aimed at providers of racial/ethnic minority patients (Evidence Grade C).
Use of Safe Times Questionnaire (STQ): One study (in the area of prevention for children) used a structured questionnaire to assess adolescent health behaviors and demonstrated a positive impact on provider counseling behaviors.35 Overall, there is poor evidence supporting the use of structured questionnaires for racial/ethnic minority patients (Evidence Grade D).
Use of Remote Simultaneous Translation (RST): One study compared the accuracy of translation and quality of patient-physician communication by using remote simultaneous and proximate consecutive interpretation and found fewer translation errors and enhanced patient and physician satisfaction by using the RST method.37 Overall, there is poor evidence supporting the use of remote simultaneous translation for racial/ethnic minority patients (Evidence Grade D).
Use of specialty consultation: One study evaluated the use of nephrology consults for patients with chronic kidney disease and found no effect on healthcare process or patient outcomes.45 Overall, there is poor evidence supporting the use of specialty consults aimed at providers of racial/ethnic minority patients (Evidence Grade D).
Use of defibrillators on emergency medical services: One study evaluated the use of defibrillators on emergency medical services and found no effect on patient outcomes.32 Overall, there is poor evidence supporting the use of defibrillators on emergency medical services (Evidence Grade D).
Only one study specifically addressed the question of whether an intervention could reduce disparities in healthcare quality between ethnic minority and white persons.53 The study, which evaluated the impact of two different culturally tailored interventions to improve the quality of depression care compared with a control group that received no intervention, had mixed results. There was no differential effect of the interventions on healthcare process for white versus ethnic minority patients; all patients (African American, Latino, and white) in the interventions groups were more likely than patients in the control group to receive appropriate therapy. However, there was a mixed effect on health outcomes: there were improvements for African American and Latino patients in the rate of depression compared to controls (with no improvement for white patients), but there were no improvements for African American and Latino patients in the intervention groups in employment rates compared with controls (with improvement for white patients). Overall, there is poor evidence to determine which interventions might reduce disparities between racial/ethnic minority patients and majority patients (Evidence Grade D).
Only one study reported on the costs of an intervention aimed at improving the quality of healthcare for racial/ethnic minority persons.45 This study, which provided case management and nephrology consultation for patients with chronic renal insufficiency, estimated that it cost a minimum US $89,355 yearly in 1998 (or $484 per intervention patient), but it found no health benefits to participants. Overall, there is poor evidence to determine the cost of strategies to improve the quality of care for racial/ethnic minorities (Evidence Grade D).
Strategies to improve the quality of care for racial/ethnic minority patients have been implemented in a variety of clinical areas by various methods. Almost all the interventions have occurred in the primary care setting, and most have focused on the provision of preventive services. There is excellent evidence that these interventions have improved the quality of providers (Evidence Grade A), good evidence that these interventions have improved appropriateness of care (Evidence Grade B), and fair evidence that these interventions have positively affected patient utilization, adherence, satisfaction, or health status (Evidence Grade C).
In terms of intervention methods, there is excellent evidence that provider tracking/reminder systems are effective in improving the quality of care for racial/ethnic minority patients (Evidence Grade A), fair evidence that multifaceted interventions, provider education interventions, and interventions which bypass the physician to offer screening services to racial/ethnic minority patients can improve quality of care (Evidence Grade C), and insufficient evidence for the use of any other of the studied interventions (Evidence Grade D).
There is poor evidence to determine which strategies are most effective in reducing disparities between minority and white populations (Evidence Grade D). The only study that was specifically designed to do this had mixed results, with improvements in only one of the two outcomes assessed.53
There is poor evidence to determine the costs of strategies to improve care and reduce disparities for minority populations (Evidence Grade D).
A total of 64 articles addressed strategies to improve the cultural competence of healthcare providers or organizations. The number of articles published has increased substantially over the last decade. Of the 64 total articles, five were published between 1980 and 1989,55–59 30 between 1990 and 1999,60–89 and 29 between 2000 and 200390–118 (see Figure 8
). Most of the articles described interventions that took place in the United States (n=52). The remainder described interventions in Australia (n=6), the United Kingdom (n=3), Canada (n=2), or New Zealand (n=1) (see Evidence Table 4).
). Most interventions targeted healthcare providers who were in their pre-professional training (n=38) or who were practicing professionals (n=17) (see Figure 10).
The content of the curricular interventions varied across the 64 studies. Using a previously developed framework to categorize cultural competence curricular content,119 we found that most interventions focused on specific cultural content (n=45), general concepts of culture (n=43), language (n=15), and patient-provider interaction (n=13). A few articles describing interventions focused on healthcare access (n=8), racism (n=3), socioeconomic status (n=2), and gender (n=1). In terms of the specific minority groups that were the focus of the interventions, e 20 studies mentioned Hispanic persons; 19 African Americans; 16 Asians/Pacific Islanders; and five, American Indians.
Most interventions used more than one training method, and no two studies used exactly the same methods. The most common training methods were group discussion (n=29), lectures (n=29), case scenarios (n=20), small group work (n=18), clinical experiences (n=17), cultural immersion (n=17), readings (n=16), role play (n=14), presentations by members of another culture (n=14), and audio/visual materials (n=14). Less common training methods were interviews of members of another culture (n=10), practice exercises (n=7), literature (e.g., poems, stories) (n=5), role play/modeling (n=5), language lessons (n=4), self-study (n=3), self-reflection/awareness (n=3), standardized patients (n=2), problem-based learning (n=1), and brainstorming (n=1).
Most articles did not specify the total contact time that the targeted learners spent in training (n=32). In those that did specify learner contact time, the majority of interventions were eight hours or less (n=12), but some were between one and five days (n=10) or longer than one week (n=10). In terms of timing of the evaluations, 13 evaluations were completed less than one day after the end of the intervention (immediate post-test), four were completed 1 to 30 days after, six were completed between one and three months after, six between four and 12 months after, five more than one year after, six had multiple evaluation time points, and 24 articles did not specify when the evaluation of the intervention was completed.
Most studies used more than one method for evaluation; the most common methods were provider self-assessment forms (used in 33 studies), participant ratings of the curriculum (n=19 studies), written exams (n=19 studies), individual provider interviews (n=6 studies), or patient ratings (n=5 studies). Less common methods of evaluation were essays (n=5 studies), group interviews (n=5 studies), direct provider observation (n=3 studies), and performance audits (n=1 study). Only four articles attempted to measure patient outcomes; most included some measure of provider outcome, either attitude (n=44), knowledge (n=30), or skills/behaviors (n=22) (see Figure 11
).
| Provider Outcomes | Patient Outcomes | |||||
|---|---|---|---|---|---|---|
| Knowledge | Attitudes | Skills | Satisfaction | Patient Adherence | Health Status | |
| Physicians | ||||||
| Beagan, 2003 | 0 / 0 / 0 | |||||
| Copeman, 1989 | ? | + / ? | ||||
| Crandall, 2003 | ++ | ++ | ++ | |||
| Culhane-Pera, 1997 | ++ | + / + | ++ / ++ | |||
| Dogra, 2001 | 0 / ? | |||||
| Farnill, 1997 | ++ / ++ | |||||
| Godkin, 2001 | ++ | ++ | ||||
| Godkin, 2003 | ++ / ? / ++ | |||||
| Haq, 2000 | ++ / + | ++ | ||||
| Mao, 1988 | + | ++ | ||||
| Mazor, 2002 | + | ++ / ? | ++ / ++ / ++ / ++ | |||
| Nora, 1994 | ++ | + | + | |||
| Rubenstein, 1992 | ++ / ++ | |||||
| Tang, 2002 | ++ / ++ / ++ | |||||
| Nurses | ||||||
| Alpers, 1996 | + / - | |||||
| Flavin, 1997 | 0 | |||||
| Frank-Stromborg, 1987 | ++ / + | + | ||||
| Frisch, 1990 | ++ / 0 / + | |||||
| Inglis, 2000 | ++ | |||||
| Jeffreys, 1999 | ++ | ++ | ++ | |||
| Lasch, 2000 | ++ | ++ | ||||
| Napholz, 1999 | + | |||||
| Scisney-Matlock, 2000 | ++ / ++ | |||||
| Smith, 2001 | ++ | ++ | ||||
| St. Clair, 1999 | ++ / ++ / + | |||||
| Underwood, 1999 | + | + / + | + | |||
| Williamson, 1996 | ++ | ++ | ++ | |||
| Mixed healthcare provider groups | ||||||
| Erkel, 1995 | + | + / + / + | ||||
| Gallagher Thompson, 2000 | ++ / ++ | ++ | ||||
| Gany, 1996 | ++ | ++ | ||||
| Way, 2002 | + | + / ++ | ++ | |||
| Other healthcare providers | ||||||
| Hansen, 2002 | ++ | |||||
| Strumphauzer, 1983 | ++ | ++ | ||||
| Wade, 1991 | ++ | ++ | ||||
“++” indicates significant positive effect, “+” indicates positive effect,“-“ indicates negative effect, “?” indicates partial/mixed effect, “?” indicates unclear, and “0” indicates no effect. If cell is left blank, then outcome type is not measured for that study. Outcomes separated by “/” indicate more than one outcome type
Studies in physicians. Six of the seven studies that evaluated changes in physician knowledge following implementation of a cultural competence curriculum found an improvement,63, 67, 74, 96, 111 and one study had mixed results.59 Of these six studies, two evaluated culture-specific knowledge,59, 67 and four evaluated knowledge about general cultural concepts.63, 74, 96, 111 There was no clear pattern regarding which type of knowledge (culture-specific versus general) was more often enhanced by cultural competence training for physicians.
Studies in nurses. Four of the seven studies that evaluated changes in nurse knowledge after implementation of a cultural competence curriculum demonstrated an improvement,72, 82, 94, 98 one had mixed results,60 and two showed no improvement.76, 91 Of these, four studies evaluated knowledge of general cultural concepts,72, 91, 94, 98 and three evaluated culture-specific knowledge.60, 76, 82 There was no clear pattern regarding which type of knowledge (culture-specific versus general) was more often affected by cultural competence training for nurses.
Studies in other providers/mixed groups. All six of the studies done in mixed groups of providers or other providers (i.e. mental health providers) demonstrated an improvement in provider knowledge.55, 69, 73, 93, 100, 108 Three of these studies evaluated knowledge of general cultural concepts,55, 69, 100 two evaluated culture-specific knowledge,73, 93 and one108 did not specify the type of knowledge evaluated. There was no clear pattern regarding which type of knowledge (culture-specific versus general) was more often expanded by cultural competence training for mixed groups of providers/other providers.
Studies in physicians. Nine of the 12 studies that evaluated changes in physician attitudes after cultural competence training demonstrated an improvement in attitudes.111, 58, 67, 74, 90, 96, 102, 106, 110These nine studies all examined different types of attitudes, such as increased confidence in communicating with Spanish-speaking patients,106 greater understanding of the effect of sociocultural issues on the patient-physician relationship,102 and more positive attitudes towards community health issues.90
Three of the 12 studies that evaluated changes in physician attitudes after cultural competence training had no change or mixed results.59, 112, 118 Notably, one of these studies demonstrated that after a curriculum that emphasized culture-specific information about Aboriginals, although students did develop more positive attitudes towards Aboriginal people, students were also more likely to agree that “in general, Aboriginal people are all pretty much alike.” 59
Studies in nurses. Nine of the 10 studies that evaluated change in nurses' attitudes after cultural competence training demonstrated some improvement in their attitudes,57, 72, 82, 84, 86, 88, 94, 95, 98 and one had mixed results.71 Although three of these studies demonstrated improvements in cultural self-efficacy,86, 88, 98 the other types of attitudes that were measured in these studies were varied and sometimes unspecific, for example, “attitudes about cultural patterns.”120
Studies in other providers/mixed groups. All three of the studies that were done in mixed groups of providers and that evaluated the effect of the intervention on provider attitudes demonstrated an improvement.69, 73, 100 These attitudes included enhanced awareness and appreciation for rural lifestyles,69 an increased interest in learning about patient and family backgrounds and increased sensitivity to cultural competence,100 and greater sensitivity to immigrant health concerns.73
Studies in physicians. All seven of the studies that evaluated the impact of cultural competence training on physician skills and/or behaviors demonstrated an improvement.58, 67, 74, 77, 90, 106, 111 In one study, participants were given 16 one-hour sessions in which they practiced communication skills with community volunteers and were subsequently shown to be significantly more competent in interviewing a non-English speaking person as rated by a masked psychologist assessing videotapes of the interviews.77 In one study, participants scored higher on a Spanish language test following 20 hours of training in Spanish,67 and in another study participants used a Spanish interpreter less often after 20 hours of training in Spanish.106 Three other studies demonstrated a positive effect on physicians' own general skills self-assessment.74, 90, 111
Studies in nurses. All five of the studies that evaluated the impact of cultural competence training on nurse skills and/or behaviors demonstrated an improvement.57, 72, 82, 84, 91 Two of these studies showed an increase in nurses' involvement in community-based cancer education programs,57, 82 and another study noted an increase in activities devoted to understanding other racial/ethnic groups and an increase in self-reported social interactions with peers of different races/ethnicities.91
Studies in other providers/mixes groups. Both of the studies that evaluated the impact of cultural competence training on the skills and/or behaviors of other providers/mixed providers demonstrated an improvement.55, 93 In one study, there was an increase in referrals of Hispanic Alzheimer's patients and families to appropriate specialized services,93 and in another study participants were better able to conduct a behavioral analysis and treatment plan.55
Only three articles evaluated patient outcomes: one that targeted physicians,106 one that targeted mental health counselors,61 and one that targeted a mixed group of providers.100 The outcome measures were not health outcomes however. All three reported favorable patient satisfaction measures (all related to the interpersonal aspects of care),61, 100, 106 and one demonstrated an improvement in adherence to follow-up among patients assigned to the intervention group providers.61
In terms of the methods used to bring about such improvements in patient satisfaction and (in one case) adherence, one study trained four mental health counselors about the attitudes that low-income African American women bring to counseling (4 hours total);61 another study trained nine physicians to speak Spanish (20 hours total);106 and the third study implemented a state-mandated three-day training program focused on team training, recipient recovery principles, clinical issues, and cultural competence for all staff who have contact with recipients of inpatient mental healthcare.100
Of the 55 articles eligible for review, only five addressed the costs of cultural competence training.67, 75, 90, 96, 106 Overall, the cost information contained in these articles was too limited to allow a comprehensive estimate of costs.
There are limited data on the costs of international cultural immersion. Four of the five articles67, 75, 90, 96 described the costs of interventions that involved international travel. Of those, three programs67, 90, 96 shared the cost of travel with the students, and one program required students to pay the entire amount.75 Two programs provided US$2000 (in 200096 and in 1995-199690) for each student to travel from the United States to South America, Asia, or Africa for either six96 or eight90 weeks. In each of these programs, the students paid the remaining costs. Another program estimated that an 8-day trip from the United States to Mexico cost US$1200 total in 1994, of which the students contributed 60 percent on average and scholarship assistance for the remainder was available through private donations.67 Another international travel program estimated that it cost each student in excess of Aust$4000 in 1997 to travel from Australia to Thailand for four weeks, and the students paid the entire amount.75
There are limited data on the costs of classroom or other types of instruction. One study estimated the cost of 20 total hours of Spanish language instruction for nine physicians to be US $2000 in 2000, not including the opportunity costs for physician time (approximately 20 hours total for each physician).106 This intervention, which had one of the best evaluations, was able to show a difference in patient satisfaction. In another program, 60 hours of classroom instruction (20 hours of Spanish-language instruction and 40 hours of cultural competence training focused on Hispanic populations) were provided for 19 students at an estimated local cost of US $3000 in 1994, of which each student contributed US $80.67 Finally, one program matched 26 students to 26 local ethnically diverse families, asked the students to visit the family six times, and paid each family US $400 in 1996 to 2000.90
There is excellent evidence to suggest that cultural competence training can favorably affect the knowledge of healthcare providers (Evidence Grade A) and good evidence that cultural competence training can improve the attitudes and skills of healthcare providers (Evidence Grade B). However, the studies are heterogeneous (perhaps reflecting the complexity of interventions), and it is difficult to conclude which specific types of training interventions are effective in improving particular outcomes. Even within an outcome category, outcome measurements are not uniform, making it difficult to determine which specific types of knowledge, attitudes, or skills are affected by cultural competence training. No studies have examined quality of care process outcomes.
There is good evidence from three studies to suggest that cultural competence training can favorably affect patient satisfaction (Evidence Grade B) and poor evidence that cultural competence training can improve patient adherence (Evidence Grade D), although the one study that examined patient adherence demonstrated a positive impact. No studies have evaluated patient health outcomes.
Evidence is insufficient to determine the cost of cultural competence training (Evidence Grade D). However, one of the studies that demonstrated an improvement in patient satisfaction also included information about cost, and so perhaps the best evidence is its estimated US $2000 to train nine emergency department physicians in the Spanish language.106
Almost all the interventions occurred in primary care settings and most of the studies occurred in the area of prevention.
Most studies targeted healthcare providers caring for a majority of African American patients.
The most common main intervention methods are tracking/reminder systems and multifaceted interventions. Most studies utilized methods that were generic quality improvement strategies, and only two studies specifically targeted the needs of racial/ethnic minority patients.
There is excellent evidence that quality improvement strategies aimed at the healthcare providers of racial/ethnic minority patients are effective in improving the quality of providers (Evidence Grade A), good evidence that these strategies are effective in improving appropriateness of care (Evidence Grade B), and fair evidence that these strategies can improve patient health service utilization, adherence, satisfaction, and health status (Evidence Grade C).
There is excellent evidence that tracking/reminder systems aimed at providers of racial/ethnic minority patients are effective in improving the quality of care (Evidence Grade A), fair evidence that multifaceted interventions, provider education interventions and interventions bypassing the physician to offer services directly to patients can improve quality of care for racial/ethnic minority patients (Evidence Grade C), and poor evidence to support the use of any of the other strategies (Evidence Grade D). Of note, however, were two types of interventions with favorable results (employed in one study each, thus receiving an evidence grade of D) that may be worth further study: use of remote simultaneous translation for patients with limited English proficiency and the use of the Safe Times Questionnaire for health behaviors risk assessment in adolescents.
There is poor evidence to determine which strategies are most effective in reducing disparities between ethnic minority and white populations (Evidence Grade D). Only one study was designed to do this (with mixed results), and this is a critical gap in the literature.
There is poor evidence to determine the costs of strategies to improve care and reduce disparities for ethnic minority populations (Evidence Grade D).
Studies have been conducted across a wide range of healthcare provider specialties and training levels.
The curricular methods utilized are heterogenous; there were no two studies that used the same combination of intervention methods.
There is excellent evidence to suggest that cultural competence training can increase the knowledge of healthcare providers (Evidence Grade A) and good evidence that cultural competence training can improve the attitudes and skills of healthcare providers (Evidence Grade B).
There is good evidence from three studies to suggest that cultural competence training can raise patient satisfaction (Evidence Grade B) and poor evidence that cultural competence training can affect patient adherence (Evidence Grade D) although the one study that was designed to measure patient adherence demonstrated favorable results.
There are no studies that have evaluated the impact of cultural competence training on patient health status outcomes.
There is insufficient evidence, because of heterogeneity of the literature, to suggest which types of cultural competence training (i.e., lecture, workshop, small group, cultural immersion) are most effective.
There is poor evidence to determine the cost of cultural competence training (Evidence Grade D).
Eligibility was limited to English language published reports of studies. There is, therefore, a possibility of publication bias. Although, our resources did not permit extensive searching of the non-English language and gray literature, recent work has suggested that results of reviews with these limits do not differ substantially from reviews with no such limits.121
Eligibility was limited to articles published after 1980.
There were small numbers of studies in each clinical category (except prevention), and many priority conditions that have had documented healthcare disparities (such as HIV/AIDS, cardiovascular disease, diabetes mellitus, infant mortality) were not represented.
Only two studies used interventions actually tailored for the healthcare needs of racial/ethnic minorities. The majority of interventions did not target aspects of care that have been demonstrated to be disparate between minority and majority patient populations; instead most were generic quality improvement interventions targeted at providers of racial/ethnic minority patients.
The targeted processes of care were not always evidence-based practices for any patient population (for example, oral cavity exams or breast self-examinations for cancer screening) and would therefore be unlikely to improve the quality of care or reduce disparities for racial/ethnic minority patients.
Few studies measured patient outcomes; most measured healthcare process. This limitation would not be as significant if the studies had targeted processes of care that were more closely linked to patient outcomes (i.e., more evidence based).
Very few studies were completed in Hispanic populations and none in American Indians/Alaska Natives or in Asians/Pacific Islanders.
Each study used slightly different intervention methods, making generalizations across studies difficult.
Studies used multicomponent interventions and did not examine separate components.
There may have been studies that had data on racial/ethnic minorities that was not presented and therefore wouldn't have qualified for our review.
Most studies did not include data on costs.
Only interventions targeting providers/organizations were included; interventions directly targeting patients may also be promising strategies to improve the quality of care and reduce racial/ethnic disparities, but they are not reflected in this report.
Only randomized controlled trials and concurrent controlled trials were included; there may be other worthwhile interventions that have been evaluated with other study designs.
Eligibility was limited to studies conducted in the United States. There may have been other promising interventions conducted in other countries that are not reflected in this report.
We made no assessment of the generalizability of the study population of targeted providers in terms of whether they were representative of the population of providers caring for racial/ethnic minorities.
Most studies were designed without a comparison group for evaluation.
There were few standardized instruments for measuring cultural competence and very few outcome assessments were objectively measured. There were often no data in the articles concerning the psychometric properties of the instruments.
Many articles did not describe the curricular interventions well enough to ensure replication.
Each curricular intervention was different, making generalizability across studies difficult.
Few studies measured patient outcomes and none measured health status. Some studies included only curriculum evaluation as an outcome.
Most studies did not include data on costs.
We made no attempt to assess the psychometric properties of the instruments used to measure cultural competence.
Our review focuses on interventions aimed at the education of healthcare providers and therefore falls more narrowly into the provider education recommendation of the National Standards for Culturally and Linguistically Appropriate Services in Healthcare published by the Office of Minority Health.17
More research is needed that is designed specifically to reduce racial/ethnic disparities in healthcare quality, for example, research that targets healthcare processes known to be a source of racial/ethnic disparities.
It is necessary to distinguish between interventions aimed at improving the quality of care for all persons and those aimed at improving quality of care for racial/ethnic minority populations specifically (such as reducing provider bias).
More quality improvement interventions in racial/ethnic minority populations should be focused on priority conditions for which there are documented health disparities such as infant mortality, cardiovascular disease, diabetes mellitus, and HIV/AIDS.
When generic quality improvement interventions are done in mixed populations, subgroup analyses should evaluate the effect of the interventions in racial/ethnic minority patients, such that we understand the effect on equality of treatment.
Studies ought to include patient outcomes, have longer follow-up, and link process of care to health outcomes.
There is a need to replicate promising intervention strategies in different healthcare settings and organizations. For example, more studies are needed in acute care and specialty settings.
More studies are needed in Asian/Pacific Islander, American Indian/Alaska Native and Hispanic populations.
More information is needed about the costs of various strategies to improve healthcare quality and reduce racial disparity.
The literature is expanding rapidly, and updated evidence assessments will be needed soon.
Funding for this research is needed.
Curricular objectives need to be measurable and linked to measured outcomes.
Outcomes should be measured objectively.
There is a need for standardized, reliable, and valid instruments to measure aspects of cultural competence.
Studies evaluating the effect of cultural competence training need to have a pre- and post-intervention evaluation and/or comparison group; there is a need for more randomized controlled trials in this area.
Studies should measure the effect of the curricular interventions on healthcare process and patient outcomes including health status.
Researchers should comprehensively describe the curricular interventions.
Studies ought to include more comprehensive information about resources needed and cost of cultural competence training.
The literature is expanding rapidly, and updated evidence assessments will be needed soon.
Funding for this research is needed.
| Selected Quality Assessment Domains and Items | Number of Articles | |
|---|---|---|
| Yes | No | |
| Representativeness | ||
 - Healthcare providers clearly described? | 20 | 7 |
| Bias/confounding | ||
| - Adequate comparison group? | 26 | 1 |
| - Randomized? | 11 | 16 |
| Intervention description | ||
| - Complete description (able to replicate)? | 24 | 3 |
| Outcome assessment | ||
| - Blinding? | 9 | 18 |
| - Pre- and post-test? | 13 | 14 |
| - Objective evaluation? | 27 | 0 |
| Analysis | ||
| - Numbers and reasons of non-inclusion? | 18 | 9 |
| - Complete statistical analysis? | 24 | 3 |
| Selected Quality Assessment Domains and Items | Number of Articles | |
|---|---|---|
| Yes | No | |
| Representativeness | ||
| - Healthcare providers clearly described? | 21 | 43 |
| Bias/confounding | ||
| - Adequate comparison group? | 8 | 56 |
| - Randomized? | 1 | 63 |
| Intervention description | ||
| - Based on theoretical model? | 17 | 47 |
| - Complete description (able to replicate)? | 21 | 43 |
| Outcome assessment | ||
| - Blinding? | 3 | 61 |
| - Pre- and post-test? | 33 | 31 |
| - Objective evaluation? | 27 | 37 |
| Analysis | ||
| - Numbers and reasons of non-inclusion? | 14 | 50 |
| - Complete statistical analysis? | 15 | 49 |
Search Strategy for MEDLINE ®
((minority groups[mh] OR ethnic groups[mh] OR urban health[mh] OR urban population[mh] OR minority[tiab] OR urban[tiab] OR inner-city[tiab] OR black*[tiab] OR african american*[tiab] OR mexican*[tiab] OR native*[tiab] OR indian*[tiab] OR latina[tiab] OR latino[tiab]) AND (nurs*[tiab] OR physician*[tiab] OR health professional*[tiab] OR health care provider*[tiab] OR health personnel[mh]) AND (randomized controlled trial [pt] OR controlled clinical trial [pt] OR randomized controlled trials [mh] OR random allocation [mh] OR double-blind method [mh] OR single-blind method [mh]) NOT (animal [mh] NOT human [mh])) OR ((cultura*[tiab] OR multicultural[tiab] OR transcultural[tiab] OR divers*[tiab] OR cultural diversity[mh] OR transcultural nursing[mh] OR ethnic[tw] OR minority[tw]) AND (competen*[tiab] OR sensitiv*[tiab] OR attitude*[tiab] OR experience[tiab] OR knowledge[tiab]) AND (education[mh] OR ed[sh] OR educat*[tiab] OR train*[tiab] OR curriculum[tiab]) AND (nurs*[tiab] OR physician*[tiab] OR health professional*[tiab] OR health care provider*[tiab] OR student*[tiab])) AND eng[la] AND 1980:2003[dp] NOT review[pt]
Search Strategy for the Cochrane CENTRAL Register of Controlled Trials
(minority or ethnic or (inner next city) or cultural) (1980 to current date)
((health next professional) or physician or nurse) (1980 to current date)
1 and 2
Search Strategy for EMBASE
cultural.mp
“ETHNIC, RACIAL AND RELIGIOUS GROUPS”/ or ETHNIC DIFFERENCE/ or “ETHNIC OR RACIAL ASPECTS”/ or “ETHNIC AND RACIAL GROUPS”/ or ETHNIC GROUP/ or ethnic.mp.
MINORITY GROUP/ or minority.mp.
1 or 2 or 3
exp EDUCATION/ or education.mp.
4 and 5
health professional.mp. or Health Practitioner/
(physician or nurse).mp.
STUDENT/
7 or 8 or 9
6 and 10
cultural competence.mp
11 or 12
limit 13 to (human and english language)
Search Strategy for the Cumulative Index of Nursing and Alliance Health Literature (CINAHL)
((minority in TI,AB or ethnic in TI,AB or urban in TI,AB or inner-city in TI,AB or “Cultural-Diversity” / all TOPICAL SUBHEADINGS / all AGE SUBHEADINGS in DE or “Minority-Groups” /all TOPICAL SUBHEADINGS /all AGE SUBHEADINGS in DE or “Ethnic-Groups” /all TOPICAL SUBHEADINGS /all AGE SUBHEADINGS in DE or “Urban-Health” /all TOPICAL SUBHEADINGS /all AGE SUBHEADINGS in DE) and (“Health-Personnel” / all TOPICAL SUBHEADINGS / all AGE SUBHEADINGS in DE or health care provider* in TI,AB or nurs* in TI,AB or physician* in TI,AB or health professional* in TI,AB) and (LA:NU = ENGLISH) and (PY:NU >= 1980) AND (“clinical trials” /all TOPICAL SUBHEADINGS /all AGE SUBHEADINGS in DE)) or ((“Education-” / all TOPICAL SUBHEADINGS / all AGE SUBHEADINGS in DE or educat* in TI,AB or train* in TI,AB or curriculum in TI,AB ) and ( “Transcultural-Nursing” / all TOPICAL SUBHEADINGS / all AGE SUBHEADINGS in DE or “Cultural-Diversity” / all TOPICAL SUBHEADINGS / all AGE SUBHEADINGS in DE or “Cultural-Sensitivity” / all TOPICAL SUBHEADINGS / all AGE SUBHEADINGS in DE or “Cultural-Competence” / all TOPICAL SUBHEADINGS / all AGE SUBHEADINGS in DE or ((cultura* in TI,AB or divers* in TI,AB) and (sensitiv* in TI,AB or attitude in TI,AB or knowledge in TI,AB or competen* in TI,AB))) and (“Health-Personnel” / all TOPICAL SUBHEADINGS / all AGE SUBHEADINGS in DE or “health care provider*” in TI,AB or nurs* in TI,AB or physician* in TI,AB or “health professional*” in TI,AB ) and (LA:NU = ENGLISH) and (PY:NU >= 1980))
Search Strategy for the Specialized Register of Effective Practice and Organization of Care Cochrane Review Group (EPOC)
(minority OR ethnic OR cultural OR urban)
Search Strategy for the Research and Development Resource Base in Continuing Medical Education (RDRB/CME)
selected indexed and non-indexed fields
OR'd terms together
minority
cultural
racial
Journals Hand Searched
Academic Medicine
Archives of Pediatric and Adolescent Medicine
Ethnicity and Disease
Health Services Research
Journal of the American Medical Association
Journal of General Internal Medicine
Journal of Health Care for the Poor and Underserved
Journal of Transcultural Nursing
Medical Care
Milbank Quarterly
New England Journal of Medicine
Pediatrics
| First Name | Last Name | Affiliation |
|---|---|---|
| Partner | ||
| Helen | Wu | National Quality Forum |
| Internal Peer Reviewer - Johns Hopkins University | ||
| Martha | Hill | Johns Hopkins University School of Nursing |
| Technical Experts | ||
| Laura | Petersen | Houston VA Medical Center, Baylor College of Medicine |
| Kevin | Fiscella | Departments of Family and Community & Preventive Medicine, University of Rochester School of Medicine |
| Robert | Beltran | Latino Medical Policy Institute |
| Anne | Beal | Commonwealth Fund |
| Marsha | Lillie-Blanton | Kaiser Family Foundation |
| Peer Reviewers | ||
| Robert | Like | Center for Healthy Families and Cultural Diversity, Department of Family Medicine, UMDNJ - Robert Wood Johnson Medical School |
| Spero | Manson | American Indian and Alaska Native Programs, University of Colorado Health Sciences Center |
| Jeanette | South-Paul | University of Pittsburgh School of Medicine, Department of Family Medicine |
| Dawn | Clancy | Department of Medicine, Medical University of South Carolina, Center for Healthcare Research |
| Debra | Danoff | American Association of Medical Colleges |
| Denice | Cora-Bramble | Children's National Medical Center |
| Pamela | Dickson | The Robert Wood Johnson Foundation |
| Internal Experts | ||
| Debra | Roter | Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health |
| Haya | Rubin | Quality of Care Research, Johns Hopkins University |
| Jon | Samet | Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health |
Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]
[PubMed]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]
