Home > Characteristics of Studies with Poor...

PubMed Health. A service of the National Library of Medicine, National Institutes of Health.

Berkman ND, Sheridan SL, Donahue KE, et al. Health Literacy Interventions and Outcomes: An Updated Systematic Review. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Mar. (Evidence Reports/Technology Assessments, No. 199.)

  • This publication is provided for historical reference only and the information may be out of date.

This publication is provided for historical reference only and the information may be out of date.

Appendix ECharacteristics of Studies with Poor Internal Validity

To assess the quality (internal validity or risk of bias) of studies, we used predefined criteria based on those described in the AHRQ Methods Guide for Comparative Effectiveness Reviews (ratings: good, fair, poor).1 Elements of quality assessment for trials included, among others, the methods used for randomization, allocation concealment, and blinding; the similarity of compared groups at baseline; maintenance of comparable groups; overall and differential loss to followup; and the use of intention-to-treat analysis. We assessed observational studies based on the potential for selection bias (methods of selection of subjects and loss to followup), potential for measurement bias (equality, validity, and reliability of ascertainment of outcomes), adjustment for potential confounders, and statistical analysis.

In general terms, a “good” study has the least bias and results are considered to be valid. A “fair” study is susceptible to some bias but probably not sufficient to invalidate its results. The fair-quality category is likely to be broad, so studies with this rating will vary in their strengths and weaknesses. A “poor” rating indicates significant bias (stemming from, e.g., serious errors in design, analysis reporting large amounts of missing information, or discrepancies in reporting) that may invalidate the study's results.

To systematically rate studies, we designed and used a structured data abstraction form. Trained reviewers abstracted data from each study and assigned an initial quality rating. A second reviewer read each abstracted article, evaluated the accuracy, completeness, and consistency of the data abstraction, and independently rated the quality. If differences in quality ratings could not be resolved by discussion, a third senior reviewer was involved. The full research team met regularly during the article abstraction period to discuss global issues related to the data abstraction process. The following lists all the studies reviewed and rated as poor quality, with their design and primary reasons for the final rating.

StudyDesignPrimary Reasons for Poor-Quality Rating
Arozullah et al., 20062Cross-sectionalHigh potential for selection biases. A convenience sample with a low participation rate was used.
Bennett et al., 20063Retrospective cohortHigh potential for selection and confounding biases. A convenience sample with no power calculation was used and there was no controlling for confounding in the analysis.
Bickmore et al., 20094RCTHigh potential for selection and measurement bias. The process of randomization was inadequate, there was no allocation concealment, groups were not comparable at baseline, and there was inadequate controlling for confounding in the analysis.
Brock et al., 20075Uncontrolled experimental study (pre/post test)This study received a fair rating for immediate outcomes but a poor rating for follow-up outcomes. There was a high risk for selection and confounding bias at followup due to high likelihood that the groups were no longer comparable and inadequate controlling for potential confounders in the analysis.
Campbell et al., 20076Cross-sectionalHigh potential for confounding and selection biases. A convenience sample was used.
Carbone et al., 20067Cross-sectionalHigh potential for measurement bias. Outcome measures were poorly described and could not be considered valid and reliable.
Clarke et al., 20058Cross-sectionalHigh potential for selection bias. Reporting of measures and statistical methods was inadequate. Important potential confounders were not considered.
Conwell et al., 20039Cross-sectionalHigh risk for confounding bias: race, socioeconomic status, parental smoking status, behavioral status, or any other potential confounder, could be responsible for association between WRAT score and smoking status.
Cordasco et al., 200910RCTFalse inclusions and attrition-introduced selection bias and residual confounding that was not controlled for in analysis.
DeWalt et al., 200711Cross-sectionalHigh potential for selection and confounding biases. A convenience sample with no power calculation was used and there was no controlling for confounding in the analysis.
DeWalt et al., 200912Uncontrolled experimental study (pre/post test)High risk of measurement bias due to social desirability. There was also inadequate controlling for confounding in the analysis.
DeWalt et al., 200413Uncontrolled experimental study (pre/post test)High risk of measurement and confounding bias. The lack of a control group carries a significant risk that any improvement in clinical symptoms was due to a Hawthorne effect or the use of cointerventions.
Donelle et al., 200814Cross-sectionalLiteracy/numeracy groups very likely to be different and only age/gender controlled for as potential confounders. Furthermore, comprehension questions were nonvalidated and not clearly appropriate.
Drainoni et al., 200815Cross-sectionalHigh potential for measurement, selection, and confounding biases. Outcome measures were poorly described and could not be considered valid and reliable. A convenience sample with no power calculation was used and there was no controlling for confounding in the analysis.
Endres et al., 200416Cross-sectionalHigh potential for selection and confounding biases. A small convenience sample was used and there was no controlling for important potential confounders in the analysis.
Garcia-Retamero and Galesic, 200917Factorial RCTThis study received a fair rating for main effect but a poor rating for subgroup analyses, with no presentation of baseline characteristics by group. There was no control of potential confounders if participants exited, making selection and confounding major issues.
Garcia-Retamero and Galesic, 201018RCTLack of adequate reporting about study, unclear what the study design is for between-group comparisons, unclear sample size and baseline numeracy/graphical literacy. No control for confounding in between-group analyses and subgroup analyses (although not clear whether needed for main group analyses).
Gazmararian et al., 201019Nonrandomized trialNonrandomized trial with no baseline differences and no control for confounding. Additionally, the author stated that the trial was underpowered, but it is not clear for what difference/outcomes.
Ginde et al., 200820Cross-sectionalHigh potential for measurement and confounding biases. Outcome measures were poorly described and could not be considered valid and reliable. There was no controlling for important potential confounders in the analysis.
Ives et al., 200621Prospective cohortHigh potential for confounding bias. Bivariate analysis was used with no controlling for important potential confounders in the analysis.
Jones et al., 200722Cross-sectionalHigh potential for measurement, selection, and confounding biases. Outcome measures were poorly described and could not be considered valid and reliable. A convenience sample with no power calculation was used and there was no controlling for confounding in the analysis.
Juzych et al., 200823Cross-sectionalHigh potential for confounding bias. Bivariate analysis was used with no controlling for important potential confounders in the analysis.
Kalichman et al., 200524Uncontrolled experimental study (pre/post test)High risk of measurement and confounding bias due to social desirability and inadequate controlling for confounding in the analysis.
Kandula et al., 200925Cross-sectional; prospective cohortHigh potential for measurement bias. Outcome measures were poorly described and could not be considered valid and reliable.
Kleinpeter, 200326Cross-sectionalHigh potential for selection and confounding biases. A small convenience sample was used and there was no controlling for important potential confounders in the analysis.
Lincoln et al., 200827Cross-sectionalHigh potential for selection biases A small convenience sample was used and participation rate was low.
Mbaezue et al., 201028Cross-sectionalHigh potential for measurement and selection bias. Descriptive data in tables do not add to the total sample. A portion of the sample population that did not check its glucose was omitted, causing the multivariate model to be misspecified.
Morrow et al., 200629Cross-sectionalHigh potential for selection and confounding bias. Health outcome measure poorly described.
Muir et al., 200630Retrospective cohortHigh potential for confounding bias. Bivariate analysis was used with no controlling for important potential confounders in the analysis.
Ntri et al., 200931Uncontrolled experimental study (pre/post test)High potential for confounding and selection biases. There was no controlling for potential confounders in the analysis and no accounting for those lost to followup. A small convenience sample was used.
Persell et al., 200732Cross-sectionalHigh potential for confounding biases. There was no controlling for important potential confounders in the analysis.
Roth et al., 200533Cross-sectionalHigh potential for selection and confounding biases. A small convenience sample was used and there was no controlling for important potential confounders in the analysis.
Rutherford et al., 200634Cross-sectionalHigh potential for measurement and confounding biases. Outcome measures were poorly described and could not be considered valid and reliable. There was inadequate controlling for important potential confounders in the analysis.
Sanders et al., 200735Retrospective cohortHigh potential for measurement bias. Outcome measures were poorly described and could not be considered valid and reliable.
Sarkar et al., 200636Cross-sectionalHigh potential for confounding biases. A convenience sample was used and there was inadequate controlling for important potential confounders in the analysis.
Sentell et al., 200337Cross-sectionalHigh potential for measurement and confounding biases. The outcome was measured by a single-item, self-reported survey question and there was inadequate controlling for important potential confounders in the analysis because only the bivariate analyses were relevant to the outcome of interest for this report.
Shieh et al., 200938Cross-sectionalHigh potential for confounding and measurement bias. Inadequate control for confounding and the outcome measure could not be considered valid and reliable.
van Servellen et al., 2003 & 200539,40RCTHigh potential for measurement and confounding biases. Inadequate reporting. Important potential confounders and multiple comparisons were not considered in the analysis and the analysis was within not between groups.
Waldrop-Valverde et al., 200841Cross-sectionalHigh potential for measurement and selection biases. The sample was divided into literacy/cognition groups so the independent effect of literacy on adherence could not be determined.
Wallace et al., 200842Cross-sectionalHigh potential for confounding bias. Bivariate analysis was used with no controlling for important potential confounders in the analysis.
Wolf et al., 200443Cross-sectionalHigh potential for measurement and confounding biases. Outcome measures were poorly described and could not be considered valid and reliable. There was inadequate controlling for important potential confounders in the analysis.
Wolf et al., 200744Cross-sectionalHigh potential for measurement and confounding biases. Outcome measures were poorly described and could not be considered valid and reliable. There was inadequate controlling for important potential confounders in the analysis.

RCT= Randomized controlled Trial

References

1.
Agency for Healthcare Research and Quality. Methods reference guide for effectiveness and comparative effectiveness reviews, version 1.0. Rockville, MD: 2007. [Draft posted Oct. 2007] Available at: http:​//effectivehealthcare​.ahrq.gov/repFiles​/2007_10DraftMethodsGuide.pdf.
2.
Arozullah AM, Lee SY, Khan T, et al. The roles of low literacy and social support in predicting the preventability of hospital admission. J Gen Intern Med. 2006 Feb;21(2):140–5. [PMC free article: PMC1484663] [PubMed: 16336616]
3.
Bennett I, Switzer J, Aguirre A, et al. ‘Breaking it down’: patient-clinician communication and prenatal care among African American women of low and higher literacy. Ann Fam Med. 2006 Jul-Aug;4(4):334–40. [PMC free article: PMC1522153] [PubMed: 16868237]
4.
Bickmore TW, Pfeifer LM, Paasche-Orlow MK. Using computer agents to explain medical documents to patients with low health literacy. Patient Educ Couns. 2009 Jun;75(3):315–20. [PMC free article: PMC2692364] [PubMed: 19297116]
5.
Brock TP, Smith SR. Using digital videos displayed on personal digital assistants (PDAs) to enhance patient education in clinical settings. Int J Med Inform. 2007 Nov-Dec;76(11-12):829–35. [PubMed: 17113345]
6.
Campbell MJ, Edwards MJ, Ward KS, et al. Developing a parsimonious model for predicting completion of advance directives. J Nurs Scholarsh. 2007;39(2):165–71. [PubMed: 17535317]
7.
Carbone ET, Lennon KM, Torres MI, et al. Testing the feasibility of an interactive learning styles measure for U.S. Latino adults with type 2 diabetes and low literacy. Int Q Comm Health Educ. 2006;25(4):315–35. [PubMed: 17686705]
8.
Clarke C, Friedman SM, Shi K, et al. Emergency department discharge instructions comprehension and compliance study. Can J Emerg Med Care. 2005 Jan;7(1):5–11. [PubMed: 17355647]
9.
Conwell LS, O'Callaghan MJ, Andersen MJ, et al. Early adolescent smoking and a web of personal and social disadvantage. J Paediatr Child Health. 2003 Nov;39(8):580–5. [PubMed: 14629522]
10.
Cordasco KM, Asch SM, Bell DS, et al. A low-literacy medication education tool for safety-net hospital patients. Am J Prev Med. 2009 Dec;37(6 Suppl 1):S209–16. [PubMed: 19896021]
11.
DeWalt DA, Boone RS, Pignone MP. Literacy and its relationship with self-efficacy, trust, and participation in medical decision making. Am J Health Behav. 2007 Sep-Oct;31 1:S27–35. [PubMed: 17931133]
12.
Dewalt DA, Davis TC, Wallace AS, et al. Goal setting in diabetes self-management: taking the baby steps to success. Patient Educ Couns. 2009 Apr 7 [PMC free article: PMC4286314] [PubMed: 19359123]
13.
DeWalt DA, Pignone M, Malone R, et al. Development and pilot testing of a disease management program for low literacy patients with heart failure. Patient Educ Couns. 2004 Oct;55(1):78–86. [PubMed: 15476993]
14.
Donelle L, Arocha JF, Hoffman-Goetz L. Health literacy and numeracy: key factors in cancer risk comprehension. Chronic Dis Can. 2008;29(1):1–8. [PubMed: 19036218]
15.
Drainoni ML, Rajabiun S, Rumptz M, et al. Health literacy of HIV-positive individuals enrolled in an outreach intervention: results of a cross-site analysis. J Health Commun. 2008 Apr-May;13(3):287–302. [PubMed: 18569359]
16.
Endres LK, Sharp LK, Haney E, et al. Health literacy and pregnancy preparedness in pregestational diabetes. Diabetes Care. 2004 Feb;27(2):331–4. [PubMed: 14747209]
17.
Garcia-Retamero R, Galesic M. Communicating treatment risk reduction to people with low numeracy skills: a cross-cultural comparison. Am J Public Health. 2009 Dec;99(12):2196–202. [PMC free article: PMC2775772] [PubMed: 19833983]
18.
Garcia-Retamero R, Galesic M. Who profits from visual aids: overcoming challenges in people's understanding of risks. Soc Sci Med. 2010 Apr;70(7):1019–25. [corrected] [PubMed: 20116159]
19.
Gazmararian J, Jacobson KL, Pan Y, et al. Effect of a pharmacy-based health literacy intervention and patient characteristics on medication refill adherence in an urban health system. Ann Pharmacother. 2010 Jan;44(1):80–7. [PubMed: 20028960]
20.
Ginde AA, Weiner SG, Pallin DJ, et al. Multicenter study of limited health literacy in emergency department patients. Acad Emerg Med. 2008 Jun;15(6):577–80. [PubMed: 18616448]
21.
Ives TJ, Chelminski PR, Hammett-Stabler CA, et al. Predictors of opioid misuse in patients with chronic pain: a prospective cohort study. BMC Health Serv Res. 2006;6:46. [PMC free article: PMC1513222] [PubMed: 16595013]
22.
Jones M, Lee JY, Rozier RG. Oral health literacy among adult patients seeking dental care. J Am Dent Assoc. 2007 Sep;138(9):1199–208. quiz 266-7. [PubMed: 17785385]
23.
Juzych MS, Randhawa S, Shukairy A, et al. Functional health literacy in patients with glaucoma in urban settings. Arch Ophthalmol. 2008 May;126(5):718–24. [PubMed: 18474786]
24.
Kalichman SC, Cherry J, Cain D. Nurse-delivered antiretroviral treatment adherence intervention for people with low literacy skills and living with HIV/AIDS. J Assoc Nurses AIDS Care. 2005 Sep-Oct;16(5):3–15. [PubMed: 16433105]
25.
Kandula NR, Nsiah-Kumi PA, Makoul G, et al. The relationship between health literacy and knowledge improvement after a multimedia type 2 diabetes education program. Patient Educ Couns. 2009 Jun;75(3):321–7. [PubMed: 19395223]
26.
Kleinpeter MA. Health literacy affects peritoneal dialysis performance and outcomes. Adv Perit Dial. 2003;19:115–9. [PubMed: 14763046]
27.
Lincoln A, Espejo D, Johnson P, et al. Limited literacy and psychiatric disorders among users of an urban safety-net hospital's mental health outpatient clinic. J Nerv Ment Dis. 2008 Sep;196(9):687–93. [PubMed: 18791430]
28.
Mbaezue N, Mayberry R, Gazmararian J, et al. The impact of health literacy on self-monitoring of blood glucose in patients with diabetes receiving care in an inner-city hospital. J Natl Med Assoc. 2010 Jan;102(1):5–9. [PMC free article: PMC2938786] [PubMed: 20158130]
29.
Morrow D, Clark D, Tu W, et al. Correlates of health literacy in patients with chronic heart failure. Gerontologist. 2006 Oct;46(5):669–76. [PubMed: 17050758]
30.
Muir KW, Santiago-Turla C, Stinnett SS, et al. Health literacy and adherence to glaucoma therapy. Am J Ophthalmol. 2006 Aug;142(2):223–6. [PubMed: 16876500]
31.
Ntiri DW, Stewart M. Transformative learning intervention: effect on functional health literacy and diabetes knowledge in older African Americans. Gerontol Geriatr Educ. 2009;30(2):100–13. [PubMed: 19440898]
32.
Persell SD, Osborn CY, Richard R, et al. Limited health literacy is a barrier to medication reconciliation in ambulatory care. J Gen Intern Med. 2007 Nov;22(11):1523–6. [PMC free article: PMC2219798] [PubMed: 17786521]
33.
Roth MT, Ivey JL. Self-reported medication use in community-residing older adults: a pilot study. Am J Geriatr Pharmacother. 2005 Sep;3(3):196–204. [PubMed: 16257822]
34.
Rutherford J, Holman R, MacDonald J, et al. Low literacy: a hidden problem in family planning clinics. J Fam Plann Reprod Health Care. 2006 Oct;32(4):235–40. [PubMed: 17032511]
35.
Sanders LM, Thompson VT, Wilkinson JD. Caregiver health literacy and the use of child health services. Pediatrics. 2007 Jan;119(1):e86–92. [PubMed: 17200263]
36.
Sarkar U, Fisher L, Schillinger D. Is self-efficacy associated with diabetes self-management across race/ethnicity and health literacy? Diabetes Care. 2006 Apr;29(4):823–9. [PubMed: 16567822]
37.
Sentell TL, Shumway MA. Low literacy and mental illness in a nationally representative sample. J Nerv Ment Dis. 2003 Aug;191(8):549–52. [PubMed: 12972860]
38.
Shieh C, Mays R, McDaniel A, et al. Health literacy and its association with the use of information sources with barriers to information seeking in clinic-based pregnant women. Health Care Women Int. 2009;30(11):971–88. [PubMed: 19809901]
39.
van Servellen G, Carpio F, Lopez M, et al. Program to enhance health literacy and treatment adherence in low-income HIV-infected Latino men and women. Aids Patient Care STDS. 2003 Nov;17(11):581–94. [PubMed: 14746666]
40.
van Servellen G, Nyamathi A, Carpio F, et al. Effects of a treatment adherence enhancement program on health literacy, patient-provider relationships, and adherence to HAART among low-income HIV-positive Spanish-speaking Latinos. Aids Patient Care STDS. 2005 Nov;19(11):745–59. [PubMed: 16283835]
41.
Waldrop-Valverde D, Jones DL, Weiss S, et al. The effects of low literacy and cognitive impairment on medication adherence in HIV-positive injecting drug users. AIDS Care. 2008 Nov;20(10):1202–10. [PubMed: 19012081]
42.
Wallace LS, Rogers ES, Weiss BD. Relationship between health literacy and health-related quality of life among Tennesseans. Tenn Med. 2008 May;101(5):35–9. [PubMed: 18507259]
43.
Wolf MS, Davis TC, Cross JT, et al. Health literacy and patient knowledge in a Southern US HIV clinic. Int J STD AIDS. 2004 Nov;15(11):747–52. [PubMed: 15537462]
44.
Wolf MS, Williams MV, Parker RM, et al. Patients' shame and attitudes toward discussing the results of literacy screening. J Health Commun. 2007 Dec;12(8):721–32. [PubMed: 18030638]
Cover of Health Literacy Interventions and Outcomes: An Updated Systematic Review
Health Literacy Interventions and Outcomes: An Updated Systematic Review.
Evidence Reports/Technology Assessments, No. 199.
Berkman ND, Sheridan SL, Donahue KE, et al.

PubMed Health Blog...

read all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...