Evidence Table 7Quality assessment of observational studies in patients with schizophrenia

Author, yearNon-biased selection?Low overall loss to follow-up?Outcomes pre-specified and defined?Ascertainment techniques adequately described?Non-biased and adequate ascertainment methods?Statistical analysis of potential confounders?Adequate duration of follow-up?Overall quality assessmentComments
Advokat, 2004No, excluded patients with incomplete dataNo withdrawals reportedYesYesNo, ratings probably unblinded because performed by psychologists/psychiatrists on staff at hospitalNo and only baseline demographic data reported; unclear if differences in prognostic factorsYesPoor
Advokat, 2004Yes for overall group; but unclear for subset for which length of stay was determined, which was only those who were discharged during study period and N was NRUnclear; implied that length of stay not available for all patients, but N=NRYes for some, no for length of stay.NoUnclearNo and there were differences between groups in rates of patents taking concomitant typical AP’s : olanzapine= 57%, risperidone=38%, quetiapine = 64%, and clozapine = 14%No; ≥ 3 monthsPoor
Agelink, 2001Method NR, unable to determine.Yes (9%)YesYesYesYesYesFair
Akkaya, 2007YesN/A: retrospective analysis excluded 32.7% of pts with an initial admission and dx but no follow-up visitYesYesPossible missing data inherent in chart review - AEs not gathered uniformly - but direction of potential bias is unknown.Yes; bivariate comparisonsN/AFair
Alvarez, 1997
No: AE withdrawals during first 3 weeks not includedNRYesYesYesNRYesFair
Al-Zakwani, 2003No, excluded patients who had a behavioral health benefit carve-out and those who were not continuously enrolled for 18 monthsNo withdrawals reported.YesYesNRYesYesFair
Ascher-Svanum, 2004
US-SCAP Study Interim Results
Not entirely clear. Broad range of patients enrolled, with few exclusion criteria but method of obtaining participants not described well enough to determine. Also, for this sub-study, patients discontinuing treatment prior to 1 year were excluded.NoneYesYesNo. Data extracted from medical records. Methods not described (e.g. blinding, validation).YesYesFair
Ascher-Svanum, 2008YesYesYesYesYesYesYesFair
Atkin, 1996
Barak, 2004No, excluded patients without treatment chartsYes (retrospective study)YesYesUnclear if database/patient chart reviewer was blind to suicide statusNo; only commented regarding similarities in gender, age, distribution of diagnosesUnclearFair
Beauclair, 2007YesYesNRYesYesYesYesFair
Bobes, 2003bUnclear if the inception cohort (n=901) represented ALL patients hospitalized for an acute psychotic episode during the specified time period; unclear how sample narrowed down to 158Unclear for the process of narrowing the sample from 901 to 158; low for LTFU among the 158YesYesUnclear if the person(s) that administered the instruments were blindedPartial; only covariates were baseline score and years since diagnosisYesPoor
Bond, 2004No, excluded patients: (1) didn’t express goal of employment; (2) were noncompliant with medications; (3) didn’t complete baseline interview; (4) discontinued early; (5) switched medications during the studyWithdrawals not reportedYesYesUnclear; no information about how the Vocational Placement Scale was administeredNo; only attempted adjustment for the few baseline differences in concomitant medication use, indicated adjustment didn’t materially change the results, so presented unadjusted resultsYesPoor
Brown, 2005No, excluded people who died during follow-upThere was differential loss to F/U
Loss to F/U reported as 6/88 (6.8%) for ziprasidone; 27/103 (26%) for olanzapine
YesYesUnclear; chart review not duplicatedNoUnclearPoor: no adjusting for confounders; F/U interval unclearretrospective, 2-group cohort
Buckman, 1999
United States
Caro, 2002
YesNRYesYesYesYesYesFairBetween-group differences in age, gender, other characteristics
Castro, 2007Yes; see comment.Yes; length of followup was significantly higher with clozapine than haloperidol or risperidoneYesYesYesYesYesFairAuthors note that patients may differ between treatment groups in their level of treatment resistance and disease severity
Castro 2007UnclearYesYesUnclearUnclearSomeYesPoor
Chen, 2008YesNRYesYesYesYesYesFairIt is not clear what % of patients included may have lost MediCal eligibility and were therefore lost to follow-up
Citrome, 2004Unclear
Lower % of males in case group vs. control
Risk factors of BMI and activity level not assessed or controlled for. No assessment of baseline risk for diabetes and how that may have influenced choice of antipsychotic medication
Conley, 1999
United States
Cooper, 2005
Cooper, 2007
Unclear: groups differed but did adjustNA (retrospective study including persons with available data only)YesYesYes; database tested for accuracyYesYes, 365-day study periodFairretrospective, 2-group cohort in pub #1
4 drugs compared in pub #2
Coulter, 2001
de Haan, 1999YesYes (retrospective study)No; not definedNoNoNo; only commented regarding between-groups comparability for sex, age at admission and diagnosisYesPoor
de Haan, 2002No; excluded 15 (6.2%) due to noncompliance and crossoverWithdrawals NRyesYesNo; raters were unblindedNo; there was no information about between-groups comparability of baseline characteristicsYesPoor
De Hert 2008Unclear;
Historical cohort: Consisted of only 148/301 (49%) of patients with complete laboratory data. But, no significant differences between patients with and without complete laboratory data.
Current cohort: No details provided on matching process. Significantly higher glucose in historic cohort (89 vs 84 mg/dl (P= 0.0055).
No; analysis excluded 22%
overall (historic=21% vs current=37%)
YesYesYes in “current” cohort of second-generation antipsychotics; unclear in historical cohort due to use of conversion factor for missing waist circumference measurementsNo and no information reported about comparability of baseline characteristics between groups of patients based on individual atypical antipsychotic agentYesPoor
Deliliers, 2000
Devinsky, 1991
United States
Dinakar, 2002Method NR, unable to determine.YesYesYesNot reported if blind or independent assessment of outcomes.NoYesPoor-no control for confounding factors, not reported if outcome assessors blinded or independent, unable to determine if selection was unbiased.
Dolder, 2002YesNA (pharmacy database with all records available)YesYesYesNo, although baseline groups were similar for known confoundersYes; 12 monthsFair2-group cohort study; appears to be retrospective
Drew, 2002
Eberhard, 2006NA (single-group study)No (completers 166/223)YesYesYes (validated rating scale for TD)NA (single-group study)Yes: 5 yearsFairthis is an observational study of AE only (not efficacy); single-group cohort
Etminan, 2003
NoNRYesYesYesYesNRPoorDiabetic events NR for 266 patients (reason NR)
Faries, 2008No
Data on 28% of patients who switched from risperidone to an antipsychotic other than olanzapine were not included
YesUnclear if outcomes were prespecified for this study at start of RCTYes“Responder day” measure of symptom severity at time of medication switch assumes linear changeYesYesFair
Feldman, 2004
Buse, 2003
No-only included patients who maintained coverage with AdvancePCS were followed-those who discontinued coverage not analyzed; also excluded those missing information on sex or year of birth.Yes (for those maintaining coverage)YesYesNot reported if independent assessment of outcomes (but outcome was new prescription, so may be objective)YesYesFair
Fuller, 2003YesNRYesNoYesYesYesFair
Ganguli, 2001Yes-consecutive patientsNot reportedYesYesNot reported if independent assessment of outcomes (outcome was weight gain from chart review, objective, but several sources used, and judgment made about which of multiple weights recorded to use)NoYes (4 months)Fair
Gianfrancesco, 2002
United States
Gianfrancesco, 2003a
United States
Gianfrancesco, 2003b
United States
Gianfrancesco, 2006YesNoneYesYesYesSomeYesFair
Gianfrancesco, 2006 (Hospitalization Risks in the Treatment of Schizophrenia)YesNA (retrospective; only patients with data were analyzed)YesYesUnclear, don’t know reliability of the databaseYesUnclear; mean treatment episode duration NRFair
Gibson, 2004Unclear: groups differed but did adjustNA (retrospective study including persons with available data only)YesYes, from Medicaid dataUnclear, don’t know reliability of the databaseNo, there were many baseline differences, but clinical significance of the differences was unclearYes, 1 yearFairretrospective, 3-group cohort
Gomez, 2000
Estudio Farmacoepidemiologico en esquizofrenia con Olanzapine (EFESO)
Hagg, 1998
YesNRYesYesYesNoN/A, cross-sectional studyFair
Haro, 2008YesNo
58.2% included
Haukka 2008YesYes (retrospective study)YesYesYesYesYesGood
Hedenmalm, 2002YesYes (retrospective study)YesYesNot stated if blinded or independent assessment of outcomesNoYesFair
Henderson, 2000
United States
Henderson, 2005Unclear; only information about sampling frame was observation periodNA (retrospective; only patients with data were analyzed)YesYesUnclear, don’t know reliability of the research psychiatrist in determining cause of death from autopsy reports and medical recordsNA (single-group study)Yes, 10 yearsPoor
Hennessy, 2002Not clearYes (retrospective study)YesYesNot reported if independent assessment of outcomesYesYesFair
Herceg 2008Not clearYes (retrospective study)YesNot clearNot clearSomeYesFair
Ho, 1999UnclearNoYesYes for group in the Longitudinal Study of Recent-Onset Psychosis, No for othersunclear, blinding NRPartially, ANCOVA analysis was done to assess impact of differences at baseline in EPS, GAS, and QOL measures but other confounders not assessed.YesPoor
Hodgson, 2005Unclear: groups differed but did adjustNA (retrospective study including persons with available data only)YesYes, from pharmacy recordsUnclearYesUnclear: study interval 1994–2001 but unclear if all three groups had same median observation periodFairretrospective, 3-group cohort
Honigfeld, 1996
United States
Hrdlicka 2009Unclear; eligibility required “medical record quality sufficient to evaluation the patient” and no information reported on comparison between patients with and without “sufficient record quality”No; 57/109 (52%) did not complete the 6-week study periodYesYesYesNoNo - 6 weeksPoor
Javitt, 2002Unclear; indicates that data was obtained but doesn’t indicate howNo loss to follow-upYesNoNoYesYesFair
Jerrell, 2007NA (single-group study)NA (retrospective; only patients with data were analyzed)YesYesYesNA (single-group study)Unclear (F/U 3 years); for vascular outcomes longer F/U would be more usefulFairthis is an observational study of AE only (not efficacy); single-group cohort (retrospective)
Jeste, 1999
United States
YesNRYesYesYesPartial: univariate regressions for baseline scores, age race, education, neuroleptic type, and daily dose on risk of TD. Subjects were matched for age, diagnosis, and length of neuroleptic exposure at study entry.YesFair
Joyce, 2005No, multiple exclusions applied depending on data most available.NoneYesYesYesNoYesPoor
Kane, 1993
United States
NoNRYesYesYesNo and there were nonsignificantly more females (38% vs 24%) and schizoaffective patients (17% vs 8%) in control group and clozapine-treated patients were significantly older (32.4 vs 26.4 years) and had significantly longer exposure to neuroleptics at baseline (6.4 vs 2.3 years),YesPoorBetween group differences in gender and diagnosis
Karagianis, 2009YesYesYesYesNo
Interrater reliability not assessed. Open label - possible rater bias
YesYesFairMore than half of included patients were using more than 1 antipsychotic medication concurrently
Kasper, 2001No; selected patients in reverse chronological order with 33 from each center; also only included data from centers that completed data collection and verification by a certain dateYes (retrospective study)YesNoUnclear; blinding NRYesYesFair
Kilzieh, 2008YesYesYesYesYesYesYesGood
Kim, 2008 (Effectiveness…)YesNot reportedYesYesYesNo analysis of treatment visit frequency as a potential confounder. Frequency for RLAI group was every 2 weeks; oral was monthlyYesFair
Kim, 2008 (Time…)YesYesYesYesInterrater reliability unclearYesYesFair
Koller, 2003YesYesYesYesNot reported if independent assessment of outcomes.No-descriptive summary statistics only.YesFair
Kopala, 2005UnclearNo (49% drop-out at 2 years)yesYesYesNoYesPoor
Koro, 2002aYesYes (retrospective study)YesYesNot reported if independent assessment of outcomesYesYes (3 at least months)Fair
Koro, 2002bYesYesYesYesNot reported if independent assessment of outcomes.YesYes (mean 5.2 years)Fair
Kozma, 2004 (poster)
United States
Kraus, 1999YesNot reportedYesYesNot reported if independent assessment of outcomes (but outcome was weight, so may be objective)No4 weeks-not surePoor: unclear if all patients analyzed at all time points (no info on dropouts), no control for confounding factors.
Lambert, 2005Yes; baseline data similar between groupsNA (retrospective; only patients with data were analyzed)YesYesUnclear: 2 authors examined charts without blinding, but did have high inter-rater reliabilityNo, although baseline groups were similar for known confoundersYes, 18 monthsFairTwo-group cohort; retrospective
Lambert, 2006YesNoneYesYesYesYesYesGood
Lambert, 2005No, excluded patients that were not continuously eligible for Medi-Cal benefitsYes: 5.4% at 24 weeks, 20.1% at 52 weeksYesYesYesNoYesPoor
Lee, 2002
United States
YesNRYesYesYesPartial: Adjusted for age, sex, geographic region, diagnosis, hypertension, heart disease, and length of AP therapy. Did not adjust for dose.YesFair79% of patients were only prescribed the index antipsychotic during the study period.
Leslie, 2004Not clearYes (retrospective study)YesNoNot reported if blind or independent assessment of outcomes.NoYes? (3 months)Poor-No control for confounding factors, not reported if outcome assessor blinded, definition of outcomes and ascertainment techniques not adequately described, unable to determine if selection was unbiased.
Lieberman, 1992
Alvir 1993
United States
Lin, 2006YesUnclearYesYesUnclear; 2 senior psychiatrists (first and second authors) verified data but no information provided about inter-rater reliability or overall reliabilityYesYesFair
Lindstrom, 1989NA (single-group study)Yes (attrition 3/96)YesNoUnclearNA (single-group study)Yes, 13 yearsFair-poorSingle-group cohort, retrospective; unclear how outcomes were ascertained
Lindstrom, 2007YesYesYesNoUnclearPartialYesPoor
Lublin, 2003YesNoneYesNoUnclearNo12 weeksPoor
Lucey, 2003Unclear. 396 patients charts reviewed, but selection of these not statedYes (retrospective study)YesYesYesPartially, analysis took into account mean dose and center.Yes, for the outcome measure of time to dischargeFair
Lund, 2001
United States
Mladsi 2004
McIntyre, 2003
Canadian National Outcomes Measurement Study in Schizophrenia (CNOMSS)
Medved 2009UnclearYesYes for metabolic features; no for metabolic syndromeYesYesYes for age or duration of illness; higher baseline triglyceride levels for olanzapine (1.91 vs 1.41 mmol; P= 0.017), but none of the clinical features tested as predictors in logistic regression on metabolic syndrome before SGA admission was significant.No-3 monthsFair
Meyer, 2002No-excluded patients with incomplete dataYes (retrospective study)YesYesNot reported if independent assessment of outcomesNoYes (one year)Poor-may be biased selection, independent outcome assessment not reported, no control for potential confounding factors.
Miller, 1998Not clear-identified patients from chart review.YesYesYesYes-blinded assessment of EPSYesYes, but time period on medications varied (45.3 months clozapine, 13.4 months risperidone, 92.5 months conventional antipsychotics)Fair
Modai, 2000
Mohamed, 2009YesNRYesYesYesPartialYesFair
Moisan, 2005YesNoneYesYesYesYes6 monthsGood
Montes, 2003
Sub-group Analysis from EFESO
Mullins, 2008YesYesYesYesUnclearPartialYesFair
Naber, 2001Method NR, unable to determine.No (4% missing SWN data, 3% missing PANSS data)YesYesNot blindedYesYesFair
Ollendorf, 2004
United States
Opolka, 2003Unclear: groups differed but did adjustNA (retrospective study including persons with available data only)YesYesUnclear, don’t know reliability of the databaseYesYes, 1 yearFairretrospective, 3-group cohort
Ostbye, 2004
United States
YesNRYesYesYesPartial: does not control for dose and duration of treatmentYesPoor
Peacock, 1996
NoNRNoNoNot clearNRYesPoor
Pelagotti, 2004YesNoneYesNoUnclearNoMinimal (4–7 months) for Primary outcome 72 months for secondary outcomesPoor
Perez 2008Unclear; groups differed but did adjust (e.g., quetiapine group had significantly greater proportions of comorbid mood disorders, previous hospitalizations, lower proportions of first episode status, and higher mean Calgary Depression Scale (CDSS) scores)No; 50% for quetiapine and 42% for risperidoneYesYesYesAdjusted means analysis using ANCOVA performed for efficacy outcomes (i.e., adjusted for unspecified clinical relevant and unbalanced baseline variables); no adjustment for weight gain or rehospitalization, but neither demonstrated a significantly significant differenceYesPoor
Peuskens 2009Unclear; some differences in baseline disease history, e.g., lower proportion of “first antipsychotic prescription” with olanzapine than risperidone (18% vs 30%)No, 33% in olanzapine group and 29% in risperidone group did not complete the studyYesNoUnclear whether weight was self-reported or measured and whether outcome assessor was blindedNoYesPoor
Phillippe, 2005YesNo, n = 3470 at enrollment, n = 1574 at analysisNot clearlySurveyNot clearYesYesFair
Procyshyn, 1998YesNone (retrospective)YesNoNo; method of determining classification as “responder” from physician note NR; blinding of chart reviewer NRNoYesFair
Ray 2009YesYesYesNo; who ascertained NRUnclear; use of blinded, independent assessment NR; reliability of assessments NRYesYesFair
Rascati, 2003Yes, Used instrumental variables to adjust for differencesNA (retrospective study including persons with available data only)YesYesUnclear, don’t know reliability of the databaseYes, used instrumental variablesYes, 365-day study periodGoodretrospective, 2- group cohort
Reid, 1998
United States
Remington, 2001UnclearNoneYesNoNoNoYesPoor
Ren, 2006Unclear: groups differed but did adjustNA (retrospective study including persons with available data only)YesYesUnclear, don’t know reliability of the databaseYesYes, 6-monthFairretrospective, 2- group cohort
Rettienbacher, 2006UnclearUnclearYesNoNoNoUnclearPoor
Sax, 1998Method NR, unable to determine.NoYesYesNot reported if blind or independent assessment of outcomes.NoYesPoor- no control for confounding factors, not reported if outcome assessors blinded or independent, unable to determine if selection was unbiased.
Schillevoort, 2001aYesYesYesYesNot reported (outcome assessor not specified)YesYesFair
Schillevoort, 2001bYesYes (retrospective study)YesYesNot reported if blind or independent assessment of outcomes.YesYesFair
Sernyak, 2002YesYesYesYesNot reported (outcome assessor not specified)YesNot sure- 4-month period studied.Fair
Shajahan, 2009YesN/A: Retrospective chart reviewYesYesProbably OK. Investigators assigned CGI scores retrospectively based on medical record notes. Author states the validity of this method has been previously established.YesYesFair
Sharif, 2000YesNone (retrospective)YesNo information about the method the research assistant used to “assess symptom domain response” when reviewing the chartsNo; after filling out structured rating forms during chart review, same unblinded research assistant blacked out identifying in formation, randomly assigned “X” or “O” to the blacked out forms and gave to research psychiatrists for interpretationNoYesPoor
Snaterse, 2000Unclear if chart review included ALL potential patients during the specified time periodNone (retrospective)YesNoUnclear; blinding NRYes; but no demographicsYesFair
Spivak, 1998
Strassnig, 2007YesNoneYesYesYesSomeYesFair
Strous, 2006Unclear; referrals from treating physicians and sampling frame time period NRNoneYesYesUnclear, details about weight measurement methods NRSomeNo - 12 weeksFair
Su, 2005Not clearUnclear - only states that 15 completed the studyNot clearYesUnclearNo3 monthsPoor
Sumiyoshi 2004Unclear; “on randomly assigned days, all patients who visited the mental health center were contacted” and ultimately, “clinical data were obtained from 116 subjects meeting the study criteria”YesYesYesYesYes for length of treatment, gender, age and raceYesFair
Swanson, 2004Unclear: groups differed but did adjust75% retention both groups over 3 years; unclear if varied between groupsYesYesYes; had multiple ascertainment methodsYesYes (3 years)FairProspective, 2-group cohort
Tadger 2008Unclear; selection methods NRYes; 4/70 excluded from analysis of increase/decrease in BMI from risperidone/olanzapine groupsNoNoUnclearNoYesPoor
Taylor, 2003Unclear if sample of charts that were reviewed represent those of ALL potentially eligible charts; also excluded 2 charts with inadequate dosing informationNone (retrospective)YesNo description of how “documented positive statement of treatment effectiveness” was definedNo, efficacy outcome very subjective and blinding NRYesYesFair
Taylor, 2005UnclearYesYesYesNoNoNo - 6 monthsPoor
Taylor, 2008YesN/A: Retrospective chart reviewYesYesYesBivariate onlyYesFairUnclear whether a patient that switched AAPs would occur multiple times in the analysis, potentially contributing discontinuation data to more than one drug.
Taylor, 2009YesYesYesYesYesInsufficient. Matched on age and gender, but was not able to adjust for smoking; there were 3 lung cancer deaths in clozapine.YesFairUnclear how meaningful the mortality difference is. In risperidone there were only 3 deaths (ages 45, 65, 81), so the 95%CI’s for observed and expected mortality were large and overlapped with the clozapine mortality estimates.
Tilhonen, 2006YesNoneYesYesYesYesYesGood
Tiihonen 2009YesYesYesYesYesYesYesGood
Umbricht, 1994
United States
Van Winkel 2008YesYesYesYesYesNo, and BMI was significantly greater for aripiprazole than olanzapine (28.4 vs 23.5 kg/m2; P< 0.05)No - 3 monthsPoor
Verma, 2001NoYesYesYesNo, unblinded ratersNoUnclear, follow-up ended at discharge, but mean duration of inpatient stay not reportedPoor
Voruganti, 2000No, convenience sample probably does not represent all of the patients among the 600 that would meet inclusion criteriaNo withdrawals reported.NoYesYesNo, and there were baseline differences in disease severity (clozapine patients were sicker)YesPoor
Wang, 2002
Yesn/aYesYesYesYesN/A (case-control)Fair
Weiser, 2000Yes ("recruited randomly")No withdrawals reported.YesYesNo- raters of ESRS not blinded; other assessments computerizedControlled for age only.YesFair
Wirshing, 2002No- included only records with adequate laboratory data, and excluded those with a lack of compliance (excluded 63.6% of charts reviewed).Yes (retrospective study)YesYesNot stated if blinded or independent assessment of outcomes (but lab test, may be objective)YesYes (tests within 2 1/2 years included)Fair
Yood 2009YesYes (retrospective study)YesUnclearUnclearYesYesFair
Yu 2008YesYesYesYesUnclearYes - propensity score matchingYesFair
Yu, 2009YesN/A: Subjects were selected on minimum 1-year enrollment after prescription dateYesYesYesYesYes; followup fixed at 12 months by designGood
Zhao, 2002Unclear: groups differed but did adjustNA (retrospective study including persons with available data only)YesYesUnclear, don't know reliability of the databaseYesYes, 1 yearFairretrospective, 2- group cohort
Zhao, 2002YesNo withdrawals reportedNoYesNoYesYesFair
Zhang 2007Yes, recruited randomlyYesNo; no specification of primary outcome variable or whether both endpoint BMI and BMI gain were pre-plannedYesYesUnclear; states "where there was a significance in ANOVA, the effect of age, sex, duration of illness and neuroleptic dose were tested by adding these variables to the analysis model as co-variate", but no mention of results of these tests of co-variate regarding impact on significance of difference in BMI and BMI change between clozapine and risperidoneYesPoor

From: Evidence Tables

Cover of Drug Class Review: Atypical Antipsychotic Drugs
Drug Class Review: Atypical Antipsychotic Drugs: Final Update 3 Report [Internet].
McDonagh M, Peterson K, Carson S, et al.
Portland (OR): Oregon Health & Science University; 2010 Jul.
Copyright © 2010 by Oregon Health & Science University, Portland, Oregon 97239. All rights reserved.

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.