Format

Send to

Choose Destination
See comment in PubMed Commons below
Med Educ. 2004 Sep;38(9):1006-12.

Reliability: on the reproducibility of assessment data.

Author information

1
Department of Medical Education, College of Medicine, University of Illinois at Chicago, 808 South Wood Street, Chicago, IL 60612-7309, USA. sdowning@uic.edu

Abstract

CONTEXT:

All assessment data, like other scientific experimental data, must be reproducible in order to be meaningfully interpreted.

PURPOSE:

The purpose of this paper is to discuss applications of reliability to the most common assessment methods in medical education. Typical methods of estimating reliability are discussed intuitively and non-mathematically.

SUMMARY:

Reliability refers to the consistency of assessment outcomes. The exact type of consistency of greatest interest depends on the type of assessment, its purpose and the consequential use of the data. Written tests of cognitive achievement look to internal test consistency, using estimation methods derived from the test-retest design. Rater-based assessment data, such as ratings of clinical performance on the wards, require interrater consistency or agreement. Objective structured clinical examinations, simulated patient examinations and other performance-type assessments generally require generalisability theory analysis to account for various sources of measurement error in complex designs and to estimate the consistency of the generalisations to a universe or domain of skills.

CONCLUSIONS:

Reliability is a major source of validity evidence for assessments. Low reliability indicates that large variations in scores can be expected upon retesting. Inconsistent assessment scores are difficult or impossible to interpret meaningfully and thus reduce validity evidence. Reliability coefficients allow the quantification and estimation of the random errors of measurement in assessments, such that overall assessment can be improved.

[Indexed for MEDLINE]
PubMed Commons home

PubMed Commons

0 comments
How to join PubMed Commons

    Supplemental Content

    Full text links

    Icon for Wiley
    Loading ...
    Support Center