Format

Send to

Choose Destination
Med Educ. 2007 Jan;41(1):32-8.

Conditional reliability of admissions interview ratings: extreme ratings are the most informative.

Author information

1
Department of Medical Education, University of Michigan, Ann Arbor, Michigan 48109, USA. rbent@umich.edu

Abstract

CONTEXT:

Admissions interviews are unreliable and have poor predictive validity, yet are the sole measures of non-cognitive skills used by most medical school admissions departments. The low reliability may be due in part to variation in conditional reliability across the rating scale.

OBJECTIVES:

To describe an empirically derived estimate of conditional reliability and use it to improve the predictive validity of interview ratings.

METHODS:

A set of medical school interview ratings was compared to a Monte Carlo simulated set to estimate conditional reliability controlling for range restriction, response scale bias and other artefacts. This estimate was used as a weighting function to improve the predictive validity of a second set of interview ratings for predicting non-cognitive measures (USMLE Step II residuals from Step I scores).

RESULTS:

Compared with the simulated set, both observed sets showed more reliability at low and high rating levels than at moderate levels. Raw interview scores did not predict USMLE Step II scores after controlling for Step I performance (additional r2 = 0.001, not significant). Weighting interview ratings by estimated conditional reliability improved predictive validity (additional r2 = 0.121, P < 0.01).

CONCLUSIONS:

Conditional reliability is important for understanding the psychometric properties of subjective rating scales. Weighting these measures during the admissions process would improve admissions decisions.

[Indexed for MEDLINE]
Free full text

Supplemental Content

Full text links

Icon for Wiley Icon for MLibrary (Deep Blue)
Loading ...
Support Center