Send to

Choose Destination
Res Eval. 2017 Jan;26(1):1-14. doi: 10.1093/reseval/rvw025. Epub 2017 Feb 14.

'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review.

Author information

Center for Women's Health Research, University of Wisconsin-Madison, 700 Regent Street, Ste. 301, Madison, WI 53715, USA.
Department of Educational Psychology, University of Wisconsin-Madison, 1025 West Johnson Street, Madison, WI 53706, USA.
Department of English, West Chester University, 700 South High Street, West Chester, PA 19383, USA.
Department of Psychology, University of Wisconsin-Madison, 1202 West Johnson Street, Madison, WI 53706, USA.
Department of English, University of Wisconsin-Madison, 600 North Park Street, Madison, WI 53706, USA.


In scientific grant peer review, groups of expert scientists meet to engage in the collaborative decision-making task of evaluating and scoring grant applications. Prior research on grant peer review has established that inter-reviewer reliability is typically poor. In the current study, experienced reviewers for the National Institutes of Health (NIH) were recruited to participate in one of four constructed peer review panel meetings. Each panel discussed and scored the same pool of recently reviewed NIH grant applications. We examined the degree of intra-panel variability in panels' scores of the applications before versus after collaborative discussion, and the degree of inter-panel variability. We also analyzed videotapes of reviewers' interactions for instances of one particular form of discourse-Score Calibration Talk-as one factor influencing the variability we observe. Results suggest that although reviewers within a single panel agree more following collaborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. We discuss implications of this variability for the scientific peer review process.


collaboration; decision making; discourse analysis; peer review

Supplemental Content

Full text links

Icon for Silverchair Information Systems Icon for PubMed Central
Loading ...
Support Center