Format

Send to

Choose Destination
Ann Emerg Med. 2017 Sep;70(3):394-401. doi: 10.1016/j.annemergmed.2016.12.025. Epub 2017 Mar 2.

Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study.

Collaborators (277)

Alexander C, Alkhalifah M, Alqahtani S, Anderson S, Anderson S, Andrews C, Andruko J, Ankel F, Antony N, Aryal D, Backus B, Baird J, Baker A, Batty S, Baylis J, Beaumont B, Belcher C, Benavides B, Benham M, Pelletier EB, Botta J, Bouchard N, Brazil V, Brumfield E, Bryson A, Bunchit W, Butler K, Buzikievich L, Calcara D, Carey R, Carroll S, Lyons C, Cassidy L, Challen K, Chaplin T, Chatham-Zvelebil N, Chen E, Chen L, Chhabra S, Chin A, Chochi E, Choudhri T, Christensen J, Colmers-Gray I, Connors K, Coppersmith V, Cosgrove A, Costello G, Cullison K, D'Alessandro A, de Wit K, Decock M, Delbani R, Denq W, Deutscher J, Devine B, Dorsett M, Duda T, Dueweke J, Dunphy T, Dyer S, Eastley KT, Edmonds M, Edwards K, Ehrman R, Elkhalidy Y, Fedor P, Ficiur B, Flynn C, Fraser B, Fu M, Fukakusa J, Funk E, Gaco D, Gawlik V, Ghaffarian K, Gharahbaghian L, Griffith P, Griffith A, Grock A, Gronowski T, Grossman C, Gucwa J, Gupta P, Gustafson A, Guy A, Haas M, Haciski S, Hajdinjak E, Hall AK, Hammock R, Hansel J, Hart A, Hattin L, Herb B, Hilbert S, Hill J, Hill J, Ho A, House E, House N, Huffman J, Inboriboon C, Ireland A, Jamal MA, Jansen V, Jarou Z, Jia V, Johnston L, Kalnow D, Kapur P, Kelly S, Kelson K, Kent W, Khakhkhar R, Khurana J, Kilp A, Knapp S, Kohler S, Kruhlak I, Lalani N, Lam S, Lank P, Laurie Z, Lea K, Leber E, Lee CH, Lenes H, Lenora N, Leontowicz J, Lien K, Lin Y, Lin M, Little A, Liu I, Liu H, Liu S, Louka S, Lovell E, Lowe D, Lubberdink A, Luc J, Ma SH, MacLeod H, Mancuso N, Maneshi A, Rosa Carrillo DM, May J, Mayo J, McDonnell M, McLellan S, McQuarrie C, Nood J, Mead T, Meeuwisse C, Meloy P, Menzies P, Messman A, Miazga S, Mills L, Milne K, Mix A, Montag S, Moore B, Morgenstern J, Mott S, Mukherj P, Mulla A, Nandalal S, Nikel T, Nugent S, Oakland M, Oberholzer W, Otugo O, Oyedokun TS, Paddock M, Pardhan A, Patel K, Paterson Q, Patocka C, Patterson C, Pearlman J, Pelletier-Bui A, Phan M, Poonja Z, Powell A, Premkumar K, Prosen G, Puri V, Quaife T, Raffel R, Raja A, Ramunno R, Rang L, Rannazzisi S, Regan S, Ridderikhof M, Rogers V, Roh C, Rosenberg K, Roure M, Rudinsky S, Rudner J, Saleh A, Sanderson W, Scheirer O, Schofield P, Schunk P, Schwarz E, Shahrabadi P, Shappell E, Sheffield J, Sherbino J, Singh M, Singson HC, Slessor D, Smith S, Sneath P, Sobehart R, Spearing K, Stempien J, Sternard B, Stratton T, Stuart K, Stuntz B, Susalla M, Sweeney C, Swisher L, Swoboda H, Syed S, Taira T, Tambe N, Tang R, Targonsky E, Taylor R, Taylor A, Taylor T, Ting P, Tiwald G, Tran K, Tran E, Trickovic J, Trinquero P, Trueger S, Tyagi A, Umana M, Vallance P, Van den Berg P, Vargas L, Verbeek R, Viggers S, Vlodaver Z, Wagner M, Walji N, Walter J, Wan M, Wang R, Wanner G, Warawa W, Ward M, Weekes J, Weersink K, Weessies C, Whalen-Browne A, Whiteside B, Willis M, Wilmer J, Wong N, Woodcroft M, Woods R, Yau L, Yee J, Yeh C, Ming Huang SY, Yurkiw K, Zaver F, Zozula A.

Author information

1
Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada; Health Professions Education Program, Massachusetts General Hospital Institute of Health Professions, Boston, MA. Electronic address: brent.thoma@usask.ca.
2
Centre for Education Research & Innovation, University of Western Ontario, London, Ontario, Canada.
3
Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
4
Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada.
5
Department of Emergency Medicine, Northwestern University, Chicago, IL.
6
Department of Emergency Medicine, University of Alberta, Edmonton, Alberta, Canada.
7
Health Professions Education Program, Massachusetts General Hospital Institute of Health Professions, Boston, MA.
8
Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, Ontario, Canada.

Abstract

STUDY OBJECTIVE:

Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs.

METHODS:

We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality.

RESULTS:

One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8).

CONCLUSION:

The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.

[Indexed for MEDLINE]

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center