Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study Academic Article uri icon

  •  
  • Overview
  •  
  • Research
  •  
  • Identity
  •  
  • Additional Document Info
  •  
  • View All
  •  

abstract

  • STUDY OBJECTIVE: Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. METHODS: We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. RESULTS: One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). CONCLUSION: The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.

authors

  • Thoma, Brent
  • Sebok-Syer, Stefanie S
  • Krishnan, Keeth
  • Siemens, Marshall
  • Trueger, N Seth
  • Colmers-Gray, Isabelle
  • Woods, Rob
  • Petrusa, Emil
  • Chan, Teresa
  • Alexander, Charlotte
  • Alkhalifah, Mohammed
  • Alqahtani, Saeed
  • Anderson, Scott
  • Anderson, Shelaina
  • Andrews, Colin
  • Andruko, Jocelyn
  • Ankel, Felix
  • Antony, Nikytha
  • Aryal, Diptesh
  • Backus, Barbra
  • Baird, Jennifer
  • Baker, Andrew
  • Batty, Sarah
  • Baylis, Jared
  • Beaumont, Braeden
  • Belcher, Chris
  • Benavides, Brent
  • Benham, Michael
  • Pelletier, Elyse Berger
  • Botta, Julian
  • Bouchard, Nicholas
  • Brazil, Victoria
  • Brumfield, Emily
  • Bryson, Anthony
  • Bunchit, Wisarut
  • Butler, Kat
  • Buzikievich, Lindy
  • Calcara, David
  • Carey, Rob
  • Carroll, Stephen
  • Lyons, Casey
  • Cassidy, Louise
  • Challen, Kirsty
  • Chaplin, Tim
  • Chatham-Zvelebil, Natasha
  • Chen, Eric
  • Chen, Lucy
  • Chhabra, Sushant
  • Chin, Alvin
  • Chochi, Eric
  • Choudhri, Tina
  • Christensen, Jeremy
  • Colmers-Gray, Isabelle
  • Connors, Kimberly
  • Coppersmith, Veronica
  • Cosgrove, Abby
  • Costello, Gregory
  • Cullison, Kevin
  • D'Alessandro, Andrew
  • de Wit, Kerstin
  • Decock, Marie
  • Delbani, Rayan
  • Denq, William
  • Deutscher, Julianna
  • Devine, Brendan
  • Dorsett, Maia
  • Duda, Taylor
  • Dueweke, Justin
  • Dunphy, Teresa
  • Dyer, Sean
  • Eastley, Kathryn T
  • Edmonds, Marcia
  • Edwards, Ken
  • Ehrman, Robert
  • Elkhalidy, Youness
  • Fedor, Preston
  • Ficiur, Brian
  • Flynn, Caley
  • Fraser, Bill
  • Fu, Meagan
  • Fukakusa, James
  • Funk, Eric
  • Gaco, Damjan
  • Gawlik, Viktor
  • Ghaffarian, Kenn
  • Gharahbaghian, Laleh
  • Griffith, Phil
  • Griffith, Andrew
  • Grock, Andrew
  • Gronowski, Tanner
  • Grossman, Cathy
  • Gucwa, Jaroslaw
  • Gupta, Pawan
  • Gustafson, Alexandra
  • Guy, Andrew
  • Haas, Mary
  • Haciski, Stanislaw
  • Hajdinjak, Emina
  • Hall, Andrew K
  • Hammock, Regina
  • Hansel, Jan
  • Hart, Alexander
  • Hattin, Larissa
  • Herb, Brandon
  • Hilbert, SueLin
  • Hill, Jesse
  • Hill, Jeff
  • Ho, Amy
  • House, Emily
  • House, Nina
  • Huffman, James
  • Inboriboon, Charlie
  • Ireland, Alex
  • Jamal, Mohammed Ali
  • Jansen, Victor
  • Jarou, Zach
  • Jia, Vivian
  • Johnston, Levi
  • Kalnow, Drew
  • Kapur, Puneet
  • Kelly, Seth
  • Kelson, Kyle
  • Kent, William
  • Khakhkhar, Rishi
  • Khurana, Jaasmit
  • Kilp, Ashley
  • Knapp, Scott
  • Kohler, Sebastian
  • Kruhlak, Ivanna
  • Lalani, Nadim
  • Lam, Samantha
  • Lank, Patrick
  • Laurie, Zander
  • Lea, Kristina
  • Leber, Ernest
  • Lee, Ching-Hsing
  • Lenes, Haakon
  • Lenora, Nilantha
  • Leontowicz, Jesse
  • Lien, Kelly
  • Lin, Yingchun
  • Lin, Michelle
  • Little, Andrew
  • Liu, Ivy
  • Liu, Harry
  • Liu, Steve
  • Louka, Stephanie
  • Lovell, Elise
  • Lowe, David
  • Lubberdink, Ashley
  • Luc, Jessica
  • Ma, Sheng-Hsiang
  • MacLeod, Hugh
  • Mancuso, Nick
  • Maneshi, Anali
  • Rosa Carrillo, Dra Maria
  • May, Jesse
  • Mayo, John
  • McDonnell, Mike
  • McLellan, Susan
  • McQuarrie, Carolyn
  • Nood, Julia
  • Mead, Therese
  • Meeuwisse, Cory
  • Meloy, Patrick
  • Menzies, Perry
  • Messman, Anne
  • Miazga, Stephen
  • Mills, Logan
  • Milne, Ken
  • Mix, Allan
  • Montag, Steve
  • Moore, Brendon
  • Morgenstern, Justin
  • Mott, Sarah
  • Mukherj, P
  • Mulla, Ali
  • Nandalal, Sheena
  • Nikel, Taylor
  • Nugent, Sean
  • Oakland, Morgan
  • Oberholzer, Werner
  • Otugo, Onyeka
  • Oyedokun, Taofiq Segun
  • Paddock, Mike
  • Pardhan, Alim
  • Patel, Kinjal
  • Paterson, Quinten
  • Patocka, Catherine
  • Patterson, Christine
  • Pearlman, James
  • Pelletier-Bui, Alexis
  • Phan, Marc
  • Poonja, Zafrina
  • Powell, Aubrey
  • Premkumar, Kamini
  • Prosen, Gregor
  • Puri, Vishal
  • Quaife, Tanis
  • Raffel, Ryan
  • Raja, Ali
  • Ramunno, Randi
  • Rang, Louise
  • Rannazzisi, Suzanne
  • Regan, Shauna
  • Ridderikhof, Milan
  • Rogers, Vanessa
  • Roh, Christine
  • Rosenberg, Keith
  • Roure, Marina
  • Rudinsky, Sherri
  • Rudner, Joshua
  • Saleh, Adeeb
  • Sanderson, Will
  • Scheirer, Owen
  • Schofield, Paul
  • Schunk, Paul
  • Schwarz, Evan
  • Shahrabadi, Parisa
  • Shappell, Eric
  • Sheffield, Julia
  • Sherbino, Jonathan
  • Singh, Manpreet
  • Singson, Hector C
  • Slessor, Dave
  • Smith, Sam
  • Sneath, Paula
  • Sobehart, Robert
  • Spearing, Kerry
  • Stempien, James
  • Sternard, Britni
  • Stratton, Tara
  • Stuart, Katherine
  • Stuntz, Bob
  • Susalla, Michael
  • Sweeney, Colleen
  • Swisher, Loice
  • Swoboda, Henry
  • Syed, Shahbaz
  • Taira, Taku
  • Tambe, Nikhil
  • Tang, Richard
  • Targonsky, Elisha
  • Taylor, Rachel
  • Taylor, Alan
  • Taylor, Todd
  • Ting, Paxton
  • Tiwald, Gerhard
  • Tran, Kelvin
  • Tran, Evelyn
  • Trickovic, Jason
  • Trinquero, Paul
  • Trueger, Seth
  • Tyagi, Aaron
  • Umana, Manrique
  • Vallance, Patrick
  • Van den Berg, Patricia
  • Vargas, Luis
  • Verbeek, Rene
  • Viggers, Sandra
  • Vlodaver, Zlata
  • Wagner, Matthew
  • Walji, Noorin
  • Walter, Joe
  • Wan, Miranda
  • Wang, Rachel
  • Wanner, Gregory
  • Warawa, Wyatt
  • Ward, Mike
  • Weekes, Jennifer
  • Weersink, Kristen
  • Weessies, Cara
  • Whalen-Browne, Anna
  • Whiteside, Brian
  • Willis, Matthew
  • Wilmer, Jonas
  • Wong, Nelson
  • Woodcroft, Mark
  • Woods, Rob
  • Yau, Lawrence
  • Yee, Jessica
  • Yeh, Calvin
  • Ming Huang, Simon York
  • Yurkiw, Katherine
  • Zaver, Fareen
  • Zozula, Alexander

publication date

  • September 2017