Journal of Educational Evaluation for Health Professions

Educational/Faculty development material

The role of large language models in the peer-review process: opportunities and challenges for medical journal reviewers and editors: Jisoo Lee, Jieun Lee, Jeong-Ju Yoo; J Educ Eval Health Prof. 2025;22:4. Published online January 16, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.4

1,755 View
162 Download

Abstract PDF Supplementary Material: The peer review process ensures the integrity of scientific research. This is particularly important in the medical field, where research findings directly impact patient care. However, the rapid growth of publications has strained reviewers, causing delays and potential declines in quality. Generative artificial intelligence, especially large language models (LLMs) such as ChatGPT, may assist researchers with efficient, high-quality reviews. This review explores the integration of LLMs into peer review, highlighting their strengths in linguistic tasks and challenges in assessing scientific validity, particularly in clinical medicine. Key points for integration include initial screening, reviewer matching, feedback support, and language review. However, implementing LLMs for these purposes will necessitate addressing biases, privacy concerns, and data confidentiality. We recommend using LLMs as complementary tools under clear guidelines to support, not replace, human expertise in maintaining rigorous peer review standards.

Research articles

Agreement between medical students’ peer assessments and faculty assessments in advanced resuscitation skills examinations in South Korea: Jinwoo Jeong, Song Yi Park, Kyung Hoon Sun; J Educ Eval Health Prof. 2021;18:4. Published online March 25, 2021; DOI: https://doi.org/10.3352/jeehp.2021.18.4

5,793 View
296 Download

Abstract PDF Supplementary Material: Purpose
In medical education, peer assessment is considered to be an effective learning strategy. Although several studies have examined agreement between peer and faculty assessments regarding basic life support (BLS), few studies have done so for advanced resuscitation skills (ARS) such as intubation and defibrillation. Therefore, this study aimed to determine the degree of agreement between medical students’ and faculty assessments of ARS examinations.
Methods
This retrospective explorative study was conducted during the emergency medicine (EM) clinical clerkship of fourth-year medical students from April to July 2020. A faculty assessor (FA) and a peer assessor (PA) assessed each examinee’s resuscitation skills (including BLS, intubation, and defibrillation) using a checklist that consisted of 20 binary items (performed or not performed) and 1 global proficiency rating using a 5-point Likert scale. The prior examinee assessed the next examinee after feedback and training as a PA. All 54 students participated in peer assessment. The assessments of 44 FA/PA pairs were analyzed using the intraclass correlation coefficient (ICC) and Gwet’s first-order agreement coefficient.
Results
The PA scores were higher than the FA scores (mean±standard deviation, 20.2±2.5 [FA] vs. 22.3±2.4 [PA]; P<0.001). The agreement was poor to moderate for the overall checklist (ICC, 0.55; 95% confidence interval [CI], 0.31 to 0.73; P<0.01), BLS (ICC, 0.19; 95% CI, -0.11 to 0.46; P<0.10), intubation (ICC, 0.51; 95% CI, 0.26 to 0.70; P<0.01), and defibrillation (ICC, 0.49; 95% CI, 0.23 to 0.68; P<0.01).
Conclusion
Senior medical students showed unreliable agreement in ARS assessments compared to faculty assessments. If a peer assessment is planned in skills education, comprehensive preparation and sufficient assessor training should be provided in advance.

Learning through multiple lenses: analysis of self, peer, nearpeer, and faculty assessments of a clinical history-taking task in Australia: Kylie Fitzgerald, Brett Vaughan; J Educ Eval Health Prof. 2018;15:22. Published online September 18, 2018; DOI: https://doi.org/10.3352/jeehp.2018.15.22

24,503 View
299 Download
5 Web of Science
6 Crossref

Abstract PDF Supplementary Material

Purpose
Peer assessment provides a framework for developing expected skills and receiving feedback appropriate to the learner’s level. Near-peer (NP) assessment may elevate expectations and motivate learning. Feedback from peers and NPs may be a sustainable way to enhance student assessment feedback. This study analysed relationships among self, peer, NP, and faculty marking of an assessment and students’ attitudes towards marking by those various groups.
Methods
A cross-sectional study design was used. Year 2 osteopathy students (n= 86) were invited to perform self and peer assessments of a clinical history-taking and communication skills assessment. NPs and faculty also marked the assessment. Year 2 students also completed a questionnaire on their attitudes to peer/NP marking. Descriptive statistics and the Spearman rho coefficient were used to evaluate relationships across marker groups.
Results
Year 2 students (n= 9), NPs (n= 3), and faculty (n= 5) were recruited. Correlations between self and peer (r= 0.38) and self and faculty (r= 0.43) marks were moderate. A weak correlation was observed between self and NP marks (r= 0.25). Perceptions of peer and NP marking varied, with over half of the cohort suggesting that peer or NP assessments should not contribute to their grade.
Conclusion
Framing peer and NP assessment as another feedback source may offer a sustainable method for enhancing feedback without overloading faculty resources. Multiple sources of feedback may assist in developing assessment literacy and calibrating students’ self-assessment capability. The small number of students recruited suggests some acceptability of peer and NP assessment; however, further work is required to increase its acceptability.

Citations

Citations to this article as recorded by

Learning how to suture: Should learners observe a demonstration of someone who is experienced or inexperienced to improve their own performance?
Portia Kalun, Ranil Sonnadara
The American Journal of Surgery.2025; 243: 116276. CrossRef
The extent and quality of evidence for osteopathic education: A scoping review
Andrew MacMillan, Patrick Gauthier, Luciane Alberto, Arabella Gaunt, Rachel Ives, Chris Williams, Dr Jerry Draper-Rodi
International Journal of Osteopathic Medicine.2023; 49: 100663. CrossRef
History and physical exam: a retrospective analysis of a clinical opportunity
David McLinden, Krista Hailstone, Sue Featherston
BMC Medical Education.2023;[Epub] CrossRef
How Accurate Are Our Students? A Meta-analytic Systematic Review on Self-assessment Scoring Accuracy
Samuel P. León, Ernesto Panadero, Inmaculada García-Martínez
Educational Psychology Review.2023;[Epub] CrossRef
Evaluating the Academic Performance of Mustansiriyah Medical College Teaching Staff vs. Final-Year Students Failure Rates
Wassan Nori, Wisam Akram , Saad Mubarak Rasheed, Nabeeha Najatee Akram, Taqi Mohammed Jwad Taher, Mustafa Ali Kassim Kassim, Alexandru Cosmin Pantazi
Al-Rafidain Journal of Medical Sciences ( ISSN 2789-3219 ).2023; 5(1S): S151. CrossRef
History-taking level and its influencing factors among nursing undergraduates based on the virtual standardized patient testing results: Cross sectional study
Jingrong Du, Xiaowen Zhu, Juan Wang, Jing Zheng, Xiaomin Zhang, Ziwen Wang, Kun Li
Nurse Education Today.2022; 111: 105312. CrossRef

First
Prev
Page of 1
Next
Last

Search