Journal of Educational Evaluation for Health Professions

Research articles

Comparison between GPT-4 and human raters in grading pharmacy students’ exam responses in Malaysia: a cross-sectional study: Wuan Shuen Yap, Pui San Saw, Li Ling Yeap, Shaun Wen Huey Lee, Wei Jin Wong, Ronald Fook Seng Lee; J Educ Eval Health Prof. 2025;22:20. Published online July 28, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.20

5,159 View
321 Download
1 Web of Science

Abstract PDF Supplementary Material: Purpose
Manual grading is time-consuming and prone to inconsistencies, prompting the exploration of generative artificial intelligence tools such as GPT-4 to enhance efficiency and reliability. This study investigated GPT-4’s potential in grading pharmacy students’ exam responses, focusing on the impact of optimized prompts. Specifically, it evaluated the alignment between GPT-4 and human raters, assessed GPT-4’s consistency over time, and determined its error rates in grading pharmacy students’ exam responses.
Methods
We conducted a comparative study using past exam responses graded by university-trained raters and by GPT-4. Responses were randomized before evaluation by GPT-4, accessed via a Plus account between April and September 2024. Prompt optimization was performed on 16 responses, followed by evaluation of 3 prompt delivery methods. We then applied the optimized approach across 4 item types. Intraclass correlation coefficients and error analyses were used to assess consistency and agreement between GPT-4 and human ratings.
Results
GPT-4’s ratings aligned reasonably well with human raters, demonstrating moderate to excellent reliability (intraclass correlation coefficient=0.617–0.933), depending on item type and the optimized prompt. When stratified by grade bands, GPT-4 was less consistent in marking high-scoring responses (Z=–5.71–4.62, P<0.001). Overall, despite achieving substantial alignment with human raters in many cases, discrepancies across item types and a tendency to commit basic errors necessitate continued educator involvement to ensure grading accuracy.
Conclusion
With optimized prompts, GPT-4 shows promise as a supportive tool for grading pharmacy students’ exam responses, particularly for objective tasks. However, its limitations—including errors and variability in grading high-scoring responses—require ongoing human oversight. Future research should explore advanced generative artificial intelligence models and broader assessment formats to further enhance grading reliability.

Effects of a cultural nursing course to enhance the cultural competence of nursing students in Korea: Hae Sook Park, Hee Jung Jang, Geum Hee Jeong; J Educ Eval Health Prof. 2019;16:39. Published online December 27, 2019; DOI: https://doi.org/10.3352/jeehp.2019.16.39

14,157 View
333 Download
10 Web of Science
12 Crossref

Purpose
With Korea’s recent rapid change into a multicultural society, cultural competence is being emphasized as a core nursing competency. This study investigated the effects of a cultural nursing course that aimed to enhance the cultural competence of nursing students in Korea.
Methods
This was a single-group pre- and post-comparison study. The subjects were 69 nursing students at Dongyang University who attended a cultural nursing course in 2015, of whom 62 students responded to the survey. The 13-week cultural nursing course was held for 2 hours a week. The methods of the course included small group activities, discussions and presentations, experiential learning, reflective activities, and lectures. Nursing students’ cultural competence was measured pre- and post-course with the Cultural Competence Scale for Korean Nurses, which contains 33 items scored on a 7-point Likert scale.
Results
After completing the cultural nursing course, students’ total cultural competence scores increased, as did their scores in each category (cultural awareness, cultural knowledge, cultural sensitivity, and cultural skills) (P<0.001). There was no significant difference in cultural competence by gender (P<0.001).
Conclusion
This cultural nursing course was found to be effective in enhancing the cultural competence of nursing students. Therefore, the educational program developed in this study can be extended to other university-level nursing programs in Korea.

Citations

Citations to this article as recorded by

Challenging the narrative to reimagine health disparities research and education: Perspectives from international scholars
Soo Hyun Kim, Celeste Chavez, Thomas Hinneh, Erh-Chi Hsu, Getachew Kassa, Sarah Won, Samuel Byiringiro
Nurse Education Today.2026; 160: 106971. CrossRef
Reflections of Cultural Differences on Nursing Care: A Systematic Review of International Quantitative Evidence
Hatice Öntürk Akyüz, Hatice Gülsoy
Holistic Nursing Practice.2026;[Epub] CrossRef
Enhancing Cultural Competence in Undergraduate Nursing Students: An Integrative Literature Review of Strategies for Institutions of Higher Education
Khumoetsile Daphney Shopo, Vistolina Nuuyoma, Leonard Chihururu
Journal of Transcultural Nursing.2025; 36(4): 412. CrossRef
Motivational Needs, Instructional Competence and Problem-Solving Skills of Teachers in Relation to Performance: Basis for Training Program
Carlo Villaflor, Avelino Santillan, Ma. Quincy Dones
Psychology and Education: A Multidisciplinary Journal.2025; 42(7): 972. CrossRef
Intercultural sensitivity among nursing students: a latent profile analysis
You-Ran Liu, Yan Wang, Tengfei Liang, Shumei Zhuang, Xi Wang, Jie Wang, Hui Xie
BMC Nursing.2025;[Epub] CrossRef
Innovative cultural care training: the impact of flipped classroom methods on critical cultural competencies in psychiatric nursing: a quasi-experimental study
Mahdiyeh Sarvarizadeh, Sakineh Miri, Fatemeh Darban, Jamileh Farokhzadian
BMC Nursing.2024;[Epub] CrossRef
Determinants of attitudes towards refugees and intercultural sensitivity of nursing students: A descriptive and correlational study
Ayşegül Akca, Sultan Ayaz-Alkaya
Nurse Education Today.2023; : 105772. CrossRef
Effect of a transcultural nursing course on improving the cultural competency of nursing graduate students in Korea: a before-and-after study
Kyung Eui Bae, Geum Hee Jeong
Journal of Educational Evaluation for Health Professions.2023; 20: 35. CrossRef
Development and Evaluation of Cultural Competence Course on Undergraduate Nursing Students in Vietnam
Trang-Thi-Thuy Ho, Jina Oh
International Journal of Environmental Research and Public Health.2022; 19(2): 888. CrossRef
Identifikacija merskih instrumentov za merjenje kulturnih kompetenc pri medicinskih sestrah in študentih zdravstvene nege
Liridon Avdylaj, Sabina Ličen
Obzornik zdravstvene nege.2022; 56(1): 49. CrossRef
Organisational and staff-related effects on cultural competence in the hospital setting: a cross-sectional online survey of nursing and medical staff
Liane Schenk, Pia-Theresa Sonntag, Patricia Beck, Zohra Khan, Lisa Peppler, Meryam Schouler-Ocak
BMC Health Services Research.2022;[Epub] CrossRef
Nursing students’ experience of learning cultural competence
Isabel Antón-Solanas, Elena Tambo-Lizalde, Nadia Hamam-Alcober, Valérie Vanceulebroeck, Shana Dehaes, Indrani Kalkan, Nuran Kömürcü, Margarida Coelho, Teresa Coelho, Antonio Casa Nova, Raul Cordeiro, Lucía Sagarra-Romero, Ana B. Subirón-Valera, Isabel Hué
PLOS ONE.2021; 16(12): e0259802. CrossRef

First
Prev
Page of 1
Next
Last

Search