Journal of Educational Evaluation for Health Professions

Research articles

Comparison between GPT-4 and human raters in grading pharmacy students’ exam responses in Malaysia: a cross-sectional study: Wuan Shuen Yap, Pui San Saw, Li Ling Yeap, Shaun Wen Huey Lee, Wei Jin Wong, Ronald Fook Seng Lee; J Educ Eval Health Prof. 2025;22:20. Published online July 28, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.20

2,439 View
239 Download
1 Web of Science

Abstract PDF Supplementary Material: Purpose
Manual grading is time-consuming and prone to inconsistencies, prompting the exploration of generative artificial intelligence tools such as GPT-4 to enhance efficiency and reliability. This study investigated GPT-4’s potential in grading pharmacy students’ exam responses, focusing on the impact of optimized prompts. Specifically, it evaluated the alignment between GPT-4 and human raters, assessed GPT-4’s consistency over time, and determined its error rates in grading pharmacy students’ exam responses.
Methods
We conducted a comparative study using past exam responses graded by university-trained raters and by GPT-4. Responses were randomized before evaluation by GPT-4, accessed via a Plus account between April and September 2024. Prompt optimization was performed on 16 responses, followed by evaluation of 3 prompt delivery methods. We then applied the optimized approach across 4 item types. Intraclass correlation coefficients and error analyses were used to assess consistency and agreement between GPT-4 and human ratings.
Results
GPT-4’s ratings aligned reasonably well with human raters, demonstrating moderate to excellent reliability (intraclass correlation coefficient=0.617–0.933), depending on item type and the optimized prompt. When stratified by grade bands, GPT-4 was less consistent in marking high-scoring responses (Z=–5.71–4.62, P<0.001). Overall, despite achieving substantial alignment with human raters in many cases, discrepancies across item types and a tendency to commit basic errors necessitate continued educator involvement to ensure grading accuracy.
Conclusion
With optimized prompts, GPT-4 shows promise as a supportive tool for grading pharmacy students’ exam responses, particularly for objective tasks. However, its limitations—including errors and variability in grading high-scoring responses—require ongoing human oversight. Future research should explore advanced generative artificial intelligence models and broader assessment formats to further enhance grading reliability.

Reliability and construct validation of the Blended Learning Usability Evaluation–Questionnaire with interprofessional clinicians in Canada: a methodological study: Anish Kumar Arora, Jeff Myers, Tavis Apramian, Kulamakan Kulasegaram, Daryl Bainbridge, Hsien Seow; J Educ Eval Health Prof. 2025;22:5. Published online January 16, 2025; DOI: https://doi.org/10.3352/jeehp.2025.22.5

3,485 View
303 Download
2 Crossref

Purpose
To generate Cronbach’s alpha and further mixed methods construct validity evidence for the Blended Learning Usability Evaluation–Questionnaire (BLUE-Q).
Methods
Forty interprofessional clinicians completed the BLUE-Q after finishing a 3-month long blended learning professional development program in Ontario, Canada. Reliability was assessed with Cronbach’s α for each of the 3 sections of the BLUE-Q and for all quantitative items together. Construct validity was evaluated through the Grand-Guillaume-Perrenoud et al. framework, which consists of 3 elements: congruence, convergence, and credibility. To compare quantitative and qualitative results, descriptive statistics, including means and standard deviations for each Likert scale item of the BLUE-Q were calculated.
Results
Cronbach’s α was 0.95 for the pedagogical usability section, 0.85 for the synchronous modality section, 0.93 for the asynchronous modality section, and 0.96 for all quantitative items together. Mean ratings (with standard deviations) were 4.77 (0.506) for pedagogy, 4.64 (0.654) for synchronous learning, and 4.75 (0.536) for asynchronous learning. Of the 239 qualitative comments received, 178 were identified as substantive, of which 88% were considered congruent and 79% were considered convergent with the high means. Among all congruent responses, 69% were considered confirming statements and 31% were considered clarifying statements, suggesting appropriate credibility. Analysis of the clarifying statements assisted in identifying 5 categories of suggestions for program improvement.
Conclusion
The BLUE-Q demonstrates high reliability and appropriate construct validity in the context of a blended learning program with interprofessional clinicians, making it a valuable tool for comprehensive program evaluation, quality improvement, and evaluative research in health professions education.

Citations

Citations to this article as recorded by

Utilizing cognitive interview in the item refinement of the Blended Teaching Assessment Tool (BTAT) for Health Professions Education
Maria Teresita B. Dalusong, Glenda Sanggalang Ogerio, Valentin C. Dones, Maria Elizabeth M. Grageda
Philippine Journal of Health Research and Development.2025; 29(2): 54. CrossRef
All providers Better Communication Skills (ABCs) program: protocol for a randomized controlled trial assessing communication training effectiveness with interprofessional clinicians
Hsien Seow, Anish K. Arora, Daryl Bainbridge, Zhimeng Jia, Leah Steinberg, Nadia Incardona, Oren Levine, Justin J. Sanders, Jessica Simon, Amanda Roze des Ordons, Karen Zhang, Jeff Myers
BMC Palliative Care.2025;[Epub] CrossRef

Inter-rater reliability and content validity of the measurement tool for portfolio assessments used in the Introduction to Clinical Medicine course at Ewha Womans University College of Medicine: a methodological study: Dong-Mi Yoo, Jae Jin Han; J Educ Eval Health Prof. 2024;21:39. Published online December 10, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.39

3,969 View
240 Download
1 Web of Science
1 Crossref

Abstract PDF Supplementary Material

Purpose
This study aimed to examine the reliability and validity of a measurement tool for portfolio assessments in medical education. Specifically, it investigated scoring consistency among raters and assessment criteria appropriateness according to an expert panel.
Methods
A cross-sectional observational study was conducted from September to December 2018 for the Introduction to Clinical Medicine course at the Ewha Womans University College of Medicine. Data were collected for 5 randomly selected portfolios scored by a gold-standard rater and 6 trained raters. An expert panel assessed the validity of 12 assessment items using the content validity index (CVI). Statistical analysis included Pearson correlation coefficients for rater alignment, the intraclass correlation coefficient (ICC) for inter-rater reliability, and the CVI for item-level validity.
Results
Rater 1 had the highest Pearson correlation (0.8916) with the gold-standard rater, while Rater 5 had the lowest (0.4203). The ICC for all raters was 0.3821, improving to 0.4415 after excluding Raters 1 and 5, indicating a 15.6% reliability increase. All assessment items met the CVI threshold of ≥0.75, with some achieving a perfect score (CVI=1.0). However, items like “sources” and “level and degree of performance” showed lower validity (CVI=0.72).
Conclusion
The present measurement tool for portfolio assessments demonstrated moderate reliability and strong validity, supporting its use as a credible tool. For a more reliable portfolio assessment, more faculty training is needed.

Citations

Citations to this article as recorded by

On the quantitative analysis of assessment scores with implicit and explicit constraints
Sanjeeb Shrestha, Xiaoying Kong, Paul Kwan
Studies in Educational Evaluation.2025; 87: 101509. CrossRef

Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study: Ariel Shana Frey-Vogel, Kristina Dzara, Kimberly Anne Gifford, Yoon Soo Park, Justin Berk, Allison Heinly, Darcy Wolcott, Daniel Adam Hall, Shannon Elliott Scott-Vernaglia, Katherine Anne Sparger, Erica Ye-pyng Chung; J Educ Eval Health Prof. 2024;21:3. Published online February 23, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.3

4,406 View
229 Download

Abstract PDF Supplementary Material: Purpose
Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into 6 elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure.
Methods
Messick’s unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from 3 pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018–2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables).
Results
Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone (r=0.34, P=0.019).
Conclusion
Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.

Review

Application of artificial intelligence chatbots, including ChatGPT, in education, scholarly work, programming, and content generation and its prospects: a narrative review: Tae Won Kim; J Educ Eval Health Prof. 2023;20:38. Published online December 27, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.38

25,191 View
1,473 Download
52 Web of Science
59 Crossref

Abstract PDF Supplementary Material

This study aims to explore ChatGPT’s (GPT-3.5 version) functionalities, including reinforcement learning, diverse applications, and limitations. ChatGPT is an artificial intelligence (AI) chatbot powered by OpenAI’s Generative Pre-trained Transformer (GPT) model. The chatbot’s applications span education, programming, content generation, and more, demonstrating its versatility. ChatGPT can improve education by creating assignments and offering personalized feedback, as shown by its notable performance in medical exams and the United States Medical Licensing Exam. However, concerns include plagiarism, reliability, and educational disparities. It aids in various research tasks, from design to writing, and has shown proficiency in summarizing and suggesting titles. Its use in scientific writing and language translation is promising, but professional oversight is needed for accuracy and originality. It assists in programming tasks like writing code, debugging, and guiding installation and updates. It offers diverse applications, from cheering up individuals to generating creative content like essays, news articles, and business plans. Unlike search engines, ChatGPT provides interactive, generative responses and understands context, making it more akin to human conversation, in contrast to conventional search engines’ keyword-based, non-interactive nature. ChatGPT has limitations, such as potential bias, dependence on outdated data, and revenue generation challenges. Nonetheless, ChatGPT is considered to be a transformative AI tool poised to redefine the future of generative technology. In conclusion, advancements in AI, such as ChatGPT, are altering how knowledge is acquired and applied, marking a shift from search engines to creativity engines. This transformation highlights the increasing importance of AI literacy and the ability to effectively utilize AI in various domains of life.

Citations

Citations to this article as recorded by

Exploring Early Perceptions and Experiences of ChatGPT in Pediatric Critical Care: A Qualitative Study Among Health Care Professionals
Mohamad-Hani Temsah, Noura Abouammoh, Mohammed Alsatrawi, Muneera Al-Jelaify, Ibraheem Altamimi, PICU ChatGPT Research Consortium, Khalid Alhasan, Khalid H. Malki, Jaffar A. Al-Tawfiq, Ayman Al-Eyadhy
Clinical Pediatrics.2026; 65(1): 11. CrossRef
Designing intelligent chatbots with ChatGPT: a framework for development and implementation
Sajjad Hyder, Javeed Kittur
Frontiers in Artificial Intelligence.2026;[Epub] CrossRef
The Development and Validation of an Artificial Intelligence Chatbot Dependence Scale
Xing Zhang, Mingyue Yin, Mingyang Zhang, Zhaoqian Li, Hansen Li
Cyberpsychology, Behavior, and Social Networking.2025; 28(2): 126. CrossRef
Readability, quality and accuracy of generative artificial intelligence chatbots for commonly asked questions about labor epidurals: a comparison of ChatGPT and Bard
D. Lee, M. Brown, J. Hammond, M. Zakowski
International Journal of Obstetric Anesthesia.2025; 61: 104317. CrossRef
ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled Trial
Christian Burisch, Abhav Bellary, Frank Breuckmann, Jan Ehlers, Serge C Thal, Timur Sellmann, Daniel Gödde
JMIR Research Protocols.2025; 14: e63887. CrossRef
The effect of incorporating large language models into the teaching on critical thinking disposition: An “AI + Constructivism Learning Theory” attempt
Peng Wang, Kexin Yin, Mingzhu Zhang, Yuanxin Zheng, Tong Zhang, Yanjun Kang, Xun Feng
Education and Information Technologies.2025; 30(9): 11625. CrossRef
The Impact of Adaptive Learning Technologies, Personalized Feedback, and Interactive AI Tools on Student Engagement: The Moderating Role of Digital Literacy
Husam Yaseen, Abdelaziz Saleh Mohammad, Najwa Ashal, Hesham Abusaimeh, Ahmad Ali, Abdel-Aziz Ahmad Sharabati
Sustainability.2025; 17(3): 1133. CrossRef
Artificial Intelligence in Nursing: New Opportunities and Challenges
Estel·la Ramírez‐Baraldes, Daniel García‐Gutiérrez, Cristina García‐Salido
European Journal of Education.2025;[Epub] CrossRef
Can ChatGPT be used as a scientific source of information on tooth extraction?
Shiori Yamamoto, Masakazu Hamada, Kyoko Nishiyama, Ayako Motoki, Yusei Fujita, Narikazu Uzawa
Journal of Oral and Maxillofacial Surgery, Medicine, and Pathology.2025; 37(5): 877. CrossRef
Exploring artificial intelligence (AI) Chatbot usage behaviors and their association with mental health outcomes in Chinese university students
Xing Zhang, Zhaoqian Li, Mingyang Zhang, Mingyue Yin, Zhangyu Yang, Dong Gao, Hansen Li
Journal of Affective Disorders.2025; 380: 394. CrossRef
The analysis of optimization in music aesthetic education under artificial intelligence
Yixuan Peng
Scientific Reports.2025;[Epub] CrossRef
The Role of Artificial Intelligence in Computer Science Education: A Systematic Review with a Focus on Database Instruction
Alkmini Gaitantzi, Ioannis Kazanidis
Applied Sciences.2025; 15(7): 3960. CrossRef
A Bibliometric Exposition and Review on Leveraging LLMs for Programming Education
Joanah Pwanedo Amos, Oluwatosin Ahmed Amodu, Raja Azlina Raja Mahmood, Akanbi Bolakale Abdulqudus, Anies Faziehan Zakaria, Abimbola Rhoda Iyanda, Umar Ali Bukar, Zurina Mohd Hanapi
IEEE Access.2025; 13: 58364. CrossRef
Can ChatGPT be trusted as a resource for a scholarly article on treatment planning implant-supported prostheses?
Steven J. Sadowsky
The Journal of Prosthetic Dentistry.2025; 134(2): 438. CrossRef
Use of machine translation in foreign language education
Blanka Klimova
Cogent Arts & Humanities.2025;[Epub] CrossRef
Comparison of triage performance among DRP tool, ChatGPT, and outpatient rehabilitation doctors
Yucong Zou, Ruixue Ye, Yan Gao, Jing Zhou, Yawei Li, Wenshi Chen, Fubing Zha, Yulong Wang
Scientific Reports.2025;[Epub] CrossRef
A Training Needs Analysis for AI and Generative AI in Medical Education: Perspectives of Faculty and Students
Lise McCoy, Natarajan Ganesan, Viswanathan Rajagopalan, Douglas McKell, Diego F. Niño, Mary Claire Swaim
Journal of Medical Education and Curricular Development.2025;[Epub] CrossRef
Application of ChatGPT 4.0 in radiological dose management: Perceptions of radiographers with varying expertise
L. Federico, D.D. Fusaro, G.C. Coppola, M. Gregori, S. Durante
Radiography.2025; 31(4): 102972. CrossRef
Status and perceptions of ChatGPT utilization among medical students: a survey-based study
Na Hu, Xiao Qin Jiang, Yi Da Wang, Yan Ming Kang, Zhen Xia, Hao Hui Chen, Sai Nan Duan, Dong Xu Chen
BMC Medical Education.2025;[Epub] CrossRef
Artificial Intelligence and the Scientific Process: A Review of ChatGPT’s Role to Foster Experimental Thinking in Physics Education
Konstantinos T. Kotsis
European Journal of Contemporary Education and E-Learning.2025; 3(3): 183. CrossRef
Performance of GPT-4 in oral and maxillofacial surgery board exams: challenges in specialized questions
Felix Benjamin Warwas, Nils Heim
Oral and Maxillofacial Surgery.2025;[Epub] CrossRef
Evaluating Authorship Guidelines of Top Medical Schools and Plastic Surgery Journals
Nicholas A. Mirsky, Sara E. Munkwitz, Wrood M. Kassira, Pawan Pathagamage, Paulo G. Coelho, Seth R. Thaller
Annals of Plastic Surgery.2025; 95(5): e53. CrossRef
How appropriately can generative artificial intelligence platforms, including GPT-4, Gemini, Bing, and Wrtn, answer questions about colon cancer in the Korean language?
Sun Huh
Annals of Coloproctology.2025; 41(3): 190. CrossRef
The application of problem-based learning (PBL) guided by ChatGPT in clinical education in the Department of Nephrology
Xiaoya Tong, Ying Hu, Yanjun Long, Qian Zhang, Yao Yang, Jing Yuan, Yan Zha
BMC Medical Education.2025;[Epub] CrossRef
Mental load, ChatGPT, and work-study balance: A TAM-based study of AI adoption by employees in non-traditional education
Humayyun Bashir, Rongting Zhou
Telematics and Informatics Reports.2025; 19: 100216. CrossRef
ChatGPT and AI Chatbots in Education: An Umbrella Review of Systematic Reviews, Scoping Reviews, and Meta-Analyses
Emmanouil D. Milakis, Constantina Corazon Argyrakou, Alexandros Melidis, John Vrettaros
International Journal of Education and Information Technologies.2025; 19: 100. CrossRef
Battle of the artificial intelligence: a comprehensive comparative analysis of DeepSeek and ChatGPT for urinary incontinence-related questions
Huawei Cao, Changzhen Hao, Tao Zhang, Xiang Zheng, Zihao Gao, Jiyue Wu, Lijian Gan, Yu Liu, Xiangjun Zeng, Wei Wang
Frontiers in Public Health.2025;[Epub] CrossRef
A review of ChatGPT in medical education: exploring advantages and limitations
Yuan Cheng, Lingling Zhu
International Journal of Surgery.2025; 111(7): 4586. CrossRef
Investigating AI Chatbot Dependence: Associations with Internet and Smartphone Dependence, Mental Health Outcomes, and the Moderating Role of Usage Purposes
Xing Zhang, Hansen Li, Mingyue Yin, Mingyang Zhang, Zhaoqian Li, Zongwei Chen
International Journal of Human–Computer Interaction.2025; : 1. CrossRef
Artificial Intelligence and Large Language Models in the Fight Against Superficial Fungal Infections: Friend or Foe?
Aditya Gupta, Vasiliki Economopoulos
Clinical, Cosmetic and Investigational Dermatology.2025; Volume 18: 1959. CrossRef
Exploring the Effects of the CER Model‐Based GenAI Learning System to Cultivate Elementary School Students' Computational Thinking Core Skills in Science Courses
Jia‐Hua Zhao, Shu‐Tao Shangguan, Ying Wang
Journal of Computer Assisted Learning.2025;[Epub] CrossRef
Artificial intelligence in preclinical epilepsy research: Current state, potential, and challenges
Jesús Servando Medel‐Matus, Cesar Santana‐Gomez, Ruby G. Escalante, Dominique Duncan, Pedro F. Viana, Giulia Sofia Cereda, Naoto Kuroda, Aristea S. Galanopoulou
Epilepsia Open.2025;[Epub] CrossRef
Exploring the Perceived Usefulness and Effect of AI Writing Tools in Enhancing the Quality of Written Outputs of Senior High School Students
Roy A. Discutido
International Journal of Multidisciplinary: Applied Business and Education Research.2025; 6(9): 4267. CrossRef
Üniversite Öğrencilerinin Yapay Zekâ Okuryazarlığı Üzerine Bir Araştırma
Ela OĞAN
Üçüncü Sektör Sosyal Ekonomi Dergisi.2025; 60(3): 2670. CrossRef
Social-Ecological Factors Associated with AI Chatbot Use and Dependence
Hansen Li, Jie Tian, Mingyue Yin, Xing Zhang
International Journal of Human–Computer Interaction.2025; : 1. CrossRef
Generative artificial intelligence in dentistry: A narrative review of current approaches and future challenges
Fabián Villena, Claudia Véliz, Rosario García-Huidobro, Sebastian Aguayo
Dentistry Review.2025; 5(4): 100160. CrossRef
A product appearance design method based on artificial intelligence generated content
Yanpu Yang, Yueming Zhuo, Zhihong Wu, Jialing Liu, Wenhao Meng
Advanced Design Research.2025; 3(1): 64. CrossRef
ChatGPT in Medical Education: Bibliometric and Visual Analysis
Yuning Zhang, Xiaolu Xie, Qi Xu
JMIR Medical Education.2025; 11: e72356. CrossRef
Collaboration, Digital Tools, and AI in Academic Writing: Student Experiences, Challenges, and Perspectives
Mehmet Kanık
Dil Eğitimi ve Araştırmaları Dergisi.2025; 11(2): 1038. CrossRef
Use of artificial intelligence and big data in transfusion medicine: An exploratory assessment of status in the Eastern Mediterranean and North Africa region
Arwa Z. Al‐Riyami, Suha Herjes
Vox Sanguinis.2025;[Epub] CrossRef
Detection of Medical Misinformation in Hemangioma Patient Education: Comparative Study of ChatGPT-4o and DeepSeek-R1 Large Language Models
Guoyong Wang, Ye Zhang, Weixin Wang, Yingjie Zhu, Wei Lu, Chaonan Wang, Hui Bi, Xiaonan Yang
JMIR AI.2025; 4: e76372. CrossRef
AI algorithm bias awareness, ethical concerns and use of AI for information retrieval among LIS students in Nigeria
Omorodion Okuonghae, Ifeyinwa Nkechi Okonkwo, Nosakhare Okuonghae, Magnus Osahon Igbinovia
Alexandria: The Journal of National and International Library and Information Issues.2025;[Epub] CrossRef
Perceptions of AI-based tools among polish medical university students
Piotr Ratajczak, Oliwia Słowik, Julia Cynar, Dorota Kopciuch, Anna Paczkowska, Tomasz Zaprutko, Krzysztof Kus
BMC Medical Education.2025;[Epub] CrossRef
Artificial intelligence: a new era in editing academic and popular science writing
Mykola Zhelezniak, Yuliya Mishura
Entsyklopedychnyi visnyk Ukrainy [The Encyclopedia Herald of Ukraine].2025; 17: 9. CrossRef
The legacy and future of recurrent neural networks in personalized medicine: A reflection on the 2024 Nobel Physics Prize
Emily Wittrup, Alan Kay, Jett Rosen, Kuan-Fu Chen, Kayvan Najarian
Biomedical Journal.2025; : 100933. CrossRef
Factors influencing users’ attitudes towards intelligent chatbots in Chinese academic libraries: the role of algorithm literacy
Heng Lu, Xin Li
Humanities and Social Sciences Communications.2025;[Epub] CrossRef
Uso de la inteligencia artificial en estrategias de repetición espaciada para la educación médica y el aprendizaje significativo: revisión sistemática.
Jennifer Carolina Duque Espinel, Andrés Leonardo García Casquete, Yoiler Batista Garcet
Revista Española de Educación Médica.2025;[Epub] CrossRef
Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review
Xiaojun Xu, Yixiao Chen, Jing Miao
Journal of Educational Evaluation for Health Professions.2024; 21: 6. CrossRef
Artificial Intelligence: Fundamentals and Breakthrough Applications in Epilepsy
Wesley Kerr, Sandra Acosta, Patrick Kwan, Gregory Worrell, Mohamad A. Mikati
Epilepsy Currents.2024;[Epub] CrossRef
A Developed Graphical User Interface-Based on Different Generative Pre-trained Transformers Models
Ekrem Küçük, İpek Balıkçı Çiçek, Zeynep Küçükakçalı, Cihan Yetiş, Cemil Çolak
ODÜ Tıp Dergisi.2024; 11(1): 18. CrossRef
Art or Artifact: Evaluating the Accuracy, Appeal, and Educational Value of AI-Generated Imagery in DALL·E 3 for Illustrating Congenital Heart Diseases
Mohamad-Hani Temsah, Abdullah N. Alhuzaimi, Mohammed Almansour, Fadi Aljamaan, Khalid Alhasan, Munirah A. Batarfi, Ibraheem Altamimi, Amani Alharbi, Adel Abdulaziz Alsuhaibani, Leena Alwakeel, Abdulrahman Abdulkhaliq Alzahrani, Khaled B. Alsulaim, Amr Jam
Journal of Medical Systems.2024;[Epub] CrossRef
Authentic assessment in medical education: exploring AI integration and student-as-partners collaboration
Syeda Sadia Fatima, Nabeel Ashfaque Sheikh, Athar Osama
Postgraduate Medical Journal.2024; 100(1190): 959. CrossRef
Comparative performance analysis of large language models: ChatGPT-3.5, ChatGPT-4 and Google Gemini in glucocorticoid-induced osteoporosis
Linjian Tong, Chaoyang Zhang, Rui Liu, Jia Yang, Zhiming Sun
Journal of Orthopaedic Surgery and Research.2024;[Epub] CrossRef
Can AI-Generated Clinical Vignettes in Japanese Be Used Medically and Linguistically?
Yasutaka Yanagita, Daiki Yokokawa, Shun Uchida, Yu Li, Takanori Uehara, Masatomi Ikusaka
Journal of General Internal Medicine.2024; 39(16): 3282. CrossRef
ChatGPT vs. sleep disorder specialist responses to common sleep queries: Ratings by experts and laypeople
Jiyoung Kim, Seo-Young Lee, Jee Hyun Kim, Dong-Hyeon Shin, Eun Hye Oh, Jin A Kim, Jae Wook Cho
Sleep Health.2024; 10(6): 665. CrossRef
Technology integration into Chinese as a foreign language learning in higher education: An integrated bibliometric analysis and systematic review (2000–2024)
Binze Xu
Language Teaching Research.2024;[Epub] CrossRef
The Transformative Power of Generative Artificial Intelligence for Achieving the Sustainable Development Goal of Quality Education
Prema Nedungadi, Kai-Yu Tang, Raghu Raman
Sustainability.2024; 16(22): 9779. CrossRef
Is AI the new course creator
Sheri Conklin, Tom Dorgan, Daisyane Barreto
Discover Education.2024;[Epub] CrossRef
Emergency Medicine Assistants in the Field of Toxicology, Comparison of ChatGPT-3.5 and GEMINI Artificial Intelligence Systems
Hatice Aslı Bedel, Cihan Bedel, Fatih Selvi, Ökkeş Zortuk, Yusuf Karanci
Acta medica Lituanica.2024; 31(2): 294. CrossRef

Technical report

Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2022, Korea: a psychometric study: Yoon Hee Kim, Bo Hyun Kim, Joonki Kim, Bokyoung Jung, Sangyoung Bae; J Educ Eval Health Prof. 2023;20:31. Published online November 22, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.31

5,650 View
266 Download
3 Web of Science
3 Crossref

Abstract PDF Supplementary Material

Purpose
This study presents item analysis results of the 26 health personnel licensing examinations managed by the Korea Health Personnel Licensing Examination Institute (KHPLEI) in 2022.
Methods
The item difficulty index, item discrimination index, and reliability were calculated. The item discrimination index was calculated using a discrimination index based on the upper and lower 27% rule and the item-total correlation.
Results
Out of 468,352 total examinees, 418,887 (89.4%) passed. The pass rates ranged from 27.3% for health educators level 1 to 97.1% for oriental medical doctors. Most examinations had a high average difficulty index, albeit to varying degrees, ranging from 61.3% for prosthetists and orthotists to 83.9% for care workers. The average discrimination index based on the upper and lower 27% rule ranged from 0.17 for oriental medical doctors to 0.38 for radiological technologists. The average item-total correlation ranged from 0.20 for oriental medical doctors to 0.38 for radiological technologists. The Cronbach α, as a measure of reliability, ranged from 0.872 for health educators-level 3 to 0.978 for medical technologists. The correlation coefficient between the average difficulty index and average discrimination index was -0.2452 (P=0.1557), that between the average difficulty index and the average item-total correlation was 0.3502 (P=0.0392), and that between the average discrimination index and the average item-total correlation was 0.7944 (P<0.0001).
Conclusion
This technical report presents the item analysis results and reliability of the recent examinations by the KHPLEI, demonstrating an acceptable range of difficulty index and discrimination index values, as well as good reliability.

Citations

Citations to this article as recorded by

The impact of traditional versus hotspot‐driven anatomy practical assessment on student cognitive processes
Shanna Williams, Helen Kaiser, Sarah Feingold
Anatomical Sciences Education.2025; 18(9): 985. CrossRef
Comparison of AI-generated and clinician-designed multiple-choice questions in emergency medicine exam: a psychometric analysis
Murtaza Kaya, Ertan Sonmez, Ali Halici, Harun Yildirim, Abdil Coskun
BMC Medical Education.2025;[Epub] CrossRef
Measuring Children’s Knowledge of Acute Rheumatic Fever and Rheumatic Heart Disease: Development and Validation of RHD Knowledge Scale in Nepal
Sandeepa Karki, Shishir Paudel, Rudra Prakash Hada, Prashant Khadka, Ganesh Shetty, Dhurba Khatri, Yamuna Chhetri, Sakar Khanal, Bhagawan Koirala
INQUIRY: The Journal of Health Care Organization, Provision, and Financing.2025;[Epub] CrossRef

Research articles

Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study: Aleksandra Ignjatović, Lazar Stevanović; J Educ Eval Health Prof. 2023;20:28. Published online October 16, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.28

7,898 View
270 Download
28 Web of Science
30 Crossref

Abstract PDF Supplementary Material

Purpose
This study aimed to assess the performance of ChatGPT (GPT-3.5 and GPT-4) as a study tool in solving biostatistical problems and to identify any potential drawbacks that might arise from using ChatGPT in medical education, particularly in solving practical biostatistical problems.
Methods
ChatGPT was tested to evaluate its ability to solve biostatistical problems from the Handbook of Medical Statistics by Peacock and Peacock in this descriptive study. Tables from the problems were transformed into textual questions. Ten biostatistical problems were randomly chosen and used as text-based input for conversation with ChatGPT (versions 3.5 and 4).
Results
GPT-3.5 solved 5 practical problems in the first attempt, related to categorical data, cross-sectional study, measuring reliability, probability properties, and the t-test. GPT-3.5 failed to provide correct answers regarding analysis of variance, the chi-square test, and sample size within 3 attempts. GPT-4 also solved a task related to the confidence interval in the first attempt and solved all questions within 3 attempts, with precise guidance and monitoring.
Conclusion
The assessment of both versions of ChatGPT performance in 10 biostatistical problems revealed that GPT-3.5 and 4’s performance was below average, with correct response rates of 5 and 6 out of 10 on the first attempt. GPT-4 succeeded in providing all correct answers within 3 attempts. These findings indicate that students must be aware that this tool, even when providing and calculating different statistical analyses, can be wrong, and they should be aware of ChatGPT’s limitations and be careful when incorporating this model into medical education.

Citations

Citations to this article as recorded by

Can Generative AI and ChatGPT Outperform Humans on Cognitive-Demanding Problem-Solving Tasks in Science?
Xiaoming Zhai, Matthew Nyaaba, Wenchao Ma
Science & Education.2025; 34(2): 649. CrossRef
From statistics to deep learning: Using large language models in psychiatric research
Yining Hua, Andrew Beam, Lori B. Chibnik, John Torous
International Journal of Methods in Psychiatric Research.2025;[Epub] CrossRef
Assessing the Current Limitations of Large Language Models in Advancing Health Care Education
JaeYong Kim, Bathri Narayan Vajravelu
JMIR Formative Research.2025; 9: e51319. CrossRef
ChatGPT for Univariate Statistics: Validation of AI-Assisted Data Analysis in Healthcare Research
Michael R Ruta, Tony Gaidici, Chase Irwin, Jonathan Lifshitz
Journal of Medical Internet Research.2025; 27: e63550. CrossRef
ChatGPT-Assisted Deep Learning Models for Influenza-Like Illness Prediction in Mainland China: Time Series Analysis
Weihong Huang, Wudi Wei, Xiaotao He, Baili Zhan, Xiaoting Xie, Meng Zhang, Shiyi Lai, Zongxiang Yuan, Jingzhen Lai, Rongfeng Chen, Junjun Jiang, Li Ye, Hao Liang
Journal of Medical Internet Research.2025; 27: e74423. CrossRef
Confirming SPSS Results With ChatGPT-4 and o3-mini Models
Frederick Strale, Isaac Riddle, Bowen Geng, Blake Oxford, Malia Kah, Robert Sherwin
Cureus.2025;[Epub] CrossRef
A whole new world, a new fantastic point of view: Charting unexplored territories in consumer research with generative artificial intelligence
Kiwoong Yoo, Michael Haenlein, Kelly Hewett
Journal of the Academy of Marketing Science.2025; 53(3): 723. CrossRef
One year in the classroom with ChatGPT: empirical insights and transformative impacts
Feng Guo, Tian Li, Christopher J. L. Cunningham
Frontiers in Education.2025;[Epub] CrossRef
A Comparative Study of the Advantages and Disadvantages of DeepSeek and SPSS in Statistical Analysis
沙沙庞
Statistics and Application.2025; 14(06): 172. CrossRef
AI‐Assisted Statistical Review: Could It Have Averted Retractions? A Case‐Based Perspective From Immunology
Michal Ordak
Allergy.2025; 80(12): 3441. CrossRef
The impact of generative AI on critical thinking skills: a systematic review, conceptual framework and future research directions
Mohamed Y. I. Helal, Ibrahim A. Elgendy, Mousa Ahmed Albashrawi, Yogesh K. Dwivedi, Mohammad S. Al-Ahmadi, Il Jeon
Information Discovery and Delivery.2025;[Epub] CrossRef
ChatGPT's performance in sample size estimation: a preliminary study on the capabilities of artificial intelligence
Paul Sebo, Ting Wang
Family Practice.2025;[Epub] CrossRef
ChatGPT in Medical Education: Bibliometric and Visual Analysis
Yuning Zhang, Xiaolu Xie, Qi Xu
JMIR Medical Education.2025; 11: e72356. CrossRef
ChatGPT’s progress over time: A longitudinal enhancing biostatistical problem-solving in medical education
Aleksandra Ignjatović, Marija Anđelković Apostolović, Lazar Stevanović, Pavle Radovanović, Marija Topalović, Tamara Filipović, Suzana Otašević
Health Informatics Journal.2025;[Epub] CrossRef
AI-assisted statistical review of 100 oncology research articles: compliance with SAMPL guidelines
Michal Ordak
Current Research in Translational Medicine.2025; 73(4): 103544. CrossRef
Applications, Challenges, and Prospects of Generative Artificial Intelligence Empowering Medical Education: Scoping Review
Yuhang Lin, Zhiheng Luo, Zicheng Ye, Nuoxi Zhong, Lijian Zhao, Long Zhang, Xiaolan Li, Zetao Chen, Yijia Chen
JMIR Medical Education.2025; 11: e71125. CrossRef
Generative Artificial Intelligence for Data Analysis: A Randomised Controlled Trial in a Public Health Research Institute
Tafadzwa Dhokotera, Nandi Joubert, Aline Veillat, Christoph Pimmer, Karin Gross, Marco Waser, Jan Hattendorf, Julia Bohlius
International Journal of Public Health.2025;[Epub] CrossRef
ChatGPT as a Tool for Biostatisticians: A Tutorial on Applications, Opportunities, and Limitations
Dennis Dobler, Harald Binder, Anne‐Laure Boulesteix, Jan‐Bernd Igelmann, David Köhler, Ulrich Mansmann, Markus Pauly, André Scherag, Matthias Schmid, Amani Al Tawil, Susanne Weber
Statistics in Medicine.2025;[Epub] CrossRef
The Pediatric Surgeon's AI Toolbox: How Large Language Models Like ChatGPT Are Simplifying Practice and Expanding Global Access
Carlos Andres Colunga Tinajero
European Journal of Pediatric Surgery.2025;[Epub] CrossRef
Reliability of ChatGPT-4o in analysing medical data: a test case study on patients at risk for limb amputation
Liat Toderis, Iris Reychav, Roger McHaney, Bernice Oberman, Chen Speter, Ronen Loebstein
Health Systems.2025; : 1. CrossRef
Statistical analysis using ChatGPT in medical research
Soo-Nyung Kim
Obstetrics & Gynecology Science.2025; 68(6): 467. CrossRef
Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review
Xiaojun Xu, Yixiao Chen, Jing Miao
Journal of Educational Evaluation for Health Professions.2024; 21: 6. CrossRef
Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy
Ambadasu Bharatha, Nkemcho Ojeh, Ahbab Mohammad Fazle Rabbi, Michael Campbell, Kandamaran Krishnamurthy, Rhaheem Layne-Yarde, Alok Kumar, Dale Springer, Kenneth Connell, Md Anwarul Majumder
Advances in Medical Education and Practice.2024; Volume 15: 393. CrossRef
Revolutionizing Cardiology With Words: Unveiling the Impact of Large Language Models in Medical Science Writing
Abhijit Bhattaru, Naveena Yanamala, Partho P. Sengupta
Canadian Journal of Cardiology.2024; 40(10): 1950. CrossRef
ChatGPT in medicine: prospects and challenges: a review article
Songtao Tan, Xin Xin, Di Wu
International Journal of Surgery.2024; 110(6): 3701. CrossRef
In-depth analysis of ChatGPT’s performance based on specific signaling words and phrases in the question stem of 2377 USMLE step 1 style questions
Leonard Knoedler, Samuel Knoedler, Cosima C. Hoch, Lukas Prantl, Konstantin Frank, Laura Soiderer, Sebastian Cotofana, Amir H. Dorafshar, Thilo Schenck, Felix Vollbach, Giuseppe Sofo, Michael Alfertshofer
Scientific Reports.2024;[Epub] CrossRef
Evaluating the quality of responses generated by ChatGPT
Danimir Mandić, Gordana Miščević, Ljiljana Bujišić
Metodicka praksa.2024; 27(1): 5. CrossRef
A Comparative Evaluation of Statistical Product and Service Solutions (SPSS) and ChatGPT-4 in Statistical Analyses
Al Imran Shahrul, Alizae Marny F Syed Mohamed
Cureus.2024;[Epub] CrossRef
ChatGPT and Other Large Language Models in Medical Education — Scoping Literature Review
Alexandra Aster, Matthias Carl Laupichler, Tamina Rockwell-Kollmann, Gilda Masala, Ebru Bala, Tobias Raupach
Medical Science Educator.2024; 35(1): 555. CrossRef
Exploring the potential of large language models for integration into an academic statistical consulting service–the EXPOLS study protocol
Urs Alexander Fichtner, Jochen Knaus, Erika Graf, Georg Koch, Jörg Sahlmann, Dominikus Stelzer, Martin Wolkewitz, Harald Binder, Susanne Weber, Bekalu Tadesse Moges
PLOS ONE.2024; 19(12): e0308375. CrossRef

Development and validation of the student ratings in clinical teaching scale in Australia: a methodological study: Pin-Hsiang Huang, Anthony John O’Sullivan, Boaz Shulruf; J Educ Eval Health Prof. 2023;20:26. Published online September 5, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.26

3,537 View
188 Download
1 Web of Science

Abstract PDF Supplementary Material: Purpose
This study aimed to devise a valid measurement for assessing clinical students’ perceptions of teaching practices.
Methods
A new tool was developed based on a meta-analysis encompassing effective clinical teaching-learning factors. Seventy-nine items were generated using a frequency (never to always) scale. The tool was applied to the University of New South Wales year 2, 3, and 6 medical students. Exploratory and confirmatory factor analysis (exploratory factor analysis [EFA] and confirmatory factor analysis [CFA], respectively) were conducted to establish the tool’s construct validity and goodness of fit, and Cronbach’s α was used for reliability.
Results
In total, 352 students (44.2%) completed the questionnaire. The EFA identified student-centered learning, problem-solving learning, self-directed learning, and visual technology (reliability, 0.77 to 0.89). CFA showed acceptable goodness of fit (chi-square P<0.01, comparative fit index=0.930 and Tucker-Lewis index=0.917, root mean square error of approximation=0.069, standardized root mean square residual=0.06).
Conclusion
The established tool—Student Ratings in Clinical Teaching (STRICT)—is a valid and reliable tool that demonstrates how students perceive clinical teaching efficacy. STRICT measures the frequency of teaching practices to mitigate the biases of acquiescence and social desirability. Clinical teachers may use the tool to adapt their teaching practices with more active learning activities and to utilize visual technology to facilitate clinical learning efficacy. Clinical educators may apply STRICT to assess how these teaching practices are implemented in current clinical settings.

What impacts students’ satisfaction the most from Medicine Student Experience Questionnaire in Australia: a validity study: Pin-Hsiang Huang, Gary Velan, Greg Smith, Melanie Fentoullis, Sean Edward Kennedy, Karen Jane Gibson, Kerry Uebel, Boaz Shulruf; J Educ Eval Health Prof. 2023;20:2. Published online January 18, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.2

4,486 View
198 Download
4 Web of Science
3 Crossref

Abstract PDF Supplementary Material

Purpose
This study evaluated the validity of student feedback derived from Medicine Student Experience Questionnaire (MedSEQ), as well as the predictors of students’ satisfaction in the Medicine program.
Methods
Data from MedSEQ applying to the University of New South Wales Medicine program in 2017, 2019, and 2021 were analyzed. Confirmatory factor analysis (CFA) and Cronbach’s α were used to assess the construct validity and reliability of MedSEQ respectively. Hierarchical multiple linear regressions were used to identify the factors that most impact students’ overall satisfaction with the program.
Results
A total of 1,719 students (34.50%) responded to MedSEQ. CFA showed good fit indices (root mean square error of approximation=0.051; comparative fit index=0.939; chi-square/degrees of freedom=6.429). All factors yielded good (α>0.7) or very good (α>0.8) levels of reliability, except the “online resources” factor, which had acceptable reliability (α=0.687). A multiple linear regression model with only demographic characteristics explained 3.8% of the variance in students’ overall satisfaction, whereas the model adding 8 domains from MedSEQ explained 40%, indicating that 36.2% of the variance was attributable to students’ experience across the 8 domains. Three domains had the strongest impact on overall satisfaction: “being cared for,” “satisfaction with teaching,” and “satisfaction with assessment” (β=0.327, 0.148, 0.148, respectively; all with P<0.001).
Conclusion
MedSEQ has good construct validity and high reliability, reflecting students’ satisfaction with the Medicine program. Key factors impacting students’ satisfaction are the perception of being cared for, quality teaching irrespective of the mode of delivery and fair assessment tasks which enhance learning.

Citations

Citations to this article as recorded by

Linking quality perception to satisfaction in private universities: a mediated marketing model
Sk. Shahabuddin Ahmmed, Md. Sharif Hassan, Mohammad Bin Amin, Julinawati Binti Suanda, Shamsad Ahmed, Veronika Fenyves
Cogent Business & Management.2025;[Epub] CrossRef
Mentor‐Student Relationship and Graduate Students' Satisfaction With Mentors: A Moderated Mediation Model
Xingzi Chen, Jiaqian Song, Hanjing Wen, Liuyi Zhang
Journal of Advanced Nursing.2025;[Epub] CrossRef
Mental health and quality of life across 6 years of medical training: A year-by-year analysis
Natalia de Castro Pecci Maddalena, Alessandra Lamas Granero Lucchetti, Ivana Lucia Damasio Moutinho, Oscarina da Silva Ezequiel, Giancarlo Lucchetti
International Journal of Social Psychiatry.2024; 70(2): 298. CrossRef

Acceptability of the 8-case objective structured clinical examination of medical students in Korea using generalizability theory: a reliability study: Song Yi Park, Sang-Hwa Lee, Min-Jeong Kim, Ki-Hwan Ji, Ji Ho Ryu; J Educ Eval Health Prof. 2022;19:26. Published online September 8, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.26

5,191 View
246 Download
1 Web of Science
1 Crossref

Abstract PDF Supplementary Material

Purpose
This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT).
Methods
This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada).
Results
All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%.
Conclusion
The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.

Citations

Citations to this article as recorded by

Applying the Generalizability Theory to Identify the Sources of Validity Evidence for the Quality of Communication Questionnaire
Flávia Del Castanhel, Fernanda R. Fonseca, Luciana Bonnassis Burg, Leonardo Maia Nogueira, Getúlio Rodrigues de Oliveira Filho, Suely Grosseman
American Journal of Hospice and Palliative Medicine®.2024; 41(7): 792. CrossRef

Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study: Janghee Park; J Educ Eval Health Prof. 2022;19:23. Published online August 31, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.23

5,106 View
188 Download
5 Web of Science
4 Crossref

Abstract PDF Supplementary Material

Purpose
The percent Angoff (PA) method has been recommended as a reliable method to set the cutoff score instead of a fixed cut point of 60% in the Korean Medical Licensing Examination (KMLE). The yes/no Angoff (YNA) method, which is easy for panelists to judge, can be considered as an alternative because the KMLE has many items to evaluate. This study aimed to compare the cutoff score and the reliability depending on whether the PA or the YNA standard-setting method was used in the KMLE.
Methods
The materials were the open-access PA data of the KMLE. The PA data were converted to YNA data in 5 categories, in which the probabilities for a “yes” decision by panelists were 50%, 60%, 70%, 80%, and 90%. SPSS for descriptive analysis and G-string for generalizability theory were used to present the results.
Results
The PA method and the YNA method counting 60% as “yes,” estimated similar cutoff scores. Those cutoff scores were deemed acceptable based on the results of the Hofstee method. The highest reliability coefficients estimated by the generalizability test were from the PA method and the YNA method, with probabilities of 70%, 80%, 60%, and 50% for deciding “yes,” in descending order. The panelist’s specialty was the main cause of the error variance. The error size was similar regardless of the standard-setting method.
Conclusion
The above results showed that the PA method was more reliable than the YNA method in estimating the cutoff score of the KMLE. However, the YNA method with a 60% probability for deciding “yes” also can be used as a substitute for the PA method in estimating the cutoff score of the KMLE.

Citations

Citations to this article as recorded by

Standard setting for dental knowledge tests: reproducibility of the modified Angoff and Ebel method across judges
Ting Khee Ho, Noor Lide Abu Kassim, Lucy O’Malley, Reza Vahid Roudsari
BMC Medical Education.2025;[Epub] CrossRef
Angoff methods in standard setting in health professional education: a systematic review and meta-analysis
Kannan Sridharan, Gowri Sivaramakrishnan
BMC Medical Education.2025;[Epub] CrossRef
Issues in the 3rd year of the COVID-19 pandemic, including computer-based testing, study design, ChatGPT, journal metrics, and appreciation to reviewers
Sun Huh
Journal of Educational Evaluation for Health Professions.2023; 20: 5. CrossRef
Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study
Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park
Journal of Educational Evaluation for Health Professions.2022; 19: 33. CrossRef

Development and validation of a measurement scale to assess nursing students’ readiness for the flipped classroom in Sri Lanka: Punithalingam Youhasan, Yan Chen, Mataroria Lyndon, Marcus Alexander Henning; J Educ Eval Health Prof. 2020;17:41. Published online December 14, 2020; DOI: https://doi.org/10.3352/jeehp.2020.17.41

9,443 View
306 Download
10 Web of Science
8 Crossref

Abstract PDF Supplementary Material

Purpose
The aim of this study was to develop and validate a scale to measure nursing students’ readiness for the flipped classroom in Sri Lanka.
Methods
A literature review provided the theoretical framework for developing the Nursing Students’ Readiness for Flipped Classroom (NSR-FC) questionnaire. Five content experts evaluated the NSR-FC, and content validity indices (CVI) were calculated. Cross-sectional surveys among 355 undergraduate nursing students from 3 state universities in Sri Lanka were carried out to assess the psychometric properties of the NSR-FC. Principal component analysis (PCA, n=265), internal consistency (using the Cronbach α coefficient, n=265), and confirmatory factor analysis (CFA, n=90) were done to test construct validity and reliability.
Results
Thirty-seven items were included in the NSR-FC for content validation, resulting in an average scale CVI of 0.94. Two items received item level CVI of less than 0.78. The factor structures of the 35 items were explored through PCA with orthogonal factor rotation, culminating in the identification of 5 factors. These factors were classified as technological readiness, environmental readiness, personal readiness, pedagogical readiness, and interpersonal readiness. The NSR-FC also showed an overall acceptable level of internal consistency (Cronbach α=0.9). CFA verified a 4-factor model (excluding the interpersonal readiness factor) and 20 items that achieved acceptable fit (standardized root mean square residual=0.08, root mean square error of approximation=0.08, comparative fit index=0.87, and χ2/degrees of freedom=1.57).
Conclusion
The NSR-FC, as a 4-factor model, is an acceptable measurement scale for assessing nursing students’ readiness for the flipped classroom in terms of its construct validity and reliability.

Citations

Citations to this article as recorded by

AI readiness scale for teachers: Development and validation
Mehmet Ramazanoglu, Tayfun Akın
Education and Information Technologies.2025; 30(6): 6869. CrossRef
Design and validation of a preliminary instrument to contextualize interactions through information technologies of health professionals
José Fidencio López Luna, Eddie Nahúm Armendáriz Mireles, Marco Aurelio Nuño Maganda, Hiram Herrera Rivas, Rubén Machucho Cadena, Jorge Arturo Hernández Almazán
Health Informatics Journal.2024;[Epub] CrossRef
Content validity of the Constructivist Learning in Higher Education Settings (CLHES) scale in the context of the flipped classroom in higher education
Turki Mesfer Alqahtani, Farrah Dina Yusop, Siti Hajar Halili
Humanities and Social Sciences Communications.2023;[Epub] CrossRef
The intensivist's assessment of gastrointestinal function: A pilot study
Varsha M. Asrani, Colin McArthur, Ian Bissett, John A. Windsor
Australian Critical Care.2022; 35(6): 636. CrossRef
Psychometric evidence of a perception scale about covid-19 vaccination process in Peruvian dentists: a preliminary validation
César F. Cayo-Rojas, Nancy Córdova-Limaylla, Gissela Briceño-Vergel, Marysela Ladera-Castañeda, Hernán Cachay-Criado, Carlos López-Gurreonero, Alberto Cornejo-Pinto, Luis Cervantes-Ganoza
BMC Health Services Research.2022;[Epub] CrossRef
Implementation of a Web-Based Educational Intervention for Promoting Flipped Classroom Pedagogy: A Mixed-Methods Study
Punithalingam Youhasan, Mataroria P. Lyndon, Yan Chen, Marcus A. Henning
Medical Science Educator.2022; 33(1): 91. CrossRef
Assess the feasibility of flipped classroom pedagogy in undergraduate nursing education in Sri Lanka: A mixed-methods study
Punithalingam Youhasan, Yan Chen, Mataroria Lyndon, Marcus A. Henning, Gwo-Jen Hwang
PLOS ONE.2021; 16(11): e0259003. CrossRef
Newly appointed medical faculty members’ self-evaluation of their educational roles at the Catholic University of Korea College of Medicine in 2020 and 2021: a cross-sectional survey-based study
Sun Kim, A Ra Cho, Chul Woon Chung
Journal of Educational Evaluation for Health Professions.2021; 18: 28. CrossRef

Development and validation of the Hocus Focus Magic Performance Evaluation Scale for health professions personnel in the United States: Kevin Spencer, Hon Keung Yuen, Max Darwin, Gavin Jenkins, Kimberly Kirklin; J Educ Eval Health Prof. 2019;16:8. Published online April 10, 2019; DOI: https://doi.org/10.3352/jeehp.2019.16.8

20,897 View
246 Download
4 Web of Science
4 Crossref

Abstract PDF Supplementary Material

Purpose
This study was conducted to describe the development and validation of the Hocus Focus Magic Performance Evaluation Scale (HFMPES), which is used to evaluate the competency of health professions personnel in delivering magic tricks as a therapeutic modality.
Methods
A 2-phase validation process was used. Phase I (content validation) involved 16 magician judges who independently rated the relevance of each of the 5 items in the HFMPES and established the veracity of its content. Phase II evaluated the psychometric properties of the HFMPES. This process involved 2 magicians using the HFMPES to independently evaluate 73 occupational therapy graduate students demonstrating 3 magic tricks.
Results
The HFMPES achieved an excellent scale-content validity index of 0.99. Exploratory factor analysis of the HFMPES scores revealed 1 distinct factor with alpha coefficients ≥0.8 across the 3 magic tricks. The construct validity of the HFMPES scores was further supported by evidence from a known-groups analysis, in which the Mann–Whitney U-test showed significant difference in HFMPES scores between participants with different levels of experience in delivering the 3 magic tricks. The inter-rater reliability coefficients were ≥0.75 across the 3 magic tricks, indicating that the competency of health professions personnel in delivering the 3 magic tricks could be evaluated precisely.
Conclusion
Preliminary evidence supported the content and construct validity of the HFMPES, which was found to have good internal consistency and inter-rater reliability in evaluating health professions personnel’s competency in delivering magic tricks.

Citations

Citations to this article as recorded by

A Magic Trick Training Program to Improve Social Skills and Self-Esteem in Adolescents With Autism Spectrum Disorder
Hon K. Yuen, Kevin Spencer, Lauren Edwards, Kimberly Kirklin, Gavin R. Jenkins
The American Journal of Occupational Therapy.2023;[Epub] CrossRef
MAGNITIVE: Effectiveness and Feasibility of a Cognitive Training Program Through Magic Tricks for Children With Attention Deficit and Hyperactivity Disorder. A Second Clinical Trial in Community Settings
Saray Bonete, Ángela Osuna, Clara Molinero, Inmaculada García-Font
Frontiers in Psychology.2021;[Epub] CrossRef
Contribution of a virtual magic camp to enhancing self-esteem in children with ADHD: A pilot study
Hon K. Yuen, Kevin Spencer, Kimberly Kirklin, Lauren Edwards, Gavin R. Jenkins
Health Psychology Research.2021; 9(1): 1. CrossRef
Evaluation of a Magic Camp for Children with Hemiparesis: A Pilot Study
Kevin Spencer, Hon K. Yuen, Gavin R. Jenkins, Kimberly Kirklin, Angla R. Griffin, Laura K. Vogtle, Drew Davis
Occupational Therapy In Health Care.2020; 34(2): 155. CrossRef

Research Articles

Psychometric properties of a novel knowledge assessment tool of mechanical ventilation for emergency medicine residents in the northeastern United States: Jeremy B. Richards, Tania D. Strout, Todd A. Seigel, Susan R. Wilcox; J Educ Eval Health Prof. 2016;13:10. Published online February 16, 2016; DOI: https://doi.org/10.3352/jeehp.2016.13.10

29,931 View
181 Download
5 Web of Science
6 Crossref

Abstract PDF

Purpose
Prior descriptions of the psychometric properties of validated knowledge assessment tools designed to determine Emergency medicine (EM) residents understanding of physiologic and clinical concepts related to mechanical ventilation are lacking. In this setting, we have performed this study to describe the psychometric and performance properties of a novel knowledge assessment tool that measures EM residents’ knowledge of topics in mechanical ventilation.
Methods
Results from a multicenter, prospective, survey study involving 219 EM residents from 8 academic hospitals in northeastern United States were analyzed to quantify reliability, item difficulty, and item discrimination of each of the 9 questions included in the knowledge assessment tool for 3 weeks, beginning in January 2013.
Results
The response rate for residents completing the knowledge assessment tool was 68.6% (214 out of 312 EM residents). Reliability was assessed by both Cronbach’s alpha coefficient (0.6293) and the Spearman-Brown coefficient (0.6437). Item difficulty ranged from 0.39 to 0.96, with a mean item difficulty of 0.75 for all 9 questions. Uncorrected item discrimination values ranged from 0.111 to 0.556. Corrected item-total correlations were determined by removing the question being assessed from analysis, resulting in a range of item discrimination from 0.139 to 0.498.
Conclusion
Reliability, item difficulty and item discrimination were within satisfactory ranges in this study, demonstrating acceptable psychometric properties of this knowledge assessment tool. This assessment indicates that this knowledge assessment tool is sufficiently rigorous for use in future research studies or for assessment of EM residents for evaluative purposes.

Citations

Citations to this article as recorded by

Management of Mechanical Ventilation in Emergency Medicine: A Scoping Review
Robert J. Klemisch, Mitchell S. Hymowitz, Ryan J. Alcantara, Brendan F. Mullan, Margaret L. Davis, Rachel Blume, Nicholas J. Johnson, Brian M. Fuller
JACEP Open.2026; 7(1): 100297. CrossRef
Educational Environment: Students’ Perceptions Using the Dundee Ready Educational Environment Measure Inventory
Sneha Sathyan, Sicy Maria, Siroslin S Mahitha, Smiji Jinny, Sonia, Soumya Dsouza, Shycil Mathew
Journal of Health and Allied Sciences NU.2025; 15: 444. CrossRef
Comparison of three methods for teaching mechanical ventilation in an emergency setting to sixth-year medical students: a randomized trial
Fernando Sabia Tallo, Letícia Sandre Vendrame, André Luciano Baitello
Revista da Associação Médica Brasileira.2020; 66(10): 1409. CrossRef
Critical Appraisal of Emergency Medicine Educational Research: The Best Publications of 2016
Nicole M. Dubosh, Jaime Jordan, Lalena M. Yarris, Edward Ullman, Joshua Kornegay, Daniel Runde, Amy Miller Juve, Jonathan Fisher, Teresa Chan
AEM Education and Training.2019; 3(1): 58. CrossRef
Mechanical Ventilation Training During Graduate Medical Education: Perspectives and Review of the Literature
Jonathan M. Keller, Dru Claar, Juliana Carvalho Ferreira, David C. Chu, Tanzib Hossain, William Graham Carlos, Jeffrey A. Gold, Stephanie A. Nonas, Nitin Seam
Journal of Graduate Medical Education.2019; 11(4): 389. CrossRef
Development and validation of a questionnaire to assess the knowledge of mechanical ventilation in urgent care among students in their last-year medical course in Brazil
Fernando Sabia Tallo, Simone de Campos Vieira Abib, Andre Luciano Baitello, Renato Delascio Lopes
Clinics.2019; 74: e663. CrossRef

Validation of a clinical critical thinking skills test in nursing: Sujin Shin, Dukyoo Jung, Sungeun Kim; J Educ Eval Health Prof. 2015;12:1. Published online January 27, 2015; DOI: https://doi.org/10.3352/jeehp.2015.12.1

37,194 View
306 Download
6 Web of Science
11 Crossref

Abstract PDF

Purpose
The purpose of this study was to develop a revised version of the clinical critical thinking skills test (CCTS) and to subsequently validate its performance. Methods: This study is a secondary analysis of the CCTS. Data were obtained from a convenience sample of 284 college students in June 2011. Thirty items were analyzed using item response theory and test reliability was assessed. Test-retest reliability was measured using the results of 20 nursing college and graduate school students in July 2013. The content validity of the revised items was analyzed by calculating the degree of agreement between instrument developer intention in item development and the judgments of six experts. To analyze response process validity, qualitative data related to the response processes of nine nursing college students obtained through cognitive interviews were analyzed. Results: Out of initial 30 items, 11 items were excluded after the analysis of difficulty and discrimination parameter. When the 19 items of the revised version of the CCTS were analyzed, levels of item difficulty were found to be relatively low and levels of discrimination were found to be appropriate or high. The degree of agreement between item developer intention and expert judgments equaled or exceeded 50%. Conclusion: From above results, evidence of the response process validity was demonstrated, indicating that subjects respondeds as intended by the test developer. The revised 19-item CCTS was found to have sufficient reliability and validity and will therefore represents a more convenient measurement of critical thinking ability.

Citations

Citations to this article as recorded by

Clinical Judgment Model‐Based Critical Reflection Program for Newly Graduated Nurses: A Nonrandomized Controlled Trial
Ae Ran Kim, Jeong Hee Hong, Kyeongsug Kim, Yuna Kim, Jung Min Lee, Heejin Lee, Ji Hyun Yoon, Mi Soon Kim
Research in Nursing & Health.2025; 48(2): 234. CrossRef
Utilizing cognitive interview in the item refinement of the Blended Teaching Assessment Tool (BTAT) for Health Professions Education
Maria Teresita B. Dalusong, Glenda Sanggalang Ogerio, Valentin C. Dones, Maria Elizabeth M. Grageda
Philippine Journal of Health Research and Development.2025; 29(2): 54. CrossRef
Meta-analysis in Physical Therapy Education Research
Mitch Wolden, Brent Hill, Sara Farquhar Voorhees
Journal of Physical Therapy Education.2019; 33(1): 78. CrossRef
Effects of a work-based critical reflection program for novice nurses
Yeon Hee Kim, Ja Min, Soon Hee Kim, Sujin Shin
BMC Medical Education.2018;[Epub] CrossRef
Measurement of critical thinking, clinical reasoning, and clinical judgment in culturally diverse nursing students – A literature review
Christine L. Sommers
Nurse Education in Practice.2018; 30: 91. CrossRef
Individual and School Factors Affecting Critical Thinking Ability among Nursing Students
Sujin Shin, Inhee Park, Eunhee Hwang, Dukyoo Jung, Kon Hee Kim
Korean Medical Education Review.2018; 20(1): 44. CrossRef
The Health Professions Education Pathway: Preparing Students, Residents, and Fellows to Become Future Educators
H. Carrie Chen, Maria A. Wamsley, Amin Azzam, Katherine Julian, David M. Irby, Patricia S. O'Sullivan
Teaching and Learning in Medicine.2017; 29(2): 216. CrossRef
Cultivating Critical Thinking Using Virtual Interactive Case Studies
Susan M. Burke
Journal of Pediatric Nursing.2017; 33: 94. CrossRef
Encouraging Critical Clinical Thinking (CCT) Skills in First-Year Veterinary Students
Duncan C. Ferguson, Leslie Klis McNeil, David J. Schaeffe, Eric M. Mills
Journal of Veterinary Medical Education.2017; 44(3): 531. CrossRef
Developing a Foundation for Interprofessional Education Within Nursing and Medical Curricula
Trisha Leann Horsley, Trent Reed, Keith Muccino, Donna Quinones, Viva Jo Siddall, Janet McCarthy
Nurse Educator.2016; 41(5): 234. CrossRef
Supervision in psychiatry
Joanna MacDonald, Pete M. Ellis
Current Opinion in Psychiatry.2012; 25(4): 322. CrossRef

Assessing the reliability and validity of the Revised Two Factor Study Process Questionnaire (R-SPQ2F) in Ghanaian medical students: Victor Mogre, Anthony Amalba; J Educ Eval Health Prof. 2014;11:19. Published online August 15, 2014; DOI: https://doi.org/10.3352/jeehp.2014.11.19

28,969 View
217 Download
9 Web of Science
12 Crossref

Abstract PDF

Purpose
We investigated the validity and reliability of the Revised Two Factor Study Process Questionnaire (R-SPQ2F) in preclinical students in Ghana. Methods: The R-SPQ2F was administered to 189 preclinical students of the University for Development Studies, School of Medicine and Health Sciences. Both descriptive and inferential statistics with Cronbach’s alpha test and factor analysis were done. Results: The mean age of the students was 22.69 ± 0.18 years, 60.8% (n = 115) were males and 42.3% (n = 80) were in their second year of medical training. The students had higher mean deep approach scores (31.23 ± 7.19) than that of surface approach scores (22.62 ± 6.48). Findings of the R-SPQ2F gave credence to a solution of two-factors indicating deep and surface approaches accounting for 49.80% and 33.57%, respectively, of the variance. The scales of deep approach (Cronbach’s alpha, 0.80) and surface approach (Cronbach’s alpha, 0.76) and their subscales demonstrated an internal consistency that was good. The factorial validity was comparable to other studies. Conclusion: Our study confirms the construct validity and internal consistency of the R-SPQ2F for measuring approaches to learning in Ghanaian preclinical students. Deep approach was the most dominant learning approach among the students. The questionnaire can be used to measure students’ approaches to learning in Ghana and in other African countries.

Citations

Citations to this article as recorded by

The Impact of Learning Approach and Study Habits on Student Performance in an Undergraduate Human Anatomy Course
Andrew R. Thompson, Amulya Vankayalapati, Neal Taliwal
Medical Science Educator.2025; 35(4): 2135. CrossRef
A comparison of two learning approach inventories and their utility in predicting examination performance and study habits
Andrew R. Thompson
Advances in Physiology Education.2024; 48(2): 164. CrossRef
Learning approaches in students of the areas of Health Sciences belonging to a regional university in Chile
Mónica Cardona-V, Aurora Henríquez-Melgarejo, Trinidad Cifuentes Muñoz, Carolina Luengo, Ángel Roco-Videla
Salud, Ciencia y Tecnología.2024;[Epub] CrossRef
Relationship between learning approach, Bloom’s taxonomy, and student performance in an undergraduate Human Anatomy course
Andrew R. Thompson, Logan P. O. Lake
Advances in Health Sciences Education.2023; 28(4): 1115. CrossRef
Evaluation of Construct Validity and Reliability of the Arabic and English Versions of Biggs Study Process Scale Among Saudi University Students
Nadeem Shafique Butt, Muhammad Abid Bashir, Sami Hamdan Alzahrani, Zohair Jamil Gazzaz, Ahmad Azam Malik
Sage Open.2023;[Epub] CrossRef
Development and Preliminary Validation of the Physical Education-Study Process Questionnaire : Insights for Physical Education University Students
Amayra Tannoubi, Noomen Guelmami, Tore Bonsaksen, Nasr Chalghaf, Fairouz Azaiez, Nicola Luigi Bragazzi
Frontiers in Public Health.2022;[Epub] CrossRef
The Association Between Learning Styles, Time Management Skills and Pharmacology Academic Performance Among First Year Medical Students in Universiti Putra Malaysia (UPM) During the COVID-19 Pandemic
Azmah Sa'at, Suryati Mohd. Thani, Safuraa Salihan, Nur Izah Ab. Razak, Siti Saleha Masrudin
Malaysian Journal of Medicine and Health Sciences.2022; 18(s14): 94. CrossRef
Outcomes of 2 Multimodal Human Anatomy Courses Among Doctor of Physical Therapy Students (Entry-Level): A Quasi-experimental Study
Sara F. Maher, Deborah J. Doherty
Journal of Physical Therapy Education.2021; 35(1): 38. CrossRef
Study Approaches of Life Science Students Using the Revised Two-Factor Study Process Questionnaire (R-SPQ-2F)
Miguel Leiva-Brondo, Jaime Cebolla-Cornejo, Rosa Peiró, Nuria Andrés-Colás, Cristina Esteras, María Ferriol, Hugo Merle, María José Díez, Ana Pérez-de-Castro
Education Sciences.2020; 10(7): 173. CrossRef
Assessing ‘approaches to learning’ in Botswana, Ghana and Kenya
Caine Rolleston, Rebecca Schendel, Ana M Grijalva Espinosa
Research in Comparative and International Education.2019; 14(1): 118. CrossRef
Psychometric properties of the revised two-factor study process questionnaire r-spq-2f - spanish version
Clara Vergara-Hernández, Miguel Simancas-Pallares, Zoila Carbonell-Muñoz
Duazary.2019; 16(2): 205. CrossRef
Enfoques de Aprendizaje según el R-SPQ-2F: Análisis de sus propiedades psicométricas en estudiantes universitarios de Buenos Aires
Agustín Freiberg Hoffmann, María Mercedes Fernández Liporace
Revista Colombiana de Psicología.2016;[Epub] CrossRef

First
Prev
Page of 1
Next
Last

Search