Skip Navigation
Skip to contents

JEEHP : Journal of Educational Evaluation for Health Professions



Page Path
HOME > J Educ Eval Health Prof > Volume 19; 2022 > Article
Application of computer-based testing in the Korean Medical Licensing Examination, the emergence of the metaverse in medical education, journal metrics and statistics, and appreciation to reviewers and volunteers
Sun Huh*orcid

Published online: January 13, 2022

Department of Parasitology and Institute of Medical Education, College of Medicine, Hallym University, Chuncheon, Korea

*Corresponding email:

Editor: A Ra Cho, The Catholic University of Korea, Korea

• Received: January 12, 2022   • Accepted: January 13, 2022

© 2022 Korea Health Personnel Licensing Examination Institute

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

  • 619 Download
  • 26 Web of Science
  • 24 Crossref
  • 26 Scopus
On January 6–7, 2022, computer-based testing (CBT) was adopted for the Korean Medical Licensing Examination (KMLE) [1] (Fig. 1). This is the first instance of CBT adoption for any of the 26 health professional licensing examinations in Korea. Of course, smart device-based testing (SBT), a variant of CBT, was already adopted for the Korea Emergency Medicine Technician Licensing Examination in 2017 [2]. Examinees’ demographic characteristics and perceived acceptability of SBT did not affect the SBT mock test scores of emergency medicine technician students in Korea [3]. After implementing CBT in the KMLE, it will be expanded to all other 24 professions, including dentists, oriental medicine doctors, and care workers, by 2025 [1].
On March 21, 2003, presenters first proposed CBT and computerized adaptive testing (CAT) for the KMLE at a seminar held by the Institute of Medical Education of Hallym University at the Chuncheon Sejong Hotel. At that time, the staff members of the Korea Health Personnel Licensing Examination Institute (KHPLEI) attended the seminar. Thus, 19 years have passed since CBT and CAT were suggested for licensing examinations in Korea.
In November 2011, the Standing Committee of the KMLE recommended introducing computerized testing to the KMLE, including CBT and CAT [4]. I also suggested improving the quality of the KMLE through the introduction of computerized testing and recommended that medical schools should prepare for the new testing environment of the KMLE [4]. The 6th president of the KHPLEI, Dr. Myung-Hyun Chung (August 1, 2012–July 31, 2015), announced that research and development on CBT and ubiquitous-based testing would be accelerated in January 2013 [5]. Research on CBT then became more active. For example, the linear programming method was suggested to construct equated-item sets that reflect each content area. The best choice was to divide the predicted correct answer rate into 2 or 3 difficulty boundaries regardless of common items [6]. Because SBT was already well implemented in the Korea Emergency Medicine Technician Licensing Examination [3], there was no need to repeat an evaluation of the examinees’ adaptability to CBT. The only step left was decision-making by the president of KHPLEI. Dr. Yoon-Seong Lee, the 8th president, finally decided to introduce CBT.
CBT has already been a very common testing platform at most medical schools. For example, at Hallym University, CBT was introduced in 1999 for clinical course testing. In my parasitology class, CBT was already introduced in 1993. My students have had no difficulties in taking CBT. A new task after adopting CBT is to guarantee item quality, including that of audio-visual materials. CBT items should be more clinically oriented than paper-and-pencil materials to test examinees’ clinical competency. Furthermore, the number of items to measure the examinee’s latent traits should be re-considered. A correct distribution of core clinical content is required, and the balance between reliability and core clinical presentations should be guaranteed to ensure desirable item numbers. The standard setting for CBT, including the modified Angoff, modified Ebel, and Hofstee methods, should be implemented soon, instead of using a cut score to ensure a success rate of 60% [7,8]. It is necessary to maintain a consistent passing rate regardless of item difficulties.
Two invited reviews were published last year: “Sample size determination and power analysis using the G*Power software” by Kang [9] and “Educational applications of metaverse: possibilities and limitations” by Kye et al. [10]. There was also an invited editorial: “Training in lung cancer surgery through the metaverse, including extended reality (XR), in the smart operating room of Seoul National University Bundang Hospital, Korea” by Koo [11].
Although Dr. Kang is an anesthesiologist, he studied statistics after graduating from Korea National Open University with a major in Statistics. I invited him to serve as a statistical editor in 2021. I asked him to submit a review on determining the study size for authors and reviewers of the journal. When I read manuscripts submitted to the journal, the most challenging task is to check the appropriateness of study size estimation, especially since many manuscripts do not describe it. If estimation before the study was not possible, it is recommended to conduct a post-hoc analysis. Dr. Kang stated that “the null and alternative hypothesis, effect size, power, alpha, type I error, and type II error should be described when calculating the sample size or power” [9]. I hope that this invited methodological review will be helpful for authors and researchers.
The year 2021 was the year of the metaverse [12]. Although the term “metaverse” was known beforehand, and it had already been developed as a technology, its use exploded in the coronavirus disease 2019 (COVID-19) pandemic era since many people could not meet face to face. This also occurred in health education. My university also trained faculty members on how to use Gather, one of the virtual educational spaces of metaverse platforms, called the virtual world. Unfortunately, I could not apply this platform for my medical students in 2021 due to my unpreparedness. I will use this platform or another virtual educational space for this year’s parasitology class. Several types of the metaverse, including augmented reality (AR), mirror world, lifelogging, and virtual reality (VR), have already been introduced in medical and health education.
I did a PubMed search with the keyword “metaverse,” and only 11 articles were found on January 8, 2022. However, this does not mean that the number of articles on metaverse is small. Before that, articles on the 4 types of the metaverse have been published frequently. These publications soared in 2021 (Fig. 2, Supplement 1) and the term “metaverse” began to appear in biomedical articles in 2020 and 2021. There are already 4 articles on this topic in the Journal of Educational Evaluation for Health Professions (JEEHP): VR endotracheal intubation training [13]; digital technologies introduced in medical and dental education [14]; a simulation-based blended training model for nurses [15], and VR training to decreases rates of needle stick/sharp injuries [16].
According to the usage trends of the term “metaverse,” a review article was invited as an abridged English translation of the issue report by the Korea Education and Research Information Service [10]. Dr. Bokyung Kye, Director of the Global Policy Research Section, kindly provided this report for the journal’s readers. This review article is a broad introduction to the educational application of the metaverse. The definition of the relevant terminology was explained, and current examples of metaverse use were introduced. It also included applications in health education. I hope that this accessible and comprehensive review article will be helpful for medical and health educators to understand the concept of the metaverse and the strengths and limitations of its application. Furthermore, Koo [11], a senior reporter from JTBC, one of the well-known broadcasting companies in Korea, wrote an editorial on the use of the metaverse to train chest surgeons at Seoul National University Bundang Hospital. This training course was an example of XR implemented by mixing AR and VR. XR refers to all VR technologies ranging from VR to mixed reality and AR.
In 2021, JEEHP received its first Journal Citation Indicator (JCI) value, 0.51, in June 2021. This score means that the citation impact of articles from 2017 to 2019 was about half of the average citation impact of Web of Science Core Collection journals. JEEHP’s JCI ranking in the scientific education category was 9th out of 35 Emerging Sources Citation Index (ESCI) journals (74.3%) and 47th out of 78 SCIE (Science Citation Index Expanded) and ESCI journals (39.8%) [17]. To receive a higher JCI value, a higher citation frequency of citable articles will be essential. The 2021 JCI may be announced in June 2022 by Clarivate. I anticipate a higher 2021 JCI than that in 2020.
Bibliometric statistics for the author’s country and total cites are presented in Figs. 3 and 4 (Supplement 2), respectively. There were authors from 15 countries, mainly from Asia, which reflects the regional scope of JEEHP. The number of total cites continuously increased. In 2021, JEEHP was cited 525 times in the Crossref metadata, 555 times in Scopus, and 461 times in Web of Science Core Collection. The manually calculated impact factor in Web of Science increased to 1.846 in 2021 from 1.254 in 2020. The CiteScore Tracker 2021 calculated by Scopus was 2.5 (366 citations 2018 to date/145 documents 2018 to data), last updated on January 5, 2022, which is available from: This value is higher than that in 2020 (1.7)
Table 1 presents journal statistics from 2021. The number of unsolicited submissions (n=296) increased from 2020 (n=275). However, the number of publications decreased from 44 in 2020 to 33 in 2021. This primarily originates from the low number of commissioned articles in 2021. The acceptance rate of unsolicited manuscripts (8.9%) was lower than that in 2002 (10.6%). In 2020, the first year of COVID-19, the number of submissions (n=286) was nearly double compared to 2019 (n=147). This sharp increase was a common phenomenon seen in other scholarly journals in Korea [18]. The amount of submissions to this journal was consistent in 2021.
I should be cautious when selecting manuscripts for review. The number of rejected or withdrawn manuscripts after review was 16 in 2020 and 10 in 2021. The number of accepted manuscripts after review was 26 out of 43 (70.3%) in 2021. In 2022, the editorial office will do its best to select acceptable manuscripts for review to save reviewers’ time.
I am deeply indebted to the reviewers who voluntarily devoted themselves to reviewing manuscripts. Their role is essential for maintaining the journal quality. I regret that I sent them some manuscripts that were finally rejected. Below are the reviewers’ names and affiliations by country.
  • Australia: Boaz Shulruf, University of New South Wales; Elio Stefan Arruzza, University of South Australia

  • Chile: Castillo Niño Manuel, University of Chile

  • Indonesia: Armyanti Ita, Tanjungpura University; Romiko, Muhammadiyah University of Palembang

  • India: Upreet Dhaliwal, University of Delhi; Kiran Goswami, All India Institute of Medical Sciences; T. S. Gugapriya, All India Institute of Medical Science, Manjiri Phansalkar, Pondicherry Institute of Medical Sciences

  • Israel: Colin Block, The Hebrew University of Jerusalem

  • Italy: Colaceci Sofia, Saint Camillus International University of Health Sciences

  • Korea: A Ra Cho, The Catholic University of Korea; Yera Hur, Hallym University; Junyong In, Dongguk University; Hyun Kang, Chung-Ang University; Jae Hyun Kim, Dankook University; Mi Kyoung Yim, Korea Health Personnel Licensing Examination Institute; Sun Kim, The Catholic University of Korea; Young Ju Kim, Ewha Womans University; Dong Kyu Lee, Dongguk University; Keumho Lee, Korea Institute for Research in the Behavioral Sciences; Younjae Oh, Hallym University; Janghee Park, Soonchunhyang University; Jeong Yun Park, University of Ulsan; Jungchul Park, Dankook University; Jungkyu Park, Kyungpook National University; Kyung Hye Park, Yonsei University; Song Yi Park, Dong-A University; Won Kyun Park, Keimyung University; Dong Gi Seo, Hallym University; Ji-Hyun Seo, Gyeongsang National University; Aeree Son, Samyook University; Sanghee Yeo, Kyungpook National University; Hyun Bae Yoon, Seoul National University; Dong-Mi Yoo, The Catholic University of Korea

  • Malaysia: Fata Nahas Abdulrahman, International Islamic University; Jamshid Shazia, Sultan Zainal Abidin University

  • Morocco: Hassouni Kenza, Mohammed VI University of Health Sciences; El Hajji Mohamed, Ibn Zohr University

  • Pakistan: Rano Mal Piryani, Liaquat University of Medical and Health Sciences

  • Portugal: José Miguel Padilha, Oporto Nursing School

  • South Africa: Richard Cooke, University of the Witwatersrand

  • Thailand: Poonpong Boonbrahm, Walailak University; Marisa Krairiksh, Khon Kaen University; Manoch Namfu, Rajamangala University of Technology Lanna

  • United Kingdom: Oliver Kemp, Gloucestershire NHS Foundation Trust; Marios Nicolaides, Queen Mary University of London; Cesar A. Orsini, Norwich Medical School, University of East Anglia; Emily Suckling, University Hospitals Bristol and Weston NHS Foundation Trust

  • United States of America: Chad Cook, Duke University; Robert Cunningham, Maryville University; Mariana D’amico, Nova Southeastern University; Jessica Fino, Midwestern State University; Sandra Groth, Midwestern State University; Junguk Hur, University of North Dakota; Myunghee Jun, University of Wisconsin-Green Bay; Hyekyung Kim, University of Wisconsin-Green Bay; Seock-Ho Kim, The University of Georgia; Amteshwar Singh, Johns Hopkins University; Jessyca Wagner, Midwestern State University

  • Vietnam: Bach Xuan Tran, Ha Noi Medical University

Tom Huh, a graduate student in the Division of Life Sciences, College of Life Sciences and Biotechnology, Korea University, Seoul, Korea, voluntarily made audio recordings of some abstracts.
In the screening process of a submitted manuscript, the fitness of the manuscript in terms of both aims and scope and style and format will be meticulously checked before dispatching it for review. Reviewers or editors cannot determine originality; this determination is dependent on readers’ experience. Therefore, the review has been focused on scientific integrity. A description according to reporting guidelines will be more strictly required. It should be demonstrated that the study design follows specific reporting guidelines. JEEHP is a non-commercial scholarly journal, so the number of published articles is not critical—but scientific integrity is essential. The editorial team and I will do our best to maintain these policies in 2022 to save reviewers’ time.

Authors’ contributions

All the work was done by Sun Huh.

Conflict of interest

Sun Huh has been the editor of the Journal of Educational Evaluation for Health Professions since 2005. Otherwise, no potential conflict of interest relevant to this article was reported.



Data availability


Supplement 1. Number of articles on the 4 types of the metaverse in PubMed by year with the search term “(((augmented reality) OR (lifelogging)) OR (mirror world)) OR (virtual reality)” [cited 2022 Jan 8], presented in Fig. 1.
Supplement 2. Total cites of Journal of Educational Evaluation for Health Professions in Crossref metadata, Scopus, and the Web of Science Core Collection (WOS) from 2015 to 2021, presented in Fig. 4.
Fig. 1.
Screenshot of the mock test of computerized adaptive testing of the Korean Medical Licensing Examination in 2021, provided by the Korea Health Personnel Licensing Examination Institute.
Fig. 2.
Number of articles on the 4 types of the metaverse in PubMed by year with the search term “(((augmented reality) OR (lifelogging)) OR (mirror world)) OR (virtual reality)” [cited 2022 Jan 8].
Fig. 3.
Number of articles in Journal of Educational Evaluation for Health Professions according to the authors’ country in 2021.
Fig. 4.
Total cites of Journal of Educational Evaluation for Health Professions in Crossref metadata, Scopus, and the Web of Science Core Collection (WOS) from 2015 to 2021.
Table 1.
Journal statistics of manuscripts submitted to Journal of Educational Evaluation for Health Professions from January 1 to December 31, 2021
No. of manuscripts Content
Manuscripts submitted 302
No. of commissioned manuscripts 6 Editorial, 4; review, 2
No. of unsolicited manuscripts 296
Manuscripts under re-submission, review, or revision 6 Under re-submission, 4; under review, 2
Manuscripts rejected without review 253 Unsuitable, 250; other reasons, 3
Manuscripts reviewed out of 290 unsolicited manuscripts 37 Accepted and published, 26; rejected 10; withdrawn, 1
No. of publications out of 296 submitted manuscripts in 2021 32 One article published in 2021 was submitted in 2020
No. of publications out of 290 unsolicited manuscripts 26 Opinion, 1; review, 4; research article, 14; brief report, 5; educational/faculty development material, 2
Acceptance rate overall (%) 10.8 32/296=0.108
Acceptance rate of unsolicited manuscripts (%) 8.9 26/290=0.089
Median time from submission to the first decision (day) 15
Median time from submission to publication (day) 43
Median time from acceptance to publication (day) 1

Figure & Data



    Citations to this article as recorded by  
    • Metaverse applications in education: a systematic review and a cost-benefit analysis
      Mark Anthony Camilleri
      Interactive Technology and Smart Education.2024; 21(2): 245.     CrossRef
    • A Bibliometric Analysis of Technology in Digital Health: Exploring Health Metaverse and Visualizing Emerging Healthcare Management Trends
      Hoang-Sy Nguyen, Miroslav Voznak
      IEEE Access.2024; 12: 23887.     CrossRef
    • Assessing the effectiveness and student perceptions of synchronous online flipped learning supported by a metaverse-based platform in medical English education: A mixed-methods study
      Emin İbi̇li̇, Melek Ölmez, Aysel Burcu İbi̇li̇, Fırat Bi̇lal, Abdullah Ci̇han, Nurullah Okumuş
      Education and Information Technologies.2024;[Epub]     CrossRef
    • Adoption of Metaverse in Healthcare Sector: An Empirical Analysis of its Enablers
      Vipulesh Shardeo, Bishal Dey Sarkar, Umar Bashir Mir, Prerna Kaushik
      IEEE Transactions on Engineering Management.2024; 71: 8671.     CrossRef
    • Facing the challenges of metaverse: a systematic literature review from Social Sciences and Marketing and Communication
      Verónica Crespo-Pereira, Eva Sánchez-Amboage, Matías Membiela-Pollán
      El Profesional de la información.2023;[Epub]     CrossRef
    • Utilizing the metaverse in anatomy and physiology
      Christian Moro
      Anatomical Sciences Education.2023; 16(4): 574.     CrossRef
    • Metaverse for Healthcare: A Survey on Potential Applications, Challenges and Future Directions
      Rajeswari Chengoden, Nancy Victor, Thien Huynh-The, Gokul Yenduri, Rutvij H. Jhaveri, Mamoun Alazab, Sweta Bhattacharya, Pawan Hegde, Praveen Kumar Reddy Maddikunta, Thippa Reddy Gadekallu
      IEEE Access.2023; 11: 12765.     CrossRef
    • Issues in the 3rd year of the COVID-19 pandemic, including computer-based testing, study design, ChatGPT, journal metrics, and appreciation to reviewers
      Sun Huh
      Journal of Educational Evaluation for Health Professions.2023; 20: 5.     CrossRef
    • Beyond Your Sight Using Metaverse Immersive Vision With Technology Behaviour Model
      Poh Soon JosephNg, Xiaoxue Gong, Narinderjit Singh, Toong Hai Sam, Hua Liu, Koo Yuen Phan
      Journal of Cases on Information Technology.2023; 25(1): 1.     CrossRef
    • Conceptions of the metaverse in higher education: A draw-a-picture analysis and surveys to investigate the perceptions of students with different motivation levels
      Gwo-Jen Hwang, Yun-Fang Tu, Hui-Chun Chu
      Computers & Education.2023; : 104868.     CrossRef
    • Presidential address: improving item validity and adopting computer-based testing, clinical skills assessments, artificial intelligence, and virtual reality in health professions licensing examinations in Korea
      Hyunjoo Pai
      Journal of Educational Evaluation for Health Professions.2023; 20: 8.     CrossRef
    • Mission and Goals of the New Editor of the Ewha Medical Journal
      Sun Huh
      The Ewha Medical Journal.2023;[Epub]     CrossRef
      Zeynep ŞAHİN
      Uluslararası İktisadi ve İdari İncelemeler Dergisi.2023; (41): 166.     CrossRef
    • La terza dimensione dell’e-learning: il metaverso
      Annamaria Cacchione
      IUL Research.2023; 4(7): 108.     CrossRef
    • Federated Learning for the Healthcare Metaverse: Concepts, Applications, Challenges, and Future Directions
      Ali Kashif Bashir, Nancy Victor, Sweta Bhattacharya, Thien Huynh-The, Rajeswari Chengoden, Gokul Yenduri, Praveen Kumar Reddy Maddikunta, Quoc-Viet Pham, Thippa Reddy Gadekallu, Madhusanka Liyanage
      IEEE Internet of Things Journal.2023; 10(24): 21873.     CrossRef
    • Assessment of the viability of integrating virtual reality programs in practical tests for the Korean Radiological Technologists Licensing Examination: a survey study
      Hye Min Park, Eun Seong Kim, Deok Mun Kwon, Pyong Kon Cho, Seoung Hwan Kim, Ki Baek Lee, Seong Hu Kim, Moon Il Bong, Won Seok Yang, Jin Eui Kim, Gi Bong Kang, Yong Su Yoon, Jung Su Kim
      Journal of Educational Evaluation for Health Professions.2023; 20: 33.     CrossRef
    • The impact of COVID-19 pandemic on hand surgery: a FESSH perspective
      Daniel B. Herren, Frederik Verstreken, Alex Lluch, Zaf Naqui, Brigitte van der Heijden
      Journal of Hand Surgery (European Volume).2022; 47(6): 562.     CrossRef
    • Public interest in the digital transformation accelerated by the COVID-19 pandemic and perception of its future impact
      Joo-Young Park, Kangsun Lee, Doo Ryeon Chung
      The Korean Journal of Internal Medicine.2022; 37(6): 1223.     CrossRef
    • The paradigm and future value of the metaverse for the intervention of cognitive decline
      Hao Zhou, Jian-Yi Gao, Ying Chen
      Frontiers in Public Health.2022;[Epub]     CrossRef
    • Advances in Metaverse Investigation: Streams of Research and Future Agenda
      Mariapina Trunfio, Simona Rossi
      Virtual Worlds.2022; 1(2): 103.     CrossRef
    • Dynamics of Metaverse and Medicine: A Review Article
      Mrudul A Kawarase, Ashish Anjankar
      Cureus.2022;[Epub]     CrossRef
    • What the Literature on Medicine, Nursing, Public Health, Midwifery, and Dentistry Reveals: An Overview of the Rapidly Approaching Metaverse
      Muhammet DAMAR
      Journal of Metaverse.2022; 2(2): 62.     CrossRef
    • Metaverse Üzerine Kapsamlı Bir Araştırma
      Çiğdem BAKIR
      European Journal of Science and Technology.2022;[Epub]     CrossRef
    • Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study
      Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park
      Journal of Educational Evaluation for Health Professions.2022; 19: 33.     CrossRef

    • PubReader PubReader
    • Cite
      export Copy
    • XML DownloadXML Download
    We recommend

    JEEHP : Journal of Educational Evaluation for Health Professions