Introduction

Background/rationale

Medical educators have developed and implemented various teaching methods with the goal of more effectively and efficiently cultivating learners’ abilities as doctors. For example, online learning has been highly anticipated since much of the medical school curriculum consists of didactic lectures that deliver knowledge. However, resistance from participants, ranging from students to professors, as well as barriers regarding institutional resources, policies, and support, slowed the transition from traditional face-to-face classes to online classes [1]. Starting in early 2020, the coronavirus disease 2019 (COVID-19) pandemic led to unexpectedly rapid changes in this domain, as it forced the full-scale implementation of online classes without a face-to-face option.

This transition to online learning can be seen as an educational intervention introduced in the pre-clerkship curriculum, where most of the learning takes place through lecture-based teaching. Program evaluation, therefore, is essential to investigate the impact of the intervention and to improve curricular outcomes. Under these circumstances, outcomes can be assessed using Kirkpatrick’s 4-level evaluation model, which is a framework widely used for program evaluation [2].

In the first level of the model (i.e., reaction), numerous findings have been reported regarding students’ perceptions of online classes and satisfaction with different teaching methods. The results of these studies are quite consistent. For example, according to a recently released meta-analysis, synchronous distance education (SDE) has higher satisfaction ratings than face-to-face classes [3]. Similar results were found in post-COVID-19 surveys, indicating that students prefer online courses over offline courses [4]. Particularly, students prefer classes offering recorded videos over live online lectures, citing “fast viewing” and “pause and resume” as the advantages of recorded videos that cannot be met in either face-to-face or live online lectures [4].

In contrast, in the second level of the model (i.e., learning), there is still insufficient evidence to draw definitive conclusions on how post-COVID-19 online classes differ from conventional ones. Although students recognized that the COVID-19 pandemic “greatly affected” or “considerably affected” their studies [5], in practice, the direction of influence is possibly both positive and negative. The fact that high satisfaction does not necessarily guarantee high academic achievement is another reason why it is difficult to make a convincing argument for academic achievement despite students’ positive responses to online courses [6]. Previous studies have demonstrated that a student’s “feeling of learning” has no association with actual academic achievement, and sometimes they are even negatively correlated [7].

Objectives

This study aims to examine how students’ academic performance changed after the COVID-19 pandemic based on the test results of 16 integrated courses conducted over 3 semesters at a single medical school in South Korea that underwent the online transition after COVID-19.

Methods

Ethics statement

This study was reviewed and approved by the Institutional Review Board of Hanyang University (approval no., HYUIRB-202105-023-1).

Study design

This was a comparative observational study, involving a comparison of 16 integrated, system-based courses before and after the transition to online classes.

Setting

This study was conducted at Hanyang University College of Medicine (HYUCM), a private medical school in Seoul, South Korea. The average number of students per year is about 100. HYUCM operates a 6-year undergraduate-entry program, consisting of a 2-year premedical course and a 4-year medical course. The 4-year medical course is divided into three phases. Phase 1 (1 semester) and phase 2 (3 semesters) correspond to the pre-clerkship period, and phase 3 is the clinical clerkship period. In HYUCM, the transition to online teaching was first implemented after COVID-19. Almost all face-to-face classroom lectures were replaced by online recorded videos, while fewer than 5% of classes were conducted as live online lectures.

In HYUCM, phase 2 is an integrated, system-based curriculum that runs for 3 semesters from the second semester of the first year. There are a total of 17 required courses in this period without any student-selected components (Table 1). The course lengths range from 1 to 7 weeks, and they are organized into blocks. To promote the integrated understanding of clinical and anatomical knowledge, most courses include some cadaver dissection sessions. Every course has at least 1 major examination as a summative assessment. Major examinations are conducted up to 3 times depending on the course length, and all examinations consist of multiple-choice questions (MCQ) items or short-answer questions. Each major examination score is added up to constitute about 80%–90% of a course’s final score. Other components such as attendance and evaluation in problem-based learning (PBL) sessions account for the remaining 10%–20%.

Variables

Students’ performance records in all courses were variables.

Data sources/measurement

The major examinations’ raw scores were collected for each student. Because the total score was different for each examination, percent-correct scores were used in subsequent analyses. For courses that conducted more than 1 major examination, student achievement was calculated as an average of the percent-correct scores obtained from the examinations. As the aim of this study was to investigate academic achievement in terms of knowledge acquisition, we did not consider other assessment components, such as attendance or PBL scores, which are more intended to assess conscientiousness, communication, or critical thinking rather than knowledge.

Bias

Given this study included 16 (out of 17) courses throughout 3 semesters of phase 2 and the examination scores from all students, the risk of selection bias was negligible.

Study size

All medical students’ records in the corresponding courses were included. There was no estimation of sample size.

Statistical methods

Data were analyzed using IBM SPSS ver. 26.0 (IBM Corp., Armonk, NY, USA). The t-test was used to compare grades between 2019 and 2020, and the Cohen D was calculated as a measure of the effect size. The correlation of scores between courses was calculated using Pearson correlation coefficients. Correlation coefficients were compared after applying the Fisher r-to-z transformation. P-values <0.05 were considered to indicate statistical significance.

Results

Difference in test scores between 2019 and 2020

When comparing the test results from the 16 courses, a significant decrease in scores was found in 10 courses (62.5%) in 2020 (Table 2, Dataset 1). There was no significant difference in 3 courses, and a significant increase was found in 3 courses. Using the Levene test, significant differences in variance were identified in 13 courses (81.3%); in all of these cases, the standard deviation was greater in 2020 than in 2019 (Table 2, Fig. 1).

Correlation of test scores between courses

For both 2019 and 2020, most of the integrated, system-based curriculum test scores showed strong correlations, with coefficients of 0.6 or higher. In 2020, the correlation coefficient was generally even higher, with 30 (85.7%) pairs out of a total of 35 having a greater correlation coefficient in 2020, of which 13 (37.1%) showed a statistically significant increase (Fig. 2). Further, when analyzing the correlation between the first- and second-semester courses for second-year students, the correlation coefficient was greater in all 30 pairs (100.0%) in 2020 than in 2019. Of them, 18 (60.0%) showed a statistically significant increase (Table 3).

Comparison between low-, middle-, and high-achieving students

After dividing students into low, middle, and high achievement groups based on overall major examination performance for a semester, we calculated the effect size of the difference between 2019 and 2020, while the sign (positive or negative) was maintained by not taking the absolute value. As a result, the average value of the effect sizes in all semesters was highest for low-achieving students, followed in descending order by middle-achieving and high-achieving students (Table 4). Specifically, low achievers showed positive values for the effect size for all 3 semesters, indicating a decline in test scores in 2020 compared to 2019.

Discussion

Key results

In this study, we compared students’ academic performance in integrated, system-based courses in the pre-clerkship curriculum before and after the COVID-19 pandemic to examine changes during the transition to full-scale online classes. In a majority of courses, the average test scores decreased, accompanied by an increase in variance compared to offline classes. The correlation of test scores between courses was mostly higher in 2020 in both intra- and inter-semester analyses. Finally, the decline in performance was most noticeable among low-achieving students compared to middle- or high-achieving students.

Interpretation

Among the 16 courses in phase 2, the average test scores in 10 courses decreased significantly, and only 3 improved significantly. In general, our findings suggest that the transition to online classes due to COVID-19 has led to an overall decline in academic performance. Interestingly, in a meta-analysis prior to the COVID-19 pandemic, which included research published from 2000 to 2017, the knowledge outcomes of online learning were found to be at least equal or superior to those of offline learning in undergraduate medical education [8]. Therefore, when interpreting the findings of this study, both the pedagogical differences between online and offline formats in delivering content and overall changes in the broader educational environment caused by COVID-19 must be taken into account.

First, we must consider that the online transition of the formal curriculum due to COVID-19 was sudden, comprehensive, and compulsory. Faculty members were required to adapt to online teaching even if they were not provided with enough institutional support or were not skilled in technology, and students likewise had no choice but to study online. Further, because not all forms of teaching can be delivered online, hands-on practice (e.g., laboratory sessions or cadaver dissection) was inevitably reduced or discontinued. Additionally, social distancing greatly reduced opportunities for informal learning.

Nevertheless, it should be noted that the degree of decline in academic performance was not uniform across students. When the average of the effect sizes was calculated for each group (low-, middle-, and high-achieving students), with “average 2019 scores–average 2020 scores” used as a numerator, low-achieving students showed the highest positive value (i.e., the largest decline in test scores among the 3 groups). In contrast, high-achieving students showed the smallest positive value, or sometimes even a negative value (i.e., an increase in test scores). This stark difference between high- and low-achieving students could be attributed to the increased isolation caused by social distancing, which further amplified the importance of self-regulation in learning. In general, it is well known that struggling learners tend to show low self-regulation such as poor motivation and inefficient resource management. On the contrary, high-achieving students could have minimized the impact or even turned this crisis into an opportunity by utilizing a variety of motivational, cognitive, and metacognitive regulation strategies as well as appropriate resource management [9]. Moreover, since some interactive learning methods, such as team-based learning, are more beneficial to lower-achieving students than to high-achieving students in terms of knowledge acquisition [10], there is a possibility that the inevitable reduction or discontinuation of certain components in the formal curriculum may have been more damaging to already-struggling students.

Comparison with previous studies

Considering that the integrated courses are relatively homogeneous in their content domains (i.e., clinical medicine) and assessment format (i.e., MCQs), it is common for multiple test scores to correlate with each other. The scores from the 16 courses consistently were highly correlated at the intra- and inter-semester levels. Moreover, unlike before the COVID-19 pandemic, these high correlations lasted until the second semester, after which students could spend about a month on vacation to review and revise their own learning strategies and behaviors. This suggests that students’ academic performance was “ossified” throughout this phase, which can be a critical problem, especially for lower-achieving students.

The negative consequences of this ossification can be viewed in 2 respects. One is an increase in the number of students who fail to progress. This is not only an unfavorable event for individual students, but also a managerial burden at the organizational level, given that remediation of struggling learners is resource-intensive work that requires significant time, performance, and expertise [11]. The other problem is that even if low achieving students progress to the clerkship phase, their weak academic foundation would pose future difficulties in learning advanced knowledge and skills. Theoretically, knowledge acquisition corresponds to the lowest level of Miller’s pyramid, “knows”, which forms the basis of subsequent higher-level performance such as “knows how”, “shows how”, and “does”. Empirically, on the medical education continuum, academic achievement in the previous phase has been identified as a major predictor of performance in the next phase, which McManus et al. [12] named the “academic backbone”. In the short term, provided that specific cognitive knowledge supports the basis of clinical reasoning as well as procedural skills, it is anticipated that students will continue to have difficulties in developing competence in the subsequent clerkship phase. Above all, in the long term, incompetent learners could pose a risk to patient safety.

Limitations

First, in terms of research design, as we compared the academic achievement in online classes between 2019 and 2020, the results of the study may have been confounded by the characteristics of the cohorts of each year. Second, although data were collected across 16 courses and over a year and a half, this study was limited to a single institution. Third, while the COVID-19 pandemic continued throughout the 3 semesters of this study, its severity varied from one period to another. As a result, there were variations in institutional policies and student behavior, but not all of these micro-level changes were considered as variables in the analysis. Finally, 2020 was the first year of the transition to online classes due to the pandemic, and neither professors nor students were fully prepared. Therefore, our findings could be attributable to students’ level of adaptation and utilization of the online curriculum rather than online classes themselves.

Suggestions

First, the overall decline in academic performance, which may occur when the curriculum is centered on recorded lectures, needs to be addressed. Improving the teaching quality—whether live or recorded—in delivering content would be the most basic and effective strategy, not only because the quality of recorded online lessons affects learners’ performance on examinations, but also because low-achieving students benefit the most from quality improvements [13]. Specifically, strategies for the effective use of videos, such as using interactive elements, managing cognitive overload, and considering technical requirements, should be considered.

Adding SDE components can be considered as a means of supplementing the impaired informal learning due to COVID-19. While asynchronous distance education is suitable for encouraging learners to cognitively participate in information processing, SDE is more advantageous for promoting psychological arousal and motivation [14]. As such, it would be most effective to use both in complementary ways. However, considering the high preference of students for recorded lectures, and the low probability of their active participation in live online lectures, it would not be appropriate to simply convert recorded lectures to a live format, even if they are delivered in a synchronous manner.

Second, the findings suggest the necessity of establishing a support system for the academic development of low-achieving students. The high correlation between tests implies that at-risk students can be predicted early with high probability. In particular, struggling in school has been linked to poor self-regulation in early medical students, and there is strong evidence for focusing remediation on assessing and improving self-regulation [11]. Therefore, preventive and proactive developmental approaches, focusing on developing students' personal and professional growth and lifelong learning skills, should be available in the early stages to prevent summative failure. More importantly, such an approach is more educationally desirable than a deficit-reactive approach, which has the risk of stigma or repeated failure even after remediation.

Third, future studies are necessary to clarify the causes and mechanisms of changes in academic performance in the online curriculum. For example, although most course components were largely the same except for the online transition between 2019 and 2020, differences in test conditions or examinee characteristics could have led to score improvements in some courses. Therefore, the use of experimental research design or statistical methods such as equating may be considered to draw more generalizable conclusions.

Conclusion

This study identified a decline in students’ academic performance in the pre-clerkship curriculum that was transitioned to online classes because of COVID-19. Further studies from other institutions that have experienced similar changes and in-depth investigations into the causes of these phenomena should be conducted.

Article information

Authors’ contributions

Conceptualization: DHK, YJK. Data curation: DHK, HJL, YL. Methodology & formal analysis: DHK, YJK. Writing–original draft: DHK. Writing–review & editing: DHK, HJL, YL, YJK.

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Funding

None.

Data availability

Data files are available from Harvard Dataverse: https://doi.org/10.7910/DVN/OBZCIT

Dataset 1. Students’ performance records of 16 courses in 2019 and 2020.

jeehp-18-24-dataset1.xlsx

Acknowledgments

Supplementary materials

Fig. 1.

Histogram of examination scores for each course. (A–E) The first year (first semester of phase 2). (F–J) The second year (second semester of phase 2). (A) Introduction to Clinical Medicine, (B) Musculoskeletal System, (C) Medical Neuroscience, (D) Neuropsychiatry and Behavioral Science, (E) Sensory System, (F) Respiratory Medicine, (G) Gastroenterology, (H) Endocrinology and Metabolism, (I) Reproductive Medicine, (J) Kidney and Urinary Tract, (K) Immunology, (L) Infectious Disease, (M) Hematology and Oncology, (N) Occupational Disease and Injury, (O) Birth and Growth, and (P) Geriatrics.

Fig. 2.

Intra-semester correlations of test scores. The values for 2020 are below the diagonal. The strength of the correlation was displayed using different shading (no shades: r <0.6, light gray: 0.6≤ r <0.7, medium gray: 0.7≤ r <0.8, dark gray: 0.8≤ r). ^a)P<0.01. ^b)Indicates a statistically significant increase between 2019 and 2020.

References

• 1. O’Doherty D, Dromey M, Lougheed J, Hannigan A, Last J, McGrath D. Barriers and solutions to online learning in medical education: an integrative review. BMC Med Educ 2018;18:130. https://doi.org/10.1186/s12909-018-1240-0 ArticlePubMedPMC
• 2. Roland D. Proposal of a linear rather than hierarchical evaluation of educational initiatives: the 7Is framework. J Educ Eval Health Prof 2015;12:35. https://doi.org/10.3352/jeehp.2015.12.35 ArticlePubMedPMC
• 3. He L, Yang N, Xu L, Ping F, Li W, Sun Q, Li Y, Zhu H, Zhang H. Synchronous distance education vs traditional education for health science students: a systematic review and meta-analysis. Med Educ 2021;55:293-308. https://doi.org/10.1111/medu.14364 ArticlePubMed
• 4. Kim JW, Myung SJ, Yoon HB, Moon SH, Ryu H, Yim JJ. How medical education survives and evolves during COVID-19: our experience and future direction. PLoS One 2020;15:e0243958. https://doi.org/10.1371/journal.pone.0243958 ArticlePubMedPMC
• 5. Mahdy MA. The impact of COVID-19 pandemic on the academic performance of veterinary medical students. Front Vet Sci 2020;7:594261. https://doi.org/10.3389/fvets.2020.594261 ArticlePubMedPMC
• 6. Bates R. A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Eval Program Plan 2004;27:341-347. https://doi.org/10.1016/j.evalprogplan.2004.04.011 Article
• 7. Deslauriers L, McCarty LS, Miller K, Callaghan K, Kestin G. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proc Natl Acad Sci U S A 2019;116:19251-19257. https://doi.org/10.1073/pnas.1821936116 ArticlePubMedPMC
• 8. Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education?: a systematic review and meta-analysis. Med Educ Online 2019;24:1666538. https://doi.org/10.1080/10872981.2019.1666538 ArticlePubMedPMC
• 9. Stegers-Jager KM, Cohen-Schotanus J, Themmen AP. Motivation, learning strategies, participation and medical school performance. Med Educ 2012;46:678-688. https://doi.org/10.1111/j.1365-2923.2012.04284.x ArticlePubMed
• 10. Kim DH, Lee JH, Kim SA. The pharmacology course for preclinical students using team-based learning. Korean J Med Educ 2020;32:35-46. https://doi.org/10.3946/kjme.2020.151 ArticlePubMedPMC
• 11. Chou CL, Kalet A, Costa MJ, Cleland J, Winston K. Guidelines: the dos, don’ts and don’t knows of remediation in medical education. Perspect Med Educ 2019;8:322-338. https://doi.org/10.1007/s40037-019-00544-5 ArticlePubMedPMC
• 12. McManus IC, Woolf K, Dacre J, Paice E, Dewberry C. The Academic Backbone: longitudinal continuities in educational achievement from secondary school and medical school to MRCP(UK) and the specialist register in UK medical students and doctors. BMC Med 2013;11:242. https://doi.org/10.1186/1741-7015-11-242 ArticlePubMedPMC
• 13. Clark AE, Nong H, Zhu H, Zhu R. Compensating for academic loss: online learning and student performance during the COVID-19 pandemic. China Econ Rev 2021;68:101629. https://doi.org/10.1016/j.chieco.2021.101629 ArticlePubMedPMC
• 14. Hrastinski S. Asynchronous and synchronous e-learning. Educ Q 2008;31:51-55.

Figure & Data

References

Citations

Citations to this article as recorded by  

• Study delay during emergency remote teaching among students at Dutch universities: the role of students’ education satisfaction and academic wellbeing
  Manja Vollmann, Renée A. Scheepers, Femke Hilverda
  European Journal of Psychology of Education.2024;[Epub]     CrossRef
• Impact of the Covid-19 pandemic and ensuing online teaching on pre-clinical medical education
  Houman Goudarzi, Masahiro Onozawa, Makoto Takahashi
  BMC Medical Education.2024;[Epub]     CrossRef
• Navigating the shift to online learning: student experiences of inclusivity, efficiency, and study efforts in Chile
  Grace Melo, Diego Monteza, Sandra Acosta
  Cogent Education.2024;[Epub]     CrossRef
• The nationwide impact of COVID-19 on life support courses. A retrospective evaluation by Resuscitation Council UK
  C.J. Thorne, P.K. Kimani, S. Hampshire, I. Hamilton-Bower, S. Begum-Ali, A. Benson-Clarke, K. Couper, J. Yeung, A. Lockey, G.D. Perkins, J. Soar
  Resuscitation Plus.2023; 13: 100366.     CrossRef
• COVID-19 and Living Situation: Effects on University of Michigan Students During the 2020-2021 Academic Year
  Barbara Tan, Alyssa Cadez-Martin, Sarah Fox Fox, Samir Gadepalli
  University of Michigan Undergraduate Research Journal.2023;[Epub]     CrossRef
• How successful the online assessment techniques in distance learning have been, in contributing to academic achievements of management undergraduates?
  Hiruni Thathsarani, Dinushika Kaushalya Ariyananda, Chalani Jayakody, Kerthiga Manoharan, A.A.S.N Munasinghe, Nilmini Rathnayake
  Education and Information Technologies.2023; 28(11): 14091.     CrossRef
• Academic performance and mental health among Chinese middle and high school students after the lifting of COVID-19 restrictions
  Hong-Jun Song, Yun-Fei Mu, Cong Wang, Jia Cai, Zhong-Yue Deng, Ai-Ping Deng, Xue-Hua Huang, Xian-Dong Meng, Lan Zhang, Yi Huang, Wei Zhang, Wen-Wu Shen, Jin Chen, Bo Liu, Ru Gao, Jun-Shu Zhao, Mao-Sheng Ran
  Frontiers in Psychiatry.2023;[Epub]     CrossRef
• HOW DID ONLINE LEARNING AFFECT PHYSIOTHERAPY STUDENTS DURING THE COVID-19 PANDEMIC?
  Muhammet Fatih BULUT, Aybüke Cansu KALKAN, Arzu GENÇ, Seher ÖZYÜREK
  SDÜ Tıp Fakültesi Dergisi.2023; 30(4): 652.     CrossRef
• The impact of asynchronous online anatomy teaching and smaller learning groups in the anatomy laboratory on medical students’ performance during the Covid‐19 pandemic
  Ming‐Fong Chang, Meng‐Lin Liao, June‐Horng Lue, Chi‐Chuan Yeh
  Anatomical Sciences Education.2022; 15(3): 476.     CrossRef
• Educational impact of an active learning session with 6-lead mobile electrocardiography on medical students’ knowledge of cardiovascular physiology during the COVID-19 pandemic in the United States: a survey-based observational study
  Alexandra Camille Greb, Emma Altieri, Irene Masini, Emily Helena Frisch, Milton Leon Greenberg
  Journal of Educational Evaluation for Health Professions.2022; 19: 12.     CrossRef
• Coronavirus Disease 2019 Cases at Universities and Colleges in Seoul Metropolitan Area
  Young June Choe, Yun-Kyung Kim
  Journal of Korean Medical Science.2021;[Epub]     CrossRef