Malaysian pharmacy students’ perspectives on the virtual objective structured clinical examination during the coronavirus disease 2019 pandemic

Background/rationale

With the global transformation of pharmacy practice to be more patient-focused, there has been a noticeable trend towards increasing the clinical components in pharmacy education programs. Consequently, the objective structured clinical examination (OSCE) has become a key assessment method used by pharmacy schools worldwide. The OSCE has also been adopted by several professional associations of pharmacists for certification and licensing purposes, as it enables the assessment of pharmacists and future pharmacists across a wide range of real-life clinical scenarios [1]. In designing OSCE cases, there should be a simulated interaction between the student and healthcare provider or patients to enhance students’ communication skills and practical application of their knowledge [1]. In the typical scenario of conducting an OSCE, a series of interactive face-to-face sessions are designed to simulate real situations in the practice settings [1,2]. However, the coronavirus disease 2019 (COVID-19) pandemic has imposed several restrictions on face-to-face interactions, creating challenges for pharmacy schools to pursue their planned conventional OSCEs. The virtual OSCE (vOSCE), as a proposed innovative method in experiential learning and assessment, then emerged as a way to proceed with the OSCE tasks without involving any face-to-face interactions with students. Careful planning for the online set-up and flow, training for students and assessors beforehand, clear instructions for the students, and simultaneous technical support are essential factors for a successful vOSCE. With slight modifications and creativity, it is possible to assess online almost all skills usually included in a face-to-face OSCE [3]. Trials also showed that good planning and training could overcome common hurdles to successful OSCEs, such as a large number of students, the use of simulated patients, and the operation of multiple concurrent stations [4].

With the unexpected COVID-19 pandemic, many emergency learning and evaluation initiatives seem to be continuing for a more extended period than previously planned. The implementation of online assessments during the pandemic has received considerable interest, and students’ feedback has been a particular focus of attention as part of lessons learned for potential improvements [5]. To further improve learning and evaluation methods, it is crucial to receive feedback from students on newly implemented online assessments.

Objectives

This study investigated Malaysian pharmacy students’ perceptions and views regarding the vOSCE conducted to assess their skills and competencies during the COVID-19 pandemic, with a specific focus on the preparation, organization, conduction, and advantages/disadvantages of the vOSCE compared with the conventional face-to-face OSCE.

Ethics statement

The study protocol obtained an ethical approval recommendation from the faculty of Pharmacy Ethics Committee (KEC) (review no., 4/2019) and obtained ethical approval from the International Islamic University Malaysia (IIUM) Research Ethics Committee (IREC 2020-074). Informed consent was obtained from participants.

Study design

A descriptive cross-sectional study was conducted to explore the views and perceptions of pharmacy students in Malaysia.

Participants

The target participants of the study were penultimate and final-year students of the Faculty of Pharmacy, IIUM in Malaysia. All 253 students (140 year 3 and 113 year 4) who took part in the vOSCE were invited to participate in the study. Participation was voluntary, and no payment was given.

Setting

The vOSCE was conducted in response to changes in the assessment methods used at IIUM during the COVID-19 pandemic. A validated questionnaire was distributed to the students 1 week after they took the vOSCE. All potential study participants were provided with a link to the online questionnaire (using a Google form), with at least 3 reminders to enhance the response rate.

In total, 3 vOSCEs were conducted in July 2020, covering different modules in clinical, hospital, and community pharmacy services courses. A similar approach was followed for all vOSCEs; each group of students was assigned to a faculty member as an examiner, and a trial-run session was arranged a day before the vOSCE. The examiners and students had the option to choose either Google Meet or Zoom as the platform of choice to conduct the assessment. The cases for assessment were shared with the students by the examiners using the screen-sharing option.

Development, validation, and reliability of the measurement tool

A self-administered 22-item structured English questionnaire was developed, as no similar tool to achieve the research objectives was found in the literature. The content validity of the developed tool was evaluated through an examination by 5 senior academic staff with experience matching the research topic. Accordingly, the content validity index for each item was calculated, and input from the validators was considered to make the necessary amendments. Based on their input, 4 items were deleted from the proposed questionnaire, which consisted of 26 items. The internal consistency and reliability of the original questionnaire were tested and reported previously, with a Cronbach’s α value of 0.87 [6].

The study instrument was divided into 3 sections. Section A comprised 5 items eliciting general information about the study participants. Section B contained 17 items used to assess students’ views and perceptions regarding the vOSCE. The items in section B were designed to encompass students’ perceptions on 5 domains of the vOSCE: the efforts made by faculty to prepare the students; the operation of the vOSCE; technical issues and faculty support in the operation of the vOSCE; normalization of the vOSCE in the future; and a comparison between conventional OSCE and vOSCE. Section C consisted of an open-ended space to provide relevant comments about the overall vOSCE experience. The items in section A were designed mainly as discrete items, whereas the items in section B were answered on a 5-point Likert scale, with a score of 1 denoting “strongly disagree” and a score of 5 indicating “strongly agree.”

Study size

A sample size of 153 students was calculated to represent the finite population of 253 pharmacy students with a response distribution of 50%, a margin of error of 5%, and a level of confidence of 95%. This sample size achieved a statistical power of 96% and 99% for gender and year of study, respectively. The sample size was calculated using the Raosoft online sample size calculator (Raosoft Inc., Seattle, WA, USA).

Statistical methods

The data were analyzed using IBM SPSS software ver. 23.0 (IBM Corp., Armonk, NY, USA). Descriptive data were expressed as numbers and percentages. The chi-square test was utilized to compare students’ responses based on their characteristics. The level of significance was set at P<0.05.

Key results

The findings highlighted the students’ overall perceptions concerning the flow of the vOSCE, including the arrangements made by faculty, their comparative views of the vOSCE versus the conventional OSCE, the technical issues associated with the virtual setting, and the adoption of the vOSCE in future assessments. More than half of the respondents were satisfied with the flow and conduct of the vOSCE. However, only one-third of them said that the tasks provided in the vOSCE were more convenient, less stressful, and easier to perform than those in the conventional OSCE. Approximately half of the students favored not using the vOSCE in the future if a conventional OSCE is available.

Interpretation

According to recently published recommendations for conducting online OSCEs, detailed planning and proper use of workforce and resources are essential for preparing a smooth and convenient assessment experience [7]. In our study, the majority of students had positive perceptions concerning the faculty initiatives in planning for the vOSCE. Our study participants commended the beneficial role of a written protocol and compulsory trial-run sessions involving all students and their proposed examiners. From students’ perspectives, the examiner’s prior communication in a mock session reduced the students’ stress level before the assessment. Moreover, more than half of the responses agreed that the overall vOSCE process flow was appropriate. However, about 40% of the participants disagreed that the time allocated for their tasks was sufficient. It seems that the vOSCE could not overcome the long-term concern of time adequacy, which has been highlighted frequently regarding conventional OSCE assessments [8]. It would be recommended to ensure time adequacy by reviewing the assessment tasks and allocated time, including the additional time needed as part of the online technical requirements.

Concerning comparative perceptions on the conventional OSCE and vOSCE, the study participants highlighted several differences. More than half of the respondents viewed the direct face-to-face interaction with the examiner in a conventional OSCE as more convenient than using an online platform in the vOSCE. Unlike our study participants, dentistry and medical students from North America agreed that their experience of vOSCE was comparable and as successful as the conventional OSCE [5,9]. This may be attributed to the low exposure of our students to online assessment activities before the pandemic. Consistently, half of the responses disagreed with maintaining the vOSCE in the post-pandemic era when the conventional OSCE could be conducted. As reported previously in the local setting, our findings indicate that the students viewed the simulation of real clinical practice involving direct interactions between healthcare providers and patients as preferable to an online assessment model [10]. In addition, only about one-third of the students agreed that the vOSCE allowed them to demonstrate their knowledge and skills to the same degree as the conventional OSCE. This result disagrees with a study of dentistry and nursing students in the United States, where the majority of the participants felt they were able to fully showcase their knowledge through the vOSCE [5]. This might also indicate the further need to expose our students to telehealthcare models in order to familiarize them with ways of demonstrating their clinical skills and providing healthcare services remotely.

Concerning the stress associated with the OSCE, it has been frequently highlighted that OSCEs are often associated with remarkably higher stress levels than are associated with other assessment types [11]. The OSCE tends to be perceived as a more difficult type of examination than others [12]. Our findings showed that about 44% and 41% of the respondents disagreed that the vOSCE was less stressful and easier, respectively, than the conventional OSCE. This finding might imply that the source of stress and difficulty levels perceived for the OSCE may be more closely related to the nature of the given tasks, rather than the medium of communication.

As expected, more than half of the responses referred to the quality of the internet connection, which impacted their vOSCE experience. This finding is not an isolated case, especially in Southeast Asia, where problems such as bandwidth issues, connectivity, financial limitations, the digital divide, learners’ engagement, technological adaptation, and capabilities exist in every level of online and remote learning [13]. Although connection problems were common in online assessments [5], most respondents were satisfied with the planned arrangements to overcome such barriers by adjusting the time or changing the online platform. In the era of online education, which started in earnest with the pandemic, establishing an alternative arrangement during online assessments is essential. It would be helpful to convey to the students that their faculty members are considering the potential limitations that they may face, and are ready to support students as needed.

Comparison with previous studies

Previous research has not provided conclusive data on the impact of students’ characteristics, such as gender, on the OSCE assessment [6]. Our findings showed a few differences among the study participants based on gender, year of study, and their interest in CPCs. Female participants tended to have better perceptions regarding the clarity of the instructions before the assessment. Thus, they showed more appreciation for the examiners’ support throughout the vOSCE experience than male participants. Year 3 students were more likely to agree than year 4 students that the overall flow of the vOSCE was smooth. This finding could be explained by the fact that the vOSCE for year 3 was designed as 1 task; meanwhile, the vOSCE for year 4 students was designed as 2 tasks to be completed consecutively. This may imply the need to consider refining the number of tasks offered in the vOSCE. Finally, students who expressed an interest in CPCs tended to have better perceptions in several items related to preparation, operation, and faculty support throughout the vOSCE experience. These findings highlight that students’ baseline clinical interest might directly impact their level of satisfaction with clinical learning activities and assessments such as the vOSCE.

Limitations

The work was conducted at a single public pharmacy school in Malaysia; therefore, it may not be generalizable to vOSCEs at other pharmacy schools in Malaysia or beyond. In addition, there was no further investigation into the possible impact of students’ grades on their satisfaction with the vOSCE.

Conclusion

This study provides a basis for the better implementation of vOSCEs in the future. Students were satisfied with the overall organization and operation of the vOSCE; however, they still preferred conventional over vOSCEs, indicating a potential need for further exposure to telehealthcare models. Stress and time limitations did not change significantly with the vOSCE compared to conventional OSCE. Students’ self-interest in clinical subjects seemed to have affected their perceptions and level of satisfaction with the experiential learning assessments.