Virtual reality (VR) is commonly used to enhance practitioners’ learning and engagement. Learning by doing is the most effective method to understand lessons; however, it is not feasible to make repeated mistakes in practice for some techniques due to risks or resource constraints. Education and training using VR can provide content in a mixed-media format based on creating a virtual world, wherein users see and interact with their surroundings virtual environment in 3 dimensions. Learners can try to perform a technique by themselves as long as they want and may make mistakes without any negative effects, allowing them to practice repeatedly and learn from their mistakes. These capabilities of VR have direct implications for medical education and training [1]. VR training provides realistic learning experiences, supporting learners who are inspired to discover the material on their own. Learners can explore a lesson’s content to see how its elements work and are more likely to engage in the lesson experientially. Moreover, VR provides an opportunity to learn by interacting with lessons instead of passively reading or listening to experts. The widespread use of simulation training is based on patient safety concerns, focusing on improving the quality of medical services and clinical outcomes. Several studies have shown that learning in VR simulations was more effective than traditional clinical training [2-4]. It has been reported that practice using VR simulations improved medical students’ learning in many domains [5].

Intubation is a standard procedure of respiratory care for patients. However, learning and practicing intubation is a risky and challenging process for beginners, because intubation involves manipulating patients’ airways using a laryngoscope and inserting an endotracheal tube into the trachea. Practicing with patients can lead to life-threatening risks when performed by inexperienced operators. Furthermore, the specific steps of intubation involve important differences in the details of the procedure. Supervisors should focus on guiding various sub-behaviors to help trainees improve their skills. Practice usually takes place using manikins; however, equipment is often insufficient for individual learners and manikins require expert guidance to use.

For these reasons, training with VR simulations is beneficial for encouraging and supporting learners as they become proficient in their skills. However, there are various forms of VR training. Interactions through controllers are at the foundation of VR [6], and VR headsets currently offer controllers as an essential accessory without additional purchases. Therefore, many VR applications have user interactions with the environment through controllers, which allow users to perform actions such as touching objects and manipulating objects according to their hand position and button commands. Another mechanism for controlling VR applications is the use of hand tracking for interactions. Hand tracking is a new technology in VR that uses a built-in camera on the VR headset to detect a user’s position and hand gestures. Therefore, VR applications can use hand tracking to control object touching and manipulation by detecting the user’s hand, similar to what is possible using controllers [7].

The present study aimed to investigate the differences in VR intubation training as a case study to explore the differences between using controllers (Figs. 1-4) and hand tracking (Fig. 5) for learning in VR (Figs. 6, 7). The following variables were compared between controller interaction and hand tracking: learning outcomes, practice scores, usability, ease of use, and satisfaction with the VR system. Feedback from real users can provide useful input for understanding the design and development of VR applications to make them more effective from a functional standpoint for learning and practice in medical training. The research question was as follows: are there any differences in learning outcomes and usability between using controllers and hand tracking in VR medical training?

This study was approved by the Ethics Committee of Human Rights Related to Research Involving Human Subjects, Walailak University, Thailand (WUEC-20-031-01). Informed consent was obtained from subjects.

This was a pre- and post-intervention comparative observational study, involving a comparison of 3 groups and interviews.

We conducted this cross-sectional study at Building B6 in the Laboratory Teaching Center of the School of Medicine, Walailak University, Nakhon Si Thammarat, Thailand, from February 17, 2021, to March 5, 2021. The interventional program is available in Supplement 1. The research protocol and experimental process for each group are presented in Table 1 and Dataset 1. None of the participants had previously studied intubation. We divided them into 3 groups, each of which contained 15 students who learned intubation in a different environment. Group 1 consisted of 15 students who received only video learning (Supplement 2). Group 2 consisted of 15 students who received both video learning and VR training with controller interactions (Supplement 3). Group 3 consisted of 15 students who received video learning and VR training with hand tracking interactions (Supplement 4).

The sample comprised participants who were 3rd-year medical students (undergraduate) taking part in clinical training at Walailak University, Nakhon Si Thammarat, Thailand. Total number of students was 48. The researchers included only data from students who completed all tasks, including the pre- and post-intervention questionnaires and practice tests (n=45), and no data were excluded for this reason. The participants all voluntarily participated in the study after providing informed consent.

The following variables were analyzed: pre-test and post-test scores for learning outcomes; usability for the System Usability Scale (SUS) tool; usefulness, ease of use, ease of learning, and satisfaction for the User Satisfaction Evaluation Questionnaire (USEQ) tool; and emotional, instrumental, and motivational experiences in the interviews.

Our experiment investigated learning outcomes and usability. Learning outcomes were assessed in 2 parts. In the first part on knowledge and understanding, pre- and post-test scores were assessed using 10 questions testing intubation knowledge with a total score of 10 (Supplement 5). The reliability of the test was pre-tested with a non-study sample using the Cronbach α coefficient. The second part related to the practice assessment (Supplement 6). The validity of the research instruments for evaluating the success of learning was checked by 3 experts to assess students’ ability to practice with the manikin. The item-objective congruence index was used to determine content validity. There were 7 steps to be followed with a total score of 14. On both VR applications, usability was assessed using the SUS [8] and USEQ [9], which are 5-point Likert-scale questionnaires. The SUS was used to evaluate the usability of the VR application, while the USEQ was used to assess its usefulness, ease of use, ease of learning, and satisfaction.

The researchers explained which protocol would be followed for each group, and each student then voluntarily chose a group according to his or her preferences. It may be possible that less self-motivated learners might not opt to try a new modality.

The sample size was chosen (45 out of a total of 48 medical students) as a convenient sample for this study. These students voluntarily participated in this study and enrolled into one of 3 groups.

Table 2 shows the means and standard derivations of the pre-test, post-test, and practice scores. The results of normality testing for all 3 groups showed that the pre-test, post-test, and practice scores had a normal distribution. One-way analysis of variance (ANOVA) was thus used as a statistical model to analyze the differences among these groups. Additionally, the SUS scores of both VR groups also exhibited a normal distribution. Therefore, the independent t-test was used to examine the differences between the VR groups. However, the usefulness, ease of use, ease of learning, and satisfaction of both VR groups showed non-parametric distributions; for this reason, the Mann-Whitney test was used to analyze the differences in those scores between the VR groups.

As shown in Table 3, the pre-test scores of all groups were not significantly different (P=0.468), implying that all 3 groups had comparable prior knowledge (Dataset 2).

As presented in Table 3, the post-test scores of all groups were significantly different (P=0.028) (Dataset 3). When using the ANOVA post hoc test (least significant difference, LSD), we found the following pairwise P-values: video versus VR controllers, P=0.012; video versus VR hand tracking, P=0.043; and VR controllers versus VR hand tracking, P=0.581. This result shows that the video group had significantly lower post-test scores than both VR groups. However, the post-test scores of both VR groups were not significantly different. Therefore, the post-test evaluation indicated that participants who learned with VR had a higher level of knowledge than participants who only watched a video.

As shown in Table 3, the practice scores of all groups were different with a high level of significance (P<0.001, Dataset 4). On the practice test, we found that participants who learned with VR had different scores from those of participants who only watched the video. Moreover, the average score on the practice test of participants who learned with VR was higher than that of participants who only watched the video. The ANOVA post hoc test (LSD) showed that the practice scores were not significantly different between the VR controller and VR hand tracking groups (P=0.168).

The SUS score was 67.17 (almost satisfactory) for the VR controllers and 60.17 (poor) for VR hand tracking. As shown in Table 4, these scores were not statistically significant, consistent with the interviews finding that there were areas for improvement in both VR applications (Dataset 5).

Similarly, as shown in Table 5, the usefulness, ease of use, ease of learning, and satisfaction scores were not significantly different between both VR applications. Therefore, to summarize, the experiment of VR medical training on intubation did not show a significant difference in usability according to the use of controllers or hand tracking.

The results from the interviews with 30 students in the 2 VR trial groups were broadly consistent, with the following details:

Although no significant differences were found between both VR groups, the results for the VR controller group were slightly better. The use of hand tracking technology is not perfect and there remains room for improvement of its accuracy. The detection of hand gestures is sometimes not stable. Handling and dropping still have some delays at times, resulting in a lower usability score on the USEQ. However, this issue related to usability did not affect learning outcomes. We found from the interviews that users who gave a low usability score would not recommend the VR system to others, but did give comments on how to improve it.

The video-only group’s average post-test and practice scores were less than those of the groups who used VR simulations. Focusing on the interactions in VR intubation training, we found that there was no significant difference according to the use of controllers or hand tracking to interact with virtual objects in the simulations. In particular, the post-test and practice scores of both VR groups, as indicators of learning outcomes, were not meaningfully different. We found that both VR groups’ SUS and USEQ scores, as measures of VR usability, were also not significantly different. However, the average SUS score of the VR controller group was slightly higher than that of the VR hand tracking group. This is consistent with the findings of the interviews that users were better at push-button interactions than at hand gestures. The interviews suggested that the participants wanted to add learning assistance systems such as sound systems, validation systems, or force feedback systems to make it easier for them to learn by themselves.

Our results on learning outcomes from the case study of intubation training using VR are consistent with the findings of other studies that VR contributes to learning and practice. Simulation training was associated to improve knowledge and skill outcomes [2,3] as well as practice performance [4,5].

Some of the equipment used for training in the VR application might have inconsistent appearances, leading to confusion among some students during the practice test.

The learning outcomes from the post-test and practice scores indicated that the 2 VR training sessions had comparable outcomes. The usability scores for all categories revealed that utilizing a VR application was similar with either controllers or hand tracking. Based on our research question, we conclude that using controllers or hand tracking for the case study of intubation training in VR made no difference in terms of learning results and usability. A future task is to develop VR applications that enable user-suggested functionalities and examine the factors that contribute to better learning results and interaction usability.