Assessment of structured physical examination skills training using a retro-pre-questionnaire
Article information
Abstract
Purpose:
The effectiveness of physical examination skills (PES) training is very rarely assessed using the “post-then-pre” approach. In this study, a retro-pre-questionnaire was used to study the effect of structured physical examination skills training (SPEST) imparted to second-year undergraduate medical students.
Methods:
KIST Medical College (KISTMC) affiliated to Tribhuvan University Nepal admitted its first batch of MBBS students in November 2008. The university curriculum recommends the involvement of Medicine and Surgery Departments in PES training, but the methods for teaching and assessment are not well defined. KISTMC has made training more structured and involved the Medicine, Surgery, Gynaecology and Obstetrics, Orthopaedics, ENT, Ophthalmology, Paediatrics, and Family Medicine Departments. SPEST includes the teaching/learning of basic PES for 210 minutes once a week for 28 weeks. Self-assessment is done by using a retro-pre-questionnaire at the end of the last session of training, and these data are analysed using SPSS.
Results:
Out of 100 students, 98 participated in the objective structured clinical examination (OSCE); 82 completed the retro-pre-questionnaire. Forty-six skills representing various systems were selected for inclusion in the retro-pre-questionnaire from among the many skills taught in different departments. The average perceived skills score (maximum score, 46×4=184) before training was 15.9 and increased to 116.5 after training. The increase was statistically significant upon the application of a paired t-test.
Conclusion:
The students perceived that their level of skills improved after the training. The retro-pre- instrument seems to be useful for assessing the learners’ self-reported changes in PES after training if a large number of skills need to be assessed. However, it should be noted that although a retro-pre-questionnaire may reveal valuable information, it is not a substitute for an objective measure or gold standard.
The retrospective post-then-pre design is widely used for assessing learners’ self-reported changes in knowledge, awareness, skills, confidence, attitudes, or behaviors. It takes less time, is less intrusive, and for self-reported change, avoids the pre-test sensitivity and the response shift bias that result from a pre-test overestimation or underestimation [1]. The effectiveness of physical examination skills (PES) training is very rarely assessed using a self-assessment instrument. In this study, a self-assessment instrument, a retro-pre-questionnaire, was used to study the perceived effect of structured physical examination skills training (SPEST) imparted to second-year undergraduate medical students as a part of early clinical exposure [2].
METHODS
This study was conducted in KIST Medical College (KISTMC), a newly established medical school in the private sector located in Lalitpur district of Nepal. The first batch of MBBS students was admitted in November 2008. The college is affiliated with Tribhuvan University Nepal. The university curriculum recommends the involvement of Medicine and Surgery Departments for PES training, but the methods for teaching and assessment are not well defined. KISTMC has made training more structured and involved the Departments of Internal Medicine, General Surgery, Gynaecology and Obstetrics, Orthopaedics, Otorhinolaryngology, Ophthalmology, Paediatrics, and Family Medicine for imparting training to secondyear MBBS students as a part of early clinical exposure. SPEST includes the teaching/learning of basic PES for 210 minutes (3.5 hours) once in a week for 28 weeks. Each SPEST session is divided into three sub-sessions: Tutorial (T), Demonstration (D), and Practice (P). The Tutorial element encompasses an overview by a faculty preceptor on the skills to be taught; the Demonstration involves an actual demonstration of each of the skills taught with a step-by-step description of the procedure; and Practice involves the performance/practice of each demonstrated skill by the students using a sequential description to be observed by the preceptor. At the end of the last session of training, a retro-pre-questionnaire was used for assessing the learners’ self-reported changes.
Forty-six skills were selected from many skills taught during training and included in the retro-pre-questionnaire (Table 1). The skills included in the questionnaire represented various systems taught in the different departments during the training. The retro-pre-questionnaire was modelled in line with the Dreyfus model of skill aquisition. Before use, the retro- pre-questionnaire was piloted with 10 students of a senior cohort. It was found to be reliable and valid. The objective item scores were added to obtain the overall score and the descriptive statistics for before and after the training. A paired t-test was used for evaluating the difference in the overall scores. The difference between the overall scores before and after was found to follow the normal distribution, as confirmed by the Shapiro-Wilk test. The data were analysed using SPSS ver. 15 (SPSS Inc., Chicago, IL, USA).
RESULTS
The students’ responses to the retro-pre-questionnaire before and after the structured physical examination skills training are given in Table 1. Out of the 100 students trained, 82 completed the retro-pre-questionnaires. The average skill score (maximum, 46×4=184) for before the SPEST module was 15.9 (median, 13.5) and that after was 116.5 (median, 116). The increase in scores was statistically significant (mean±SD, 100.5±23; 95% confidence interval, 95.45 to 105.59), which implied that the students perceived that they learned most of the skills after the SPEST module and that the course was effective. Further, the paired correlation was not very high (r= 0.240), but it was in the positive direction and was statistically significant (P=0.03), implying that the questionnaire before score had a low impact on the questionnaire after score.
DISCUSSION
The most widely used evaluation tool is a traditional “prethen- post” test, where participants are asked a series of questions at both the start of a program (pre-test) and then again at the end of program (post-test). This tool is believed to measure changes in participant knowledge, attitudes, or behaviours regarding whatever the program content is [3,4]. The criticism of the traditional pre/post tool has led to the use of a “retrospective pre-test” tool. This type of tool is well accepted because it is implemented at only one point in time. The theory behind designing this type of tool is that by testing what participants believe about the program content after program completion, their standard of assessing the changes in knowledge, skills, or attitudes is consistent and thus, not subject to a response shift bias [3,5,6]. Others note that the post-then-pre design reduces incomplete data sets and is convenient to administer given the time constraints that many program providers face, and is also easier for the program participants to complete [3,7,8].
If the objective is to identify how participants perceive the changes that they have made in knowledge, skills, attitudes, or behaviour, then a retro-pre method may be adequate to capture information on this type of data [3].
Our objective was to study the perceived impact of SPEST imparted to second-year undergraduate medical students. The results from the paired t-test showed that the difference between before and after the SPEST was statistically significant, which implied that the students did learn most of the skills after the implementation of the SPEST module and that the training was effective. Further, the paired correlation was not very high, but it was in the positive direction and was statistically significant, implying that the retro-pre before score had a low impact on the retro-pre after score. It is important to acknowledge that all self-reported information could be considered somewhat subjective.
In summary, because the students perceived that their level of skills improved after the training, we concluded that the retro-pre instrument is useful for assessing learners’ self-reported changes in PES after training if a large number of skills are to be assessed. However, it should be noted that although a retro-pre instrument may reveal valuable information, it is not a substitute for an objective measure or gold standard.
Notes
CONFLICT OF INTEREST
No potential conflict of interest relevant to this article was reported