Assessment of structured physical examination skills training using a retro-pre-questionnaire

Article information

J Educ Eval Health Prof. 2013;10.13
Publication date (electronic) : 2013 December 4
doi : https://doi.org/10.3352/jeehp.2013.10.13
1KIST Medical College, Lalitpur, Nepal
2Patan Academy of Health Sciences, Lalitpur, Nepal

Present address: Rano Mal Piryani and Mahesh Prasad Khakurel, Nepal Police Hospital, Kathmandu, Nepal; P. Ravi Shankar, Xavier University School of Medicine, Aruba, Kingdom of the Netherlands; Suneel Piryani, Institute of Medicine, Kathmandu, Nepal.

*Corresponding email: r_piryani@hotmail.com
Received 2013 September 17; Accepted 2013 December 3.

Abstract

Purpose:

The effectiveness of physical examination skills (PES) training is very rarely assessed using the “post-then-pre” approach. In this study, a retro-pre-questionnaire was used to study the effect of structured physical examination skills training (SPEST) imparted to second-year undergraduate medical students.

Methods:

KIST Medical College (KISTMC) affiliated to Tribhuvan University Nepal admitted its first batch of MBBS students in November 2008. The university curriculum recommends the involvement of Medicine and Surgery Departments in PES training, but the methods for teaching and assessment are not well defined. KISTMC has made training more structured and involved the Medicine, Surgery, Gynaecology and Obstetrics, Orthopaedics, ENT, Ophthalmology, Paediatrics, and Family Medicine Departments. SPEST includes the teaching/learning of basic PES for 210 minutes once a week for 28 weeks. Self-assessment is done by using a retro-pre-questionnaire at the end of the last session of training, and these data are analysed using SPSS.

Results:

Out of 100 students, 98 participated in the objective structured clinical examination (OSCE); 82 completed the retro-pre-questionnaire. Forty-six skills representing various systems were selected for inclusion in the retro-pre-questionnaire from among the many skills taught in different departments. The average perceived skills score (maximum score, 46×4=184) before training was 15.9 and increased to 116.5 after training. The increase was statistically significant upon the application of a paired t-test.

Conclusion:

The students perceived that their level of skills improved after the training. The retro-pre- instrument seems to be useful for assessing the learners’ self-reported changes in PES after training if a large number of skills need to be assessed. However, it should be noted that although a retro-pre-questionnaire may reveal valuable information, it is not a substitute for an objective measure or gold standard.

The retrospective post-then-pre design is widely used for assessing learners’ self-reported changes in knowledge, awareness, skills, confidence, attitudes, or behaviors. It takes less time, is less intrusive, and for self-reported change, avoids the pre-test sensitivity and the response shift bias that result from a pre-test overestimation or underestimation [1]. The effectiveness of physical examination skills (PES) training is very rarely assessed using a self-assessment instrument. In this study, a self-assessment instrument, a retro-pre-questionnaire, was used to study the perceived effect of structured physical examination skills training (SPEST) imparted to second-year undergraduate medical students as a part of early clinical exposure [2].

METHODS

This study was conducted in KIST Medical College (KISTMC), a newly established medical school in the private sector located in Lalitpur district of Nepal. The first batch of MBBS students was admitted in November 2008. The college is affiliated with Tribhuvan University Nepal. The university curriculum recommends the involvement of Medicine and Surgery Departments for PES training, but the methods for teaching and assessment are not well defined. KISTMC has made training more structured and involved the Departments of Internal Medicine, General Surgery, Gynaecology and Obstetrics, Orthopaedics, Otorhinolaryngology, Ophthalmology, Paediatrics, and Family Medicine for imparting training to secondyear MBBS students as a part of early clinical exposure. SPEST includes the teaching/learning of basic PES for 210 minutes (3.5 hours) once in a week for 28 weeks. Each SPEST session is divided into three sub-sessions: Tutorial (T), Demonstration (D), and Practice (P). The Tutorial element encompasses an overview by a faculty preceptor on the skills to be taught; the Demonstration involves an actual demonstration of each of the skills taught with a step-by-step description of the procedure; and Practice involves the performance/practice of each demonstrated skill by the students using a sequential description to be observed by the preceptor. At the end of the last session of training, a retro-pre-questionnaire was used for assessing the learners’ self-reported changes.

Forty-six skills were selected from many skills taught during training and included in the retro-pre-questionnaire (Table 1). The skills included in the questionnaire represented various systems taught in the different departments during the training. The retro-pre-questionnaire was modelled in line with the Dreyfus model of skill aquisition. Before use, the retro- pre-questionnaire was piloted with 10 students of a senior cohort. It was found to be reliable and valid. The objective item scores were added to obtain the overall score and the descriptive statistics for before and after the training. A paired t-test was used for evaluating the difference in the overall scores. The difference between the overall scores before and after was found to follow the normal distribution, as confirmed by the Shapiro-Wilk test. The data were analysed using SPSS ver. 15 (SPSS Inc., Chicago, IL, USA).

Students’ responses to the retro-pre-questionnaire before and after structured physical examination skills training in KIST Medical College (KISTMC) affiliated to Tribhuvan University Nepal

RESULTS

The students’ responses to the retro-pre-questionnaire before and after the structured physical examination skills training are given in Table 1. Out of the 100 students trained, 82 completed the retro-pre-questionnaires. The average skill score (maximum, 46×4=184) for before the SPEST module was 15.9 (median, 13.5) and that after was 116.5 (median, 116). The increase in scores was statistically significant (mean±SD, 100.5±23; 95% confidence interval, 95.45 to 105.59), which implied that the students perceived that they learned most of the skills after the SPEST module and that the course was effective. Further, the paired correlation was not very high (r= 0.240), but it was in the positive direction and was statistically significant (P=0.03), implying that the questionnaire before score had a low impact on the questionnaire after score.

DISCUSSION

The most widely used evaluation tool is a traditional “prethen- post” test, where participants are asked a series of questions at both the start of a program (pre-test) and then again at the end of program (post-test). This tool is believed to measure changes in participant knowledge, attitudes, or behaviours regarding whatever the program content is [3,4]. The criticism of the traditional pre/post tool has led to the use of a “retrospective pre-test” tool. This type of tool is well accepted because it is implemented at only one point in time. The theory behind designing this type of tool is that by testing what participants believe about the program content after program completion, their standard of assessing the changes in knowledge, skills, or attitudes is consistent and thus, not subject to a response shift bias [3,5,6]. Others note that the post-then-pre design reduces incomplete data sets and is convenient to administer given the time constraints that many program providers face, and is also easier for the program participants to complete [3,7,8].

If the objective is to identify how participants perceive the changes that they have made in knowledge, skills, attitudes, or behaviour, then a retro-pre method may be adequate to capture information on this type of data [3].

Our objective was to study the perceived impact of SPEST imparted to second-year undergraduate medical students. The results from the paired t-test showed that the difference between before and after the SPEST was statistically significant, which implied that the students did learn most of the skills after the implementation of the SPEST module and that the training was effective. Further, the paired correlation was not very high, but it was in the positive direction and was statistically significant, implying that the retro-pre before score had a low impact on the retro-pre after score. It is important to acknowledge that all self-reported information could be considered somewhat subjective.

In summary, because the students perceived that their level of skills improved after the training, we concluded that the retro-pre instrument is useful for assessing learners’ self-reported changes in PES after training if a large number of skills are to be assessed. However, it should be noted that although a retro-pre instrument may reveal valuable information, it is not a substitute for an objective measure or gold standard.

Notes

CONFLICT OF INTEREST

No potential conflict of interest relevant to this article was reported

SUPPLEMENTARY MATERIAL

Audio recording of the abstract.

References

1. Program Development and Evaluation. Using the Retrospective Post then-Pre Design, Quick Tips #27 [Internet]. Madison, WI:University of Wisconsin Extension; 2005 [updated 2005 Jul 25; cited 2013 Sep 17]. Available from: http://www.uwex.edu/ces/pdande/resources/pdf/Tipsheet27.pdf.
2. Piryani RM, Shankar PR, Thapa TP, Karki BM, Kafle RK, Khakurel MP, Bhandary S. Introduction of structured physical examination skills to second year undergraduate medical students. F1000Research 2013;2:16. http://dx.doi.org/10.12688/f1000research.2-16.v1.
3. Colosi L, Dunifon R. What’s the difference?: “post then pre” & “pre then post” [Internet]. New York: Cornell University Cooperative Extension; 2006[cited 2013 Sep 17]. Available from:http://www.human.cornell.edu/pam/outreach/parenting/research/upload/What-s-20the-20Difference-20Post-20then-20Pre-20and-20Pre-20then-20Post.pdf.
4. Zaidi Z, Zaidi SM, Razzaq Z, Luqman M, Moin S. Training workshops in problem-based learning: changing faculty attitudes and perceptions in a Pakistani medical college. Educ Health 2010;23:440. http://www.educationforhealth.net/text.asp?2010/23/3/440/101470.
5. Rockwell SK, Kohn H. Post-then-pre evaluation. J Ext 1989;27:2FEA5. http://www.joe.org/joe/1989summer/a5.php.
6. Davis GA. Using retrospective pre-post questionnaire to determine program impact. J Ext 2003;41:4TOT4. http://www.joe.org/joe/2003august/tt4.php/.
7. Raidl M, Johnson S, Gardiner K, Denham M, Spain K, Lanting R, Jayo C, Liddil A, Barron K. Use retrospective surveys to obtain complete data sets and measure impact in extension programs. J Ext 2004;42:2RIB2. http://www.joe.org/joe/2004april/rb2.php.
8. Lamb T. The retrospective pretest: an imperfect but useful tool . Eval Exchange 2005;11:18. http://www.hfrp.org/evaluation/theevaluation-exchange/issue-archive/evaluation-methodology/theretrospective-pretest-an-imperfect-but-useful-tool.

Article information Continued

Table 1.

Students’ responses to the retro-pre-questionnaire before and after structured physical examination skills training in KIST Medical College (KISTMC) affiliated to Tribhuvan University Nepal

Skill (item) Response (%)
Not confident Somewhat confident Very confident Able to do independently
Before training After training Before training After training Before training After training Before training After training
1 Measuring body temperature 12.5 1.2 48.8 13.4 28.8 25.6 10.0 59.8
2 Measuring respiratory rate 55.1 2.4 38.8 18.3 06.1 37.8 0.0 41.5
3 Recording pulse 58.2 1.2 32.8 6.2 07.5 46.3 0 1.5 46.3
4 Measuring blood pressure 56.1 2.4 33.3 18.4 09.1 39.0 0 1.5 40.2
5 Detecting pedal edema 70.4 2.4 25.9 25.6 3.7 35.4 0.0 36.6
6 Detecting cyanosis 60.9 2.4 34.8 18.3 4.3 53.7 0.0 25.6
7 Detecting clubbing 80.0 4.9 13.3 18.3 6.7 43.9 0.0 32.9
8 Palpating cervical lymph nodes 75.0 7.3 25.0 48.8 0.0 28.0 0.0 15.9
9 Demonstrating Range of Motion at Elbow Joint 73.9 14.6 26.1 46.4 0.0 24.4 0.0 14.6
10 Demonstrating range of motion at hip joint 83.3 14.6 16.7 45.1 0.0 24.4 0.0 15.9
11 Palpating peripheral pulses 75.0 7.3 25.0 46.3 0.0 22.0 0.0 24.4
12 Assessing chest expansion 88.9 11.0 11.1 37.8 0.0 32.9 0.0 18.3
13 Determining position of trachea or palpation for trachea 100.0 14.6 0.0 42.7 0.0 24.4 0.0 18.3
14 Percussion of anterior chest 91.7 11.0 8.3 30.5 0.0 35.4 0.0 23.2
15 Auscultation of posterior chest 93.8 15.9 6.2 46.3 0.0 26.8 0.0 11.0
16 Eliciting vocal resonance on anterior chest 91.7 23.2 8.3 39.0 0.0 24.4 0.0 13.4
17 Measuring jugular venous pressure 100.0 41.5 0.0 39.0 0.0 13.4 0.0 6.1
18 Localizing apex beat 80.0 9.8 20.0 42.7 0.0 32.9 0.0 14.6
19 Auscultation of precordium 100.0 40.2 0.0 34.1 0.0 14.6 0.0 11.0
20 Using otoscope for examining ears 100.0 23.2 0.0 41.5 0.0 23.2 0.0 12.1
21 Using nasal speculum for examining ears 100.0 43.9 0.0 31.7 0.0 19.5 0.0 4.9
22 Using head mirror with light for examining ears 100.0 24.4 0.0 54.9 0.0 12.2 0.0 8.5
23 Measuring visual acuity 69.6 11.0 30.4 48.9 0.0 24.4 0.0 15.9
24 Measuring visual fields by confrontation method 80.0 20.7 20.0 37.8 0.0 30.5 0.0 11.0
25 Examining uniocular eye movement (muscles) 83.3 24.4 16.7 43.9 0.0 18.3 0.0 13.4
26 Examining pupillary light reaction 89.3 4.9 10.7 35.4 0.0 39.0 0.0 20.7
27 Palpation of liver 91.7 11.0 8.3 32.9 0.0 34.1 0.0 22.0
28 Palpation of kidneys 100.0 11.0 0.0 31.7 0.0 36.6 0.0 20.7
29 Palpation of spleen 100.0 12.2 0.0 31.7 0.0 34.1 0.0 22.0
30 Eliciting shifting dullness 100.0 20.7 0.0 34.1 0.0 30.5 0.0 14.6
31 Measuring liver span 100.0 43.9 0.0 32.9 0.0 13.4 0.0 9.8
32 Examining facial nerves 100.0 22.0 0.0 45.1 0.0 22.0 0.0 11.0
33 Examining accessory nerves 100.0 32.9 0.0 37.8 0.0 20.7 0.0 8.6
34 Eliciting tone of muscles in upper limbs 100.0 22.0 0.0 50.0 0.0 18.2 0.0 9.8
35 Eliciting reflexes in lower limbs 100.0 12.2 0.0 50.0 0.0 25.6 0.0 12.2
36 Eliciting planter response 100.0 19.5 0.0 37.8 0.0 34.2 0.0 8.5
37 Eliciting position sense in lower limbs 100.0 26.8 0.0 42.7 0.0 23.2 0.0 7.3
38 Assessing orientation to time, place, and person 100.0 3.7 0.0 17.0 0.0 47.6 0.0 31.7
39 Measuring height in children 61.9 1.2 38.1 19.5 0.0 46.3 0.0 32.9
40 Measuring mid-arm circumference in children 60.5 2.4 39.5 23.2 0.0 42.7 0.0 31.7
41 Demonstration of dehydration in infants 100.0 7.3 0.0 29.3 0.0 42.7 0.0 20.7
42 Counting respiratory rate in children under five years of age 100.0 13.4 0.0 46.4 0.0 25.6 0.0 14.6
43 Demonstration of pallor in children 100.0 6.1 0.0 29.3 0.0 50.0 0.0 14.6
44 Demonstration of jaundice in children 100.0 6.1 0.0 29.2 0.0 48.8 0.0 15.9
45 Measuring height of fundus 100.0 22.0 0.0 35.4 0.0 30.4 0.0 12.2
46 Eliciting obstetrical grips or Leopold’s manoeuvre 100.0 48.8 0.0 24.4 0.0 15.8 0.0 11.0