Introduction

In Korea, there are 2 kinds of emergency medical technicians: class I and II. Class I emergency medical technicians are comparable to paramedics, while class II includes basic emergency medical technicians. Examinees of the class I Korean emergency medicine technician licensing examination (KEMTLE) should be college or university graduates who have completed a 3- or 4-year emergency medicine technician course. Graduates of the fire service academies, which have a six-month training period, are eligible to take the class II KEMTLE. When a class II emergency medical technician has acquired more than 3 years of experience in firefighting, he or she is eligible to take the class I KEMTLE. The KEMTLE is supervised by the National Health Personnel Licensing Examination Board (NHPLEB) of Korea. Both the class I and II KEMTLE are composed of written and skill-based tests. The skills tests of both examinations are composed of a basic skills test, an advanced skills test, and physical test items. Examinees randomly select 1 item each from the basic and advanced skills test item pools. The skills test for each examinee is scored by a grader according to a checklist. Table 1 presents the composition of the KEMTLE and the topics of the skills tests. Concerns have been raised regarding the reliability of item selection by examinees. Administering the same test items to all examinees is ideal from the point of view of test reliability; however, this approach has not been applied to the KEMTLE due to limited time, human resources, and budget. This study aimed to determine whether the skills test items that examinees randomly select differ in difficulty, by analyzing examination data from the 2013 class I and II KEMTLE examinations provided by the NHPLEB.

Methods

Study design

This study involved the cross-sectional analysis of examination data without any personal identifiers.

Materials and subjects

Skills test data from 1,309 class I and 1,801 class II examinees from the 19th KEMTLE in 2013 were obtained from the NHPLEB. These data were used to compare the difficulty index of each item in the basic and advanced skills tests. The difficulty index of each item was calculated as each examinee’s scores divided by the maximum score; therefore, a high difficulty index means that examinees performed well on the question.

Statistical analysis

The difficulty index of items was compared using SPSS version 18.0 (SPSS Inc., Chicago, IL, USA). One-way analysis of variance and the t-test were used to compare the means. The cutoff for statistical significance was set at P<0.05. Items that exhibited significant intergroup differences were further analyzed using the post-hoc Scheffe test.

Ethical approval

This study was conducted after receiving individual consent from all subjects, each of whom had previously received an explanation of the study’s objectives.

Results

Class I KEMTLE

The average difficulty index of the 4 basic skills test items was 0.87 (Table 2). Item D (the assessment of internal medicine patients) showed the highest difficulty index (0.91), while item A (endotracheal intubation) showed the lowest (0.85). The 4 items of the basic skills test significantly varied in terms of difficulty index (P<0.01). Post-hoc testing indicated that the difficulty index of item D was different from that of all other items: A, B (automated external defibrillator therapy), and C (intravenous injection) (P<0.01). The average difficulty index of 4 advanced skills items was 0.55. Item H (laryngeal tube intubation) showed the highest average score (0.56). Item I (electrocardiogram check and reading) showed the lowest item difficulty index (0.53). The 4 items of the advanced skills test showed significant differences in difficulty index (P<0.05) (Table 2). Post-hoc testing indicated that the difficulty index of item I was different than that of items E (infant airway obstruction therapy), F (primary assessment of trauma), and G (traction splint application) (P<0.01).

Class II KEMTLE

The average difficulty of the 5 items of the basic skills test was 0.77 (Table 3). Item J (automated external defibrillator therapy) was the least difficult (0.85), while item K (primary assessment of trauma) was the most difficult (0.73). The 4 items in the basic skills test showed significantly different levels of difficulty index (P<0.01). Post-hoc testing indicated significant differences between items J and K, J and L (traction splint application), J and M (infant airway obstruction therapy), J and N (laryngeal tube intubation), and K and M in the basic skills test items (P<0.05). The average difficulty index of the 3 advanced skills test items was 0.76. Item G (suction and oxygen therapy) was the least difficult (0.84), while item O (infant cardiopulmonary resuscitation) was the most difficult (0.68). The 3 advanced skills test items significantly varied in difficulty index (P<0.01). Post-hoc testing indicated significant differences among items O, P (suction and oxygen therapy), and Q (vacuum splint application) in the advanced skills test (P<0.01) (Table 3).

Discussion

The KEMTLE permits examinees to randomly choose test items, which they then solve in front of the graders. This study demonstrated that the random selection of items by examinees requires reform, since differences in item difficulty index among items can affect examinees’ total scores on the licensing examination. In our comparison of the difficulty index of skills test items, we made the assumption that the ability of the examinees who selected each item was evenly distributed. If an examinee selects a skills test item with a high item difficulty, he or she can achieve a better score. Ideally, implementing fixed items would solve this problem, as is already done on the national certification examination for emergency medicine technician-paramedics in the United States [2]. Out of the 25 national health personnel licensing examinations supervised by the NHPLEB of the Republic of Korea, the random selection of skills test items by examinees is present in two: the KEMTLE and the Korean dental hygienist licensing examination. If a skills test center is established for the health personnel licensing examination, the present performance evaluation system allowing examinees to randomly select skills test items could easily be reformed. We believe that the Korean government should support the establishment of skills test centers that can be used for skills tests in a variety of health personnel licensing examinations. Fortunately, the city of Daegu announced that a health skills test training center would be constructed in the Daegu Innovation City Cluster by 2020, since the plans of this center passed the preliminary validity evaluation by the Ministry of Strategy and Finance of the Korean government (http://www.daegu.go.kr/Contents/Content.aspx?cid= 109:17 [cited 2016 Jan 1]). We hope that the construction of this center is completed in the near future and that it enters into active use.

Improvement of the content of the KEMTLE is also required, as the current examination technique is simple and does not evaluate the problem-solving and comprehensive thinking skills that emergency medicine technicians must possess in emergency medical situations [3]. Previous research has also argued for the inclusion of integrated management questions in skills tests [1]. Measures for developing sophisticated yet simple administration formats for skills tests should therefore be studied. Recently, the importance of on-site skills test has been increasingly emphasized. Assessment methods using simulations have proven effective for performance-based evaluations [4]. Similarly, simulation- and scenario-based skills test items should be developed and used in national licensing examinations in order to adequately reflect the changing environment of emergency medical technician services.

In conclusion, a fixed set of items should be administered in order to overcome the aforementioned limitations posed by varying levels of difficulty index in the test items that are randomly selected by examinees.

Supplementary materials

References

• 1. Yoou SK, Choi ES, Sin DM, Jo JM, Lee JE, Uhm TH, Roh SG, Kim YS. Proposal of a new emergency medical technician national practical examination. Korean J Emerg Med Serv 2012;16:45-62.
• 2. National Registry of Emergency Medical Technicians. Paramedic national registry skills test [Internet] Ohio (USA): NREMT; 2015 [cited 2015 September 1]. Available from: https://www.nremt.org/nremt/about/reg_para_history.asp
• 3. National Health Personnel Licensing Examination Board. EMT practical examination notice [Internet] Seoul (KR): National Health Personnel Licensing Examination Board; 2015 [cited 2015 September 1]. Available from: http://www.kuksiwon.or.kr/index.aspx
• 4. Tavares W, LeBlanc VR, Mausz J, Sun V, Eva KW. Simulation-based assessment of paramedics and performance in real clinical contexts. Prehosp Emerg Care 2013;18:116-122. http://dx.doi.org/10.3109/10903127.2013.818178 ArticlePubMed

Figure & Data

References

Citations

Citations to this article as recorded by  

• Multimedia-Based online Test on Indonesian Language Receptive Skills Development
  M Sudaryanto, D Mardapi, S Hadi
  Journal of Physics: Conference Series.2019; 1339(1): 012120.     CrossRef