1Department of Medical Education, Soonchunhyang University College of Medicine, Asan, Korea
2Research and Development, Korea Health Personnel Licensing Examination Institute, Seoul, Korea
3Master Center for Medical Education Support, College of Medicine, The Catholic University of Korea, Seoul, Korea
4Korea University College of Medicine, Seoul, Korea
5Department of Emergency Medicine, College of Medicine, The Catholic University of Korea, Seoul, Korea
6Department of Medical Education, College of Medicine, The Catholic University of Korea, Seoul, Korea
© 2020, Korea Health Personnel Licensing Examination Institute
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Authors’ contributions
Conceptualization: JHP, MKY, NJK, YMK, DSA. Data curation and analysis: JHP, MKY, NJK. Funding acquisition & project administration: JHP. Writing–original draft: JHP, MKY, NJK. Writing–review & editing: JHP, MKY, NJK, YMK, DSA.
Conflict of interest
Mi Kyoung Yim is a staff of the Korea Health Personnel Licensing Examination Institute; however, this study is not an official opinion of the Institute. No potential conflict of interest relevant to this article was reported.
Funding
This work was supported by the Korea Health Personnel Licensing Examination Institute of Korea in 2019 (No. RE02-2007-00).
Data availability
Data files are available from Harvard Dataverse: https://doi.org/10.7910/DVN/CBOUPU
Dataset 1. Panel rating data.
Dataset 2. Panel survey data.
Characteristic | No. (%) |
---|---|
Specialty | |
Internal medicine | 2 (13.3) |
Surgery | 2 (13.3) |
Obstetrics and gynecology | 2 (13.3) |
Pediatrics | 2 (13.3) |
Psychiatry | 2 (13.3) |
Family medicine | 2 (13.3) |
Preventive medicine | 2 (13.3) |
Emergency medicine | 1 (6.7) |
Gender | |
Male | 10 (66.7) |
Female | 5 (33.3) |
Age group (yr) | |
40s | 5 (33.3) |
50s | 8 (53.4) |
≥60s | 2 (13.3) |
Overall | 15 (100) |
Step 1: specialty | Step 2: item discrimination index | Step 3: item difficulty index | Step 4: split-half sets 1 (50% for each group) | Step 5: split-half sets 2 (25% for each group) |
---|---|---|---|---|
1 | <0.2 | 0–0.39 | A, B, A, B, ... | A1, A2, A1, A2, ... |
B1, B2, B1, B2, ... | ||||
0.40–0.89 | A, B, A, B, ... | A1, A2, A1, A2, ... | ||
B1, B2, B1, B2, ... | ||||
0.90–1 | A, B, A, B, ... | A1, A2, A1, A2, ... | ||
B1, B2, B1, B2, ... | ||||
≥0.2 | 0–0.39 | A, B, A, B, ... | A1, A2, A1, A2, ... | |
B1, B2, B1, B2, ... | ||||
0.40–0.89 | A, B, A, B, ... | A1, A2, A1, A2, ... | ||
B1, B2, B1, B2, ... | ||||
0.90–1 | A, B, A, B, ... | A1, A2, A1, A2, ... | ||
B1, B2, B1, B2, ... | ||||
2–8 |
Test | Type | No. (%) | Mean % |
---|---|---|---|
KMLE 2017 | A1 | 90 (25) | 62.5 |
A2 | 90 (25) | 63.1 | |
B1 | 90 (25) | 64.3 | |
B2 | 90 (25) | 64.2 | |
Mean | 63.5 | ||
KMLE 2018 | A1 | 90 (25) | 62.8 |
B2 | 94 (26) | 61.1 | |
Mean | 62.0 | ||
KMLE 2019 | A1+B2 | 182 (51) | 65.1 |
(A1) | 107 (30) | 64.1 | |
(B2) | 75 (21) | 66.2 | |
Mean | 65.1 |
Relevance | Difficulty | No. of items |
Expected correct answer rate of borderline group | No. of items×expected correct answer rate of borderline group |
||||
---|---|---|---|---|---|---|---|---|
KMLE 2017 | KMLE 2018 | KMLE 2019 | KMLE 2017 | KMLE 2018 | KMLE 2019 | |||
Essential | Easy | 2 | 1 | 80.67 | 1.6 | 0.8 | ||
Medium | 215 | 136 | 117 | 69.33 | 148.4 | 93.8 | 80.7 | |
Hard | 2 | 3 | 2 | 53.67 | 1.1 | 1.6 | 1.1 | |
Important | Easy | 75.33 | ||||||
Medium | 135 | 41 | 49 | 63 | 85.1 | 25.8 | 30.9 | |
Hard | 2 | 2 | 47.33 | 0.9 | 0.9 | |||
Additional | Easy | 64 | ||||||
Medium | 6 | 2 | 10 | 48 | 2.9 | 1 | 4.8 | |
Hard | 1 | 33.47 | 0.3 | |||||
Total | - | 360 | 184 | 182 | Cut score | 239 | 123.2 | 119.6 |
Score out of 100 points | - | 66.4 | 67 | 65.7 |
Test | Type | No. (%) | Modified Angoff | Modified Ebel | Hofstee |
---|---|---|---|---|---|
KMLE 2017 | A1 | 90 (25) | 62.5 | 66 | 61.9 |
A2 | 90 (25) | 63.1 | 65.9 | ||
B1 | 90 (25) | 64.3 | 67.1 | ||
B2 | 90 (25) | 64.2 | 66.6 | ||
Mean±SD | 63.5±0.8 | 66.4±0.5 | |||
KMLE 2018 | A1 | 90 (25) | 62.8 | 66.5 | 67.8 |
B2 | 94 (26) | 61.1 | 67.4 | ||
Mean±SD | 62.0±0.9 | 67.0±0.5 | |||
KMLE 2019 | A1+B2 | 182 (51) | 65.1 | 65.7 | 65.8 |
(A1) | 107 (30) | 64.1 | 65.9 | ||
(B2) | 75 (21) | 66.2 | 65.4 | ||
Mean±SD | 65.1±1.1 | 65.7±0.3 |
Panelist | KMLE 2017 vs. KMLE 2018 | KMLE 2017 vs. KMLE 2019 |
---|---|---|
P1 | 0.902 | - |
P2 | 0.874 | - |
P6 | 0.912 | - |
P9 | 0.051 | - |
P10 | 0.994 | - |
P11 | 0.755 | - |
P14 | 0.966 | - |
P7 | 0.794 | 0.770 |
P3 | - | 0.729 |
P4 | - | 0.919 |
P5 | - | 0.753 |
P8 | - | 0.561 |
P12 | - | 0.645 |
P13 | - | 0.909 |
P15 | - | 0.572 |
Test | No. | Mean±standard deviation | t-value | Degrees of freedom | P-value |
---|---|---|---|---|---|
KMLE 2017 vs. 2018 | -5.158 | 6 | 0.002 | ||
KMLE 2017 | 7 | 0.519±0.095 | |||
KMLE 2018 | 7 | 0.633±0.113 | |||
KMLE 2017 vs. KMLE 2019 | -6.072 | 6 | 0.001 | ||
KMLE 2017 | 7 | 0.471±0.071 | |||
KMLE 2019 | 7 | 0.636±0.076 |
Test | Question | Mean±standard deviation |
---|---|---|
Orientation | 1. The orientation provided adequate information on the purpose of the standard-setting process for a cut score. | 4.6±0.5 |
2. The concept of the minimum-competency physician was clear. | 4.2±0.8 | |
3. It was expedient to assume the correct answer rate of minimum-competency physician. | 3.1±0.9 | |
4. I could fill in the assessment form based on the guidelines. | 4.4±0.6 | |
KMLE 2017 | 1. The individual assessment went smoothly. | 4.3±0.5 |
2. The overall assessment procedure went smoothly. | 4.4±0.5 | |
3. An appropriate rest period was given. | 4.4±0.5 | |
4. The cut score I submitted was appropriate. | 3.6±0.6 | |
5. The final cut score determined by the panel was appropriate. | 4.1±0.6 | |
KMLE 2018 and 2019 | 1. The individual assessments went smoothly. | 4.4±0.5 |
2. The overall assessment procedure went smoothly. | 4.3±0.6 | |
3. An appropriate rest period was given. | 4.5±0.5 | |
4. The cut score I submitted was appropriate. | 3.7±0.6 | |
5. The final cut score determined by the panel was appropriate. | 4.1±0.5 |
Response | Before workshop | After workshop | After the result analysis process |
---|---|---|---|
Strongly disagree | - | 1 (6.7) | - |
Disagree | 2 (14.3) | 6 (40) | 2 (20) |
Neutral | 2 (14.3) | 2 (13.3) | 3 (30) |
Agree | 8 (57.1) | 5 (33.3) | 3 (30) |
Strongly agree | 2 (14.3) | 1 (6.7) | 2 (20) |
Overall | 14 (100) | 15 (100) | 10 (100) |
KMLE, Korean Medical Licensing Examination.
KMLE, Korean Medical Licensing Examination; SD, standard deviation.
KMLE, Korean Medical Licensing Examination.
KMLE, Korean Medical Licensing Examination.
KMLE, Korean Medical Licensing Examination.
Values are presented as number (%).