Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2023, Korea: a psychometric study

Article information

J Educ Eval Health Prof. 2024;21.40

Publication date (electronic) : 2024 December 11

doi : https://doi.org/10.3352/jeehp.2024.21.40

Yoon Hee Kim^,, Bo Hyun Kim, Joonki Kim, Bokyoung Jung, Sangyoung Bae

Research and Development Division, Korea Health Personnel Licensing Examination Institute, Seoul, Korea

Editor: A Ra Cho, The Catholic University of Korea, Korea

^*Corresponding email: yhkim@kuksiwon.or.kr

Received 2024 November 28; Accepted 2024 December 5.

The Korea Health Personnel Licensing Examination Institute conducts annual national licensing examinations to assess whether candidates possess the minimum competencies required for healthcare practice. Post-examination analyses of item difficulty, discrimination, and reliability are necessary to ensure the validity and appropriateness of these tests. Items with excessively high or low difficulty or discrimination indices may need review and revision, offering valuable feedback to improve future examinations.

This study analyzed the item difficulty, discrimination, and reliability of the 26 health personnel licensing examinations in Korea conducted in 2023 to evaluate their appropriateness. Additionally, it examined the correlation between the average item difficulty index and the average item discrimination index. It is a follow-up report from the 2022 licensing examinations [1].

This was not a human population study, but an analysis of test results. Therefore, institutional review board approval and informed consent were not required.

Item analysis was conducted for the 26 licensing examinations using classical test theory. Item difficulty (P) was calculated as the proportion of examinees selecting the correct answer, ranging from 0 to 1. Item discrimination was assessed through the upper and lower 27% rule, measuring the difference in difficulty between the top 27% and bottom 27% of examinees and the correlation coefficient between the item score and the total test score. The discrimination indices and reliability were calculated only for tests with more than 100 examinees. Statistical analyses were performed using IBM SPSS ver. 21.0 (IBM Corp.).

Pass rates for the 26 examinations varied widely, from 55.2% to 100%. The number of examinees ranged from 6 (health educators–level 1) to 339,417 (care workers). Examinations for physicians, dentists, midwives, nurses, doctors of Korean medicine, and pharmacists had pass rates above 90%. Of note, the midwives’ examination had a 100% pass rate, although only 8 candidates participated (Table 1).

Table 1.

Pass rates for examinees in the 26 national health professions licensing examinations in Korea, 2023

The results of the item analysis for the 2023 examinations are summarized in Table 2.

Table 2.

Difficulty index, upper and lower group discrimination index, item-total correlation, and reliability index (Cronbach α) of the 26 health professionals licensing examinations in Korea, 2023

All examinations demonstrated high reliability, with the lowest reliability observed (0.855) for the health educator (level 3) examination. The average difficulty indices ranged between 62.2% (health educators–level 2) and 81.3% (nurses), while the average item-total correlation spanned from 0.17 (doctors of Korean medicine) to 0.40 (medical technologists and opticians). A negative correlation was observed between the average difficulty index and the average discrimination index (r=–0.3910, P=0.0395), indicating that easier items were less effective at discriminating between high and low performers.

According to classical test theory, average difficulty indices are categorized as moderate (50%–60%), somewhat easy (60%–70%), easy (70%–80%), and very easy (80% or higher). Based on these criteria, the difficulty indices of the 2023 examinations were classified as very easy (nurses and physicians), easy (18 exams), and somewhat easy (10 exams). For medical professionals such as physicians (average difficulty index 80.9%), dentists (77.0%), nurses (81.3%), and doctors of Korean medicine (77.9%), the difficulty indices were in the very easy or easy categories, with pass rates exceeding 90%.

Discrimination indices were interpreted as low (<0.25), average (0.25–0.30), and good (>0.30). Eight examinations showed low discrimination, 9 had average discrimination, and 11 showed good discrimination. For example, the doctors of Korean medicine examination had a low discrimination index (average item-total correlation of 0.17), whereas the optician’s examination showed good discrimination (average item-total correlation of 0.40). All tests demonstrated high reliability, with Cronbach’s α values of 0.855 or higher, ensuring consistent assessment across the examinations.

In 2022, the national licensing examinations had 468,352 examinees with an 89.4% pass rate. In 2023, there were decreases in both examinees (446,054) and passers (388,067), yielding an 87.0% pass rate. The pass rate range shifted from 29.7%–97.1% in 2022 to 52.2%–100% in 2023. An analysis of difficulty indices for both years revealed that most examinations were relatively easy. Thirteen examinations in 2023 exhibited decreased difficulty indices compared to 2022, primarily transitioning among the “easy,” “very easy,” and “somewhat easy” categories, without significant overall shifts. Additionally, the Care Workers Qualification Examination’s transition to computer-based testing in 2023 resulted in non-disclosure of test items and analysis results, limiting direct comparisons.

For discrimination indices, 23 examinations showed lower indices in 2023, and 17 exhibited decreased item-total correlation coefficients. Despite these declines, the overall discrimination indices for the 2023 examinations did not significantly differ from 2022, suggesting no substantial reduction in discriminative power. Confidentiality policies limit the disclosure of detailed item analyses and test results, hindering comparisons with previous analyses and restricting academic discussions on these examinations.

In conclusion, Korea’s 2023 national health personnel licensing examinations demonstrated acceptable ranges of difficulty, discrimination, and reliability indices. However, 8 out of 28 examinations showed low discrimination indices despite appropriate difficulty levels.