Graphical abstract

Introduction

Background

The Dr. LEE Jong-wook Fellowship Program is a project being implemented by the Korea Foundation for International Healthcare (KOFIH), and it is also part of the memorial project to honor the late Dr. Lee Jong-Wook, who served as secretary-general of the World Health Organization (WHO) from January 2003 to May 2006. This program is a mid- to long-term invitational training project for healthcare personnel in partner countries to improve the level of healthcare in partner countries and promote international cooperation, supported by the KOFIH. Since its launch in 2007, 1,504 healthcare practitioners from 30 countries have participated as of 2023. This program helps practitioners to learn about the healthcare technology and system of Korea while contributing in the improvement of global healthcare services and building healthcare systems.

As the program continues, the importance and internal/external needs for educational evaluation also increase. However, it is difficult to measure long-term effects with current performance indicators other than achieving short-term goals, and it is also challenging to collect, analyze, and interpret information that matches various expertise, such as that of doctors, nurses, and biomedical engineers. Additionally, current performance indicators have already reached their highest measurable levels.

Therefore, we planned to develop objective performance indicators for evaluating and monitoring educational effectiveness by course and job group by reviewing and analyzing existing performance indicators and sought to improve the program’s evaluation system by developing new indicators that can measure the effectiveness of training in the long term. Furthermore, we aimed to contribute to promoting the revision of the KOFIH’s training projects and revitalizing international exchange through a systematic evaluation system.

Objectives

The purpose of this study was to develop a new performance evaluation indication for the LEE Jong-wook Fellowship Program through a 3-stage process: (1) literature review of established models, (2) validation using a Delphi survey, and (3) importance assessment using the analytic hierarchy process (AHP).

Methods

Ethics statement

We obtained consent from the 18 experts who participated in the Delphi panel.

Study design

We used the Delphi method to evaluate the appropriateness of the assessment tool developed through a literature review. In order to develop criteria for evaluating the LEE Jong-wook Fellowship Program, we designed this study in 3 stages: devising evaluation criteria, validating the evaluation criteria, and calculating the relative importance of the evaluation criteria. First, we reviewed research related to the program and drafted program evaluation criteria. Second, we conducted 2-stage Delphi surveys on the developed evaluation criteria, targeting a panel of experts to evaluate the validity of the evaluation criteria. Third, we calculated the relative importance of the evaluation criteria using the AHP method [1-3].

The Delphi surveys were designed in 3 rounds to confirm the validity of the evaluation indicators. In the first 2 rounds, content validity ratio (CVR) values were calculated to check whether the indicators were valid. In the last of the 3 rounds, the weight for each evaluation indicator was calculated using the AHP method.

Setting

Researchers developed criteria and evaluation indicators after negotiating a proposal based on a literature review with staff and team leaders involved in the LEE Jong-wook Fellowship Program. Then, a finalized survey paper was written for the Delphi survey.

Participants

Respondent group

Delphi members consisted of a total of 18 people (Table 1). Delphi members formed a panel of academic experts related to training projects, experts in international development cooperation projects, and medical education experts who had training experience at the KOFIH in order to synthesize various expert opinions. Experts working at public institutions were also included upon recommendation by the KOFIH.

Workgroup

This study involved 8 researchers, 5 of whom had experience in qualitative research, 2 of whom had extensive experience in qualitative research and Delphi research, and the other had extensive experience in Official Development Assistance projects.

Instrumentation

An email requesting participation in the Delphi panel was sent to experts to confirm their willingness to participate. This Delphi study was conducted from May 2021 to June 2021. It took approximately 30–40 minutes to respond to each survey. Then, the first, second, and third Delphi surveys were conducted on a weekly basis. The survey questions were delivered by email for each round. The collected survey questionnaires were analyzed, and the survey questionnaire was revised to reflect the results. In the third AHP survey, in cases where inconsistent results occurred, this was explained to the relevant panel member, and the survey paper was re-distributed and collected for revision and re-analysis.

Protocol

The study protocol is available in Fig. 1.

Statistical methods

The data were summarized using descriptive statistics, and the CVR was calculated for the Delphi panel. The CVR is the proportional level of agreement regarding how many experts within a panel rate an item as “essential” and is calculated in the following equation [4,5]:

CVR=ne−N2N2

where n_e=number of panel members indicating an item as “essential” and N=number of panel members. Since the number of experts in the Delphi panel was 18, validity was considered to have been confirmed if the CVR was 0.444 or higher. The weight for each evaluation indicator was calculated using the AHP. The equation for the eigenvector method to find the priority vector in AHP is as follows:

A⋅W=λmax⋅W

where A is a pairwise comparison matrix for evaluation indicators; λ_max is the maximum eigenvalue; and W is the eigenvector. The consistency index (CI) was calculated to determine how consistently respondents responded to relative importance:

CI=λmax−nn−1

The consistency ratio (CR) was calculated using the Randon index (RI) as follows:

CR=CIRI

If the CR value was less than 0.1, it was judged that consistency had been ensured, and if it exceeded 0.1, a pairwise comparison was performed again or the questionnaire was revised.

Results

This study was conducted in 3 stages, and the research content and research methods of each stage were as follows (Fig. 2).

Devising the evaluation criteria based on the literature review

We reviewed the literature to examine several models for evaluating training programs. Kirkpatrick’s 4-level model is a famous and widely used model for evaluating outcomes in training programs and this model can be used to specifically evaluate program outcomes [6]. Level 1 is “response,” which involves evaluating the response to the training and satisfaction with the training; level 2 is “learning,” where the knowledge, skills, and attitude changes gained from the training are evaluated; level 3 is “behavior,” which involves evaluating the behavioral changes of the trainees, and level 4 is “result,” where the final outcomes of the training are evaluated. However, Kirkpatrick’s 4-level model does not include trainees’ motivation, variables of knowledge and skills, important relationships between program elements and situations, or the effectiveness of resources.

The context/input/process/product (CIPP) evaluation model focuses on improving programs. This model accommodates the ever-changing nature of educational programs, as well as educators’ needs [7]. This model can evaluate all stages of an educational program, including planning, implementation, overall process, and final results, in accordance with the flow of situations, inputs, processes, and outputs. However, to evaluate a program using the CIPP evaluation model, care must be taken from the program planning stage.

The LOGIC model approach is primarily used when evaluating the relationships between program components and program context [8]. This is a logic model consisting of inputs, activities, outputs, and outcomes. Its format is not complicated, but it may oversimplify the program and prevent evaluators from producing what they need. This model shares similar characteristics with the CIPP evaluation model, but focuses on systems that embody change processes and innovation in education.

Based on the literature review of reports, papers and expert advice [9], CIPP and Kirkpatrick’s 4-level model were evaluated as appropriate. In addition to these models, evaluation criteria were selected, including 6 evaluation criteria (consistency, sustainability, efficiency, relevance, impact, and effectiveness) recommended by the Organization for Economic Cooperation and Development (OECD) Development Assistance Committee (DAC) [10,11]. Based on this evaluation model, evaluation criteria were derived by dividing them into input, activity, and output performance (Fig. 3).

Validating the evaluation criteria

We developed a draft of evaluation criteria and evaluation indicators and conducted validity verification through a Delphi survey targeting a panel of experts. We sent an email requesting participation in the Delphi panel to all experts to confirm their willingness to participate, and then conducted the first, second, and third Delphi surveys on a weekly basis (Datasets 1–3). The collected questionnaires for each round were analyzed, and the next questionnaire was revised based on these. A workshop was conducted to complete the analysis of the Delphi results. It was established that the CVR for each evaluation index must be at least 0.444 to ensure validity. The first round Delphi survey consisted of 4 evaluation areas, 20 evaluation items, and 85 evaluation indicators. The evaluation criteria were well organized and the indicators in each area were evaluated as valid by the Delphi panel members. Comments were provided on items that required detailed evaluation or had ambiguous aspects of measurement. The second round of the survey was conducted by revising the indicators based on the quantitative and qualitative opinions of the Delphi panel. The second Delphi survey consisted of 4 evaluation areas, 20 evaluation items, and 92 evaluation indicators. The CVR of the items and indicators across all areas was 0.444 or higher, indicating that content validity was secured (Table 2).

Calculating the relative importance of the evaluation criteria

We surveyed experts on the relative importance of the finalized evaluation criteria. In this study, a CR analysis was conducted on all response values for domains, items, and indicators based on the results of the expert panel’s responses, and it was determined that consistency was maintained if the value of the CR did not exceed 0.1 (Table 3). The final weights of the areas, items, and indicators were calculated by taking the arithmetic mean of the individual weights of the panel, where consistency was ensured (Table 4). The final evaluation indicators were confirmed through a workshop with the KOFIH and selected as the evaluation tool (Table 5).

Discussion

Key results

The study developed a new set of performance evaluation indicators for the Dr. LEE Jong-wook Fellowship Program, addressing the need for objective and long-term measures of educational effectiveness. Through a comprehensive 3-stage process—reviewing the literature, validating with a Delphi panel of 18 experts, and using the AHP to determine the relative importance of the indicators—the researchers integrated elements from Kirkpatrick’s 4 levels, the CIPP model, and OECD DAC criteria.

Interpretation

Devising evaluation criteria

The study developed evaluation criteria by reviewing established models such as Kirkpatrick’s 4-level framework, the CIPP model, and the OECD DAC criteria. This combination ensured a holistic evaluation approach, addressing both short-term outcomes like training satisfaction and long-term aspects such as sustainability and impact.

Validation through the Delphi study

The Delphi survey involved 18 experts who reviewed and validated the proposed evaluation indicators over 2 rounds. In the second round, 92 indicators were confirmed, achieving a minimum CVR of 0.444, reflecting the result of consensus among experts. This validation process confirmed that the indicators were suitable and reliable for evaluating the fellowship program’s diverse training objectives and outcomes.

Calculating importance using AHP

The importance of each evaluation criterion was ranked using the AHP. The consistency of expert feedback was ensured, and the CR was less than 0.1, indicating that the developed evaluation index was reliable.

This innovative hybrid model ensures a robust, nuanced framework that strengthens the program’s evaluation system. The evaluation model used in this study can ensure a robust and detailed framework for improving the evaluation system of educational programs.

Comparison with previous studies

Compared to previous studies that typically use only a single model [12,13], this study attempted to provide a more comprehensive and robust evaluation framework by integrating multiple models (Kirkpatrick’s 4-level model, the CIPP model, and the OECD DAC criteria). In addition, by using both the Delphi method and the AHP to validate and prioritize evaluation indicators, it provides a level of rigor that has often been lacking in previous studies that have relied more on subjective or less formal methods. Unlike past studies that primarily evaluated immediate outcomes such as satisfaction and learning [14], this study provides a more holistic approach to evaluating the effectiveness of educational programs by emphasizing long-term impacts such as behavior change and sustainability.

Limitations

This study’s specific focus on the Dr. LEE Jong-wook Fellowship Program means that while the methodology can be applied to other contexts, the criteria and indicators developed may require adjustments for applicability in different contexts or programs. Furthermore, the inherent difficulty in measuring long-term impacts remains a challenge, as the study primarily focused on immediate and intermediate outcomes, which may not fully capture the long-term effectiveness and sustainability of the training programs.

Suggestions

The performance indicators developed in this study are educational program evaluation indicators. In particular, since they are used to evaluate the performance of a training program, it is more meaningful to discuss the cycle of evaluating performance indicators and how to use them while establishing an overall program evaluation plan. For further study, it would be beneficial to apply the developed evaluation framework to a broader range of healthcare training programs across different countries and contexts to test its adaptability and generalizability. Additionally, conducting a longitudinal study to track the long-term impacts of the training programs on participants’ professional development and healthcare outcomes in their home countries would provide deeper insights into the sustained effectiveness and impact of the training.

Conclusion

We effectively developed a comprehensive set of performance evaluation indicators for the Dr. LEE Jong-wook Fellowship Program, addressing the need for objective and long-term assessment measures. By integrating multiple evaluation models and validating them through a Delphi panel of experts, the study produced a robust framework that enhances the program’s evaluation system and contributes significantly to global healthcare capacity building. Despite limitations related to the panel size and the challenge of measuring long-term impacts, these indicators can serve as a valuable tool for improving healthcare training programs and lay the groundwork for future research to further refine and test them across diverse contexts.

Supplementary materials

Fig. 1.

Flow chart of the study protocol. KOFIH, Korea Foundation for International Healthcare.

Fig. 2.

Research process and design.

Fig. 3.

Evaluation model and evaluation criteria.

References

• 1. Saaty TL. A scaling method for priorities in hierarchical structures. J Math Psychol 1977;15:234-281. https://doi.org/10.1016/0022-2496(77)90033-5 Article
• 2. Saaty TL. What is the analytic hierarchy process? In: Mitra G, Greenberg HJ, Lootsma FA, Rijkaert MJ, Zimmermann HJ, editors. Mathematical models for decision support. Springer; 1988. p. 109-121. https://doi.org/10.1007/978-3-642-83555-1_5
• 3. Saaty TL. Fundamentals of decision making and priority theory with the analytic hierarchy process. RWS Publications; 1994.
• 4. Ayre C, Scally AJ. Critical values for Lawshe’s content validity ratio: revisiting the original methods of calculation. Meas Eval Counsel Dev 2014;47:79-86. https://doi.org/10.1177/0748175613513808 Article
• 5. Lawshe CH. A quantitative approach to content validity. Pers Psychol 1975;28:563-575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x Article
• 6. Kirkpatrick D, Kirkpatrick J. Evaluating training programs: the four levels. Berrett-Koehler Publishers; 2006.
• 7. Kellaghan T, Stufflebeam DL. International handbook of educational evaluation: part one: perspectives. Springer; 2003. https://doi.org/10.1007/978-94-010-0309-4
• 8. Bickman L. The functions of program theory. New Dir Eval 1987;1987:5-18. https://doi.org/10.1002/ev.1443 Article
• 9. Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE guide no. 67. Med Teach 2012;34:e288-e299. https://doi.org/10.3109/0142159X.2012.668637 ArticlePubMed
• 10. Chianca T. The OECD/DAC criteria for international development evaluations: an assessment and ideas for improvement. J Multidiscip Eval 2008;5:41-51. https://doi.org/10.56645/jmde.v5i9.167 Article
• 11. Ju H, Oh M, Lee JT, Yoon BY. Adapting an integrated program evaluation for promoting competency‐based medical education. Korean Med Educ Rev 2021;23:56-67. https://doi.org/10.17496/kmer.2021.23.1.56 Article
• 12. Lee C, Jyung CY, Choi YJ, Moon Y. Performance analysis of National Human Resource Development Consortium Project using logic model. J Vocat Educ Res [Internet] 2012;[cited 2024 Sep 20];31:195-216. Available from: https://www.kci.go.kr/kciportal/ci/sereArticleSearch/ciSereArtiView.kci?sereArticleSearchBean.artiId=ART001650231
• 13. Lee Y, Choi YS, Hong SM. The diagnostic analysis of consortium training programs for small and middle-sized corporations, using CIPP evaluation model. Korean J Hum Resour Dev Q 2009;11:91-113. https://doi.org/10.18211/kjhrdq.2009.11.2.005 Article
• 14. Bae J, No W. Evaluation of effectiveness on KOICAs training program: focus on KOICAs Country Training Program for Tanzania. Int Dev Coop Rev [Internet] 2011 [cited 2024 Sep 20];3:228-255. Available from: https://scholar.kyobobook.co.kr/article/detail/4010024979080

Figure & Data

References

Citations

Citations to this article as recorded by  

• Halted medical education and medical residents’ training in Korea, journal metrics, and appreciation to reviewers and volunteers
  Sun Huh
  Journal of Educational Evaluation for Health Professions.2025; 22: 1.     CrossRef