Journal of Educational Evaluation for Health Professions

Research article

Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study: Ariel Shana Frey-Vogel, Kristina Dzara, Kimberly Anne Gifford, Yoon Soo Park, Justin Berk, Allison Heinly, Darcy Wolcott, Daniel Adam Hall, Shannon Elliott Scott-Vernaglia, Katherine Anne Sparger, Erica Ye-pyng Chung; J Educ Eval Health Prof. 2024;21:3. Published online February 23, 2024; DOI: https://doi.org/10.3352/jeehp.2024.21.3

967 View
193 Download

Abstract PDF Supplementary Material: Purpose
Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into 6 elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure.
Methods
Messick’s unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from 3 pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018–2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables).
Results
Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone (r=0.34, P=0.019).
Conclusion
Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.

Review

Application of artificial intelligence chatbots, including ChatGPT, in education, scholarly work, programming, and content generation and its prospects: a narrative review: Tae Won Kim; J Educ Eval Health Prof. 2023;20:38. Published online December 27, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.38

5,213 View
657 Download
4 Web of Science
5 Crossref

This study aims to explore ChatGPT’s (GPT-3.5 version) functionalities, including reinforcement learning, diverse applications, and limitations. ChatGPT is an artificial intelligence (AI) chatbot powered by OpenAI’s Generative Pre-trained Transformer (GPT) model. The chatbot’s applications span education, programming, content generation, and more, demonstrating its versatility. ChatGPT can improve education by creating assignments and offering personalized feedback, as shown by its notable performance in medical exams and the United States Medical Licensing Exam. However, concerns include plagiarism, reliability, and educational disparities. It aids in various research tasks, from design to writing, and has shown proficiency in summarizing and suggesting titles. Its use in scientific writing and language translation is promising, but professional oversight is needed for accuracy and originality. It assists in programming tasks like writing code, debugging, and guiding installation and updates. It offers diverse applications, from cheering up individuals to generating creative content like essays, news articles, and business plans. Unlike search engines, ChatGPT provides interactive, generative responses and understands context, making it more akin to human conversation, in contrast to conventional search engines’ keyword-based, non-interactive nature. ChatGPT has limitations, such as potential bias, dependence on outdated data, and revenue generation challenges. Nonetheless, ChatGPT is considered to be a transformative AI tool poised to redefine the future of generative technology. In conclusion, advancements in AI, such as ChatGPT, are altering how knowledge is acquired and applied, marking a shift from search engines to creativity engines. This transformation highlights the increasing importance of AI literacy and the ability to effectively utilize AI in various domains of life.

Citations

Citations to this article as recorded by

Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review
Xiaojun Xu, Yixiao Chen, Jing Miao
Journal of Educational Evaluation for Health Professions.2024; 21: 6. CrossRef
Artificial Intelligence: Fundamentals and Breakthrough Applications in Epilepsy
Wesley Kerr, Sandra Acosta, Patrick Kwan, Gregory Worrell, Mohamad A. Mikati
Epilepsy Currents.2024;[Epub] CrossRef
A Developed Graphical User Interface-Based on Different Generative Pre-trained Transformers Models
Ekrem Küçük, İpek Balıkçı Çiçek, Zeynep Küçükakçalı, Cihan Yetiş, Cemil Çolak
ODÜ Tıp Dergisi.2024; 11(1): 18. CrossRef
Art or Artifact: Evaluating the Accuracy, Appeal, and Educational Value of AI-Generated Imagery in DALL·E 3 for Illustrating Congenital Heart Diseases
Mohamad-Hani Temsah, Abdullah N. Alhuzaimi, Mohammed Almansour, Fadi Aljamaan, Khalid Alhasan, Munirah A. Batarfi, Ibraheem Altamimi, Amani Alharbi, Adel Abdulaziz Alsuhaibani, Leena Alwakeel, Abdulrahman Abdulkhaliq Alzahrani, Khaled B. Alsulaim, Amr Jam
Journal of Medical Systems.2024;[Epub] CrossRef
Authentic assessment in medical education: exploring AI integration and student-as-partners collaboration
Syeda Sadia Fatima, Nabeel Ashfaque Sheikh, Athar Osama
Postgraduate Medical Journal.2024;[Epub] CrossRef

Technical report

Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2022, Korea: a psychometric study: Yoon Hee Kim, Bo Hyun Kim, Joonki Kim, Bokyoung Jung, Sangyoung Bae; J Educ Eval Health Prof. 2023;20:31. Published online November 22, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.31

1,144 View
90 Download

Abstract PDF Supplementary Material: Purpose
This study presents item analysis results of the 26 health personnel licensing examinations managed by the Korea Health Personnel Licensing Examination Institute (KHPLEI) in 2022.
Methods
The item difficulty index, item discrimination index, and reliability were calculated. The item discrimination index was calculated using a discrimination index based on the upper and lower 27% rule and the item-total correlation.
Results
Out of 468,352 total examinees, 418,887 (89.4%) passed. The pass rates ranged from 27.3% for health educators level 1 to 97.1% for oriental medical doctors. Most examinations had a high average difficulty index, albeit to varying degrees, ranging from 61.3% for prosthetists and orthotists to 83.9% for care workers. The average discrimination index based on the upper and lower 27% rule ranged from 0.17 for oriental medical doctors to 0.38 for radiological technologists. The average item-total correlation ranged from 0.20 for oriental medical doctors to 0.38 for radiological technologists. The Cronbach α, as a measure of reliability, ranged from 0.872 for health educators-level 3 to 0.978 for medical technologists. The correlation coefficient between the average difficulty index and average discrimination index was -0.2452 (P=0.1557), that between the average difficulty index and the average item-total correlation was 0.3502 (P=0.0392), and that between the average discrimination index and the average item-total correlation was 0.7944 (P<0.0001).
Conclusion
This technical report presents the item analysis results and reliability of the recent examinations by the KHPLEI, demonstrating an acceptable range of difficulty index and discrimination index values, as well as good reliability.

Research articles

Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study: Aleksandra Ignjatović, Lazar Stevanović; J Educ Eval Health Prof. 2023;20:28. Published online October 16, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.28

2,971 View
193 Download
5 Web of Science
7 Crossref

Abstract PDF Supplementary Material

Purpose
This study aimed to assess the performance of ChatGPT (GPT-3.5 and GPT-4) as a study tool in solving biostatistical problems and to identify any potential drawbacks that might arise from using ChatGPT in medical education, particularly in solving practical biostatistical problems.
Methods
ChatGPT was tested to evaluate its ability to solve biostatistical problems from the Handbook of Medical Statistics by Peacock and Peacock in this descriptive study. Tables from the problems were transformed into textual questions. Ten biostatistical problems were randomly chosen and used as text-based input for conversation with ChatGPT (versions 3.5 and 4).
Results
GPT-3.5 solved 5 practical problems in the first attempt, related to categorical data, cross-sectional study, measuring reliability, probability properties, and the t-test. GPT-3.5 failed to provide correct answers regarding analysis of variance, the chi-square test, and sample size within 3 attempts. GPT-4 also solved a task related to the confidence interval in the first attempt and solved all questions within 3 attempts, with precise guidance and monitoring.
Conclusion
The assessment of both versions of ChatGPT performance in 10 biostatistical problems revealed that GPT-3.5 and 4’s performance was below average, with correct response rates of 5 and 6 out of 10 on the first attempt. GPT-4 succeeded in providing all correct answers within 3 attempts. These findings indicate that students must be aware that this tool, even when providing and calculating different statistical analyses, can be wrong, and they should be aware of ChatGPT’s limitations and be careful when incorporating this model into medical education.

Citations

Citations to this article as recorded by

Can Generative AI and ChatGPT Outperform Humans on Cognitive-Demanding Problem-Solving Tasks in Science?
Xiaoming Zhai, Matthew Nyaaba, Wenchao Ma
Science & Education.2024;[Epub] CrossRef
Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review
Xiaojun Xu, Yixiao Chen, Jing Miao
Journal of Educational Evaluation for Health Professions.2024; 21: 6. CrossRef
Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy
Ambadasu Bharatha, Nkemcho Ojeh, Ahbab Mohammad Fazle Rabbi, Michael Campbell, Kandamaran Krishnamurthy, Rhaheem Layne-Yarde, Alok Kumar, Dale Springer, Kenneth Connell, Md Anwarul Majumder
Advances in Medical Education and Practice.2024; Volume 15: 393. CrossRef
Revolutionizing Cardiology With Words: Unveiling the Impact of Large Language Models in Medical Science Writing
Abhijit Bhattaru, Naveena Yanamala, Partho P. Sengupta
Canadian Journal of Cardiology.2024;[Epub] CrossRef
ChatGPT in medicine: prospects and challenges: a review article
Songtao Tan, Xin Xin, Di Wu
International Journal of Surgery.2024;[Epub] CrossRef
In-depth analysis of ChatGPT’s performance based on specific signaling words and phrases in the question stem of 2377 USMLE step 1 style questions
Leonard Knoedler, Samuel Knoedler, Cosima C. Hoch, Lukas Prantl, Konstantin Frank, Laura Soiderer, Sebastian Cotofana, Amir H. Dorafshar, Thilo Schenck, Felix Vollbach, Giuseppe Sofo, Michael Alfertshofer
Scientific Reports.2024;[Epub] CrossRef
Evaluating the quality of responses generated by ChatGPT
Danimir Mandić, Gordana Miščević, Ljiljana Bujišić
Metodicka praksa.2024; 27(1): 5. CrossRef

Development and validation of the student ratings in clinical teaching scale in Australia: a methodological study: Pin-Hsiang Huang, Anthony John O’Sullivan, Boaz Shulruf; J Educ Eval Health Prof. 2023;20:26. Published online September 5, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.26

1,505 View
141 Download

Abstract PDF Supplementary Material: Purpose
This study aimed to devise a valid measurement for assessing clinical students’ perceptions of teaching practices.
Methods
A new tool was developed based on a meta-analysis encompassing effective clinical teaching-learning factors. Seventy-nine items were generated using a frequency (never to always) scale. The tool was applied to the University of New South Wales year 2, 3, and 6 medical students. Exploratory and confirmatory factor analysis (exploratory factor analysis [EFA] and confirmatory factor analysis [CFA], respectively) were conducted to establish the tool’s construct validity and goodness of fit, and Cronbach’s α was used for reliability.
Results
In total, 352 students (44.2%) completed the questionnaire. The EFA identified student-centered learning, problem-solving learning, self-directed learning, and visual technology (reliability, 0.77 to 0.89). CFA showed acceptable goodness of fit (chi-square P<0.01, comparative fit index=0.930 and Tucker-Lewis index=0.917, root mean square error of approximation=0.069, standardized root mean square residual=0.06).
Conclusion
The established tool—Student Ratings in Clinical Teaching (STRICT)—is a valid and reliable tool that demonstrates how students perceive clinical teaching efficacy. STRICT measures the frequency of teaching practices to mitigate the biases of acquiescence and social desirability. Clinical teachers may use the tool to adapt their teaching practices with more active learning activities and to utilize visual technology to facilitate clinical learning efficacy. Clinical educators may apply STRICT to assess how these teaching practices are implemented in current clinical settings.

What impacts students’ satisfaction the most from Medicine Student Experience Questionnaire in Australia: a validity study: Pin-Hsiang Huang, Gary Velan, Greg Smith, Melanie Fentoullis, Sean Edward Kennedy, Karen Jane Gibson, Kerry Uebel, Boaz Shulruf; J Educ Eval Health Prof. 2023;20:2. Published online January 18, 2023; DOI: https://doi.org/10.3352/jeehp.2023.20.2

2,039 View
149 Download
1 Web of Science
1 Crossref

Abstract PDF Supplementary Material

Purpose
This study evaluated the validity of student feedback derived from Medicine Student Experience Questionnaire (MedSEQ), as well as the predictors of students’ satisfaction in the Medicine program.
Methods
Data from MedSEQ applying to the University of New South Wales Medicine program in 2017, 2019, and 2021 were analyzed. Confirmatory factor analysis (CFA) and Cronbach’s α were used to assess the construct validity and reliability of MedSEQ respectively. Hierarchical multiple linear regressions were used to identify the factors that most impact students’ overall satisfaction with the program.
Results
A total of 1,719 students (34.50%) responded to MedSEQ. CFA showed good fit indices (root mean square error of approximation=0.051; comparative fit index=0.939; chi-square/degrees of freedom=6.429). All factors yielded good (α>0.7) or very good (α>0.8) levels of reliability, except the “online resources” factor, which had acceptable reliability (α=0.687). A multiple linear regression model with only demographic characteristics explained 3.8% of the variance in students’ overall satisfaction, whereas the model adding 8 domains from MedSEQ explained 40%, indicating that 36.2% of the variance was attributable to students’ experience across the 8 domains. Three domains had the strongest impact on overall satisfaction: “being cared for,” “satisfaction with teaching,” and “satisfaction with assessment” (β=0.327, 0.148, 0.148, respectively; all with P<0.001).
Conclusion
MedSEQ has good construct validity and high reliability, reflecting students’ satisfaction with the Medicine program. Key factors impacting students’ satisfaction are the perception of being cared for, quality teaching irrespective of the mode of delivery and fair assessment tasks which enhance learning.

Citations

Citations to this article as recorded by

Mental health and quality of life across 6 years of medical training: A year-by-year analysis
Natalia de Castro Pecci Maddalena, Alessandra Lamas Granero Lucchetti, Ivana Lucia Damasio Moutinho, Oscarina da Silva Ezequiel, Giancarlo Lucchetti
International Journal of Social Psychiatry.2024; 70(2): 298. CrossRef

Acceptability of the 8-case objective structured clinical examination of medical students in Korea using generalizability theory: a reliability study: Song Yi Park, Sang-Hwa Lee, Min-Jeong Kim, Ki-Hwan Ji, Ji Ho Ryu; J Educ Eval Health Prof. 2022;19:26. Published online September 8, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.26

2,705 View
215 Download
1 Web of Science
1 Crossref

Abstract PDF Supplementary Material

Purpose
This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT).
Methods
This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada).
Results
All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%.
Conclusion
The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.

Citations

Citations to this article as recorded by

Applying the Generalizability Theory to Identify the Sources of Validity Evidence for the Quality of Communication Questionnaire
Flávia Del Castanhel, Fernanda R. Fonseca, Luciana Bonnassis Burg, Leonardo Maia Nogueira, Getúlio Rodrigues de Oliveira Filho, Suely Grosseman
American Journal of Hospice and Palliative Medicine®.2024; 41(7): 792. CrossRef

Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study: Janghee Park; J Educ Eval Health Prof. 2022;19:23. Published online August 31, 2022; DOI: https://doi.org/10.3352/jeehp.2022.19.23

3,024 View
171 Download
2 Web of Science
2 Crossref

Abstract PDF Supplementary Material

Purpose
The percent Angoff (PA) method has been recommended as a reliable method to set the cutoff score instead of a fixed cut point of 60% in the Korean Medical Licensing Examination (KMLE). The yes/no Angoff (YNA) method, which is easy for panelists to judge, can be considered as an alternative because the KMLE has many items to evaluate. This study aimed to compare the cutoff score and the reliability depending on whether the PA or the YNA standard-setting method was used in the KMLE.
Methods
The materials were the open-access PA data of the KMLE. The PA data were converted to YNA data in 5 categories, in which the probabilities for a “yes” decision by panelists were 50%, 60%, 70%, 80%, and 90%. SPSS for descriptive analysis and G-string for generalizability theory were used to present the results.
Results
The PA method and the YNA method counting 60% as “yes,” estimated similar cutoff scores. Those cutoff scores were deemed acceptable based on the results of the Hofstee method. The highest reliability coefficients estimated by the generalizability test were from the PA method and the YNA method, with probabilities of 70%, 80%, 60%, and 50% for deciding “yes,” in descending order. The panelist’s specialty was the main cause of the error variance. The error size was similar regardless of the standard-setting method.
Conclusion
The above results showed that the PA method was more reliable than the YNA method in estimating the cutoff score of the KMLE. However, the YNA method with a 60% probability for deciding “yes” also can be used as a substitute for the PA method in estimating the cutoff score of the KMLE.

Citations

Citations to this article as recorded by

Issues in the 3rd year of the COVID-19 pandemic, including computer-based testing, study design, ChatGPT, journal metrics, and appreciation to reviewers
Sun Huh
Journal of Educational Evaluation for Health Professions.2023; 20: 5. CrossRef
Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study
Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park
Journal of Educational Evaluation for Health Professions.2022; 19: 33. CrossRef

Development and validation of a measurement scale to assess nursing students’ readiness for the flipped classroom in Sri Lanka: Punithalingam Youhasan, Yan Chen, Mataroria Lyndon, Marcus Alexander Henning; J Educ Eval Health Prof. 2020;17:41. Published online December 14, 2020; DOI: https://doi.org/10.3352/jeehp.2020.17.41

6,824 View
264 Download
8 Web of Science
7 Crossref

Abstract PDF Supplementary Material

Purpose
The aim of this study was to develop and validate a scale to measure nursing students’ readiness for the flipped classroom in Sri Lanka.
Methods
A literature review provided the theoretical framework for developing the Nursing Students’ Readiness for Flipped Classroom (NSR-FC) questionnaire. Five content experts evaluated the NSR-FC, and content validity indices (CVI) were calculated. Cross-sectional surveys among 355 undergraduate nursing students from 3 state universities in Sri Lanka were carried out to assess the psychometric properties of the NSR-FC. Principal component analysis (PCA, n=265), internal consistency (using the Cronbach α coefficient, n=265), and confirmatory factor analysis (CFA, n=90) were done to test construct validity and reliability.
Results
Thirty-seven items were included in the NSR-FC for content validation, resulting in an average scale CVI of 0.94. Two items received item level CVI of less than 0.78. The factor structures of the 35 items were explored through PCA with orthogonal factor rotation, culminating in the identification of 5 factors. These factors were classified as technological readiness, environmental readiness, personal readiness, pedagogical readiness, and interpersonal readiness. The NSR-FC also showed an overall acceptable level of internal consistency (Cronbach α=0.9). CFA verified a 4-factor model (excluding the interpersonal readiness factor) and 20 items that achieved acceptable fit (standardized root mean square residual=0.08, root mean square error of approximation=0.08, comparative fit index=0.87, and χ2/degrees of freedom=1.57).
Conclusion
The NSR-FC, as a 4-factor model, is an acceptable measurement scale for assessing nursing students’ readiness for the flipped classroom in terms of its construct validity and reliability.

Citations

Citations to this article as recorded by

Design and validation of a preliminary instrument to contextualize interactions through information technologies of health professionals
José Fidencio López Luna, Eddie Nahúm Armendáriz Mireles, Marco Aurelio Nuño Maganda, Hiram Herrera Rivas, Rubén Machucho Cadena, Jorge Arturo Hernández Almazán
Health Informatics Journal.2024;[Epub] CrossRef
Content validity of the Constructivist Learning in Higher Education Settings (CLHES) scale in the context of the flipped classroom in higher education
Turki Mesfer Alqahtani, Farrah Dina Yusop, Siti Hajar Halili
Humanities and Social Sciences Communications.2023;[Epub] CrossRef
The intensivist's assessment of gastrointestinal function: A pilot study
Varsha M. Asrani, Colin McArthur, Ian Bissett, John A. Windsor
Australian Critical Care.2022; 35(6): 636. CrossRef
Psychometric evidence of a perception scale about covid-19 vaccination process in Peruvian dentists: a preliminary validation
César F. Cayo-Rojas, Nancy Córdova-Limaylla, Gissela Briceño-Vergel, Marysela Ladera-Castañeda, Hernán Cachay-Criado, Carlos López-Gurreonero, Alberto Cornejo-Pinto, Luis Cervantes-Ganoza
BMC Health Services Research.2022;[Epub] CrossRef
Implementation of a Web-Based Educational Intervention for Promoting Flipped Classroom Pedagogy: A Mixed-Methods Study
Punithalingam Youhasan, Mataroria P. Lyndon, Yan Chen, Marcus A. Henning
Medical Science Educator.2022; 33(1): 91. CrossRef
Assess the feasibility of flipped classroom pedagogy in undergraduate nursing education in Sri Lanka: A mixed-methods study
Punithalingam Youhasan, Yan Chen, Mataroria Lyndon, Marcus A. Henning, Gwo-Jen Hwang
PLOS ONE.2021; 16(11): e0259003. CrossRef
Newly appointed medical faculty members’ self-evaluation of their educational roles at the Catholic University of Korea College of Medicine in 2020 and 2021: a cross-sectional survey-based study
Sun Kim, A Ra Cho, Chul Woon Chung
Journal of Educational Evaluation for Health Professions.2021; 18: 28. CrossRef

Development and validation of the Hocus Focus Magic Performance Evaluation Scale for health professions personnel in the United States: Kevin Spencer, Hon Keung Yuen, Max Darwin, Gavin Jenkins, Kimberly Kirklin; J Educ Eval Health Prof. 2019;16:8. Published online April 10, 2019; DOI: https://doi.org/10.3352/jeehp.2019.16.8

18,684 View
233 Download
4 Web of Science
4 Crossref

Abstract PDF Supplementary Material

Purpose
This study was conducted to describe the development and validation of the Hocus Focus Magic Performance Evaluation Scale (HFMPES), which is used to evaluate the competency of health professions personnel in delivering magic tricks as a therapeutic modality.
Methods
A 2-phase validation process was used. Phase I (content validation) involved 16 magician judges who independently rated the relevance of each of the 5 items in the HFMPES and established the veracity of its content. Phase II evaluated the psychometric properties of the HFMPES. This process involved 2 magicians using the HFMPES to independently evaluate 73 occupational therapy graduate students demonstrating 3 magic tricks.
Results
The HFMPES achieved an excellent scale-content validity index of 0.99. Exploratory factor analysis of the HFMPES scores revealed 1 distinct factor with alpha coefficients ≥0.8 across the 3 magic tricks. The construct validity of the HFMPES scores was further supported by evidence from a known-groups analysis, in which the Mann–Whitney U-test showed significant difference in HFMPES scores between participants with different levels of experience in delivering the 3 magic tricks. The inter-rater reliability coefficients were ≥0.75 across the 3 magic tricks, indicating that the competency of health professions personnel in delivering the 3 magic tricks could be evaluated precisely.
Conclusion
Preliminary evidence supported the content and construct validity of the HFMPES, which was found to have good internal consistency and inter-rater reliability in evaluating health professions personnel’s competency in delivering magic tricks.

Citations

Citations to this article as recorded by

A Magic Trick Training Program to Improve Social Skills and Self-Esteem in Adolescents With Autism Spectrum Disorder
Hon K. Yuen, Kevin Spencer, Lauren Edwards, Kimberly Kirklin, Gavin R. Jenkins
The American Journal of Occupational Therapy.2023;[Epub] CrossRef
MAGNITIVE: Effectiveness and Feasibility of a Cognitive Training Program Through Magic Tricks for Children With Attention Deficit and Hyperactivity Disorder. A Second Clinical Trial in Community Settings
Saray Bonete, Ángela Osuna, Clara Molinero, Inmaculada García-Font
Frontiers in Psychology.2021;[Epub] CrossRef
Contribution of a virtual magic camp to enhancing self-esteem in children with ADHD: A pilot study
Hon K. Yuen, Kevin Spencer, Kimberly Kirklin, Lauren Edwards, Gavin R. Jenkins
Health Psychology Research.2021;[Epub] CrossRef
Evaluation of a Magic Camp for Children with Hemiparesis: A Pilot Study
Kevin Spencer, Hon K. Yuen, Gavin R. Jenkins, Kimberly Kirklin, Angla R. Griffin, Laura K. Vogtle, Drew Davis
Occupational Therapy In Health Care.2020; 34(2): 155. CrossRef

Research Articles

Psychometric properties of a novel knowledge assessment tool of mechanical ventilation for emergency medicine residents in the northeastern United States: Jeremy B. Richards, Tania D. Strout, Todd A. Seigel, Susan R. Wilcox; J Educ Eval Health Prof. 2016;13:10. Published online February 16, 2016; DOI: https://doi.org/10.3352/jeehp.2016.13.10

28,028 View
179 Download
4 Web of Science
4 Crossref

Abstract PDF

Purpose
Prior descriptions of the psychometric properties of validated knowledge assessment tools designed to determine Emergency medicine (EM) residents understanding of physiologic and clinical concepts related to mechanical ventilation are lacking. In this setting, we have performed this study to describe the psychometric and performance properties of a novel knowledge assessment tool that measures EM residents’ knowledge of topics in mechanical ventilation.
Methods
Results from a multicenter, prospective, survey study involving 219 EM residents from 8 academic hospitals in northeastern United States were analyzed to quantify reliability, item difficulty, and item discrimination of each of the 9 questions included in the knowledge assessment tool for 3 weeks, beginning in January 2013.
Results
The response rate for residents completing the knowledge assessment tool was 68.6% (214 out of 312 EM residents). Reliability was assessed by both Cronbach’s alpha coefficient (0.6293) and the Spearman-Brown coefficient (0.6437). Item difficulty ranged from 0.39 to 0.96, with a mean item difficulty of 0.75 for all 9 questions. Uncorrected item discrimination values ranged from 0.111 to 0.556. Corrected item-total correlations were determined by removing the question being assessed from analysis, resulting in a range of item discrimination from 0.139 to 0.498.
Conclusion
Reliability, item difficulty and item discrimination were within satisfactory ranges in this study, demonstrating acceptable psychometric properties of this knowledge assessment tool. This assessment indicates that this knowledge assessment tool is sufficiently rigorous for use in future research studies or for assessment of EM residents for evaluative purposes.

Citations

Citations to this article as recorded by

Comparison of three methods for teaching mechanical ventilation in an emergency setting to sixth-year medical students: a randomized trial
Fernando Sabia Tallo, Letícia Sandre Vendrame, André Luciano Baitello
Revista da Associação Médica Brasileira.2020; 66(10): 1409. CrossRef
Critical Appraisal of Emergency Medicine Educational Research: The Best Publications of 2016
Nicole M. Dubosh, Jaime Jordan, Lalena M. Yarris, Edward Ullman, Joshua Kornegay, Daniel Runde, Amy Miller Juve, Jonathan Fisher, Teresa Chan
AEM Education and Training.2019; 3(1): 58. CrossRef
Mechanical Ventilation Training During Graduate Medical Education: Perspectives and Review of the Literature
Jonathan M. Keller, Dru Claar, Juliana Carvalho Ferreira, David C. Chu, Tanzib Hossain, William Graham Carlos, Jeffrey A. Gold, Stephanie A. Nonas, Nitin Seam
Journal of Graduate Medical Education.2019; 11(4): 389. CrossRef
Development and validation of a questionnaire to assess the knowledge of mechanical ventilation in urgent care among students in their last-year medical course in Brazil
Fernando Sabia Tallo, Simone de Campos Vieira Abib, Andre Luciano Baitello, Renato Delascio Lopes
Clinics.2019; 74: e663. CrossRef

Validation of a clinical critical thinking skills test in nursing: Sujin Shin, Dukyoo Jung, Sungeun Kim; J Educ Eval Health Prof. 2015;12:1. Published online January 27, 2015; DOI: https://doi.org/10.3352/jeehp.2015.12.1

35,514 View
290 Download
5 Web of Science
9 Crossref

Abstract PDF

Purpose
The purpose of this study was to develop a revised version of the clinical critical thinking skills test (CCTS) and to subsequently validate its performance. Methods: This study is a secondary analysis of the CCTS. Data were obtained from a convenience sample of 284 college students in June 2011. Thirty items were analyzed using item response theory and test reliability was assessed. Test-retest reliability was measured using the results of 20 nursing college and graduate school students in July 2013. The content validity of the revised items was analyzed by calculating the degree of agreement between instrument developer intention in item development and the judgments of six experts. To analyze response process validity, qualitative data related to the response processes of nine nursing college students obtained through cognitive interviews were analyzed. Results: Out of initial 30 items, 11 items were excluded after the analysis of difficulty and discrimination parameter. When the 19 items of the revised version of the CCTS were analyzed, levels of item difficulty were found to be relatively low and levels of discrimination were found to be appropriate or high. The degree of agreement between item developer intention and expert judgments equaled or exceeded 50%. Conclusion: From above results, evidence of the response process validity was demonstrated, indicating that subjects respondeds as intended by the test developer. The revised 19-item CCTS was found to have sufficient reliability and validity and will therefore represents a more convenient measurement of critical thinking ability.

Citations

Citations to this article as recorded by

Meta-analysis in Physical Therapy Education Research
Mitch Wolden, Brent Hill, Sara Farquhar Voorhees
Journal of Physical Therapy Education.2019; 33(1): 78. CrossRef
Effects of a work-based critical reflection program for novice nurses
Yeon Hee Kim, Ja Min, Soon Hee Kim, Sujin Shin
BMC Medical Education.2018;[Epub] CrossRef
Measurement of critical thinking, clinical reasoning, and clinical judgment in culturally diverse nursing students – A literature review
Christine L. Sommers
Nurse Education in Practice.2018; 30: 91. CrossRef
Individual and School Factors Affecting Critical Thinking Ability among Nursing Students
Sujin Shin, Inhee Park, Eunhee Hwang, Dukyoo Jung, Kon Hee Kim
Korean Medical Education Review.2018; 20(1): 44. CrossRef
The Health Professions Education Pathway: Preparing Students, Residents, and Fellows to Become Future Educators
H. Carrie Chen, Maria A. Wamsley, Amin Azzam, Katherine Julian, David M. Irby, Patricia S. O'Sullivan
Teaching and Learning in Medicine.2017; 29(2): 216. CrossRef
Cultivating Critical Thinking Using Virtual Interactive Case Studies
Susan M. Burke
Journal of Pediatric Nursing.2017; 33: 94. CrossRef
Encouraging Critical Clinical Thinking (CCT) Skills in First-Year Veterinary Students
Duncan C. Ferguson, Leslie Klis McNeil, David J. Schaeffe, Eric M. Mills
Journal of Veterinary Medical Education.2017; 44(3): 531. CrossRef
Developing a Foundation for Interprofessional Education Within Nursing and Medical Curricula
Trisha Leann Horsley, Trent Reed, Keith Muccino, Donna Quinones, Viva Jo Siddall, Janet McCarthy
Nurse Educator.2016; 41(5): 234. CrossRef
Supervision in psychiatry
Joanna MacDonald, Pete M. Ellis
Current Opinion in Psychiatry.2012; 25(4): 322. CrossRef

Assessing the reliability and validity of the Revised Two Factor Study Process Questionnaire (R-SPQ2F) in Ghanaian medical students: Victor Mogre, Anthony Amalba; J Educ Eval Health Prof. 2014;11:19. Published online August 15, 2014; DOI: https://doi.org/10.3352/jeehp.2014.11.19

27,011 View
210 Download
8 Web of Science
10 Crossref

Abstract PDF

Purpose
We investigated the validity and reliability of the Revised Two Factor Study Process Questionnaire (R-SPQ2F) in preclinical students in Ghana. Methods: The R-SPQ2F was administered to 189 preclinical students of the University for Development Studies, School of Medicine and Health Sciences. Both descriptive and inferential statistics with Cronbach’s alpha test and factor analysis were done. Results: The mean age of the students was 22.69 ± 0.18 years, 60.8% (n = 115) were males and 42.3% (n = 80) were in their second year of medical training. The students had higher mean deep approach scores (31.23 ± 7.19) than that of surface approach scores (22.62 ± 6.48). Findings of the R-SPQ2F gave credence to a solution of two-factors indicating deep and surface approaches accounting for 49.80% and 33.57%, respectively, of the variance. The scales of deep approach (Cronbach’s alpha, 0.80) and surface approach (Cronbach’s alpha, 0.76) and their subscales demonstrated an internal consistency that was good. The factorial validity was comparable to other studies. Conclusion: Our study confirms the construct validity and internal consistency of the R-SPQ2F for measuring approaches to learning in Ghanaian preclinical students. Deep approach was the most dominant learning approach among the students. The questionnaire can be used to measure students’ approaches to learning in Ghana and in other African countries.

Citations

Citations to this article as recorded by

A comparison of two learning approach inventories and their utility in predicting examination performance and study habits
Andrew R. Thompson
Advances in Physiology Education.2024; 48(2): 164. CrossRef
Relationship between learning approach, Bloom’s taxonomy, and student performance in an undergraduate Human Anatomy course
Andrew R. Thompson, Logan P. O. Lake
Advances in Health Sciences Education.2023; 28(4): 1115. CrossRef
Evaluation of Construct Validity and Reliability of the Arabic and English Versions of Biggs Study Process Scale Among Saudi University Students
Nadeem Shafique Butt, Muhammad Abid Bashir, Sami Hamdan Alzahrani, Zohair Jamil Gazzaz, Ahmad Azam Malik
SAGE Open.2023;[Epub] CrossRef
Development and Preliminary Validation of the Physical Education-Study Process Questionnaire : Insights for Physical Education University Students
Amayra Tannoubi, Noomen Guelmami, Tore Bonsaksen, Nasr Chalghaf, Fairouz Azaiez, Nicola Luigi Bragazzi
Frontiers in Public Health.2022;[Epub] CrossRef
The Association Between Learning Styles, Time Management Skills and Pharmacology Academic Performance Among First Year Medical Students in Universiti Putra Malaysia (UPM) During the COVID-19 Pandemic
Azmah Sa'at, Suryati Mohd. Thani, Safuraa Salihan, Nur Izah Ab. Razak, Siti Saleha Masrudin
Malaysian Journal of Medicine and Health Sciences.2022; 18(s14): 94. CrossRef
Outcomes of 2 Multimodal Human Anatomy Courses Among Doctor of Physical Therapy Students (Entry-Level): A Quasi-experimental Study
Sara F. Maher, Deborah J. Doherty
Journal of Physical Therapy Education.2021; 35(1): 38. CrossRef
Study Approaches of Life Science Students Using the Revised Two-Factor Study Process Questionnaire (R-SPQ-2F)
Miguel Leiva-Brondo, Jaime Cebolla-Cornejo, Rosa Peiró, Nuria Andrés-Colás, Cristina Esteras, María Ferriol, Hugo Merle, María José Díez, Ana Pérez-de-Castro
Education Sciences.2020; 10(7): 173. CrossRef
Assessing ‘approaches to learning’ in Botswana, Ghana and Kenya
Caine Rolleston, Rebecca Schendel, Ana M Grijalva Espinosa
Research in Comparative and International Education.2019; 14(1): 118. CrossRef
Psychometric properties of the revised two-factor study process questionnaire r-spq-2f - spanish version
Clara Vergara-Hernández, Miguel Simancas-Pallares, Zoila Carbonell-Muñoz
Duazary.2019; 16(2): 205. CrossRef
Enfoques de Aprendizaje según el R-SPQ-2F: Análisis de sus propiedades psicométricas en estudiantes universitarios de Buenos Aires
Agustín Freiberg Hoffmann, María Mercedes Fernández Liporace
Revista Colombiana de Psicología.2016;[Epub] CrossRef

First
Prev
Page of 1
Next
Last

Search