Skip Navigation
Skip to contents

JEEHP : Journal of Educational Evaluation for Health Professions

OPEN ACCESS
SEARCH
Search

Search

Page Path
HOME > Search
13 "Reproducibility of results"
Filter
Filter
Article category
Keywords
Publication year
Authors
Funded articles
Research article
Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study  
Ariel Shana Frey-Vogel, Kristina Dzara, Kimberly Anne Gifford, Yoon Soo Park, Justin Berk, Allison Heinly, Darcy Wolcott, Daniel Adam Hall, Shannon Elliott Scott-Vernaglia, Katherine Anne Sparger, Erica Ye-pyng Chung
J Educ Eval Health Prof. 2024;21:3.   Published online February 23, 2024
DOI: https://doi.org/10.3352/jeehp.2024.21.3
  • 615 View
  • 157 Download
AbstractAbstract PDFSupplementary Material
Purpose
Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into 6 elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure.
Methods
Messick’s unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from 3 pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018–2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables).
Results
Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone (r=0.34, P=0.019).
Conclusion
Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.
Review
Application of artificial intelligence chatbots, including ChatGPT, in education, scholarly work, programming, and content generation and its prospects: a narrative review
Tae Won Kim
J Educ Eval Health Prof. 2023;20:38.   Published online December 27, 2023
DOI: https://doi.org/10.3352/jeehp.2023.20.38
  • 2,577 View
  • 393 Download
  • 3 Web of Science
  • 4 Crossref
AbstractAbstract PDFSupplementary Material
This study aims to explore ChatGPT’s (GPT-3.5 version) functionalities, including reinforcement learning, diverse applications, and limitations. ChatGPT is an artificial intelligence (AI) chatbot powered by OpenAI’s Generative Pre-trained Transformer (GPT) model. The chatbot’s applications span education, programming, content generation, and more, demonstrating its versatility. ChatGPT can improve education by creating assignments and offering personalized feedback, as shown by its notable performance in medical exams and the United States Medical Licensing Exam. However, concerns include plagiarism, reliability, and educational disparities. It aids in various research tasks, from design to writing, and has shown proficiency in summarizing and suggesting titles. Its use in scientific writing and language translation is promising, but professional oversight is needed for accuracy and originality. It assists in programming tasks like writing code, debugging, and guiding installation and updates. It offers diverse applications, from cheering up individuals to generating creative content like essays, news articles, and business plans. Unlike search engines, ChatGPT provides interactive, generative responses and understands context, making it more akin to human conversation, in contrast to conventional search engines’ keyword-based, non-interactive nature. ChatGPT has limitations, such as potential bias, dependence on outdated data, and revenue generation challenges. Nonetheless, ChatGPT is considered to be a transformative AI tool poised to redefine the future of generative technology. In conclusion, advancements in AI, such as ChatGPT, are altering how knowledge is acquired and applied, marking a shift from search engines to creativity engines. This transformation highlights the increasing importance of AI literacy and the ability to effectively utilize AI in various domains of life.

Citations

Citations to this article as recorded by  
  • Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review
    Xiaojun Xu, Yixiao Chen, Jing Miao
    Journal of Educational Evaluation for Health Professions.2024; 21: 6.     CrossRef
  • Artificial Intelligence: Fundamentals and Breakthrough Applications in Epilepsy
    Wesley Kerr, Sandra Acosta, Patrick Kwan, Gregory Worrell, Mohamad A. Mikati
    Epilepsy Currents.2024;[Epub]     CrossRef
  • A Developed Graphical User Interface-Based on Different Generative Pre-trained Transformers Models
    Ekrem Küçük, İpek Balıkçı Çiçek, Zeynep Küçükakçalı, Cihan Yetiş, Cemil Çolak
    ODÜ Tıp Dergisi.2024; 11(1): 18.     CrossRef
  • Art or Artifact: Evaluating the Accuracy, Appeal, and Educational Value of AI-Generated Imagery in DALL·E 3 for Illustrating Congenital Heart Diseases
    Mohamad-Hani Temsah, Abdullah N. Alhuzaimi, Mohammed Almansour, Fadi Aljamaan, Khalid Alhasan, Munirah A. Batarfi, Ibraheem Altamimi, Amani Alharbi, Adel Abdulaziz Alsuhaibani, Leena Alwakeel, Abdulrahman Abdulkhaliq Alzahrani, Khaled B. Alsulaim, Amr Jam
    Journal of Medical Systems.2024;[Epub]     CrossRef
Technical report
Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2022, Korea: a psychometric study
Yoon Hee Kim, Bo Hyun Kim, Joonki Kim, Bokyoung Jung, Sangyoung Bae
J Educ Eval Health Prof. 2023;20:31.   Published online November 22, 2023
DOI: https://doi.org/10.3352/jeehp.2023.20.31
  • 848 View
  • 73 Download
AbstractAbstract PDFSupplementary Material
Purpose
This study presents item analysis results of the 26 health personnel licensing examinations managed by the Korea Health Personnel Licensing Examination Institute (KHPLEI) in 2022.
Methods
The item difficulty index, item discrimination index, and reliability were calculated. The item discrimination index was calculated using a discrimination index based on the upper and lower 27% rule and the item-total correlation.
Results
Out of 468,352 total examinees, 418,887 (89.4%) passed. The pass rates ranged from 27.3% for health educators level 1 to 97.1% for oriental medical doctors. Most examinations had a high average difficulty index, albeit to varying degrees, ranging from 61.3% for prosthetists and orthotists to 83.9% for care workers. The average discrimination index based on the upper and lower 27% rule ranged from 0.17 for oriental medical doctors to 0.38 for radiological technologists. The average item-total correlation ranged from 0.20 for oriental medical doctors to 0.38 for radiological technologists. The Cronbach α, as a measure of reliability, ranged from 0.872 for health educators-level 3 to 0.978 for medical technologists. The correlation coefficient between the average difficulty index and average discrimination index was -0.2452 (P=0.1557), that between the average difficulty index and the average item-total correlation was 0.3502 (P=0.0392), and that between the average discrimination index and the average item-total correlation was 0.7944 (P<0.0001).
Conclusion
This technical report presents the item analysis results and reliability of the recent examinations by the KHPLEI, demonstrating an acceptable range of difficulty index and discrimination index values, as well as good reliability.
Research articles
Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study  
Aleksandra Ignjatović, Lazar Stevanović
J Educ Eval Health Prof. 2023;20:28.   Published online October 16, 2023
DOI: https://doi.org/10.3352/jeehp.2023.20.28
  • 2,183 View
  • 173 Download
  • 3 Web of Science
  • 6 Crossref
AbstractAbstract PDFSupplementary Material
Purpose
This study aimed to assess the performance of ChatGPT (GPT-3.5 and GPT-4) as a study tool in solving biostatistical problems and to identify any potential drawbacks that might arise from using ChatGPT in medical education, particularly in solving practical biostatistical problems.
Methods
ChatGPT was tested to evaluate its ability to solve biostatistical problems from the Handbook of Medical Statistics by Peacock and Peacock in this descriptive study. Tables from the problems were transformed into textual questions. Ten biostatistical problems were randomly chosen and used as text-based input for conversation with ChatGPT (versions 3.5 and 4).
Results
GPT-3.5 solved 5 practical problems in the first attempt, related to categorical data, cross-sectional study, measuring reliability, probability properties, and the t-test. GPT-3.5 failed to provide correct answers regarding analysis of variance, the chi-square test, and sample size within 3 attempts. GPT-4 also solved a task related to the confidence interval in the first attempt and solved all questions within 3 attempts, with precise guidance and monitoring.
Conclusion
The assessment of both versions of ChatGPT performance in 10 biostatistical problems revealed that GPT-3.5 and 4’s performance was below average, with correct response rates of 5 and 6 out of 10 on the first attempt. GPT-4 succeeded in providing all correct answers within 3 attempts. These findings indicate that students must be aware that this tool, even when providing and calculating different statistical analyses, can be wrong, and they should be aware of ChatGPT’s limitations and be careful when incorporating this model into medical education.

Citations

Citations to this article as recorded by  
  • Can Generative AI and ChatGPT Outperform Humans on Cognitive-Demanding Problem-Solving Tasks in Science?
    Xiaoming Zhai, Matthew Nyaaba, Wenchao Ma
    Science & Education.2024;[Epub]     CrossRef
  • Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review
    Xiaojun Xu, Yixiao Chen, Jing Miao
    Journal of Educational Evaluation for Health Professions.2024; 21: 6.     CrossRef
  • Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy
    Ambadasu Bharatha, Nkemcho Ojeh, Ahbab Mohammad Fazle Rabbi, Michael Campbell, Kandamaran Krishnamurthy, Rhaheem Layne-Yarde, Alok Kumar, Dale Springer, Kenneth Connell, Md Anwarul Majumder
    Advances in Medical Education and Practice.2024; Volume 15: 393.     CrossRef
  • Revolutionizing Cardiology with Words: Unveiling the Impact of Large Language Models in Medical Science Writing
    Abhijit Bhattaru, Naveena Yanamala, Partho P. Sengupta
    Canadian Journal of Cardiology.2024;[Epub]     CrossRef
  • ChatGPT in medicine: prospects and challenges: a review article
    Songtao Tan, Xin Xin, Di Wu
    International Journal of Surgery.2024; 110(6): 3701.     CrossRef
  • In-depth analysis of ChatGPT’s performance based on specific signaling words and phrases in the question stem of 2377 USMLE step 1 style questions
    Leonard Knoedler, Samuel Knoedler, Cosima C. Hoch, Lukas Prantl, Konstantin Frank, Laura Soiderer, Sebastian Cotofana, Amir H. Dorafshar, Thilo Schenck, Felix Vollbach, Giuseppe Sofo, Michael Alfertshofer
    Scientific Reports.2024;[Epub]     CrossRef
Development and validation of the student ratings in clinical teaching scale in Australia: a methodological study  
Pin-Hsiang Huang, Anthony John O’Sullivan, Boaz Shulruf
J Educ Eval Health Prof. 2023;20:26.   Published online September 5, 2023
DOI: https://doi.org/10.3352/jeehp.2023.20.26
  • 1,162 View
  • 120 Download
AbstractAbstract PDFSupplementary Material
Purpose
This study aimed to devise a valid measurement for assessing clinical students’ perceptions of teaching practices.
Methods
A new tool was developed based on a meta-analysis encompassing effective clinical teaching-learning factors. Seventy-nine items were generated using a frequency (never to always) scale. The tool was applied to the University of New South Wales year 2, 3, and 6 medical students. Exploratory and confirmatory factor analysis (exploratory factor analysis [EFA] and confirmatory factor analysis [CFA], respectively) were conducted to establish the tool’s construct validity and goodness of fit, and Cronbach’s α was used for reliability.
Results
In total, 352 students (44.2%) completed the questionnaire. The EFA identified student-centered learning, problem-solving learning, self-directed learning, and visual technology (reliability, 0.77 to 0.89). CFA showed acceptable goodness of fit (chi-square P<0.01, comparative fit index=0.930 and Tucker-Lewis index=0.917, root mean square error of approximation=0.069, standardized root mean square residual=0.06).
Conclusion
The established tool—Student Ratings in Clinical Teaching (STRICT)—is a valid and reliable tool that demonstrates how students perceive clinical teaching efficacy. STRICT measures the frequency of teaching practices to mitigate the biases of acquiescence and social desirability. Clinical teachers may use the tool to adapt their teaching practices with more active learning activities and to utilize visual technology to facilitate clinical learning efficacy. Clinical educators may apply STRICT to assess how these teaching practices are implemented in current clinical settings.
What impacts students’ satisfaction the most from Medicine Student Experience Questionnaire in Australia: a validity study  
Pin-Hsiang Huang, Gary Velan, Greg Smith, Melanie Fentoullis, Sean Edward Kennedy, Karen Jane Gibson, Kerry Uebel, Boaz Shulruf
J Educ Eval Health Prof. 2023;20:2.   Published online January 18, 2023
DOI: https://doi.org/10.3352/jeehp.2023.20.2
  • 1,698 View
  • 131 Download
  • 1 Web of Science
  • 1 Crossref
AbstractAbstract PDFSupplementary Material
Purpose
This study evaluated the validity of student feedback derived from Medicine Student Experience Questionnaire (MedSEQ), as well as the predictors of students’ satisfaction in the Medicine program.
Methods
Data from MedSEQ applying to the University of New South Wales Medicine program in 2017, 2019, and 2021 were analyzed. Confirmatory factor analysis (CFA) and Cronbach’s α were used to assess the construct validity and reliability of MedSEQ respectively. Hierarchical multiple linear regressions were used to identify the factors that most impact students’ overall satisfaction with the program.
Results
A total of 1,719 students (34.50%) responded to MedSEQ. CFA showed good fit indices (root mean square error of approximation=0.051; comparative fit index=0.939; chi-square/degrees of freedom=6.429). All factors yielded good (α>0.7) or very good (α>0.8) levels of reliability, except the “online resources” factor, which had acceptable reliability (α=0.687). A multiple linear regression model with only demographic characteristics explained 3.8% of the variance in students’ overall satisfaction, whereas the model adding 8 domains from MedSEQ explained 40%, indicating that 36.2% of the variance was attributable to students’ experience across the 8 domains. Three domains had the strongest impact on overall satisfaction: “being cared for,” “satisfaction with teaching,” and “satisfaction with assessment” (β=0.327, 0.148, 0.148, respectively; all with P<0.001).
Conclusion
MedSEQ has good construct validity and high reliability, reflecting students’ satisfaction with the Medicine program. Key factors impacting students’ satisfaction are the perception of being cared for, quality teaching irrespective of the mode of delivery and fair assessment tasks which enhance learning.

Citations

Citations to this article as recorded by  
  • Mental health and quality of life across 6 years of medical training: A year-by-year analysis
    Natalia de Castro Pecci Maddalena, Alessandra Lamas Granero Lucchetti, Ivana Lucia Damasio Moutinho, Oscarina da Silva Ezequiel, Giancarlo Lucchetti
    International Journal of Social Psychiatry.2023;[Epub]     CrossRef
Acceptability of the 8-case objective structured clinical examination of medical students in Korea using generalizability theory: a reliability study  
Song Yi Park, Sang-Hwa Lee, Min-Jeong Kim, Ki-Hwan Ji, Ji Ho Ryu
J Educ Eval Health Prof. 2022;19:26.   Published online September 8, 2022
DOI: https://doi.org/10.3352/jeehp.2022.19.26
  • 2,385 View
  • 211 Download
  • 1 Web of Science
  • 1 Crossref
AbstractAbstract PDFSupplementary Material
Purpose
This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT).
Methods
This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada).
Results
All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%.
Conclusion
The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.

Citations

Citations to this article as recorded by  
  • Applying the Generalizability Theory to Identify the Sources of Validity Evidence for the Quality of Communication Questionnaire
    Flávia Del Castanhel, Fernanda R. Fonseca, Luciana Bonnassis Burg, Leonardo Maia Nogueira, Getúlio Rodrigues de Oliveira Filho, Suely Grosseman
    American Journal of Hospice and Palliative Medicine®.2024; 41(7): 792.     CrossRef
Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study  
Janghee Park
J Educ Eval Health Prof. 2022;19:23.   Published online August 31, 2022
DOI: https://doi.org/10.3352/jeehp.2022.19.23
  • 2,723 View
  • 168 Download
  • 2 Web of Science
  • 2 Crossref
AbstractAbstract PDFSupplementary Material
Purpose
The percent Angoff (PA) method has been recommended as a reliable method to set the cutoff score instead of a fixed cut point of 60% in the Korean Medical Licensing Examination (KMLE). The yes/no Angoff (YNA) method, which is easy for panelists to judge, can be considered as an alternative because the KMLE has many items to evaluate. This study aimed to compare the cutoff score and the reliability depending on whether the PA or the YNA standard-setting method was used in the KMLE.
Methods
The materials were the open-access PA data of the KMLE. The PA data were converted to YNA data in 5 categories, in which the probabilities for a “yes” decision by panelists were 50%, 60%, 70%, 80%, and 90%. SPSS for descriptive analysis and G-string for generalizability theory were used to present the results.
Results
The PA method and the YNA method counting 60% as “yes,” estimated similar cutoff scores. Those cutoff scores were deemed acceptable based on the results of the Hofstee method. The highest reliability coefficients estimated by the generalizability test were from the PA method and the YNA method, with probabilities of 70%, 80%, 60%, and 50% for deciding “yes,” in descending order. The panelist’s specialty was the main cause of the error variance. The error size was similar regardless of the standard-setting method.
Conclusion
The above results showed that the PA method was more reliable than the YNA method in estimating the cutoff score of the KMLE. However, the YNA method with a 60% probability for deciding “yes” also can be used as a substitute for the PA method in estimating the cutoff score of the KMLE.

Citations

Citations to this article as recorded by  
  • Issues in the 3rd year of the COVID-19 pandemic, including computer-based testing, study design, ChatGPT, journal metrics, and appreciation to reviewers
    Sun Huh
    Journal of Educational Evaluation for Health Professions.2023; 20: 5.     CrossRef
  • Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study
    Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park
    Journal of Educational Evaluation for Health Professions.2022; 19: 33.     CrossRef
Development and validation of a measurement scale to assess nursing students’ readiness for the flipped classroom in Sri Lanka  
Punithalingam Youhasan, Yan Chen, Mataroria Lyndon, Marcus Alexander Henning
J Educ Eval Health Prof. 2020;17:41.   Published online December 14, 2020
DOI: https://doi.org/10.3352/jeehp.2020.17.41
  • 6,466 View
  • 254 Download
  • 7 Web of Science
  • 6 Crossref
AbstractAbstract PDFSupplementary Material
Purpose
The aim of this study was to develop and validate a scale to measure nursing students’ readiness for the flipped classroom in Sri Lanka.
Methods
A literature review provided the theoretical framework for developing the Nursing Students’ Readiness for Flipped Classroom (NSR-FC) questionnaire. Five content experts evaluated the NSR-FC, and content validity indices (CVI) were calculated. Cross-sectional surveys among 355 undergraduate nursing students from 3 state universities in Sri Lanka were carried out to assess the psychometric properties of the NSR-FC. Principal component analysis (PCA, n=265), internal consistency (using the Cronbach α coefficient, n=265), and confirmatory factor analysis (CFA, n=90) were done to test construct validity and reliability.
Results
Thirty-seven items were included in the NSR-FC for content validation, resulting in an average scale CVI of 0.94. Two items received item level CVI of less than 0.78. The factor structures of the 35 items were explored through PCA with orthogonal factor rotation, culminating in the identification of 5 factors. These factors were classified as technological readiness, environmental readiness, personal readiness, pedagogical readiness, and interpersonal readiness. The NSR-FC also showed an overall acceptable level of internal consistency (Cronbach α=0.9). CFA verified a 4-factor model (excluding the interpersonal readiness factor) and 20 items that achieved acceptable fit (standardized root mean square residual=0.08, root mean square error of approximation=0.08, comparative fit index=0.87, and χ2/degrees of freedom=1.57).
Conclusion
The NSR-FC, as a 4-factor model, is an acceptable measurement scale for assessing nursing students’ readiness for the flipped classroom in terms of its construct validity and reliability.

Citations

Citations to this article as recorded by  
  • Content validity of the Constructivist Learning in Higher Education Settings (CLHES) scale in the context of the flipped classroom in higher education
    Turki Mesfer Alqahtani, Farrah Dina Yusop, Siti Hajar Halili
    Humanities and Social Sciences Communications.2023;[Epub]     CrossRef
  • The intensivist's assessment of gastrointestinal function: A pilot study
    Varsha M. Asrani, Colin McArthur, Ian Bissett, John A. Windsor
    Australian Critical Care.2022; 35(6): 636.     CrossRef
  • Psychometric evidence of a perception scale about covid-19 vaccination process in Peruvian dentists: a preliminary validation
    César F. Cayo-Rojas, Nancy Córdova-Limaylla, Gissela Briceño-Vergel, Marysela Ladera-Castañeda, Hernán Cachay-Criado, Carlos López-Gurreonero, Alberto Cornejo-Pinto, Luis Cervantes-Ganoza
    BMC Health Services Research.2022;[Epub]     CrossRef
  • Implementation of a Web-Based Educational Intervention for Promoting Flipped Classroom Pedagogy: A Mixed-Methods Study
    Punithalingam Youhasan, Mataroria P. Lyndon, Yan Chen, Marcus A. Henning
    Medical Science Educator.2022; 33(1): 91.     CrossRef
  • Assess the feasibility of flipped classroom pedagogy in undergraduate nursing education in Sri Lanka: A mixed-methods study
    Punithalingam Youhasan, Yan Chen, Mataroria Lyndon, Marcus A. Henning, Gwo-Jen Hwang
    PLOS ONE.2021; 16(11): e0259003.     CrossRef
  • Newly appointed medical faculty members’ self-evaluation of their educational roles at the Catholic University of Korea College of Medicine in 2020 and 2021: a cross-sectional survey-based study
    Sun Kim, A Ra Cho, Chul Woon Chung
    Journal of Educational Evaluation for Health Professions.2021; 18: 28.     CrossRef
Development and validation of the Hocus Focus Magic Performance Evaluation Scale for health professions personnel in the United States  
Kevin Spencer, Hon Keung Yuen, Max Darwin, Gavin Jenkins, Kimberly Kirklin
J Educ Eval Health Prof. 2019;16:8.   Published online April 10, 2019
DOI: https://doi.org/10.3352/jeehp.2019.16.8
  • 18,465 View
  • 229 Download
  • 4 Web of Science
  • 4 Crossref
AbstractAbstract PDFSupplementary Material
Purpose
This study was conducted to describe the development and validation of the Hocus Focus Magic Performance Evaluation Scale (HFMPES), which is used to evaluate the competency of health professions personnel in delivering magic tricks as a therapeutic modality.
Methods
A 2-phase validation process was used. Phase I (content validation) involved 16 magician judges who independently rated the relevance of each of the 5 items in the HFMPES and established the veracity of its content. Phase II evaluated the psychometric properties of the HFMPES. This process involved 2 magicians using the HFMPES to independently evaluate 73 occupational therapy graduate students demonstrating 3 magic tricks.
Results
The HFMPES achieved an excellent scale-content validity index of 0.99. Exploratory factor analysis of the HFMPES scores revealed 1 distinct factor with alpha coefficients ≥0.8 across the 3 magic tricks. The construct validity of the HFMPES scores was further supported by evidence from a known-groups analysis, in which the Mann–Whitney U-test showed significant difference in HFMPES scores between participants with different levels of experience in delivering the 3 magic tricks. The inter-rater reliability coefficients were ≥0.75 across the 3 magic tricks, indicating that the competency of health professions personnel in delivering the 3 magic tricks could be evaluated precisely.
Conclusion
Preliminary evidence supported the content and construct validity of the HFMPES, which was found to have good internal consistency and inter-rater reliability in evaluating health professions personnel’s competency in delivering magic tricks.

Citations

Citations to this article as recorded by  
  • A Magic Trick Training Program to Improve Social Skills and Self-Esteem in Adolescents With Autism Spectrum Disorder
    Hon K. Yuen, Kevin Spencer, Lauren Edwards, Kimberly Kirklin, Gavin R. Jenkins
    The American Journal of Occupational Therapy.2023;[Epub]     CrossRef
  • MAGNITIVE: Effectiveness and Feasibility of a Cognitive Training Program Through Magic Tricks for Children With Attention Deficit and Hyperactivity Disorder. A Second Clinical Trial in Community Settings
    Saray Bonete, Ángela Osuna, Clara Molinero, Inmaculada García-Font
    Frontiers in Psychology.2021;[Epub]     CrossRef
  • Contribution of a virtual magic camp to enhancing self-esteem in children with ADHD: A pilot study
    Hon K. Yuen, Kevin Spencer, Kimberly Kirklin, Lauren Edwards, Gavin R. Jenkins
    Health Psychology Research.2021;[Epub]     CrossRef
  • Evaluation of a Magic Camp for Children with Hemiparesis: A Pilot Study
    Kevin Spencer, Hon K. Yuen, Gavin R. Jenkins, Kimberly Kirklin, Angla R. Griffin, Laura K. Vogtle, Drew Davis
    Occupational Therapy In Health Care.2020; 34(2): 155.     CrossRef
Research Articles
Psychometric properties of a novel knowledge assessment tool of mechanical ventilation for emergency medicine residents in the northeastern United States  
Jeremy B. Richards, Tania D. Strout, Todd A. Seigel, Susan R. Wilcox
J Educ Eval Health Prof. 2016;13:10.   Published online February 16, 2016
DOI: https://doi.org/10.3352/jeehp.2016.13.10
  • 27,832 View
  • 175 Download
  • 4 Web of Science
  • 4 Crossref
AbstractAbstract PDF
Purpose
Prior descriptions of the psychometric properties of validated knowledge assessment tools designed to determine Emergency medicine (EM) residents understanding of physiologic and clinical concepts related to mechanical ventilation are lacking. In this setting, we have performed this study to describe the psychometric and performance properties of a novel knowledge assessment tool that measures EM residents’ knowledge of topics in mechanical ventilation.
Methods
Results from a multicenter, prospective, survey study involving 219 EM residents from 8 academic hospitals in northeastern United States were analyzed to quantify reliability, item difficulty, and item discrimination of each of the 9 questions included in the knowledge assessment tool for 3 weeks, beginning in January 2013.
Results
The response rate for residents completing the knowledge assessment tool was 68.6% (214 out of 312 EM residents). Reliability was assessed by both Cronbach’s alpha coefficient (0.6293) and the Spearman-Brown coefficient (0.6437). Item difficulty ranged from 0.39 to 0.96, with a mean item difficulty of 0.75 for all 9 questions. Uncorrected item discrimination values ranged from 0.111 to 0.556. Corrected item-total correlations were determined by removing the question being assessed from analysis, resulting in a range of item discrimination from 0.139 to 0.498.
Conclusion
Reliability, item difficulty and item discrimination were within satisfactory ranges in this study, demonstrating acceptable psychometric properties of this knowledge assessment tool. This assessment indicates that this knowledge assessment tool is sufficiently rigorous for use in future research studies or for assessment of EM residents for evaluative purposes.

Citations

Citations to this article as recorded by  
  • Comparison of three methods for teaching mechanical ventilation in an emergency setting to sixth-year medical students: a randomized trial
    Fernando Sabia Tallo, Letícia Sandre Vendrame, André Luciano Baitello
    Revista da Associação Médica Brasileira.2020; 66(10): 1409.     CrossRef
  • Critical Appraisal of Emergency Medicine Educational Research: The Best Publications of 2016
    Nicole M. Dubosh, Jaime Jordan, Lalena M. Yarris, Edward Ullman, Joshua Kornegay, Daniel Runde, Amy Miller Juve, Jonathan Fisher, Teresa Chan
    AEM Education and Training.2019; 3(1): 58.     CrossRef
  • Mechanical Ventilation Training During Graduate Medical Education: Perspectives and Review of the Literature
    Jonathan M. Keller, Dru Claar, Juliana Carvalho Ferreira, David C. Chu, Tanzib Hossain, William Graham Carlos, Jeffrey A. Gold, Stephanie A. Nonas, Nitin Seam
    Journal of Graduate Medical Education.2019; 11(4): 389.     CrossRef
  • Development and validation of a questionnaire to assess the knowledge of mechanical ventilation in urgent care among students in their last-year medical course in Brazil
    Fernando Sabia Tallo, Simone de Campos Vieira Abib, Andre Luciano Baitello, Renato Delascio Lopes
    Clinics.2019; 74: e663.     CrossRef
Validation of a clinical critical thinking skills test in nursing  
Sujin Shin, Dukyoo Jung, Sungeun Kim
J Educ Eval Health Prof. 2015;12:1.   Published online January 27, 2015
DOI: https://doi.org/10.3352/jeehp.2015.12.1
  • 35,348 View
  • 287 Download
  • 5 Web of Science
  • 9 Crossref
AbstractAbstract PDF
Purpose
The purpose of this study was to develop a revised version of the clinical critical thinking skills test (CCTS) and to subsequently validate its performance. Methods: This study is a secondary analysis of the CCTS. Data were obtained from a convenience sample of 284 college students in June 2011. Thirty items were analyzed using item response theory and test reliability was assessed. Test-retest reliability was measured using the results of 20 nursing college and graduate school students in July 2013. The content validity of the revised items was analyzed by calculating the degree of agreement between instrument developer intention in item development and the judgments of six experts. To analyze response process validity, qualitative data related to the response processes of nine nursing college students obtained through cognitive interviews were analyzed. Results: Out of initial 30 items, 11 items were excluded after the analysis of difficulty and discrimination parameter. When the 19 items of the revised version of the CCTS were analyzed, levels of item difficulty were found to be relatively low and levels of discrimination were found to be appropriate or high. The degree of agreement between item developer intention and expert judgments equaled or exceeded 50%. Conclusion: From above results, evidence of the response process validity was demonstrated, indicating that subjects respondeds as intended by the test developer. The revised 19-item CCTS was found to have sufficient reliability and validity and will therefore represents a more convenient measurement of critical thinking ability.

Citations

Citations to this article as recorded by  
  • Meta-analysis in Physical Therapy Education Research
    Mitch Wolden, Brent Hill, Sara Farquhar Voorhees
    Journal of Physical Therapy Education.2019; 33(1): 78.     CrossRef
  • Effects of a work-based critical reflection program for novice nurses
    Yeon Hee Kim, Ja Min, Soon Hee Kim, Sujin Shin
    BMC Medical Education.2018;[Epub]     CrossRef
  • Measurement of critical thinking, clinical reasoning, and clinical judgment in culturally diverse nursing students – A literature review
    Christine L. Sommers
    Nurse Education in Practice.2018; 30: 91.     CrossRef
  • Individual and School Factors Affecting Critical Thinking Ability among Nursing Students
    Sujin Shin, Inhee Park, Eunhee Hwang, Dukyoo Jung, Kon Hee Kim
    Korean Medical Education Review.2018; 20(1): 44.     CrossRef
  • The Health Professions Education Pathway: Preparing Students, Residents, and Fellows to Become Future Educators
    H. Carrie Chen, Maria A. Wamsley, Amin Azzam, Katherine Julian, David M. Irby, Patricia S. O'Sullivan
    Teaching and Learning in Medicine.2017; 29(2): 216.     CrossRef
  • Cultivating Critical Thinking Using Virtual Interactive Case Studies
    Susan M. Burke
    Journal of Pediatric Nursing.2017; 33: 94.     CrossRef
  • Encouraging Critical Clinical Thinking (CCT) Skills in First-Year Veterinary Students
    Duncan C. Ferguson, Leslie Klis McNeil, David J. Schaeffe, Eric M. Mills
    Journal of Veterinary Medical Education.2017; 44(3): 531.     CrossRef
  • Developing a Foundation for Interprofessional Education Within Nursing and Medical Curricula
    Trisha Leann Horsley, Trent Reed, Keith Muccino, Donna Quinones, Viva Jo Siddall, Janet McCarthy
    Nurse Educator.2016; 41(5): 234.     CrossRef
  • Supervision in psychiatry
    Joanna MacDonald, Pete M. Ellis
    Current Opinion in Psychiatry.2012; 25(4): 322.     CrossRef
Assessing the reliability and validity of the Revised Two Factor Study Process Questionnaire (R-SPQ2F) in Ghanaian medical students  
Victor Mogre, Anthony Amalba
J Educ Eval Health Prof. 2014;11:19.   Published online August 15, 2014
DOI: https://doi.org/10.3352/jeehp.2014.11.19
  • 26,818 View
  • 207 Download
  • 8 Web of Science
  • 10 Crossref
AbstractAbstract PDF
Purpose
We investigated the validity and reliability of the Revised Two Factor Study Process Questionnaire (R-SPQ2F) in preclinical students in Ghana. Methods: The R-SPQ2F was administered to 189 preclinical students of the University for Development Studies, School of Medicine and Health Sciences. Both descriptive and inferential statistics with Cronbach’s alpha test and factor analysis were done. Results: The mean age of the students was 22.69 ± 0.18 years, 60.8% (n = 115) were males and 42.3% (n = 80) were in their second year of medical training. The students had higher mean deep approach scores (31.23 ± 7.19) than that of surface approach scores (22.62 ± 6.48). Findings of the R-SPQ2F gave credence to a solution of two-factors indicating deep and surface approaches accounting for 49.80% and 33.57%, respectively, of the variance. The scales of deep approach (Cronbach’s alpha, 0.80) and surface approach (Cronbach’s alpha, 0.76) and their subscales demonstrated an internal consistency that was good. The factorial validity was comparable to other studies. Conclusion: Our study confirms the construct validity and internal consistency of the R-SPQ2F for measuring approaches to learning in Ghanaian preclinical students. Deep approach was the most dominant learning approach among the students. The questionnaire can be used to measure students’ approaches to learning in Ghana and in other African countries.

Citations

Citations to this article as recorded by  
  • A comparison of two learning approach inventories and their utility in predicting examination performance and study habits
    Andrew R. Thompson
    Advances in Physiology Education.2024; 48(2): 164.     CrossRef
  • Relationship between learning approach, Bloom’s taxonomy, and student performance in an undergraduate Human Anatomy course
    Andrew R. Thompson, Logan P. O. Lake
    Advances in Health Sciences Education.2023; 28(4): 1115.     CrossRef
  • Evaluation of Construct Validity and Reliability of the Arabic and English Versions of Biggs Study Process Scale Among Saudi University Students
    Nadeem Shafique Butt, Muhammad Abid Bashir, Sami Hamdan Alzahrani, Zohair Jamil Gazzaz, Ahmad Azam Malik
    SAGE Open.2023;[Epub]     CrossRef
  • Development and Preliminary Validation of the Physical Education-Study Process Questionnaire : Insights for Physical Education University Students
    Amayra Tannoubi, Noomen Guelmami, Tore Bonsaksen, Nasr Chalghaf, Fairouz Azaiez, Nicola Luigi Bragazzi
    Frontiers in Public Health.2022;[Epub]     CrossRef
  • The Association Between Learning Styles, Time Management Skills and Pharmacology Academic Performance Among First Year Medical Students in Universiti Putra Malaysia (UPM) During the COVID-19 Pandemic
    Azmah Sa'at, Suryati Mohd. Thani, Safuraa Salihan, Nur Izah Ab. Razak, Siti Saleha Masrudin
    Malaysian Journal of Medicine and Health Sciences.2022; 18(s14): 94.     CrossRef
  • Outcomes of 2 Multimodal Human Anatomy Courses Among Doctor of Physical Therapy Students (Entry-Level): A Quasi-experimental Study
    Sara F. Maher, Deborah J. Doherty
    Journal of Physical Therapy Education.2021; 35(1): 38.     CrossRef
  • Study Approaches of Life Science Students Using the Revised Two-Factor Study Process Questionnaire (R-SPQ-2F)
    Miguel Leiva-Brondo, Jaime Cebolla-Cornejo, Rosa Peiró, Nuria Andrés-Colás, Cristina Esteras, María Ferriol, Hugo Merle, María José Díez, Ana Pérez-de-Castro
    Education Sciences.2020; 10(7): 173.     CrossRef
  • Assessing ‘approaches to learning’ in Botswana, Ghana and Kenya
    Caine Rolleston, Rebecca Schendel, Ana M Grijalva Espinosa
    Research in Comparative and International Education.2019; 14(1): 118.     CrossRef
  • Psychometric properties of the revised two-factor study process questionnaire r-spq-2f - spanish version
    Clara Vergara-Hernández, Miguel Simancas-Pallares, Zoila Carbonell-Muñoz
    Duazary.2019; 16(2): 205.     CrossRef
  • Enfoques de Aprendizaje según el R-SPQ-2F: Análisis de sus propiedades psicométricas en estudiantes universitarios de Buenos Aires
    Agustín Freiberg Hoffmann, María Mercedes Fernández Liporace
    Revista Colombiana de Psicología.2016;[Epub]     CrossRef

JEEHP : Journal of Educational Evaluation for Health Professions