Skip Navigation
Skip to contents

JEEHP : Journal of Educational Evaluation for Health Professions

OPEN ACCESS
SEARCH
Search

Author index

Page Path
HOME > Browse articles > Author index
Search
Martha Ainslie 1 Article
Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions  
Jimmy Bourque, Haley Skinner, Jonathan Dupré, Maria Bacchus, Martha Ainslie, Irene W. Y. Ma, Gary Cole
J Educ Eval Health Prof. 2020;17:12.   Published online April 20, 2020
DOI: https://doi.org/10.3352/jeehp.2020.17.12
  • 7,029 View
  • 175 Download
  • 8 Web of Science
  • 7 Crossref
AbstractAbstract PDFSupplementary Material
Purpose
This study aimed to assess the performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions. Specifically, the following parameters were evaluated: inter-rater agreement, the correlations between Ebel scores and item facility indices, the impact of raters’ knowledge of correct answers on the Ebel score, and the effects of raters’ specialty on inter-rater agreement and Ebel scores.
Methods
Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. The Ebel method was applied to 203 multiple-choice questions by 49 raters. Facility indices came from 194 candidates. We computed the Fleiss kappa and the Pearson correlations between Ebel scores and item facility indices. We investigated differences in the Ebel score according to whether correct answers were provided or not and differences between internists and other specialists using the t-test.
Results
The Fleiss kappa was below 0.15 for both facility and relevance. The correlation between Ebel scores and facility indices was low when correct answers were provided and negligible when they were not. The Ebel score was the same whether the correct answers were provided or not. Inter-rater agreement and Ebel scores were not significantly different between internists and other specialists.
Conclusion
Inter-rater agreement and correlations between item Ebel scores and facility indices were consistently low; furthermore, raters’ knowledge of the correct answers and raters’ specialty had no effect on Ebel scores in the present setting.

Citations

Citations to this article as recorded by  
  • Competency Standard Derivation for Point-of-Care Ultrasound Image Interpretation for Emergency Physicians
    Maya Harel-Sterling, Charisse Kwan, Jonathan Pirie, Mark Tessaro, Dennis D. Cho, Ailish Coblentz, Mohamad Halabi, Eyal Cohen, Lynne E. Nield, Martin Pusic, Kathy Boutis
    Annals of Emergency Medicine.2023; 81(4): 413.     CrossRef
  • The effects of a land-based home exercise program on surfing performance in recreational surfers
    Jerry-Thomas Monaco, Richard Boergers, Thomas Cappaert, Michael Miller, Jennifer Nelson, Meghan Schoenberger
    Journal of Sports Sciences.2023; 41(4): 358.     CrossRef
  • Medical specialty certification exams studied according to the Ottawa Quality Criteria: a systematic review
    Daniel Staudenmann, Noemi Waldner, Andrea Lörwald, Sören Huwendiek
    BMC Medical Education.2023;[Epub]     CrossRef
  • A Target Population Derived Method for Developing a Competency Standard in Radiograph Interpretation
    Michelle S. Lee, Martin V. Pusic, Mark Camp, Jennifer Stimec, Andrew Dixon, Benoit Carrière, Joshua E. Herman, Kathy Boutis
    Teaching and Learning in Medicine.2022; 34(2): 167.     CrossRef
  • Pediatric Musculoskeletal Radiographs: Anatomy and Fractures Prone to Diagnostic Error Among Emergency Physicians
    Winny Li, Jennifer Stimec, Mark Camp, Martin Pusic, Joshua Herman, Kathy Boutis
    The Journal of Emergency Medicine.2022; 62(4): 524.     CrossRef
  • Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study
    Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park
    Journal of Educational Evaluation for Health Professions.2022; 19: 33.     CrossRef
  • Image interpretation: Learning analytics–informed education opportunities
    Elana Thau, Manuela Perez, Martin V. Pusic, Martin Pecaric, David Rizzuti, Kathy Boutis
    AEM Education and Training.2021;[Epub]     CrossRef

JEEHP : Journal of Educational Evaluation for Health Professions
TOP