Participants

This study was cross-sectional in nature. The participants were first, second, and third post-graduate year (PGY 1–3) emergency medicine residents of Geisinger Medical Center, in the United States (PGY 1, n=17; PGY 2/3, n=13). Due to the relatively small sample size, PGY 2 and PGY 3 residents were combined. The data were collected in July 2016 and July 2017, the beginning of each training year. Table 1 presents the demographic and training variables.

Procedures

This study was reviewed by the local Institutional Review Board of Geisinger Medical Center and determined to be exempt (IRB # 2016-0283) after receiving informed consent from the subjects. Surveys were administered to emergency medicine residents at the beginning of the training year. The instructions were read by a research assistant, and included information about the voluntary nature of this study, how data would be de-identified, and how data would not be used for formal performance evaluations. Once completed, names were replaced with a study identification number. Only the research assistant had access to the link between the name and identification number.

Measure

Two domains were assessed: residents’ self-reported confidence and their competence/accuracy when answering knowledge-based questions. A 19-item survey was developed by study investigators due to the lack of validated surveys in the literature (Appendix 1). Items 1–10 consisted of questions pertaining to demographics and training history, and items 11–14 consisted of residents’ self-reported confidence in 4 domains rated on a Likert-type scale of 1–10 (1= not at all confident to 10= very confident). Item 15 was an open-ended question asking residents what they would like to learn more about to effectively address psychiatric issues. Items 16–19 consisted of open-ended questions regarding competence in mental health-related issues. The competence items were scored based on the American Academy of Child and Adolescent Psychiatry’s practice parameters for the assessment and treatment of children and adolescents with suicidal behavior [4] and local and hospital laws. Doctorate-level clinical psychologists assisted in the generation and scoring of this instrument. Since a scoring rubric was used, no biases were anticipated in scoring. Raw data are available in Supplement 1.

The 2 groups (PGY 1 versus PGY 2/3) were evaluated separately to allow for comparisons and to determine the degree to which learning improved through residency. The independent-samples t-test was used to examine differences

The independent-samples t-test was used to examine residents’ confidence in their mental health knowledge and skills. Confidence was assessed for the following 4 domains: conducting a mental status exam, identifying provisional mental health disorders, conducting risk assessment evaluations, and developing a follow-up plan. Residents’ confidence ratings ranged between 6.43 and 7.13 on a scale of 0 to 10 (Table 2). Differences were found in respondents’ confidence related to developing a follow-up plan (t [28] = 2.77, P= 0.01), with PGY 2/3 residents endorsing more confidence than PGY 1s.

The independent-samples t-test was used to examine competence related to mental health crises (Table 3). For identifying risk factors related to intentional self-harm, out of a possible score of 12, residents identified 3–8 risk factors, with an average of 4.90 factors. In terms of developing a safety plan, out of a possible 4 points, scores ranged from 0 to 3, with an average score of 1.90. PGY 2/3 residents could identify more pertinent elements of a safety plan than PGY 1 residents (t [28] = 2.26, P= 0.03). When differentiating between voluntary and involuntary admissions paperwork, out of a possible 3 points, residents scored 0–2 points, with an average of 0.60 points. PGY 2/3 residents more accurately identified who signs hospital admission paperwork, the length of hold, and the consequences of these admissions than PGY 1 residents (t [28] = 7.51, P< 0.001). In practice vignettes related to signing paperwork, out of a possible score of 5 points, scores ranged from 0 to 5, with an average of 2.13 points. On the item requiring respondents to describe the importance of toxicology screenings, scores ranged from 0 to 2 out of a possible 2 points, and the average score was 0.93 points.

From an internal quality improvement perspective, emergency medicine departments can use the instrument described herein in a number of useful ways to inform their training efforts. Training faculty can ascertain the specific skills in which trainees demonstrate marked improvement and which skills remain stagnant. We suggest that these data may be useful for generating targeted training initiatives to remediate areas of low performance that do not sufficiently improve. Of course, decisions regarding what constitutes a sufficient score in a given competency will be made on an individual program basis.

As an example, the data described in this paper show that PGY 2/3 residents displayed significantly higher levels of confidence than PGY 1 residents in developing a follow-up plan in response to patients endorsing suicidality. However, confidence in other areas related to psychiatric crises did not significantly differ between the groups. This data, along with other sources of information such as feedback solicited from residents and faculty, may indicate that innovations in the training curriculum are necessary, such as more frequent experiential encounters with experienced faculty. As another example, our data demonstrated that there were no between-group differences in competencies relating to signing psychiatric admissions paperwork as assessed through vignettes, but differences were found when participants were asked to list a set of facts. As such, we believe that our data support the use of observations and live performance feedback for our residents. Therefore, our hypothesis that we could obtain a breadth of useful information to make training decisions was confirmed. In addition to identifying areas for improvements in training, programs can quantitatively evaluate the effectiveness of various training interventions by comparing results after a training intervention with a baseline pre-training score or with the scores of a group that did not receive the intervention.

The utility of this training evaluation approach should be viewed in the context of a key limitation. Without psychometric validation of the assessment measure, firm conclusions regarding the efficacy of the training curriculum cannot be made. Our data may reflect differences between cohorts rather than true improvement over time. The existence of similar scores between groups does not necessarily mean that no learning/improvement in mental health took place. Rather, the domains that we specifically assessed may not have improved. Nonetheless, this assessment approach can be used with ongoing iterative feedback to track responses to changes in the curriculum. Although not generalizable in the sense of allowing assumptions to be made about how residents would perform at other sites, the data from this assessment may be useful for programmatic decision-making purposes to ensure that training opportunities are standardized and structured in response to identified needs.

Supplementary materials

References

• 1. Dolan MA, Fein JA; Committee on Pediatric Emergency Medicine. Pediatric and adolescent mental health emergencies in the emergency medical services system. Pediatrics 2011;127:e1356-e1366. https://doi.org/10.1542/peds.2011-0522 ArticlePubMed
• 2. Douglass AM, Luo J, Baraff LJ. Emergency medicine and psychiatry agreement on diagnosis and disposition of emergency department patients with behavioral emergencies. Acad Emerg Med 2011;18:368-373. https://doi.org/10.1111/j.1553-2712.2011.01024.x ArticlePubMed
• 3. Melton BB, Coverdale JH. What do we teach psychiatric residents about suicide?: a national survey of chief residents. Acad Psychiatry 2009;33:47-50. https://doi.org/10.1176/appi.ap.33.1.47 ArticlePubMed
• 4. American Academy of Child and Adolescent Psychiatry. Practice parameter for the assessment and treatment of children and adolescents with suicidal behavior. J Am Acad Child Adolesc Psychiatry 2001;40(7 Suppl):24S-51S. https://doi.org/10.1097/00004583-200107001-00003 ArticlePubMed

Appendix

Figure & Data

References

Citations

Citations to this article as recorded by