Skip Navigation
Skip to contents

JEEHP : Journal of Educational Evaluation for Health Professions

OPEN ACCESS
SEARCH
Search

Articles

Page Path
HOME > J Educ Eval Health Prof > Volume 13; 2016 > Article
Brief report
Text messaging versus email for emergency medicine residents’ knowledge retention: a pilot comparison in the United States
Wirachin Hoonpongsimanont1*orcid, Miriam Kulkarni2orcid, Pedro Tomas-Domingo1orcid, Craig Anderson1orcid, Denise McCormack2orcid, Khoa Tu3orcid, Bharath Chakravarthy1orcid, Shahram Lotfipour1orcid

DOI: https://doi.org/10.3352/jeehp.2016.13.36
Published online: October 26, 2016

1Department of Emergency Medicine, University of California, Irvine School of Medicine, Orange, CA, USA

2Department of Emergency Medicine, Rutgers New Jersey Medical School, Newark, NJ, USA

3Department of Emergency Medicine, Kern Medical Center, Bakersfield, CA, USA

*Corresponding email: whoonpon@uci.edu

: 

• Received: September 4, 2016   • Accepted: October 25, 2016

© 2016, Korea Health Personnel Licensing Examination Institute

This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

prev next
  • 41,672 Views
  • 217 Download
  • 6 Web of Science
  • 6 Crossref
  • 7 Scopus
  • We evaluated the effectiveness of text messaging versus email, as a delivery method to enhance knowledge retention of emergency medicine (EM) content in EM residents. We performed a multi-centered, prospective, randomized study consisting of postgraduate year (PGY) 1 to PGY 3 & 4 residents in three United States EM residency programs in 2014. Fifty eight residents were randomized into one delivery group: text message or email. Participants completed a 40 question pre- and post-intervention exam. Primary outcomes were the means of pre- and post-intervention exam score differences. Data were analyzed using descriptive statistics, paired t-test, and multiple linear regressions. No significant difference was found between the primary outcomes of the two groups (P=0.51). PGY 2 status had a significant negative effect (P=0.01) on predicted exam score difference. Neither delivery method enhanced resident knowledge retention. Further research on implementation of mobile technology in residency education is required.
A majority of residents are part of a techno-literate generation known as Millennials [1]. Traditional classroom lectures and readings are not preferable learning methods for this generation. Millennials are accustomed to using technology and mobile phones in their everyday lives, including their education. A Pew Research Center study found that 95% of 18 to 29 year-old American cell phone owners reported using text messaging services daily [2]. A subsequent study found that 77% of the college students reported sending and receiving emails on a typical day [3]. These technologies could be an efficient way to enhance residency education.
Integrating text messaging and email into curriculums could improve students’ learning efficiency. One study reported that sending concise exercises to students via mobile devices demonstrated favorable results in class attendance, students’ performance, and motivation [4]. Using text messaging to deliver medical knowledge also provided a positive influence in nursing academic studies [5]. Hassini found that asynchronous communication through email could lead to a richer learning experience with students [6].
With the significant use of text messaging and mobile devices by millennials, our study compared the effectiveness of text messaging versus email to improve resident knowledge retention of emergency medicine (EM) core content. An email delivery method, not a formal classroom lecture, was chosen as a comparison group due to its practicality and common use by the millennial population. We hypothesized that EM residents who received daily text messages would have better knowledge retention than those who received an email. The study was a multi-centered, prospective, randomized study conducted in three United States EM residency programs accredited by the Accreditation Council for Graduate Medical Education over three different, two-month, periods in 2014. All three institutions are university-based residency programs; one three-year program, and two four-year programs.
The study subjects were EM residents from postgraduate year (PGY) 1 to PGY 3 & 4. Inclusion criteria included residents who possessed a mobile phone and were familiar with text messaging. We excluded residents who opted out, and those who were co-investigators of the study. Participants created Study Identifier Numbers (SIN) that included their training year and gender. SINs were linked to their email addresses, phone numbers, and pre- and post-intervention exam scores. Residents were randomized within PGY into two groups; text or email. PGY 3 & 4 were grouped together due to their similar roles and responsibilities in the emergency department.
Researchers created EM-related educational material in text messaging format. The material was derived from the American Academy of Emergency Medicine written board review textbook, “Emergency Medicine: A Focused Review of the Core Curriculum,” which is used to prepare residents for the American Board of Emergency Medicine (ABEM) written qualification exam, a required exam for board certification. A group text messaging system, Celly available from: https://cel.ly/, was used to deliver scheduled text messages twice a day. One hundred messages were sent in a two-month period to the text group. The email group received a one-time email in the middle of the two-month period, which contained the exact material that was sent to the text group (Supplement 1).
All participants completed a pre- and post-intervention exam, a 40 question exam derived from the “Physician’s Evaluation and Educational Review VII and VIII.” The pre-intervention exams assessed residents’ baseline knowledge, while post-intervention exams assessed the amount of retained knowledge after exposure to the interventions. Residents still had access to multiple resources to enhance their education, including material that was used to create the education material and exams. We only tested the effectiveness of the delivery method, text or email, to increase knowledge retention of EM content (Supplements 2, 3).
Our main outcome was the difference between pre- and post-intervention exam scores (exam score difference). We used descriptive statistics to analyze the demographic data, and paired t-test to compare the mean of exam score differences between delivery groups. We collected subject characteristics, gender and PGY, to perform an exploratory secondary analysis using multiple linear regressions to evaluate the effect of subject characteristics and intervention group on knowledge retention (predicted exam score difference). The data were analyzed using Stata ver. 10.1 (Stata Co., College Station, TX, USA).
The study was approved by the institutional review board of the University of California, Irvine, the University of Medicine and Dentistry of New Jersey, and Kern Medical Center (IRB number: HS#: 2012-8946).
A total of 58 EM residents, 41 male and 17 female, participated in the study. We enrolled 16 of PGY 1, 16 of PGY 2, and 26 of PGY 3 & 4 (Table 1). Although we observed a better knowledge retention in the text group, we did not find any significant difference in knowledge retention between text and email groups (P=0.51). We found no difference in the means of exam score differences between the two groups when we compared among gender or PGY (Table 2). A multiple linear regression showed that gender had no significant effect on the predicted exam score difference as seen in Table 3 (P>0.05). PGY 1, PGY 3 & 4 also had no statistically significant effects on predicted exam score difference (Table 4). However, we found that PGY 2 status had a statistically significant negative effect on the predicted exam score difference (P=0.01). Data file of intervention exam are available from Supplement 4.
These findings suggest that using text messaging to deliver educational content may benefit resident education. However, educators need to be aware of factors that affect text messaging implementation. Because busy schedules could prevent messages from being viewed by residents, finding optimal timing of message delivery is essential [7]. Studies showed a greater benefit when text messages were sent at optimal intervals with predetermined times and repetition, since repetition of pertinent material continually exposes residents to the knowledge that can become better retained [8,9]. In our study, the text messaging system had a maximum character restriction. We used symbols and abbreviations in text messages to minimize the character counts, while managing to deliver meaningful concepts in their entirety. For example, arrows were used to represent ‘increase’ or ‘decrease.’ This character restriction may lead to de-contextualization and misinterpretation of the material [10]. Therefore, text messaging might be of better use in delivering concise and simple educational content.
In our exploratory secondary analysis, we found that PGY 2 status predicted lower knowledge retention. This finding may be confounded by the nature of EM residency training curriculums. PGY 1 residents are able to navigate their clinical and educational experience at their own pace as they are introduced into their residency programs. PGY 2 residents are expected to see a higher volume of patients, and many critical care rotations are often included in this year. Focus on clinical workload could distract these residents from their learnings. PGY 3 & 4 residents are also responsible for a high volume of patients, but their experiences may enable them to efficiently manage their time. Upcoming ABEM qualifying examination also encourages senior residents to spend more time on test preparation. Program directors should closely monitor the learning behavior of PGY 2 and prepare to implement additional education strategies to maintain their appropriate progress.
Overall, the means of exam score differences were negative in value, meaning that residents had higher pre-intervention exam scores than post-intervention exam scores. We administered a set of pre-intervention exams shortly after an EM ABEM in-training exam, an exam used to assess an individual resident’s progress towards obtaining ABEM certification. Negative exam score differences may be accounted for by residents’ efforts in preparing for the in-training exam. Future studies should look into administering exams at a time where underlying interests would not influence the outcome of scores.
We noticed that exam score differences in the text group were less negative than those of the email group, alluding to more knowledge retention in the text group. However, this finding was not statistically significant even though we obtained an adequate sample size. We calculated that 34 subjects, 17 subjects per intervention group, were needed to achieve a power study of 80% to detect a 15% difference in exam scores between the two interventions. A larger sample size may yield significant and more generalizable results. Additionally, we did not calculate a sample size for our exploratory secondary analysis, nor stratified the randomization according to gender.
It is possible that residents shared their text messages or email with other residents, however, this would not affect the outcome of the study since the educational content of email and text messages were exact. We also did not track whether the text messages or emails were ever viewed by the residents. A crossover design could yield more significant findings by testing the capacity to achieve better knowledge retention with one intervention over the other, in the same participant. Alternatively, future studies could seek to use both interventions as complementary tools of education delivery to enhance knowledge retention.
In conclusion, use of text messaging to deliver EM education material did not enhance resident knowledge retention when compared to email. PGY 2 status predicted lower knowledge retention. Further studies about text messaging effectiveness and its implementation in residency education are still required.

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Funding

The funding of this project was provided by internal funds from the Department of Emergency Medicine at the University of California, Irvine.

Audio recording of abstract
jeehp-13-36-abstract-recording.avi
Supplement 1. Data file of text message and email content.
jeehp-13-36-supple-1.docx
Supplement 2. Data file of pre-intervention exam.
jeehp-13-36-supple-2.docx
Supplement 3. Data file of post-intervention exam.
jeehp-13-36-supple-3.docx
Supplement 4. Data file of intervention exam.
jeehp-13-36-supple-4.xlsx
Table 1.
Emergency medicine resident demographics
Characteristic Intervention group
Total no. (%)
Text no.(%) Email no. (%)
Overall 28 (48.3) 30 (51.7) 58 (100.0)
Gender
 Male 17 (41.5) 24 (58.5) 41 (100.0)
 Female 11 (64.7) 6 (35.3) 17 (100.0)
Postgraduate year
 1 7 (43.7) 9 (56.3) 16 (100.0)
 2 7 (43.7) 9 (56.3) 16 (100.0)
 3 & 4 14 (53.8) 12 (46.2) 26 (100.0)
Table 2.
Comparing means of pre- and post-intervention exam score difference between intervention groups (using paired t-test analysis)
Characteristic Intervention group
P-value
Text mean (95% CI) Email mean (95% CI)
Overall -1.82 (-6.41 to 2.76) -3.77 (-7.72 to 0.19) 0.51
Gender
 Male -3.44 (-9.84 to 2.95) -3.15 (-7.33 to 1.04) 0.93
 Female 0.68 (-6.64 to 8.01) -6.25 (-20.69 to 8.19) 0.27
Postgraduate year
 1 -0.29 (-17.34 to 16.77) -1.94 (-6.75 to 2.86) 0.80
 2 -3.50 (-13.41 to 6.31) -10.33 (-15.16 to -5.51) 0.13
 3 & 4 -1.75 (-6.88 to 3.38) -0.21 (-8.88 to 8.47) 0.73

CI, confidence interval.

Table 3.
Multiple linear regression of exam score difference by gender and intervention group
Variable Coefficient Standard error 95% Confidence interval P-value
Female 1.1 3.33 -5.58 to 7.77 0.74
Text 1.73 3.03 -4.35 to 7.81 0.57

Predicted exam score difference=-3.99+(1.10×female)+(1.73×text).

Table 4.
Multiple linear regression of exam score difference by PGY and intervention group
Variable Coefficient Standard error 95% Confidence interval P-value
PGY 1 -1.94 3.04 -8.04 to 4.17 0.53
PGY 2 -8.06 3.04 -14.16 to -1.96 0.01
PGY 3 & 4 -1.92 2.68 -7.29 to 3.45 0.48
Text 1.64 2.92 -4.21 to 7.49 0.56

Predicted exam score difference=(-1.94×PGY 1)-(8.06×PGY 2)-(1.92×PGY 3 & 4)+(1.64×text).

PGY, postgraduate year.

Figure & Data

References

    Citations

    Citations to this article as recorded by  
    • The #ObGynInternChallenge: Reach, Adoption, Implementation, and Effectiveness of a Microlearning SMS-Distributed Curriculum
      Fei Cai, Sarah Santiago, Elizabeth Southworth, Alyssa Stephenson-Famy, Emily Fay, Eileen Y. Wang, R. Nicholas Burns
      Academic Medicine.2023; 98(8): 917.     CrossRef
    • Birth of a Social Mediatrician: Adopting Slack, Twitter, and Instagram for Residents
      July Lee, Bradford Nguyen, Samantha Scanlon, Caroline Rassbach
      Cureus.2022;[Epub]     CrossRef
    • Mobile app technology is associated with improved otolaryngology resident in‐service performance
      Jacob E. Kuperstock, Michal Horný, Michael P. Platt
      The Laryngoscope.2019;[Epub]     CrossRef
    • Critical Appraisal of Emergency Medicine Educational Research: The Best Publications of 2016
      Nicole M. Dubosh, Jaime Jordan, Lalena M. Yarris, Edward Ullman, Joshua Kornegay, Daniel Runde, Amy Miller Juve, Jonathan Fisher, Teresa Chan
      AEM Education and Training.2019; 3(1): 58.     CrossRef
    • Bolstering diagnostic reasoning skills with adaptive learning
      Miranda Huffman, Sarah Gustafson, Souvik Chatterjee, Marc Rabner, Shantanu Nundy, Mary M. Gerkovich, Scott M. Wright
      Medical Teacher.2018; 40(8): 845.     CrossRef
    • Mobile Learning in Medical Education
      Blanka Klímová
      Journal of Medical Systems.2018;[Epub]     CrossRef

    Text messaging versus email for emergency medicine residents’ knowledge retention: a pilot comparison in the United States
    Text messaging versus email for emergency medicine residents’ knowledge retention: a pilot comparison in the United States
    Characteristic Intervention group
    Total no. (%)
    Text no.(%) Email no. (%)
    Overall 28 (48.3) 30 (51.7) 58 (100.0)
    Gender
     Male 17 (41.5) 24 (58.5) 41 (100.0)
     Female 11 (64.7) 6 (35.3) 17 (100.0)
    Postgraduate year
     1 7 (43.7) 9 (56.3) 16 (100.0)
     2 7 (43.7) 9 (56.3) 16 (100.0)
     3 & 4 14 (53.8) 12 (46.2) 26 (100.0)
    Characteristic Intervention group
    P-value
    Text mean (95% CI) Email mean (95% CI)
    Overall -1.82 (-6.41 to 2.76) -3.77 (-7.72 to 0.19) 0.51
    Gender
     Male -3.44 (-9.84 to 2.95) -3.15 (-7.33 to 1.04) 0.93
     Female 0.68 (-6.64 to 8.01) -6.25 (-20.69 to 8.19) 0.27
    Postgraduate year
     1 -0.29 (-17.34 to 16.77) -1.94 (-6.75 to 2.86) 0.80
     2 -3.50 (-13.41 to 6.31) -10.33 (-15.16 to -5.51) 0.13
     3 & 4 -1.75 (-6.88 to 3.38) -0.21 (-8.88 to 8.47) 0.73
    Variable Coefficient Standard error 95% Confidence interval P-value
    Female 1.1 3.33 -5.58 to 7.77 0.74
    Text 1.73 3.03 -4.35 to 7.81 0.57
    Variable Coefficient Standard error 95% Confidence interval P-value
    PGY 1 -1.94 3.04 -8.04 to 4.17 0.53
    PGY 2 -8.06 3.04 -14.16 to -1.96 0.01
    PGY 3 & 4 -1.92 2.68 -7.29 to 3.45 0.48
    Text 1.64 2.92 -4.21 to 7.49 0.56
    Table 1. Emergency medicine resident demographics

    Table 2. Comparing means of pre- and post-intervention exam score difference between intervention groups (using paired t-test analysis)

    CI, confidence interval.

    Table 3. Multiple linear regression of exam score difference by gender and intervention group

    Predicted exam score difference=-3.99+(1.10×female)+(1.73×text).

    Table 4. Multiple linear regression of exam score difference by PGY and intervention group

    Predicted exam score difference=(-1.94×PGY 1)-(8.06×PGY 2)-(1.92×PGY 3 & 4)+(1.64×text).

    PGY, postgraduate year.


    JEEHP : Journal of Educational Evaluation for Health Professions
    TOP