jeehp Search

CLOSE


J Educ Eval Health Prof > Volume 13; 2016 > Article
Engelhard, Leugers, and Stephan: Effectiveness of pre-admission data and letters of recommendation to predict students who will need professional behavior intervention during clinical rotations in the United States

Abstract

The study aimed at finding the value of letters of recommendation in predicting professional behavior problems in the clinical portion of a Doctor of Physical Therapy program learning cohorts from 2009-2014 in the United States. De-identified records of 137 Doctor of Physical Therapy graduates were examined by the descriptive statistics and comparison analysis. Thirty letters of recommendation were investigated based on grounded theory from 10 student applications with 5 randomly selected students of interest and 5 non-students of interest. Critical thinking, organizational skills, and judgement were statistically significant and quantitative differentiating characteristics. Qualitatively, significant characteristics of the student of interest included effective communication and cultural competency. Meanwhile, those of nonstudents of interest included conflicting personality descriptor, commitment to learning, balance, teamwork skills, potential future success, compatible learning skills, effective leadership skills, and emotional intelligence. Emerged significant characteristics did not consistently match common non-professional behavior issues encountered in clinic. Pre-admission data and letters of recommendation appear of limited value in predicting professional behavior performance in clinic.

Almost 17,000 individuals were seeking admission to Doctor of Physical Therapy (DPT) programs in 2014-2015. Over 95,000 applications were processed by the Physical Therapy Centralized Application System (PTCAS) [1, 2], and ultimately, by DPT admission committees. These applications are being processed in a variety of ways, often involving lengthy and time consuming procedures to procure what they believe to be a cohort of students that will have success in both the academic and clinical portions of the program [3,4,5]. As with other healthcare admissions processes, there appears to be minimal evidence available that provides information related to the predictive value of much of the data/information that is requested of DPT applicants and which is subsequently processed by DPT programs during an admission cycle [6,7,8]. The purpose of this study was to examine the value of letters of recommendation (LORs) in predicting a professional behavior problem in the professional clinical portion of a DPT program. Since the LORs from PTCAS include insights regarding professional behaviors, we hypothesized that the ratings and qualitative language in LORs would predict students who required major interventions in the clinic.
In this mixed methods study, de-identified records of 137 DPT graduates of the University of Cincinnati were examined in this retrospective cohort study to determine if predictive quantitative and qualitative indicators of clinical professional behavior problems could be identified. Records of graduates of the classes of 2009-2014 revealed that 21 graduates had at least one major clinical intervention or two minor clinical interventions requiring an onsite visits and/or more than two in-depth phone communications and follow-up involving the Director of Clinical Education (DCE). These students were defined as cases (students of interest, SIs). In each of these 21 cases, all SIs had professional behavior issues and some additionally had clinical skills deficits. Considering Fig. 1, there is an illustration of the frequency of the professional behavior issues of these students in clinic. The remaining 116 student records served as controls (non-students of interest, NSIs).
With respect to the quantitative aspect of the study, the background demographic status (age, gender, marital status, and responsibility for dependent children in the home), academic factors (undergraduate major, overall Grade Point Average (GPA), required pre-requisite science GPA, anatomy, physics, chemistry, and biology GPA’s, and Graduate Record Examination scores), pre-admission LOR’s and number of observation hours were studied. Additionally, rankings of professional behavior characteristics from LOR’s and the professional identification of the recommender (Physical Therapist (PT) or Non-PT) were collected. Once collected, investigation of the data was done by descriptive statistics to ascertain if demographic and academic differences existed. Wilcoxon Rank Sum Test was used to know if there were statistical differences with LORs between SIs and NSIs. All analyses were performed using R version 3.2.3 (2015-12-10) (R Foundation for Statistical Computing Platform). Statistical significance was determined by P-values < 0.05.
With respect to the qualitative aspect of the study, Strauss and Corbin’s grounded theory [9] was used to investigate 30 LORs. These LORs originated from randomly selected 5 SIs and 5 NSIs, all of which were de-identified. Themes emerged using open and then axial coding. The researchers selectively coded the data to create a qualitative data table, which consisted of SIs and NSIs characteristics. After constant comparison, the coders reduced the data table to determine significant and non-significant characteristics. Data were then reduced again to allow for comparison to determine if the LORs could predict SIs. Individual characteristics were defined as significant for the group of SIs if ≥ 4 or ≤ 1 students possessed the characteristic. This same process was used to identify significant characteristics for NSIs. However, if the characteristic was deemed significant for both the NSIs and SIs then the characteristic was deemed non-predictive. Only the characteristics that were deemed significant and were unique to the SIs were considered to be predictive. Prior to initiating the qualitative aspect of the study, the researchers sought to insure dependability which translates to reliability for quantitative studies. One trial LOR was reviewed and coded by both researchers simultaneously for coding training and confirming expectations and methodology. Both researchers independently open and axial coded all LORs. Analysis of 10% of the qualitative data set resulted in 87% coding agreement between researchers. With negotiation, the researchers came to 100% agreement.
Quantitatively, demographic differences were observed between SIs vs. NSIs in age at the start of the program (µ= 25.76 SIs, µ= 23.67 NSIs), gender (SIs= 87% female, 13% male; NSIs = 76% female, 24% male), the number of children at the start of the program (SIs 4.76%, NSIs 5.17%) and children during the program (SIs 9.5%,NSIs 5.2%). Academic differences in biology GPA (µ= 3.46 SIs, µ= 3.39 NSIs), physics GPA (µ= 3.48 SIs, µ= 3.39 NSIs), and verbal GRE percentile scores (µ= 59.6 SIs, µ= 53.7 NSIs) were noted. Other differences between the groups were negligible. On average, SIs were two years older, had more observation hours, have higher biology GPA (+0.07), higher physics GPA (+0.09), higher verbal GRE scores (+5.91%) and were less likely to take the GRE more than once in comparison to NSIs. The Wilcoxon rank sum test (Mann-Whitney u-test) was utilized in analyzing the differences in numeric ratings on LORs between the SIs and NSIs. Analysis of each of 11 professional behavior characteristics rated on LOR’s by PT’s showed three characteristics to be significantly different between the SIs and the NSIs; critical thinking (P= 0.0029), organizational skills (P = 0.0363) and judgement (P = 0.0001). None of the other professional behavior characteristics rated by PT’s in LORs were significant. PT ratings on LORs of communication (P= 0.0731) and professionalism (P= 0.0789) indicate marginal findings and may deserve further attention. Additionally, none of the professional behavior characteristics rated by Non-PTs in LOR’s were found to be significant.
Qualitatively, in order to establish the dependability of the study, the researchers individually, hypothesized which of the 10 students were SIs. Upon comparison of the researcher’s independent hypotheses, each had selected the same five students as SIs. However, only 2/5 hypothesized SIs were correct (40% accuracy). Using the previously stated criteria for significance, the following characteristics were determined to be significant for SIs: communicates effectively (4/5) and cultural competency (4/5), as seen in Table 1. NSIs characteristics included: conflicting personality descriptor (4/5), commitment to learning (5/5), balance (4/5), teamwork skills (0/5), potential future success (5/5), compatible learning skills (0/5), effective leadership skills (4/5), and emotional intelligence (4/5) as seen in Table 2. To clarify, an example of a conflicting descriptor is as following: One applicant being described by a recommender as “introverted” in 1 LOR; while, another LOR describes the applicant as “extroverted.” Quantitatively, we found significant differences in PT ratings on pre-admission LORs between SIs and NSIs for the characteristics of critical thinking, organizational skills, and judgement. However, we found no significant differences in pre-admission ratings by PTs in other professional behavior characteristics on LORs between SIs and NSIs. No significant differences were found in LOR ratings by Non-PTs in all professional behavior characteristics.
Therefore, our quantitative findings suggest our PT program may benefit from using PT ratings in pre-admission letters in the critical thinking, organizational skills, and judgement characteristics as predictors of future professional behaviors in the educational clinical setting. Use of LORs provided by Non-PTs did not appear to offer any predictive value in estimating a potential student’s professional behaviors in clinical education. Although critical thinking, judgement, and organizational skills were found to be significant predictors from LORs, our DCE spends the majority of time dealing with issues related to communication, clinical skills, confidence, commitment to learning and interpersonal skills which were not found to be significant predictors from either PTs or Non-PTs. As Table 3 illustrated, despite having followed a trustworthy qualitative method to analyze the data, the data did not yield any new insights to indicate that LORs could be used to predict clinical performance as it relates to professional behaviors. Thus, LORs appear to be of limited value in predicting professional behavior performance in clinic.
Limitations for this study included the sample size which was based upon one DPT program with a sample size of 137 students. Another limitation to our study was the skewed data from LORs and a ceiling effect on the rating scale for professional behaviors, as evidenced by overall positive scores on LOR’s and very few low ratings on a 5 point scale. Future research should include multiple DPT programs for a larger sample size. Additionally, an investigation of pre-admission characteristics for the development of a prediction rule for specific professional behaviors that relate to employability skills must occur as our profession goes forward [10].

Notes

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Acknowledgments

The authors acknowledge Dr. Andrew Tran, University of Cincinnati, Cincinnati, OH, USA, who assisted with data analysis for this study.

Supplementary material

Audio recording of the abstract.
jeehp-13-26-abstract-recording.avi
Supplementary raw data of quantitative analysis:
jeehp_13_26_suppl_1.xlsx
Supplementary raw data of qualitative analysis:
jeehp_13_26_suppl_2.xlsx

References

1. Commission on Accreditation of Physical Therapy Programs. 2014-2015 fact sheet physical therapist education programs [Internet]. Alexandria (VA): Commission on Accreditation in Physical Therapy Education; 2015 [cited 2016 Feb 4]. Available from: http://www.capteonline.org/uploadedfiles/capteorg/about_capte/resources/aggregate_program_data/aggregateprogramdata_ptprograms.pdf.

2. American Physical Therapy Association. Physical therapy centralized application service 2014-2015 applicant data report [Internet]. Alexandria (VA): American Physical Therapy Association; 2015 [cited 2016 Feb 4]. Available from: http://www.ptcas.org/uploadedFiles/PTCASorg/About_PTCAS/PTCASApplicantDataRpt.pdf.

3. Jette DU, Bertoni A, Coots R, Johnson H, Mclaughlin C, Weisbach C. Clinical instructors’ perceptions of behaviors that comprise entry-level clinical performance in physical therapist students: a qualitative study. Phys Ther 2007;87:833-843. http://dx.doi.org/10.2522/ptj.20070003
crossref pmid
4. Utzman RR, Riddle DL, Jewell DV. Use of demographic and quantitative admissions data to predict academic difficulty among professional physical therapist students. Phys Ther 2007;87:1164-1180.
crossref pmid
5. Galleher C, Rundquist P, Barker D, Chang W. Determining cognitive and non-cognitive predictors of success on the National Physical Therapy Examination. Internet J Allied Health Sci Pract 2012;10(4):7. http://dx.doi.org/10.2522/ptj.20060221

6. Bathje MS, Ozelie DHS, BCPD R, Deavila MS. The relationship between admission criteria and fieldwork performance in a masters-level OT program: implications for admissions. Open J Occup Ther 2014;2:6. http://dx.doi.org/10.15453/2168-6408.1110
crossref
7. Nelson LP, Maramaldi P, Kinnunen TH, Kalenderian E. Early performance in a humanistic medicine course as a predictor of dental students’ later clinical performance. J Dent Educ 2013;77:1006-1012.
crossref pmid
8. Lemay JF, Lockyer JM, Collin VT, Brownell AK. Assessment of non-cognitive traits through the admissions multiple mini-interview. Med Educ 2007;41:573-9. http://dx.doi.org/10.1111/j.1365-2923.2007.02767.x
crossref pmid
9. Strauss A, Corbin J. Grounded theory methodology. In: Denzin NK, Lincoln YS, editor. s. The Sage handbook of qualitative research. 4th ed. Newbury Park (CA): Sage Publications; 1994. p. 273-285.

10. Lazarus A. Soften up: the importance of soft skills for job success. Physician Exec 2013;39.5:40-45.

Fig. 1.
Frequency of professional behavior issues demonstrated during clinical internships of 21 students of interest who were admitted to the Doctor of Physical Therapy program in 2009-2014 at the University of Cincinnati, OH, USA.
jeehp-13-26f1.gif
Table 1.
Significant characteristics of 5 randomly selected students of interest who were admitted to the Doctor of Physical Therapy program in 2009-2014 at the University of Cincinnati, OH, USA based on grounded theory
Student's No. Communicates effectively Cultural competency
1 X X
2 X X
3 X
4 X
5 X X
Table 2.
Significant characteristics of 5 randomly selected non-students of interest who admitted to Doctor of Physical Therapy program in 2009-2014 at the University of Cincinnati, OH, USA based on grounded theory
Student's No. Personality conflicting descriptors Commitment to learning Team work skills Potential future success Compatible learning skills Effective leadership skills Emotional intelligence
1 X X X X X
2 X X X X
3 X X X X X
4 X X X
5 X X X X X
Table 3.
Communication predictor from letters of recommendation (LOR) and actual clinical outcome of 5 randomly selected students of interest who were admitted to Doctor of Physical Therapy program in 2009-2014 at the University of Cincinnati, OH, USA based on grounded theory
Student's No LORs communication predictor Clinical intervention
1 X
2 X X
3 X X
4 X
5 X
TOOLS
Share :
Facebook Twitter Linked In Google+
METRICS Graph View
  • 1 Crossref
  • 0 Scopus
  • 24,097 View
  • 261 Download
We recommend
Related articles in JEEHP

Construct validity test of evaluation tool for professional behaviors of entry-level occupational therapy students in the United States2016 ;13

MeSH terms
Cultural Competency
Emotional Intelligence
Grounded Theory
Humans
Leadership
Learning
Problem Behavior
School Admission Criteria
Students
United States


BROWSE ARTICLES
PUBLICATION ETHICS
FOR CONTRIBUTORS
ARTICLE CATEGORY
ABOUT
Editorial Office
Institute of Medical Education, College of Medicine, Hallym Unversity, Hallymdaehak-gil 1, Chuncheon 24252, Korea
TEL: +82-33-248-2652   

Copyright © 2020 by Korea Health Personnel Licensing Examination Institute. All rights reserved.

Developed in M2community

Close layer
prev next