Using Word Clouds to Re-envision Letters of Recommendation for Residency Applicants Sinehan B. Bayrak, M.D.; Jennifer A.Villwock, M.D.; Mark R.Villwock, M.S.; Alexander G. Chiu, M.D.; Kevin J. Sykes, Ph.D., M.P.H. INTRODUCTION RESULTS DISCUSSION CONCLUSION REFERENCES • Letters of recommendation (LORs) are an important element of the residency application process. 1–3 • Previous publications have identified specific disadvantages of these narrative letters of recommendation (NLORs), including: inability to predict future clinical performance, 4 suboptimal inter-rater reliability, 5 and intrinsic gender bias. 6 • A standardized letter of recommendation (SLOR) was developed in 1995 by the Council of Emergency Medicine Residency Directors. 7 • Advantages of the SLOR: less time to evaluate applicants and better inter-rater reliability compared to the traditional NLOR. 8 • Disadvantages: lack of variation in ratings preventing differentiation amongst applicants 9 and the questionable utility of individual domains within the SLOR. 10 METHODS • Applicants included 23 (48%) males and 25 (52%) females • 187 blinded NLORs and 48 VLORS were reviewed and rated • No significant difference in the distribution of median scores for VLORs and NLORs (p=0.289; Figure 1) • Significantly more time required to review the NLORs vs VLORs 61 seconds IQR: 48-81 vs 18 seconds IQR: 15-19 (p<0.001) • Review time and VLOR score correlated positively (ρ=0.407, p=0.004) o Positive correlation appeared with NLORs, but not statistically significant (ρ=0.268, p=0.066) • Median NLOR and VLOR scores were statistically equivalent for both genders • All reviewers found VLORs easy to use and moderately/extremely useful • Previous research has shown that the current “standard of practice” regarding letters of recommendation for otolaryngology residency applicants fails to fairly convey distinct characteristics (i.e., the SLOR) and is time-consuming (i.e., the NLOR). • The objective of this study was to propose and evaluate a new additive tool for letters of recommendation utilized in applications for residency programs. • While existing literature has highlighted the shortcomings of SLORs, it has not yet proposed a solution to the problem. • We developed a “word cloud”-based VLOR that preserves the descriptive context of the NLOR without sacrificing the efficiency of the SLOR. • We evaluated the VLOR’s efficiency in discerning applicant quality and compared it to the standard NLOR’s. • We found that NLOR and VLOR scores are not significantly different from one another and that VLORs are significantly faster to evaluate. • We have created a novel letter of recommendation that combines the best elements of both SLORs (i.e., efficiency) and NLORs (i.e., meaningful content). • We found that VLOR ratings did not significantly vary from NLOR ratings. • Additionally,VLORs had the added benefit of significantly decreasing evaluative time. • Future research on this topic will focus on prospective VLOR data generation for medical students and residents to create more effective “snapshots” of their achievements and strengths. • Collected NLORs for 48 otolaryngology residency applicants interviewed in the 2016 application cycle • NLORs were mined for descriptors (e.g., adjectives, adverbs, etc.) • QSR NVivo 11 used to narrow descriptors by synonym and to query them for a “word cloud” VLOR for each applicant • Blinded NLORs and VLORs were reviewed and rated by eight individuals within the department • Evaluators surveyed to assess the usefulness and ease of use 11 of the VLORs Department of Otolaryngology – Head and Neck Surgery B C Figure 2. Distribution of median scores for VLORs versus NLORs. “[He] is simply a great candidate. I met and worked with him closely during his 3 rd and 4 th years on his clinical rotations. I found him very friendly; he was respectful of staff and patients alike and just a pleasant person to work with daily. Not surprisingly, [he] got along very well with our faculty and residents, who were all quite fond of his attitude, compassion, work ethic, humble nature, and overall competence. His zeal for our specialty was typified by a ‘first to arrive, last to leave’ work ethic. I found him prepared in the OR and an able surgical assistant. I really see no red flags of concern for this young man; he is academically very sound, and I feel certain he will do quite well in our field.” Figure 1. Sample NLOR with a corresponding VLOR. 1. Hillebrand K, Leinum CJ, Desai S, Pettit NN, Fuller PD. Residency Application Screening Tools: A Survey of Academic Medical Centers. Am J Health Syst Pharm. 2015;72(11 Supplement 1):S16-S19. doi:10.2146/ajhp150093. 2. Liang F, Rudnicki PA, Prince NH, Lipsitz S, May Jr. JW, Guo L. An Evaluation of Plastic Surgery Resident Selection Factors. J Surg Educ. 2015;72(1):8-15. doi:10.1016/j.jsurg.2014.07.013. 3. Nies MS, Bollinger AJ, Cassidy C, Jebson PJL. Factors Used by Program Directors to Select Hand Surgery Fellows. J Hand Surg. 2014;39(11):2285-2288.e5. doi:10.1016/j.jhsa.2014.07.012. 4. DeZee KJ, Thomas MR, Mintz M, Durning SJ. Letters of Recommendation: Rating, Writing, and Reading by Clerkship Directors of Internal Medicine. Teach Learn Med. 2009;21(2):153-158. doi:10.1080/10401330902791347. 5. Dirschl DR, Adams GL. Reliability in Evaluating Letters of Recommendation. Acad Med J Assoc Am Med Coll. 75(10). https://insights.ovid.com/pubmed?pmid=11031153. Accessed September 18, 2017. 6. Messner AH, Shimahara E. Letters of Recommendation to an Otolaryngology/Head and Neck Surgery Residency Program: Their Function and the Role of Gender. The Laryngoscope. 2008;118(8):1335-1344. doi:10.1097/MLG.0b013e318175337e. 7. Keim SM, Rein JA, Chisholm C, et al. A Standardized Letter of Recommendation for Residency Application. Acad Emerg Med. 1999;6(11):1141-1146. doi:10.1111/j.1553-2712.1999.tb00117.x. 8. Girzadas DV, Harwood RC, Dearie J, Garrett S. A Comparison of Standardized and Narrative Letters of Recommendation. Acad Emerg Med. 1998;5(11):1101-1104. doi:10.1111/j.1553-2712.1998.tb02670.x. 9. Kominsky AH, Bryson PC, Benninger MS, Tierney WS. Variability of Ratings in the Otolaryngology Standardized Letter of Recommendation. Otolaryngol-Head Neck Surg. 2016;154(2):287-293. doi:10.1177/0194599815623525. 10. Kimple AJ, McClug SW, Del Signore AG, Tomoum MO, Lin F-C, Senior BA. Standardized Letters of Recommendation and Successful Match into Otolaryngology. The Laryngoscope. 2016;126(5):1071-1076. doi:10.1002/lary.25637. 11. Laitenberger O, Dreyer H. Evaluating the Usefulness and the Ease of Use of a Web-based Inspection Data Collection Tool. Softw Metr Symp. 1998. RESEARCH QUESTION Does a “word cloud”-based visual letter of recommendation (VLOR) provide an efficient and effective alternative to standard narrative letters of recommendation (NLORs) in discerning applicant qualities?