Factors influencing the number of applications submitted per applicant to orthopedic residency programs

Background From 2002 to 2014, the orthopedic surgery residency applicant pool increased by 25% while the number of applications submitted per applicant rose by 69%, resulting in an increase of 109% in the number of applications received per program. Objective This study aimed to identify applicant factors associated with an increased number of applications to orthopedic surgery residency programs. Design An anonymous survey was sent to all applicants applying to the orthopedic surgery residency program at Loyola University. Questions were designed to define the number of applications submitted per respondent as well as the strength of their application. Of 733 surveys sent, 140 (19.1%) responses were received. Setting An academic institution in Maywood, IL. Participants Fourth-year medical students applying to the orthopedic surgery residency program at Loyola University. Results An applicant's perception of how competitive he or she was (applicants who rated themselves as ‘average’ submitted more applications than those who rated themselves as either ‘good’ or ‘outstanding’, p=0.001) and the number of away rotations (those who completed >2 away rotations submitted more applications, p=0.03) were significantly associated with an increased number of applications submitted. No other responses were found to be associated with an increased number of applications submitted. Conclusion Less qualified candidates are not applying to significantly more programs than their more qualified counterparts. The increasing number of applications represents a financial strain on the applicant, given the costs required to apply to more programs, and a time burden on individual programs to screen increasing numbers of applicants. In order to stabilize or reverse this alarming trend, orthopedic surgery residency programs should openly disclose admission criteria to prospective candidates, and medical schools should provide additional guidance for candidates in this process.

O rthopedic surgery remains one of the most competitive residencies to match for students participating in the National Residency Matching Program (NRMP) (1Á4). Over the past 12 years, the number of US and Canadian applicants to US orthopedic residency programs increased by 25% (from 901 to 1,140) while at the same time, the number of applications submitted per applicant rose by 69% (40.6Á70.2) (5,6). As a result of these increases, the average number of applications received by orthopedic residency programs increased by 109% from 248.9 to 519.5 (5,6). This dramatic increase in the number of applications submitted per applicant and the number of applications received by orthopedic residency programs has increased the financial burden and time management concerns on the applicants and programs alike. Although there is currently no information on the financial and time burden of an increasing volume of applications, the time requirements and ability to appropriately and efficiently Medical Education Online ae Medical Education Online 2016. # 2016 Elissa S. Finkler et al. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.
screen an enlarging pool of orthopedic surgery applicants have been a concern at Loyola University.
Factors that determine a successful match in orthopedic surgery have been well studied and published by NRMP (7). For example, successful match applicants in the 2014 match were offered an average of 12.1 interviews and had average United States Medical Licensing Exam (USMLE) Step 1 and Step 2 scores of 245 and 251, respectively. Confirmatory data for these metrics are the fact that probability of matching is 90% when 12 interviews are obtained and 80% when USMLE Step 1 score is 240.
The highly competitive nature of the orthopedic surgery application process and the perceived need to interview at 12 programs or more to have a !90% chance to successfully match have presumably contributed to the increasing number of applications submitted. In addition, other studies have suggested that the implementation of the 80-hour work restriction has led to an increase in applications (8,9). The NRMP and others have previously reported on factors such as class rank, USMLE 1 scores, and the number of publications that correlate with a successful match. However, recently the effect of applicant publication volume on orthopedic residency match has been called into question. Campbell et al. concluded that 'the average matched applicant to an orthopedic residency program publishes in the peer-reviewed literature less frequently than previously reported' (10). In addition, factors that influence applicants' rankings of orthopedic residency programs have been studied. In the study by Huntington et al., the most important factors affecting rank lists were perceived happiness/quality of life of current residents, resident camaraderie, and impression after an away rotation (11). However, to our knowledge, there have been no investigations to determine the rationale for the number of applications submitted to orthopedic residency programs. We hypothesized that lesser qualified applicants apply to more residency programs. Current investigation is designed to delineate which factors are associated with the number of applications submitted per applicant.

Materials and methods
This study was reviewed by our institutional review board and classified as exempt.

Study participants and survey administration
Three orthopedic surgery attending physicians, one statistician, and three orthopedic surgery resident physicians developed a 22-question survey (Table 1). Although not independently validated, the survey obtained basic demographic information as well as data regarding specific factors that have been correlated with a successful match according to the NRMP data; Alpha Omega Alpha (AOA) status, USMLE scores, number of publications, honors status in medicine/surgery clerkships, and the number of research/work experiences. Furthermore, candidates were asked to rate their 'application competitiveness' based on their own assessment as poor, average, good, or outstanding. We intentionally did not define these categories for the applicants but did evaluate their objective assessment of their 'competitiveness' by cross referencing factors known to be associated with highly competitive applications. Qualitative data were grouped into major categories by two of the others (EF and SK), and consensus was obtained to pool data into five categories including 'other' for statistical analysis to ensure adequate sample size in each category. Questions 21 and 22 were grouped together due to the low number of PhD applicants (n02). The survey was sent via SurveyMonkey (Survey Monkey, Palo Alto, CA) to all applicants of our institution's 2015 orthopedic surgery residency program with a cover letter that explained the purpose of the study, emphasizing the voluntary nature and anonymity of the study. The survey was sent thrice over an 8-week period to the applicant pool in an attempt to maximize response rate.

Data collection and statistical analysis
Completed surveys were compiled into a single database, and all analyses were conducted using SAS v9.4 (SAS Institute Cary, NC). Continuous variables were made into categorical variables based on cut points of interest or quantiles to ensure adequate sample sizes in each category. One-way ANOVA and Independent t-tests were used to compare average number of residency applications for each variable collected. In cases of low cell counts or nonnormality, non-parametric analyses (KruskalÁWallis tests and Wilcoxon Rank Sum tests) were used. Post-hoc tests, adjusted for multiple comparisons, were conducted on significant associations to determine specific associations between levels of categorical variables. Significance was assessed at a 00.05 level.

Results
Of 733 anonymous electronic surveys sent, 140 responses were obtained for a 19% response rate (Table 1). Of the 140 responders, 116 (82.9%) were male and 24 (17.1%) were female. Ages ranged from 24 to 33 years. The compiled results are shown in Table 2. This entire group of 140 submitted an average of 80 applications per applicant to orthopedic residency programs.
Our objective analysis of the candidates' perception of their 'competitiveness' revealed that higher USMLE Step 1 scores, higher USMLE Step 2 Clinical Knowledge (CK) scores, honors in medicine rotation, honors in surgery rotation, and AOA designation were significantly associated (p B0.0001 for each) with candidates rating their application competitiveness as 'outstanding' (results not shown).
Several factors were found to be significantly associated with the number of applications submitted per applicant, including candidates' own perceptions of how competitive they were. Specifically, applicants who considered their competitiveness level as 'average' submitted more applications than those that ranked their application competitiveness as 'poor', 'good', or 'outstanding' (p B0.001). Candidates who ranked themselves as 'average' on the competitive application scale submitted an average of 94.6 (926.7) applications per applicant compared with those who considered their application competitiveness as 'poor', 'good', or 'outstanding' who submitted an average of 81. In addition, the number of away orthopedic surgery rotations completed was also associated with the number of applications submitted. Applicants who completed more than two away rotations submitted significantly more applications per applicant (85.9 (927.4)) than those who had completed none or one rotation (58.4 applications/ applicant (940.8)) or two away rotations (76.4 applications per applicant (924.6), pB0.03).
'Reasons for not applying to more programs' also revealed statistically significant findings in the 'other' category. Specifically, candidates who selected 'other' as a reason for not applying to more programs submitted an average of 111.8 applications per applicant, which was significantly higher than any other reason listed in the category (pB0.004). Specific comments listed included items such as 'would have applied to more programs but was not eligible for military programs'.
As seen in Table 2, no other factors queried for were significantly associated with the number of applications submitted per applicant. Interestingly, 'reasons for not  applying to fewer programs' did not reveal any statistically significant findings. However, the major theme centers around obtaining enough interviews to successfully match.

Discussion
Findings of this study are reflective of the Electronic Residency Application Service (ERAS) data demonstrating the increasing numbers of applications per applicant to orthopedic residency programs (6). As previously noted, there are both an increasing number of applicants to orthopedic residency programs and an increasing number of applications per applicant to orthopedic surgery residency programs. The NRMP surveyed program directors of orthopedic residency programs in an effort to determine factors considered important to the programs (12,13). In 2010, orthopedic residency programs received an average of 457 applications, invited 58 resident applicants for interviews, and ranked 45 candidates to fill four PGY-1 positions. At that time, 85% of program directors cited USMLE Step 1 target scores for interview invitation and 40% cited USMLE Step 2 (CK) target scores for interviews (12). Using a five-point ranking, many other factors (e.g., clinical rotation grades, letters of reference, and MSPE/Dean Letter) were considered important in determining interview invites. In the last program directors' survey in 2014, 93% of program directors reported that they have target USMLE Step 1 scores when considering which applicants to interview, and 52% of program directors cited using USMLE Step 2 target scores in a similar fashion. In that survey, programs received an average of 549 applications (N084), sent 72 applicants an invitation to interview, interviewed an average of 59 applicants, and ranked 51 candidates (13). Another study compared the perspectives on resident selection criteria solicited from orthopedic program directors and orthopedic residency applicants. They found that program directors valued an applicant performing a rotation at their institution, USMLE 1 score, and rank in medical school to be most important. In contrast, applicants felt the most important criteria to be a letter of recommendation from an orthopedic surgeon, USMLE 1 score, and rank in medical school (14).
Our results showed 79.5 applications submitted per applicant, consistent with 2014 ERAS data where the mean number of applications submitted per applicant was 70.1 (15). Results of this investigation refute our hypothesis that less qualified candidates (based on objective measures) apply to more programs than their more qualified counterparts. Factors usually associated with a more qualified candidate such as AOA status, honors in medicine/surgery rotation, USMLE Step 1 and Step 2 scores, number of abstracts/presentations/publications, and number of research experiences were not associated with fewer applications per applicant. We also investigated other factors that could potentially affect the number of applications submitted per applicant such as couples matching, sex, age, and region of origin, and there was no significant correlation between any of these factors and the number of applications submitted. Only completion of more than two away rotations in orthopedic surgery, an applicant's perception of his/her 'competitiveness', and 'other' reasons for not applying to more programs were associated with significantly more applications submitted per applicant. Although objective measures were associated with candidates' perceived 'competitiveness' level, and perceived competitiveness was associated with the number of applications submitted, there was no significant association between these objective measures (other than number of away rotations completed) and the number of applications submitted. It is not overly surprising that candidates who completed more than two away rotations in orthopedic surgery applied to a higher number of orthopedic surgery residency programs than those who completed fewer rotations. A likely explanation would be that if candidates view themselves as less competitive, they would likely attempt to 'audition' at more programs in an effort to gain access to more institutions and geographies and potentially programs that would have not otherwise offered them an interview. Similarly, if applicants view themselves as less competitive than their counterparts, they will tend to apply to more programs in an attempt to find programs that may not use conventional selection 'cut off' criteria during the process of deciding whom to invite for an interview. This may explain why candidates who rated their application competitiveness as 'average' submitted statistically more applications per applicant than those who rated their application competitiveness as 'good' or 'outstanding'. Although not statistically different, candidates who considered their application competitiveness as 'average' submitted more applications per applicant than those who rated their application competitiveness as 'poor'. We speculate that candidates who ranked their application competitiveness as 'poor' applied to more than one specialty.
The issue of applicants applying to multiple specialties in the same match year (residency cross-specialty applications) was not addressed in this study. Historically, for all applicants applying to orthopedic surgery residency programs, the cross-specialty application rate has been constant at approximately 20%, and the majority of crossspecialty applications are submitted to general surgical Á preliminary and categorical Á programs (16).
The increasing number of applicants and applications per applicant to highly competitive residency programs is not an issue isolated solely to orthopedic surgery programs. Over the past 6 years ending in 2015, for example, dermatology has seen an increase in the average number of applications per applicant for US and Canadian graduates from 53 to 66. Over the same time period, otolaryngology saw an increase from 43 to 52 applications per applicant (17). In the preliminary data recently released by ERAS, the average number of applications per applicant for other competitive specialties such as dermatology and otolaryngology appear to have decreased in 2016 (dermatology 59.2 and otolaryngology 49.0). However, the average number of applications per applicant to orthopedic surgery continues to increase and is now at 79 (18).
Although we expected to see a decrease in the number of applications submitted by better qualified candidates, this was not the case. Even well-qualified candidates submitted large numbers of applications, presumably to ensure a threshold number of interviews that statistically provide a !90% chance of a successful match based on the NRMP data (7). This phenomenon is creating an environment of increased financial and time strain for both the applicants and programs. We believe that solutions must be sought to stabilize or reverse this trend.
Medical school deans, orthopedic program directors, and orthopedic faculty who have contact with medical students must become more transparent and take a more active role in counseling candidates who have little chance of successfully matching to pursue alternative fields. Based on the 2014 NRMP survey of residency program directors, another strategy for decreasing the number of applicants is transparency of programs that use target numbers and publishing of their cut off targets in a manner that is available to all potential candidates. This would allow applicants to understand at which programs an interview and subsequent successful match is highly unlikely.
This study has several limitations, most notably the 19% response rate. It is possible that a higher response rate may have yielded different results. Furthermore, only applicants to a single institution were surveyed, thus leaving open the potential for bias based on single institution applicants. Our program received 733 applicants of the 1,062 total applicant pool in academic year 2015 applying to orthopedic surgery (19). Although we had a low response rate, we believe we had sufficient penetration of the orthopedic residency applicant pool. However, future studies should consider a multi-institutional approach to the applicant pool in order to increase the response rate. In addition, the survey was anonymous which limited the ability to obtain further clarification in some areas, particularly the 'other' category of reasons for not applying to more programs. Since it was anonymous, we also did not have the opportunity to determine the final match status of these applicants and could not correlate an applicant's competitiveness, number of programs applied to, and the success of matching.

Conclusion
Our investigation suggests that applicants are submitting large number of applications per applicant, irrespective of the competitiveness of the individual application. In light of the enormous financial and time burden placed on applicants and programs, we suggest that our specialty, with the support of our medical schools, seek innovative and creative ways to stabilize or reverse this trend.