Different approaches to selection of surgical trainees in the European Union

Abstract

Background

There is an increasing global interest in selection processes for candidates to surgical training. The aim of the present study is to compare selection processes to specialist surgeon training in the European Union (EU). A secondary goal is to provide guidance for evidence-based methods by a proposed minimum standard that would align countries within the EU.

Methods

Publications and grey literature describing selection strategies were sought. Correspondence with Union Européenne des Médecins Specialists (UEMS) Section of Surgery delegates was undertaken to solicit current information on national selection processes. Content analysis of 13 semi-structured interviews with experienced Swedish surgeons on the selection process. Two field trips to Ireland, a country with a centralized selection process were conducted. Based on collated information typical cases of selection in a centralized and decentralized setting, Ireland and Sweden, are described and compared.

Results

A multitude of methods for selection to surgical training programs were documented in the 27 investigated countries, ranging from locally run processes with unstructured interviews to national systems for selection of trainees with elaborate structured interviews, and non-technical and technical skills assessments. Associated with the difference between centralized and decentralized selection systems is whether surgical training is primarily governed by an employment or educational logic. Ireland had the most centralized and elaborate system, conducting a double selection process using evidence-based methods along an educational logic. On the opposite end of the scale Sweden has a decentralized, local selection process with a paucity of evidence-based methods, no national guidelines and operates along an employment logic, and Spain that rely solely on examination tests to rank candidates.

Conclusion

The studied European countries all have different processes for selection of surgical trainees and the use of evidence-based methods for selection is variable despite similar educational systems. Selection in decentralized systems is currently often conducted non-transparent and subjectively. A suggested improvement towards an evidence-based framework for selection applicable in centralized and decentralized systems as well as educational and employer logics is suggested.

Introduction

The selection of candidates to surgical training has been called a “missing link” in surgical patient safety work [1]. Regardless of how good the education and training system is, selection of candidates to undergo education and training is crucial. An important question is thus if the right candidates are trained to become future surgeons. Currently, there is a lack of evidence on which instruments can predict future competence [2]. However, there is an increased global interest in evidence-based recruitment and selection processes for specialist surgical training [3,4,5,6].

Assessment tools for selection have mainly been investigated in Anglo-Saxon countries such as Australia, Canada, USA, New Zealand, Ireland and the UK concomitant with profound reforms of the recruitment and selection processes to improve surgical training [2]. Most of these countries use centralized systems for selection. This is combined with structured interviews, psychological and aptitude tests based on national templates with exact grading according to specific criteria to predict clinical performance [7,8,9]. In contrast some countries such as Sweden apply a traditional, locally-based, decentralized system for selection to surgical trainee positions without formal guidelines or firm selection criteria [10].

The purpose of this report is to delineate current selection procedures in EU member-states and identify salient bases for differentiating these. Cognizant of the heterogeneity that also exists within centralized and decentralized systems, the advantages and disadvantages of these two paradigmatic systems are discussed. Avenues for improving selection processes with evidence-based tools independent of the underlying structure are suggested.

Material and methods

Information on bodies in EU countries responsible for surgical training programs and assessment tools for selection were obtained through Google searches and PubMed. A string of keywords was used including name of country, “surgical”, “trainee”, “selection”, “specialization” or “application.” Websites and “grey literature” [11] on the application and selection processes to surgical training were identified. Subsequently, a letter was sent to 16 Board Delegates in the Section of Surgery of the Union Européenne des Médecins Specialists (UEMS) to obtain information concerning the respective country’s selection process. One reminder was sent to non-responders. The data for this article derives from an ongoing multi-disciplinary intervention project on recruitment and selection processes to specialist surgical training in Sweden. All data pertaining to human subjects was obtained in adherence with relevant guidelines and regulations for the wider project’s ethical clearance obtained from the Swedish Ethical Review Authority (case number 2016–1050). The application and selection processes were sorted into three categories: centralized selection was defined as having a single national application, selection, and admission/employment process; hybrid as having one or two of the three; decentralized selection process as having none of the above undertaken at the national level.

Comparative case study of a decentralized (Sweden) and centralized (Ireland) process for trainee selection

Flyvebjerg argues [12] comparative case studies provide deeper and more sophisticated insight into complex, context-dependent phenomena than searching for context-independent, aggregate-based generalizations (rules). Sweden was selected as an archetypical case of a decentralized selection system. Data on selection procedures in Sweden was obtained via available literature [10] together with semi-structured interviews with 13 experienced surgeons, as previously described [13] and knowledge within the research group. Oral informed consent as per Swedish Ethical Review Authority (case number 2016–1050) was obtained from all informants, who were informed of the purpose of the study, that their contributions would be analysed and can come to comprise part of published results of the study, and that they would not be cited by name in any public documents. Permission to record interviews was obtained at the opening of all recorded interviews. One interviewee did not consent to be recorded and notes were taken in this interview. The theme of selection emerged in the interview material and served as a starting point for the current study.

Ireland was selected as a polar opposite case to Sweden, being the most centralized and formalized system for selection of surgical trainees in the EU. To obtain comparable insight, the research team conducted two study-visits to the Royal College of Surgery Ireland (RCSI) to observe the current testing and selection procedures. Visit 1 (2018) consisted of observation of interviews and selection of candidates to Core Surgical Training. Visit 2 (2019) consisted of observation of the interviewing and selection of candidates for the cut to Higher Surgical Training which was performed after the 2 years of basic training.

During the visits, relevant material describing the selection process was shared by RCSI faculty and staff, fieldnotes were taken during observation of the selection process and combined with spontaneous interviews with faculty members and involved staff [14]. Fieldnotes were pooled after each of the two visits to the RCSI and analysed by a team of two sociologists and four surgeons [14]. Focus during the study-visits was on the operationalization of the selection instruments and procedure, discussions of intended and unintended effects, challenges and benefits of the selection system in general and specific instruments, and the preconditions for their utilization. Data from the site visits and supporting documents provided the core background for Ireland as archetypical of the centralized system is based.

Results

Mapping

Selection processes in 27 countries were analysed and divided into three types of selection on a scale from centralized to hybrid to decentralized (Fig. 1, Table 1). Slovakia was excluded due to lack of information in English. Answers from seven UEMS Board delegates were obtained (Additional file 1). A wide range of selection processes using different assessment tools were documented, from a strict national and centralized process to a completely decentralized system down to the local hospital level. Only a few countries conducted transparent, standardized selection with clear statements of the weight of each criterion. Some countries have standardized selection, but with no clear statements on the weighting of criteria, while other countries, like Sweden, have no official standardization or guidelines for selection of trainees.

Fig. 1 The selection process to surgical training programs in the EU is organized in a centralized, hybrid or decentralized fashion. See Table 1 for explanations on the associated educational or employment logic

Table 1 Selection processes to surgical training programs in EU (refences in Additional file 2)

Interviews were found to be the most frequently used assessment tool, but type of interview ranged from ad hoc to structured Multiple Mini Interviews (MMI). The use of evidence-based assessment tools in the selection process also varies, from combinations of a multitude of validated tools to only test results or an interview (Table 1).

Twelve countries have decentralized selection processes with recruitment, evaluation and selection decision-making by local universities and medical schools or hospitals. Two countries have hybrid systems and 14 countries have national centralized systems. None of the processes are identical across countries, and the elaborateness of the centralized systems vary considerably.

Associated with the centralization-decentralization scale which constitutes a significant difference among countries is whether surgical training is primarily governed by an employment or educational logic. In some countries where a college or university is the responsible organizing entity, the applicant to a trainee position is considered a “student”, and a centralized admissions or selection process similar to other higher studies is performed. This is exemplified by the Irish case. Other systems, usually decentralized, where hospitals select candidates to surgical training, and do so as immediate or future employers, act according to an employment logic.

Ireland has the most elaborate centralized selection process, and the only country formally testing technical skills and aptitude. Spain has a centralized recruitment process where written exam scores are translated into a rank and hospital positions are based on this rank only. In other countries, for example Poland with a national selection process and guidelines for selection, there are fewer applicants than positions, meaning that more or less all applicants are accepted (written communication UEMS delegate, Oct 24, 2017) (Additional file 1). Some countries such as Estonia and Malta have de facto centralized systems due to country size, with only one teaching hospital.

Denmark is an example of a hybrid centralized-decentralized system, consisting of a national application system whereafter regional healthcare organizations perform the final selection of candidates. Likewise, in the Netherlands, after applying through a centralized national system, candidates choose the regions they wish to work in and varied regional selection procedures ensue. Finland changed its system during the study period, from a locally run and less structured process to a centralized system with selection through the universities with structured interviews and clear criteria for evaluation.

Belgium represents a country with decentralized local selection but a standardized consensus selection process across each of the seven university hospitals (Additional file 1). In Greece, training positions only exist in public hospitals and are generally advertised on the official website of respective local authority for Public Health. There are no specific application deadlines for applying for a training position and selection process works on a “first come, first served” basis, i.e. priority is given to an individual only according to the date of submission of their applications.

Comparative case study of two representatives of archetypes: Sweden and Ireland

Comparison of the decentralized Swedish and centralized Irish selection processes is visualized in Table 2. Sweden conducts strictly local selection processes at each hospital without standardized guidelines or consensus on selection criteria. An informal system exists with a 6–12 month locum position in the surgical department for vacant or temporary positions. The head of department and program director then decides whether or not a permanent position as a surgical trainee is offered based on employment needs. The lack of transparency and the unstructured process of assessment during the locum was considered problematic by the interviewed surgeons. However, support for the locum system was quite strong as it was deemed appropriate to evaluate a candidate’s fit for the job. A common theme for those not entirely satisfied with the current system concerned the need to apply structured and validated assessment tools as they had experienced unsuitable candidates slipping through the system [13]. A trainee usually receives a permanent position in the department when training is completed. This was stated as the main reason why the surgeons felt concerned about unsuitable surgical trainees in order to avoid a dysfunctional, ill-suited colleague.

Table 2 Comparison of selection process in a centralized vs decentralized country

In Ireland, the selection process is well-described on the RCSI website for potential candidates [15]. Ireland’s surgical training takes place in two steps with a selection process for each. Selection for Core Surgical Training is included in Table 2. After 2 years a second selection is made to Higher Surgical Training. During the first selection, around 100 candidates compete for 60 positions, and candidates are tested for technical and non-technical skills. After technical testing, a session of four consecutive eight-minutes Multiple Mini Interviews (MMIs) with two assessors from different surgical specialties are conducted. The four stations focus on different professional areas and the assessors independently electronically rate the candidates on a Likert scale. The members of the faculty were aligned through a common information meeting in the morning before the interview and testing process started. Instructions and rules for how to conduct and document the MMI assessment were provided. Furthermore, the Department for Quality Assurance at RCSI monitors the process in real time to assure alignment in judgement of the assessors during the MMI sessions. Personal knowledge of a candidate was not an exclusion criterion for being an assessor.

The RCSI has developed a metricized system for observations during the MMI and assessments of academic records and aptitude tests to rank the candidates. The highest ranked candidate chooses hospital and training position first. The metrics from the first selection process are adjusted to a maximum of 100 points. A majority of the assessors were satisfied with the process, and support for the system is strong, though a few faculty members mentioned concerns about the lack of possibility of identifying unsuitable candidates per se during the short interviews with standardized questions.

After 2 years of training, with a variety of compulsory academic, pedagogical and practical activities, a selection takes place to Higher Surgical Training. For this round of selection, a maximum of 1000 points could be collected through the above-described activities and another round of MMIs, where the interviews comprised 400 points of the total score. Some candidates missed the final selection by less than 10 out of 1000 points [16].

After scores are compiled, the assessors were able to discuss the candidates and scores but are not allowed to make adjustments to the final result, even in cases where they were frustrated over exclusion of a candidate they found very promising by a slim margin. The lack of possibility to directly impact the choice of a future colleague was clear and some faculty members stated that the only way to have any impact was to participate in the process. This, along with a collegial sense of duty ensures a sufficient number of quality assessors to maintain the very personnel-intensive system. A further systemic benefit according to the RCSI program director is that the attrition rate has been reduced substantially after this rigorous process for selection has been applied in tandem with follow-up on skills during the first two trainee years.

Discussion

This study shows great variations in how surgical trainees are selected and in the application of validated assessment tools.

A comparison of centralized and decentralized systems, especially the specific cases of Sweden and Ireland, shows particular advantages and disadvantages associated with each selection system. The Irish centralized system benefits from running an annual and open competitive national recruitment process that increases the number of candidates to choose from. The selection process is systematic, transparent and concentrated, and all candidates are compared in line with the BEME (best evidence medical education) guide no 45 [2]. A centralized process with a high degree of standardization mitigates the risk of decisive individual bias and secures a transparent and equal process for all applicants [17].

A further advantage of the current centralized standardized system in Ireland is that it reportedly lowered the attrition rate in the training programs (Prof. O.Traynor, Dean Of Postgraduate Surgical Education &Training, personal communication, March 20, 2019). A low attrition indicates efficacy of the instruments used for assessment, i.e., how well they test for capabilities needed for completing the educational program. Declining attrition rates also have clear economic benefits. However, it is difficult to compare attrition rates across centralized and decentralized selection systems, due to the lack of data in decentralized systems. The case in point: in Sweden the total number of surgical trainees at any given time is not recorded. Keeping track of the trainees over the course of their training allows for evaluation of cost, attrition, and effectiveness of the training program in each hospital. This is more feasible and practiced in centralized systems. However, it is not impossible to implement in a decentralized context if drop-outs are reported to a central entity charged with collecting and analysing such data.

Centralized systems for selection with systematic assessment tools are also the ones producing knowledge on the subject of selection with sufficient volumes of observations in controlled contexts, as shown by the Anglo-Saxon countries [5, 6, 18,19,20]. Extensive selection systems receive more attention, resources and scrutiny devoted to their operation, maintenance and improvement. Likewise, the more centralized the system, the more transparency and accountability is required for legitimacy. On the other hand, more sporadically used systems implemented on fewer numbers of candidates and involving fewer individuals in the recruitment evaluation and selection process, garner less attention, critical assessment and produce less documentation. Thus, centralized systems more frequently produce evidence-based data on instruments and procedures, data that can also be utilized by decentralized systems. This is reflected in that almost all studies on assessment tools for selection were conducted in the centralized Anglo-Saxon contexts [2, 21].

A central virtue of local or decentralized systems is flexibility, responsiveness and adaptation to local needs [22]. Depending on how processes are implemented the flexibility of decentralized systems can be advantageous or disadvantageous.

Recruitment from the locum system was deemed by the experienced Swedish surgeons to be positive as it allowed decisions to be made after observing the candidates´ work in situ [23]. Previous job performance in the department is considered a valid way to assess a candidate if done in a structured way by peer ratings [24,25,26]. Such a system allows for a holistic assessment over time compared with a “collection of indicators” rendered through a series of discrete assessments produced by multi-instrument centralized systems. This decentralized selection process benefits from being flexible and allows for locum positions whenever need arise, e. g. parental leave, turn-over or health issues. It also increases the control over the process for the local head of department and allows young doctors to try several different specialties. On the other hand, a disadvantage is, as the case for Sweden, that there is currently no national consensus on assessment methods for selection of candidates to surgical training programs.

The decentralized and unstandardized processes can lead to happenstance playing a significant role in the selection process with locals or “insiders” having undue advantage especially if the trainee positions are not announced externally. Also, having more than one candidate to choose from affords comparative evaluative opportunities and reduces happenstance playing a role.

Most selection systems are oriented towards finding the top candidates, but an important consideration when selecting to training positions is also to identify unsuitable candidates to mitigate the multiple detrimental effects of enrolling a potentially problematic trainee and future colleague [13, 27]. Trainees that need remediation will inevitably affect patient safety, work climate, and are costly for the department [28,29,30,31]. On the one hand, local selection processes are particularly susceptible in that they de facto narrow the candidate field with an increased risk for selecting a person with a slow or dysfunctional learning curve or hazardous traits [13, 27]. On the other hand, prolonged exposure of a candidate to multiple senior colleagues and members of other professions in a variety of settings would result in deeper familiarity and offer greater opportunities to detect both unsuitability and fitness for the job [13, 32]. Hazardous traits may be detected through validated tools or clinical job performance if appropriate attention is paid, albeit currently not performed to the best of the authors knowledge [13, 27, 33, 34].

As shown in Table 1, a majority of the countries rely on academic examination (six) or non-MMI job interviews (ten) in their selection process. These methods have relatively low predictivity for work performance according to literatures in industrial organizational psychology and current evidence in post-graduate medical education [2, 6, 25]. As pointed out in BEME guide no 45, test scores and grades are insufficient in selecting applicants for a complex and multidimensional role like specialty training. Likewise, traditional CVs and letter of recommendations are inconsistent in predicting performance [2, 6].

Regardless of having a centralized, decentralized or hybrid selection system, the interview is the key assessment tool. The current study indicates that structured interviews are not often used, although they are considered to be one of the most reliable tools for assessment with twice to three times the predictive value compared with unstructured interviews [25, 35, 36]. Even with only a handful of applicants for each position, MMIs have also been shown to be reliable and well suited to evaluate non-cognitive skills [19, 37, 38]. However, conducting MMIs may be too resource-demanding in decentralized systems with few applicants [39], whereas the cost of implementing a structured interview would be reasonable and advantageous. Structured interviews allow probing for non-technical skills, warning signs for unsuitability and simultaneously finding top achievers [40].

With the current pandemic of Covid-19 there are new challenges arising when interviewing candidates. The transition to video-mediated communication (VMC – Skype / Zoom) recruitment interviewing instead of face-to-face (FTF) interviews can save time and decrease cost for the applicant, but concerns have been expressed if there are disadvantages [41]. Research has been conducted for over a decade on VMC versus FTF interviewing and early studies of those actively engaged in interactive interviewing tend to show little difference between FTF and VMC on the impressions of assessors of job candidates [42,43,44]. However, a recent study found significant differences in assessments of passive assessors, i.e. those not actively participating in the conversation, but watching it, indicating that there are varying impacts on assessors depending on their roles in interviewing [45].

Although alluring to surgeons, evidence on selection using assessment instruments and procedures specific for surgical training, for instance assessing technical aptitude and visuospatial ability, has yet to be proven. In Ireland aptitude testing is conducted only for the first round of selection, i.e., for core surgical training [4, 46,47,48]. Aptitude testing is costly and resource demanding. Its importance and optimal weighting between different assessment instruments is unknown, as is correlation to later surgical performance.

Situational judgment tests (SJTs) have been found to reflect job performance and tend to discriminate accurately between average and low performing participants [49, 50]. Situational judgment tests could be used as an initial screening method before conducting more costly and time-consuming assessments, especially in situations with a low applicant to vacancy ratio [19, 20, 51]. Other assessment tools that have been investigated as suggested screening tools for successful residents are personality testing, emotional intelligence (EI) and grit, but without proving a clear benefit like the SJTs in distinguishing between candidates at the level of selection [52, 53]. Grit is interesting since it has been found to be higher amongst consultants than trainees, and proposed to be associated with attrition amongst residents, but only few and underpowered studies exist and there is a lack of large and longitudinal studies [54,55,56].

Surgery is a high-stakes occupation. Insufficient critical attention to selection, conserved traditional practices, and reluctance to use decision aids and systematic procedures is not unique to this field [35]. However, the consequences are uniquely inappropriate due to high risks to patient safety and work environment. Current practice in most countries is sub-optimal with room for improvement in both centralized and decentralized systems.

Many centralized systems would benefit from utilizing more validated instruments, especially combinations thereof, and have volumes of applicants to feasibly do so. Decentralized systems can benefit from the collective and cumulative learning of developing and deploying more standardized systems across local units. Standardization is not the same as centralization. In countries where a hospital is the organizing entity for recruitment and selection of surgical trainees, there is often an employment logic, i.e., the applicant considered a part of the workforce, and thus selection for the job is conducted like any other apprentice position. In contexts where surgical training is associated with more or less permanent employment at the unit where training takes place, transition to a centralized system is unlikely. Even in such contexts, a standardized selection procedure preferably composed of validated or evidence-based components, and sharing data across like units, is possible to develop and would be beneficial [57].

Limitations

There are several limitations in our study. The countries selected were limited by language and responding UEMS surgical board members, leading to a possible selection bias. Despite these differences, the material was considered valuable in describing the overall features for selection of trainees to surgical programs across the EU in a comparative manner. A limitation is that researchers were only able to study two prototypical examples of a centralized and decentralized system in depth.

A strength of the conducted study is that the research group consisted of social scientists and surgeons, which allowed for researcher triangulation and contributions on qualitative research competence, human resource management competence and the experience of heads of department with personnel and hiring responsibilities in Sweden. Another strength is that data was collected over an extended period of time, which gave the opportunity for iterative data collection and discussion within the group.

Conclusion

A wide variety of selection processes to surgical training programs and limited use of evidence-based assessment tools were found across the EU. Comparing systems reveals advantages and disadvantages of different selection systems with modern evidence-based processes found in reformed centralized rather than traditional systems. Based on these findings and current evidence, recruitment and selection to surgical training in many European countries has great room for improvement.

A standardized procedure composed of validated evidence-based components could be developed and distributed across decentralized units as well as a fixture of centralized systems, independent of employment or educational logics without increasing cost significantly.

A selection framework adopting current evidence-based instruments would include 1) ensuring multiple candidates to select from per position through wide advertisement and recruitment efforts; 2) conducting multiple mini interviews (MMI) or structured interviews by at least two assessors; 3) applying situational judgment testing (SJT); 4) conduct structured validated assessments during job trial periods with SJTs (if trial work periods are employed). An unresolved issue is that the optimal weighting between these different instruments. Likewise, even though surgery is a technical craft, the evidence is still equivocal concerning technical skill and aptitude assessment at the point of selection.