Optimum number of procedures required to achieve procedural skills competency in internal medicine residents

Abstract

Background

Procedural skills training forms an essential, yet difficult to assess, component of an Internal Medicine Residency Program. We report the development of process of documentation and assessment of procedural skills training.

Method

An explanatory sequential mixed methods design was adopted where both quantitative and qualitative information was collected sequentially. A survey was conducted within the Department of Internal Medicine at The Aga Khan University Hospital, Karachi, Pakistan to determine the optimum number of procedures needed to be performed by residents at each year of residency. Respondents included both faculty and the residents in the Department. Thereafter, all responses were compiled and later scrutinized by a focus group comprising of a mix of faculty from various subspecialties and resident representatives.

Results

A total of 64 responses were obtained. A significant difference was found in eight procedural skills’ status between residents and faculty, though none of these were significant after accounting for multiple consecutive testing. However, the results were reviewed and a consensus for the procedures needed was developed through a focus group. A finalized procedural list was generated to determine: (a) the minimum number of times each procedure needed to be performed by the resident before deemed competent; (b) the level of competency for each procedure for respective year of residency.

Conclusion

We conclude that the opinion of both the residents and the faculty as key stakeholders is vital to determine the number of procedures to be performed during an Internal Medicine Residency. Documentation of procedural competency development during the training would make the system more objective and hence reproducible. A log book was designed consisting of minimum number of procedures to be performed before attaining competency.

Background

Internal Medicine Residency Programs are responsible for identifying and implementing the requirements to ensure comprehensive training of the residents enrolled in the program. This includes procedural skills training, and the mandate for a competency-based postgraduate medical training requires all residency programs to teach clinical skills formally [1, 2]. Studies have shown that physicians report the procedure skills learned during the residency as the most important skills which have helped them in their career [3, 4]. Also, as depicted in one study, prospective residents prefer programs with more procedural training [5] and defining standards has been shown to be challenging yet beneficial for both patients and the physicians [6, 7]. Many physicians could depend on specialists to perform a procedure, but keeping in mind the unavailability of such expertise or inability of patients to move to such a setting frequently necessitates the Internist to perform these procedures, and even learn or master these procedures on their own due to lack of sophisticated training facilities for formal postgraduate training [8–10], hence the job of devising a comprehensive program is all the more important.

The residency program needs to identify specific procedures in which competency is to be expected of the graduating residents as a response to its specific context, and furthermore determine after performing what number of procedures under supervision are the residents deemed to be competent enough to perform them independently. However, determination of such numbers remains a challenge despite work being done in the past to unveil this dilemma [8, 11, 12]. Furthermore, experience of programs differs according to the their health care settings and requirements, and while some work from the developed world is available on the issue of imparting procedural skills competency to residents, the developing world still lags behind in its assessment of this issue.

We, at the Aga Khan University have taken the systematic instructional design process which has the core elements of analysis, design, development, implementation and evaluation (ADDIE). Instructional design is a set of procedures for developing education and training programs in a systematic, reliable and consistent manner.

The Aga Khan University Hospital (AKUH), Karachi, Pakistan is a major tertiary care hospital catering to more than 18 million people of Karachi and the surrounding region. With an operational strength of 545 beds, the facility serves over 42,000 inpatients and over 500,000 outpatients annually. Established since 1985, it is one of the few teaching hospitals in South Asia accredited by the Joint Commission for International Accreditation [13].

The Internal Medicine residency program at AKUH, comprising a total of 50 residents, is a 4 year program during which residents rotate through General Internal Medicine as well as all other medicine sub-specialties. The faculty members for the department have received training from programs in the United States of America, the United Kingdom as well as Pakistan.

Objective

This study was conducted to identify the number of times a procedure needs to be performed by residents at different procedure status levels, during a residency program in the developing world, in order for them to achieve sufficient competency in their technical performance. We aim to formulate a set of guidelines to be implemented in our institute, as well as having applicability internationally, in our region and beyond.

Method

This was an explanatory sequential mixed methods design [14–16], where we had collected quantitative and qualitative information sequentially. We had first collected quantitative data through cross-sectional study and then qualitative data to help refine the quantitative results, so that the study design should capture the best of both quantitative and qualitative data. We obtained quantitative data from questionnaires filled out by the faculty and the residents, and then elaborated on these findings through in-depth qualitative exploration focus group discussions (Additional file 1).

A written informed consent for participation in the study was obtained from the participants after explaining the research study and design to them.

In order to identify the optimum number of procedures required to achieve procedural skills competency a comprehensive list of procedures was identified according to the program’s contextual and certification requirements, which included peritoneal paracentesis, pleural paracentesis, urethral catheterization, lumbar puncture, CVP/JO Cath insertion into femoral vein, CVP/JO CATH insertion into Internal Jugular vein, CVP insertion into subclavian vein, Internal Jugular Vein, Temporary Pace Maker placement, Arterial Line placements, drawing of Arterial Blood gases (ABGs), Cardiopulmonary resuscitation (CPR), Bone Marrow aspiration, Joint aspiration, Chest Tube insertion, Endotracheal intubation, Swan Ganz catheterization, Pericardial paracentesis and Pleural biopsy. This list was then sent to all faculty members of the department as well as all residents enrolled in the program. The list was designed in the form of a questionnaire in order to yield two different pieces of information from faculty and residents separately: (1) what the residents and faculty felt was the adequate number of times each procedure listed is to be performed in order to achieve competency and (2) the status of the individual resident while performing the procedure in their opinion in the residency program. Four different statuses were determined as follows:

Procedure status definitions

Observer status: Procedure observed without any active involvement in the intervention.

Assistant status: Assisted the procedure which was performed by a trained Post Graduate/Faculty

Performed under supervision: Performed procedure under direct supervision of a trained Post Graduate/Faculty

Independently performed: Perform a particular procedure independently, in consultation with the Faculty/Consultant

Statistical analysis

All results from the above mentioned questionnaire were compiled and analyzed using SPSS Ver. 17.0 (SPPS Inc, Chicago, IL). Basic descriptive statistics (medians and interquartile ranges) were generated. The Kolmogorov-Smirnov test identified that the data was non-parametric; hence, the Mann-Whitney test was used to compare differences in the responses of the faculty and the residents at significance level of .05. Due to multiple Mann-Whitney tests applied, a Bonferroni correction was applied to the significance level to deal with the potential problems with an inflated Type I error. Results were tabulated for presentation.

Focus group discussions

The preliminary responses were gathered and discussed within a focus group comprising of 12 faculty members, including the program director and coordinator of Internal Medicine residency program, ex-Program Director of internal medicine, faculty representatives of all the medical sub-specialties, and two Chief Residents of Internal Medicine. The chief residents represent the opinion of residents while the sub specialty faculty provided the faculty’s perspective ensuring equal and unbiased viewpoint from all the stakeholders. Furthermore, a list of certain basic procedures (e.g. ECG recording, venupuncture, proctoscopy etc.), which were not included in the questionnaire, and certain advance procedures (e.g. Upper and Lower GI Endoscopy etc.) which were supposed to be only observed or assisted by the residents, was finalized.

This group evaluated the responses of the faculty and residents, and was given the responsibility to approve the optimal number of procedures, the year in residency when the procedure must be performed and procedure status of the residents for different procedural skills in light of the earlier conducted survey. Furthermore, wherever there was a significant difference in the opinion of the faculty and residents regarding specific procedures, the focus group gave its expert opinion regarding the final number of procedures to be recommended in the program guidelines. These consensus guidelines for the program were then approved by the Chair of the Department of Medicine and the Quality Improvement Committee of the hospital chaired by the Medical Director of the hospital. The compiled results after thorough debate and consensus were given the form of a log book to facilitate documentation and evaluation of procedures performed by a resident (Additional file 2).

Ethical approval

Ethical approval was obtained from the Aga Khan University’s Ethics Review Committee.

Results

A total of 64 responses were obtained from the questionnaire, in which 44 were residents while 20 were faculty members. Table 1 lists the median (and IQR) of the numbers of each procedure as suggested by both the faculty and residents. It also lists the differences between each of these observations according to p-value generated through the Mann-Whitney test, as well as the overall medians of faculty and residents for each procedure.

Table 1 No. of procedures required to be completed according to the responses of faculty and residents with differences according to p-value

In general, similar responses were obtained from both faculty and residents for each procedure. Statistically significant differences were seen only for CVP insertion into subclavian vein (supervised), blood drawing for arterial blood gases (independent), cardiopulmonary resuscitation (observed and independent), joint aspiration (assisted and supervised), and pleural biopsy (assisted and supervised). None of these results were however significant with a Bonferroni correction applied which reduced the p-value to 0.0007. Table 2 summarizes the above mentioned procedures, depicting responses of faculty, residents, their overall median (and IQR), as well as the subsequent focus group recommendations for these procedures. Focused group was ultimately responsible for generating the final number of procedures in light of the suggestions provided by the stake holders. For example, the faculty and residents suggested that pericardial paracentesis be performed independently by residents as depicted in Table 1. However, after careful consideration by the focus group members, involved in post graduate medical education, and with the consent of chief residents it was decided that such a procedure be performed only under supervision of cardiologist and not by internal medicine residents alone irrespective of year of training.

Table 2 Procedures with significant differences in faculty and resident responses along with focus groups recommendations

All the results depicted in Table 1 were subsequently discussed under the focus group; Table 3 summarizes the final focus group recommendations about the number of times the procedures needed to be performed in order to achieve competency at different procedure status and the minimum residency level when these competencies should be obtained.

Table 3 Procedural skills required to be completed according to status and program level as determined by the consensus of focus group

Discussion

While identification of specific procedures and number of times they need to be repeated to achieve a level of competency is a matter of debate [12, 17], methods to determine this optimal set of numbers are also contentious. Expert consensus guidelines, although widely reported in literature [6, 11, 12, 18–22], have been questioned due to their inherent subjectivity [18, 23, 24], leading to a need for more standardized and vigorous system [23, 25]. Setting these criteria and standards has been shown to positively impact training of post graduate trainees [6, 26, 27]. Furthermore, when these recommendations are exposed to formal testing in terms of the skills imparted, these numbers may not seem sufficient to impart the competence in procedures deemed generally advanced and specialized [17, 24]. This, however may not always be the case, especially in the more routine procedures of the internal medicine residency training [12].

Therefore, while developing these guidelines, equal weightage was given to the residents’ and the faculty’s opinion to arrive at the optimal number, while the focus group served to streamline their opinions in cases where the opinions diverged significantly. Taking residents’ opinion to form these tools has been suggested by earlier literature [28–30]. Lack of funding and resources even at places with specialized training programs have been identified as possible causes for inadequate procedural training [31]. We must come up with ways to overcome these hurdles and by making this tool we have tried, at least in part, to increase the competence of our trainees, while using the limited recourses available to us.

Residents in our country have to undergo a mandatory internship year before joining a residency program and have already given the first part of their accreditation exam (Fellowship of College of Physicians and Surgeons—FCPS); they are adequately exposed to the ground realities of training in order to give them an informed opinion. Taking responses from faculty is more intuitive as these are the ones who perform these procedures themselves or supervise and train others.

The focus group was necessary to factor in the expectations from individuals who have actually designed or are responsible for academic and administrative affairs of the program. The individuals comprised, thus remained aware of the realities of the society in which graduates of the program are expected to serve. Hence, their expert consensus was important in bringing the expectations of the patients and society into account while designing these guidelines. This system evokes experience published for other systems and programs such as that of the Accreditation Council for Graduate Medical Education (ACGME) of the United States [32] translated in our particular context.

We also believe that supervision and assessment at different competence levels or procedure status levels for necessary skill acquirement could have better objectivity, than only direct observation to acquire competency [12, 33]. Furthermore, this may serve to limit the traditional “see one, do one, teach one” model which has been called into question due to the inherent risks of complication and incompetence associated with it [18, 21], while not resorting to the over use of simulations and models in resource poor settings.

We, at the Aga Khan University have a trainee centered program and strongly believe in the transition of trainer based to trainee based curriculum (Table 4). We train our residents in areas they feel they need most assistance and where they have lagged in their previous years during undergraduate or postgraduate training. It must also be kept in mind that none of the curriculum development is done without a rigorous evaluation of suggestions that are put forward by the residents, as done in this exercise of log book development. To put it aptly, we direct and not dictate the training of our residents, keeping in mind the changing trends and upcoming need of skills in a physician.

Table 4 Procedures in which performance competency is required by the Aga Khan University, Karachi, Pakistan

As a result of this exercise, a log book has been designed, wherein the residents are required to log all procedures performed during each residency year. Figure 1 shows a sample page of the log book. Table 4 lists the names of the procedures performed by residents at the Aga Khan University. This will facilitate in reporting complications, if any, encountered during or after the performance of the procedure. This is to ensure that optimal recommendations could be arrived at after field testing the new recommendations, while also serving the long term aims of improving the practices of post graduate medical education for our institutional program.

Fig. 1

Sample logbook page

The development of this log book has been placed at the “implementation” element of the ADDIE model as explained earlier. We have analyzed the need and importance of the issue, designed and developed a log book which is currently being filled by the residents at their respective levels of training. We believe that this tool will help bring a uniform consistency and competency to the training of the resident, which unfortunately has not been achieved earlier [24]. The following part of the study will be the evaluation part where we would be able to determine the number of procedures performed and competency achievement.

Study limitation

Firstly, the initial survey was performed on the faculty and residents of only one institution, further multi-centered studies must be performed in order to generalize its applicability. Secondly, time for competency level accomplishment may vary according to residents’ rotation, personal motivation, availability of cases and their learning abilities.

We feel that one potential bias could be information bias as data obtained was subjective and dependent upon the individual faculty/residents’ understanding of obtaining expertise in a certain procedure.

Future studies done on the same topic may rectify the number of procedures to be performed by residents to become competent in a particular procedure. Log book is designed in such a way that a resident is being observed through various stages and finally performs supervision.

It will be valuable to also see which procedures are performed and also the attainment of competence in performing these procedures in programs globally and compare the training methodologies between the programs in this part of the world to those in the West.

The ultimate value of these set of numbers compiled in a log book lies in the cumulative effort and input of the residents and faculty of a teaching hospital to define competence of a trainee in procedural skills in the Internal Medicine Residency Program. These numbers can never be of any value unless practically implemented, monitored and regularly updated.

Conclusion

It has remained a challenge to identify the precise number and procedures to achieve procedural skill competence in an Internal Medicine Residency Program. It is vital to consider the opinions of all stakeholders including both the post graduate trainees and faculty before any guidelines are formulated. Documentation of each skill developed and accountability for each mistake made during the training would make the system more objective and hence reproducible globally. A general consensus should be sought to eliminate the difference of region or country where the training is provided regarding Procedural Skills Competency.

This study suggests that residency programs in different parts of the world have different requirements regarding procedural skills. It also adds to the literature in terms of an illustrative exercise for development of guidelines in a developing world setting, which is responsive to its national healthcare context. Further assessment of the logbook developed will benefit in streamlining these proposed guidelines and may serve as a model for other programs in similar settings. These can thus form the basis and provide the tools for conducting, potentially large scale, multicenter studies to promulgate a set of such precise guidelines, having a much wider applicability.

Through this study, we have identified different ‘numbers’ of procedures and their respective status quantitatively, and compiled them in a log book form. Further work is required to fine tune this effort and to add a qualitative aspect to determine the efficiency and effectiveness of a resident while performing, assisting or even observing these procedures. Ultimately, the next step for the University is to evaluate how this documentation of acquisition of procedural skills can change the quality of care in terms of fewer complications due to appropriate supervision and better skills thus enriching the quality of residents we produce and eventually impact patient care.