Skip to main content

Measuring change in knowledge acquisition of Rwandan residents: using the American Board of Pediatrics International In-Training Examination (I-ITE) as an independent tool to monitor individual and departmental improvements during the Human Resources for Health program: an observational study

Abstract

Background

Rwanda is the only African country to use the pediatric International In-Training Examination (I-ITE). The objectives of this study were to use the scores from the I-ITE to outline the baseline level of knowledge of Rwandan residents entering the pediatric residency and the trends in knowledge acquisition from 2012 to 2018, during the Human Resources for Health (HRH) Program, an education partnership between the Rwanda Ministry of Health and a consortium of US universities.

Methods

A retrospective descriptive analysis of the I-ITE exam scores, taken by all Rwandan pediatric residents for five of the six academic years of the study period. Individual resident scores were weighted using the non-Rwandan I-ITE sites to minimise confounding from annual variations in exam difficulty. Statistical analysis included descriptives with ANOVA to compare variation in annual mean scores.

Results

Eighty-four residents took 213 I-ITE exam sittings over the five exam cycles. The mean weighted I-ITE score of all residents increased from 34% in 2013 to 49% (p < 0.001) in 2018. The 32-point gap between the mean US-ITE and Rwandan I-ITE score in 2012–2013 was reduced to a 16-point gap in 2017–2018. First year resident (PG1) scores, which likely reflect the knowledge level of undergraduate medical students entering the residency program, increased from 34.8 to 44.3% (p = 0.002) between 2013 and 2018.

Conclusions

The I-ITE is an independent, robust tool, measuring both learners and the institutional factors supporting residents. This is the first study to demonstrate that the I-ITE can be used to monitor resident knowledge acquisition in resource-limited settings, where assessment of resident knowledge can be a major challenge facing the academic medicine community. The significant increase in I-ITE scores between 2012 and 18 reflects the substantial curricular reorganisation accomplished through collaboration between Rwandan and US embedded faculty and supports the theory that programs such as HRH are highly effective at improving the quality of residency programs and undergraduate medical education.

Peer Review reports

Background

Twenty-three African countries offered paediatric postgraduate training programs in 2013 [1]. Paediatric residents throughout the world are trained to gain and acquire skills, attitudes, and clinical judgment [2]; knowledge acquisition is also essential, and assessment using certified exams is an important tool in assessing this latter competency [3].

In the United States (US) monitoring this knowledge acquisition among pediatric residents has been accomplished over the last four decades through the use of an annual In-Training Examination (ITE) as a formative self-assessment instrument [4]. Although participation in the US-ITE is voluntary, program participation has increased from 55% in 2003 to 89% in 2015 demonstrating its formative value to US-residents and their programs [5, 6]. The objective of the ITE is to assess general pediatric knowledge, to track progress year-to-year and to compare scores with national peers. Research has shown that the pediatric ITE is a valid and reliable estimate of resident knowledge and can predict final performance in the General Pediatric Certifying Board examination [7].

The International In-Training Examination (I-ITE) is offered annually by the American Board of Pediatrics (ABP) to training institutions outside of North America. The I-ITE is a subset of 200 multiple-choice questions (MCQs) developed by board-certified subject matter experts [7]. ABP psychometric and test development experts match the multiple-choice questions (MCQs) to the current General Pediatrics Content Outline, and one or more medical editors review the I-ITE to ensure current/relevant information and to remove any Americanisms before each year’s administration. The I-ITE is constructed from ITE questions given previously to all US residents. Therefore the US comparison data is for US residents who took that specific, previous ITE. If questions are removed from the I-ITE, the US-ITE score is recalibrated with those items removed. The I-ITE was piloted with one training program in Lebanon in 2008, and made available to all countries in 2009. Ten countries used the I-ITE during our study period, namely: Bahamas, Italy, Kuwait, Lebanon, Oman, Qatar, Saudi Arabia, Singapore and the United Arab Emirates (UAE).

Rwanda is a small country in East Africa with a population of 10.5 million people according to the last census in 2012, a GDP of 787 USD per capita in 2018 [8]. Health expenditure per capita is 57 USD, which less than 1% of US health expenditure of 9536 per capita [9]. There is a significant shortage of healthcare workers, with only 15 paediatricians in the public sector in 2012, representing one Paediatrician per 700,000 population [10]. The Rwandan residency program at the University of Rwanda was strengthened by the technical partnership with the Human Resources for Health (HRH) Program, launched in August 2012 as a $150million US Government and Global Fund-funded initiative to address the lack of qualified health professionals in Rwanda. The long-term goal of the HRH Program is to build the health education infrastructure and health workforce needed to create high quality sustainable healthcare in Rwanda. In this program, an unprecedented consortium of over 20 respected US Universities partnered with the Government of Rwanda and deployed US-trained faculty with advanced clinical training and experience [11,12,13]. The HRH program supports faculty in the medical and surgical specialities, nursing, midwifery public health, and hospital management [14,15,16]. A recent systematic review of strategies to improve health-care provider practices in low-income and middle-income countries (LMICs) demonstrated that training or supervision programs and group-problem solving programs were the most likely to see improvements in healthcare provider practices [17]. The HRH program models these strategies.

Within paediatrics, the US institution faculty making up the HRH team has included general paediatricians and paediatric subspecialists who have worked alongside local University of Rwanda (UR) academic and Ministry of Health clinical staff to develop and support the educational, research and administrative activities of the Department of Paediatrics [11, 13]. During the period of the study, 84 residents undertook one or more I-ITEs. Forty-four graduated with a further 34 HRH-trained residents expected to graduate in the future, which will increase local paediatric workforce by close to 500% and hopefully make a difference to the community.

Methods

Aims

This study aimed to use a completely independent high-quality tool, the I-ITE, to measure the baseline level of paediatric knowledge of new residents entering the paediatric residency, and the trends in knowledge acquisition by Rwandan paediatric residents, from 2012 to 2018.

Study design and site

We performed a retrospective analysis of I-ITE scores of all Postgraduate (PG) paediatric residents enrolled from 2012 to 2018 in the University of Rwanda (UR), the only university in Rwanda with a paediatric residency program. Our reporting of this observational study is in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist [18].

Implementation of I-ITE in Rwanda

No Rwandan resident had previously taken the I-ITE before the HRH Program. We contracted with the American Board of Pediatrics to utilise the I-ITE in Rwanda for five of the six academic years of the study period, namely: November 2012, March 2014, April 2016, March 2017 and May 2018. However, we were unable to administer the I-ITE in 2015 due to financial and organisational barriers.

Participants

All Postgraduate (PG) paediatric residents enrolled in the program from 2012 to 2018 sat the I-ITE.

Data collection and management

Following standard ABP protocol, we administered the best-of-five MCQ I-ITE electronically using password protected computers connected to the internet and proctored by faculty. The ABP provided us with summary data for the US ITE and non-Rwandan International-ITE scores in electronic format along with raw scores for all the Rwandan PG residents, and data was password protected in an Excel spreadsheet.

Sample size calculation

No sample size calculation was undertaken as all residents in the program sat the I-ITE throughout the study period, and we analysed all results.

Weighting annual scores

We weighted exam scores for variation in difficulty. Although the difficulty of the questions in the I-ITE are relatively stable, there is still statistically significant variation year-to-year [19]. The ABP provided summary data in the form of counts, means and standard deviations for the US-ITE and non-Rwandan I-ITE for each year. We did not include an analysis of the raw US and non-Rwandan I-ITE sites but rather undertook a post-hoc analysis of the summary data of the US-ITE and non-Rwandan I-ITE scores. This analysis demonstrated that there was a significant difference in the difficulty year-to-year (ANOVA, p < 0.0001, Table 1). Therefore, to minimise confounding from this annual variation in exam difficulty, we weighted individual resident scores using the results from other non-Rwandan I-ITE sites. We calculated an annual weighting factor by dividing the mean non-Rwandan I-ITE score for each year by the mean of the non-Rwandan I-ITE scores for all five exam sittings of the study period (Table 1). Each resident score was then weighted by dividing their I-ITE score by the annual weighting factor. We chose to use the non-Rwandan I-ITE scores to calculate the weighting (rather than the larger US cohorts) as other International-ITE training sites would have been more likely to encounter similar challenges (e.g., language, pathology relevance, etc.). Each resident examinee I-ITE score was also converted into a z-score based on the mean and standard deviation of the non-Rwandan I-ITE scores for that year.

Table 1 Weighting of annual mean I-ITE scores based on other International exam centers and Annual Mean Weighted Rwandan I-ITE scores

Statistical analysis

We used the Statistical Package for the Social Sciences (SPSS) (IBM SPSS Statistics for Windows, Version 24, Armonk, NY: IBM Corp). In order to analyse increases in weighted scores and z-scores during the five sittings, we conducted an ANOVA and used a repeated-measures ANOVA to analyse if there was an increase in scores within each class.

Ethical considerations

We paid particular attention to potential ethical issues and resident confidentiality. Faculty discussed the rationale and context for the exam before providing each resident with their score and ABP-provided subject breakdown. Residents’ results are otherwise only available to faculty and used for resident development and advising purposes. No incentives were offered, no potential risks were identified, and participation was deemed as consent. The Institutional review board (IRB) of the University of Rwanda (UR) approved the project proposal. Ref: 027/CMHS_IRB/2018.

Results

Participants

Eighty-four residents completed a total of 213 I-ITE sittings (Table 1) during the study period. Each resident sitting the I-ITE earned a score out of 100 (percentage), and scores are described as percentages throughout. No demographic data (age, gender, etc.) is available from the ABP.

Annual variation and weighting

We observed a significant fluctuation in the difficulty of the I-ITE in both the US-ITE and the International-ITE (Fig. 1) manifested a significant variation (p < 0.0001) in the mean scores from these sites. Therefore, we used the mean non-Rwandan I-ITE score to weight each Rwandan resident’s I-ITE score (Table 1).

Fig. 1
figure 1

Annual variation in US-ITE, Non-Rwandan International-ITE and Rwandan Mean I-ITE scores. p-values represent ANOVA within category

Changes in I-ITE scores

We demonstrate a significant increase in the performance of residents during the period of the HRH program with mean weighted-scores of all the residents increasing from 34.3 in 2013 to 48.6 (p < 0.001) in 2018 (Table 1). US-ITE scores were higher than the other non-Rwandan I-ITE scores (Fig. 1), but the 32-point gap between the mean US-ITE and Rwandan I-ITE score in 2012–2013 was reduced to a 16-point gap in 2017–2018. The pairwise increase was particularly significant between PG1 and PG2 and for the resident cohorts with more years of exposure to the HRH program during their training (Table 2 & Fig. 1). We surmise that PG1 scores are likely to reflect the knowledge level of undergraduate medical students (Table 1, Fig. 2) and demonstrate that the mean score of these entering residents increased from 34.8 in 2012 to 44.3 in 2018 (p = 0.002).

Table 2 Annual Mean Weighted Rwandan I-ITE scores by graduation class
Fig. 2
figure 2

Box-plot of weighted Rwandan scores by level of PG training. Box-plot represents: Medians, interquartile ranges, min and max scores. *There were no PG4 residents in the program in 2012-13 due to no recruitment of residents

Progression of year-groups

In order to analyse progression throughout time in the program, we created resident cohorts that were “tracked” by year of predicted graduation (Table 2). We observe a significant increase of the mean I-ITE score for the two I-ITE sittings taken by the class of 2014 (PG3/PG4 only) from 37.1 to 46.7 for the two I-ITE sittings taken by the class of 2020 (PG1/PG2 only), (Table 2, ANOVA, p < 0.001). We also demonstrate a greater progression in the increase of scores among residents who began residency in 2012 and later, while HRH was present.

Discussion

Our objective was to use a globally available, independent monitoring tool of high quality that could objectively measure trends in knowledge acquisition by Rwandan paediatric residents from 2012 to 2018. With this study, we demonstrate a significant increase in I-ITE scores during this period, along with a decrease in the gap between the mean US US ITE score and mean Rwandan I-ITE score. In addition to overall improvement of the mean scores for all Rwandan residents, we find that the mean Rwandan PG1 I-TE results has increased from − 2 z-scores to − 0.98 z-score, below the non-Rwandan International PG1 scores. This means that the top performing Rwandan post-graduates are now performing at the level of 16% of the other International post-graduates taking the I-ITE. If the rate of improvement was sustained, equity with these other international countries could be obtained within 5–6 years.

US residency programs have implemented different strategies for improving ITE and board performance by residents, including web-based curriculum [20], email-based board review [21], protected block curriculum [22], structured reading and mandatory remediation [23], and accountability program with incentives and consequences [24]. Most of these interventions are not currently utilised in Rwanda. However, there are several plausible explanations for increased Rwandan I-ITE scores, which we outline below:

Implementation of a formal paediatric curriculum

Prior to the HRH program, residents were taught primarily by means of week-long clusters of lectures delivered a few times per year in an ad-hoc fashion according to local, and primarily Belgian, lecturer availability. Subsequently, a formal curriculum based on the Global Pediatrics Education Consortium was developed and implemented by the HRH and UR faculty during the 2012–2013 academic year [25]. This curriculum was used to guide the development of rotation objectives and of a formal core didactic program that cycled every 2 years. The addition of weekly didactic sessions has been shown to correlate with improved ITE performance in the domains covered by the didactics [26].

Additionally, other modalities of teaching were implemented both at the hospital and university level and included daily morning reports, daily bedside teaching rounds, simulation sessions, practical skill teaching, morbidity, and mortality meetings, research training, and leadership development workshops.

Improved level of knowledge among clinicians entering the residency program

The mean score of PG1 residents increased during this period. Though PGs sit the I-ITE late in the academic year, the PG1 score is also a surrogate marker for the level of knowledge of residents entering into the residency program. The PG1’s increased level of knowledge suggests that undergraduate education has improved during the period of this study. Implementation of a structured educational curriculum during clinical clerkship has been shown to improve medical student performance on end-of-clerkship shelf exams [27]. In addition to contributing to post-graduate education, the HRH Faculty have also supported undergraduate teaching by reorganizing the paediatric clerkships to include weekly didactics and teaching of practical skills, In addition, HRH Faculty has been involved in undergraduate curriculum revision, in designing new assessment strategies and in supporting undergraduate research projects.

Change of the official language in Rwanda

Kinyarwanda, English, French, and Kiswahili are the current official languages of Rwanda. English became an official language in Rwanda in 2008. Until then, Kinyarwanda and French were the primary languages of instruction in primary, secondary, and tertiary education. Most of the residents first encountered English as the language of instruction at the tertiary university level. This is similar in Saudi Arabia where most high schools are instructed in Arabic, but the language of instruction in Saudi medical schools is English. A study of Saudi students enrolled in a premedical preparatory course found a correlation between English examination scores and performance on medical written and oral examinations [28]. Therefore, the improved I-ITE scores could be related to improved English proficiency of the Rwandan residents as they advanced in training. In addition, the ABP noticed that some residents might not have been able to complete the I-ITE in the time given and this could have impacted the performance of the residents taking the I-ITE in 2013. In response to this, the ABP increased the time permitted to complete the I-ITE from three to 4.5 h in 2014. This increased time could have contributed to improved scores for the residents taking the I-ITE after 2013.

Competency in using multiple-choice style questions

Longitudinal exposure to MCQ testing has been shown to improve ITE performance [29]. The first I-ITE in 2012 was the first time any Rwandan resident had taken an examination on a computer, and the MCQ format was not completely familiar to them. In subsequent years there was a greater proportion of residents within the cohort who were previously exposed to the unique computerised format. Therefore, it is possible that the increased scores were due to improved resident familiarity with the electronic platform and question structure format. In addition, since the beginning of HRH, the residents have had access to MCQ practice resources, allowing them to practice taking questions in the same MCQ format as the I-ITE.

Collaboration and intervention by the HRH faculty

Prior to the start of the HRH program in 2012, there were15 paediatricians in the Rwandan public sector. During the program, there have been 36 expatriate paediatricians working full time, in-country for periods between of 2–3 months (subspecialists) to multiple years (a 1 year minimum for generalists). HRH faculty are integrated as members of the Department of Pediatrics, to work full time in the training environment, supporting clinical and didactic activities at both the undergraduate and postgraduate level. I-ITE scores increased most significantly for the classes that had more years of exposure to the curriculum and teaching and curriculum developed by the Department of Paediatrics Faculty and their embedded HRH colleagues.

Implication of our results

Using the I-ITE

The I-ITE is administered in an online, secure, and proctored manner. The individual questions are of uniformly high quality and are standardised by trained question writers following a standardised curriculum. The entire I-ITE is analysed and modified by skilled psychometric analysts. Therefore, we believe that the I-ITE should be highly attractive to paediatric residencies internationally as few paediatric departments have the infrastructure or the faculty with this breadth of technical skills.

The I-ITE can, therefore, be used to provide consistent and detailed formative feedback to individual residents on their pediatric knowledge throughout their residency. The I-ITE is independent of faculty group, in contrast to programs where local faculty design end-of-year internal exams which determine promotion to the next year of training. This lack of reliance on local design leads to recognition of the I-ITE as a tool to monitor the ability of a residency program in general to meet its training requirements over time.

International training collaboration

These results are highly suggestive that residency organisation and restructuring, through collaborative efforts between local and international faculty, can significantly improve the knowledge competency of residents. The HRH Program aims not only to create a larger, diversified health workforce capable of providing the people of Rwanda with high quality healthcare, but also, through strengthening of academic institutions to develop capacity within Rwanda to carry forward with their own high-quality programs. This study demonstrates that the HRH Program can have an impact on narrowing the gap between resource rich countries and low-resource countries that are facing a shortage of healthcare workforce (including Faculty). However, other competencies cannot be measured with a standardised computer-based testing mechanism and must still be assessed on site. Most importantly, whether and how the increase in scores ultimately impacts clinical practice and patient care remains to be studied.

Strengths of the study

This is the first study to report the use the I-ITE to measure performance in a resource-limited setting. The results are based on a standardised assessment and therefore void of local bias.

Limitations of the I-ITE

There are several limitations to our study. The I-ITE is best taken in the first few months of the academic calendar in order to more closely match scores with US counterparts: this was never accomplished due to organisational and financial constraints. Because of administrative and financial issues, we were unable to implement the I-ITE in the 2014–15 academic year, and therefore one data cycle is absent.

The I-ITE only assesses medical knowledge, which is only one of the competencies assessed in a high-quality residency training program. However, the I-ITE has given us a standardised format within which to measure resident medical knowledge. Our personal experience is that clinical skills, attitudes, communication, etc. have also seen parallel improvements during the course of the HRH program.

It is unclear how improved I-ITE scores will impact patient care or residents in the long run. In the US and Canada, ITE scores are used as a predictor of passing internal medicine speciality and subspecialty board examinations [5, 30,31,32,33]. There is no objective way to judge if the higher I-ITE scores translate to improved local examination pass rates as there are no standardised speciality certification examinations in Rwanda.

Conclusions

We demonstrate that the use of a standardised, well-designed, and independent exam available globally can be a powerful tool to monitor the knowledge acquisition of individual residents throughout training. Moreover, we believe it can reflect the capability of residency programs to provide the requisite training. Rwandan pediatric residents improved their pediatric knowledge scores on the I-ITE over a six-year period, both upon entry to residency and throughout their training, possibly reflecting the significant curricular reorganisation accomplished through the HRH Program.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available but are available from the corresponding author on reasonable request and consultation with the authorising IRB.

Abbreviations

ABP:

American Board of Pediatrics

CHAI:

Clinton Health Access Initiative

HRH:

Human Resources for Health

I-ITE:

International In-Training Examination

IRB:

Institutional review board

ITE:

In-Training Examination

LMICs:

Low-income and middle-income countries

MCQs:

Multiple-choice questions

PG:

Postgraduate

STROBE:

Strengthening the Reporting of Observational Studies in Epidemiology

UAE:

United Arab Emirates

UR:

University of Rwanda

US:

United States

References

  1. Cartledge P, Wachira J, Fagan C. Paediatric associations and postgraduate training programmes in Africa. Arch Dis Child. 2013;98:745–6.

    Article  Google Scholar 

  2. Freed GL, Dunham KM, Switalski KE, Jones MD, McGuinness GA. Recently trained general pediatricians: perspectives on residency training and scope of practice. Pediatrics. 2009;123:S38–43.

    Article  Google Scholar 

  3. Burg FD, Wright FH. Evaluation of pediatric residents and their training programs. J Pediatr. 1972;80:183–9.

    Article  Google Scholar 

  4. The American Board of Pediatrics. General pediatrics in-training examination. In: Am. Board. Pediatr; 2018. Available from: https://www.abp.org/content/general-pediatrics-training-examination. Cited 10 Dec 2018.

    Google Scholar 

  5. Du Y, Furter RT, Tan RJB, Althouse LA. In-training examinations: All about predictive validity and meaningful feedback. Paper presented at: American Educational Research Association Meeting 2016 Annual Conference; Washington, DC.

  6. The American Board of Pediatrics. ABP report: is there an effect on predictive validity when shortening an in-training examination? 2008.

    Google Scholar 

  7. Ham HP. Assessing general pediatric knowledge during post-graduate training - Report. Am Board Pediatr. 

  8. Rwanda National Institute of Statistics. Population size , structure and distribution - report; 2012. p. 1–86. Available from: http://statistics.gov.rw/publication/rphc4-thematic-report-population-size-structure-and-distribution

    Google Scholar 

  9. The World Bank. Current health expenditure per capita (current US$); 2019. p. 1–18. Available from: https://data.worldbank.org/indicator/SH.XPD.CHEX.PC.CD?end=2015&name_desc=true&start=2015&view=map&year=2015. Cited 2 May 2019

    Google Scholar 

  10. Rwanda National Institute of Statistics. Gross domestic product and its structure - report; 2018. p. 1–11. Available from: http://www.statistics.gov.rw/publication/gdp-national-accounts-2018

    Google Scholar 

  11. Binagwaho A, Kyamanywa P, Farmer PE, Nuthulaganti T, Umubyeyi B, Sc MN, et al. The human resources for health program in Rwanda — a new partnership. NEJM. 2010;369:2054–9.

    Article  Google Scholar 

  12. Cancedda C, Riviello R, Wilson K, Scott KW, Tuteja M, Barrow JR, et al. Building workforce capacity abroad while strengthening Global Health programs at home. Acad Med. 2017;92:649–58.

    Article  Google Scholar 

  13. Cancedda C, Cotton P, Shema J, Rulisa S, Riviello R, Adams LV, et al. Health professional training and capacity strengthening through international academic partnerships: the first five years of the human resources for health program in Rwanda. Int J Heal Policy Manag. 2018;7:1024–39.

    Article  Google Scholar 

  14. Kabakambira J, Mugeni R, Ho J, Sofair A, Walker T. Improvement of medical education through the establishment of a medical chief resident position in Rwanda. Rwanda Med J. 2017;74:17–20.

    Google Scholar 

  15. Ewing H, Pyburn R, Asingizwe D, Asiimwe-Kateera BA, Njunwa K, Kim MJ. Facilitating health professional scholarship through a workshop series at the University of Rwanda. Rwanda Med J. 2017;74:1–5.

    Google Scholar 

  16. Ndayambaje A, Anderson R, Yoder CMK, Ewing H, Thomson DR. Human resources for health (HRH) midwives mentoring program and episiotomy rates at muhima hospital, Rwanda: a retrospective cross-sectional study. Rwanda Med J. 2017;74:7–11.

    Google Scholar 

  17. Rowe AK, Rowe SY, Peters DH, Holloway KA, Chalker J, Ross-degnan D. Articles effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review. Lancet Glob Heal. 2018;6:e1163–75 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license.

    Article  Google Scholar 

  18. Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4:1623–7.

    Google Scholar 

  19. Althouse LA, McGuinness GA. The in-training examination: an analysis of its predictive value on performance on the general pediatrics certification examination. J Pediatr. 2008;153:425–8.

    Article  Google Scholar 

  20. Boody B, Johnson P, Pugely A, Miller D, Geller J, Payne W, et al. Validation of a web-based curriculum for resident education in orthopedic surgery. J Surg Educ. 2016;73:1060–5 Elsevier.

    Article  Google Scholar 

  21. Langenau EE, Fogel J, Schaeffer HA. Correlation between an email based board review program and American Board of Pediatrics General Pediatrics Certifying Examination scores. Med Educ Online. 2009;14:1–8.

    Article  Google Scholar 

  22. Webb TP, Paul J, Treat R, Codner P, Anderson R, Redlich P. Surgery residency curriculum examination scores predict future American board of surgery in-training examination performance. J Surg Educ. 2014;71:743–7 Elsevier.

    Article  Google Scholar 

  23. Kim RH, Tan TW. Interventions that affect resident performance on the american board of surgery in-training examination: a systematic review. J Surg Educ. 2015;72:418–29 Elsevier.

    Article  Google Scholar 

  24. Ferrell BT, Tankersley WE, Morris CD. Using an accountability program to improve psychiatry resident scores on in-service examinations. J Grad Med Educ. 2015;7:555–9.

    Article  Google Scholar 

  25. Global Pediatric Education Consortium. Global pediatric curriculum and guidelines for residency training, assessment, certification, and continuous professional development. Available from: http://www.globalpediatrics.org/globalcurriculum/curriculumdownloadpdf.html. Cited 23 Aug 2018.

  26. Franklin CC, Bosch PP, Grudziak JS, Dede O, Ramirez RN, Mendelson SA, et al. Does a weekly didactic conference improve resident performance on the pediatric domain of the Orthopaedic in-training examination? J Pediatr Orthop. 2017;37:149–53.

    Article  Google Scholar 

  27. Wirth K, Malone B, Turner C, Schulze R, Widmann W, Sanni A. A structured teaching curriculum for medical students improves their performance on the national board of medical examiners shelf examination in surgery. Am J Surg. 2015;209:765–70 Elsevier Inc.

    Article  Google Scholar 

  28. Kaliyadan F, Thalamkandathil N, Parupalli S, Amin T, Balaha M, Al Bu Ali W. English language proficiency and academic performance: a study of a medical preparatory year program in Saudi Arabia. Avicenna J Med. 2015;5:140–4.

    Article  Google Scholar 

  29. Mathis BR, Warm EJ, Schauer DP, Holmboe E, Rouan GW. A multiple choice testing program coupled with a year-long elective experience is associated with improved performance on the internal medicine in-training examination. J Gen Intern Med. 2011;26:1253–7.

    Article  Google Scholar 

  30. Grabovsky I, Hess BJ, Haist SA, Lipner RS, Hawley JL, Woodward S, et al. The relationship between performance on the infectious diseases in-training and certification examinations. Clin Infect Dis. 2015;60:677–83.

    Article  Google Scholar 

  31. Flannery MT. Trends in the history of certification and recertification of the American board of internal medicine. Am J Med Sci. 2014;347:74–7 Elsevier Masson SAS.

    Article  Google Scholar 

  32. O’Neill TR, Li Z, Peabody M, Lybarger M, Royal K, Puffer JC. The predictive validity of the ABFM’s in-training examination. Fam Med. 2015;47:349–56.

    Google Scholar 

  33. Yen D, Athwal GS, Cole G. The historic predictive value of Canadian orthopedic surgery residents’ orthopedic in-training examination scores on their success on the RCPSC certification examination. Can J Surg. 2014;57:260–2.

    Article  Google Scholar 

Download references

Acknowledgments

The authors wish to thank Valerie Haig, Dr. Robert Furter and Donna Crisp of the ABP; Pr. Narcisse Muganga, Dr. Jean-Claude Kabayiza, Dr. Aimable Musafili, Dr. Emmanuel Rusingiza of the University of Rwanda and the Pediatric Post-graduates from the University of Rwanda.

Funding

Authors have no financial relationships relevant to this study to disclose. No external funding was sought nor received for this study or the manuscript. The I-ITE fees were paid to the ABP. The first I-ITE exam fee was paid to the ABP by the Clinton Health Access Initiative (CHAI) who managed the first year of the HRH program on behalf of the Rwandan Ministry of Health. The second I-ITE cycle was paid by one of the co-investigators personal funds (CO). The University of Rwanda paid the remaining sittings through the HRH budgets. The corresponding author had full access to all the data in the study and hold the final responsibility for the decision to submit for publication.

Author information

Authors and Affiliations

Authors

Contributions

NMC conceptualised and designed the study, oversaw data collection and analysis, wrote the initial draft of the manuscript and reviewed and revised the final manuscript. CU conceptualised and designed the study, oversaw data collection and analysis, drafted the initial manuscript and reviewed and revised the final manuscript. AK consulted on study design, reviewed and revised the final manuscript. TR analysed the result of the literature search and reviewed and revised the manuscript. DS conceptualised the study, contributed and revised the manuscript. CO conceptualised and designed the study, drafted the initial manuscript and reviewed and revised the manuscript. PTC conceptualised and designed the study, carried out the literature search and data analysis, drafted the initial manuscript and reviewed and revised the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Natalie McCall.

Ethics declarations

Ethics approval and consent to participate

Ethical approval has been granted by the Institutional review board (IRB) of the University of Rwanda (UR) on 02/14/2018. Ref: 027/CMHS_IRB/2018.

Consent for publication

Not applicable.

Competing interests

We declare no competing interest: One author (CO) is an unpaid member of the ABP Global Health taskforce. All authors are members of the HRH program and Rwandan academic staff of the University of Rwanda.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

McCall, N., Umuhoza, C., O’Callahan, C. et al. Measuring change in knowledge acquisition of Rwandan residents: using the American Board of Pediatrics International In-Training Examination (I-ITE) as an independent tool to monitor individual and departmental improvements during the Human Resources for Health program: an observational study. BMC Med Educ 19, 217 (2019). https://0-doi-org.brum.beds.ac.uk/10.1186/s12909-019-1617-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12909-019-1617-8

Keywords