Setting and Problem
In the new accreditation system, scholarly activity by faculty and residents is a requirement for residency program accreditation by the Accreditation Council for Graduate Medical Education (ACGME). Residency programs are required to report information annually via the ACGME's Accreditation Data System (ADS), through tables in the ADS update that document faculty and resident scholarly activity for the previous completed academic year (AY). PubMed IDs (PMIDs) are used as 1 form of objective evidence for measuring scholarly activity. Often, PMIDs are flagged as invalid because the publication date was outside the previous AY, an outcome we termed “out-of-range.” We sought to improve the efficiency of the annual ADS update by developing a method for reducing errors in scholarly activity submissions.
Intervention
Given the large volume of data submitted to the ACGME, we developed an automated web service that could be used by the graduate medical education office or program coordinators to screen resident- and faculty-reported PMIDs for errors. The annual update scholarly activity monitoring (AUSAM) tool was developed to verify the accuracy of PMID submissions for each training program, using data on PMIDs recorded by PubMed. AUSAM prioritizes dates of publication in the following order: Date of Electronic Publication (DEP; also known as the ePub date); Date of Publication (DP; the date on which the article was published in print format); and, if the DP is ambiguous (eg “Summer 2014”), the Entrez Date (EDAT; the date the citation was added to PubMed). AUSAM was deployed in August 2015, and all ADS update submissions for AY 2014–2015 were prescreened using the tool. Our study evaluating AUSAM was exempt from Institutional Review Board approval.
We initially collected annual ADS update data for AY 2013–2014 for all accredited residency and fellowship programs sponsored by Henry Ford Hospital in Detroit, Michigan. Faculty and resident scholarly activity tables were copied and pasted into the AUSAM tool, which automatically flagged invalid and out-of-range PMIDs (outside July 1, 2013 to June 30, 2014). The process was repeated after AUSAM deployment for ADS updates submitted for AY 2014–2015 (July 1, 2014 to June 30, 2015). Pearson χ2 was used to compare the proportion of errors before and after deployment of AUSAM, and a P value less than .05 was considered significant.
Outcomes to Date
The study included ADS updates from 49 programs that were utilized for each academic year. For AY 2013–2014 prior to AUSAM's deployment, 662 unique PMIDs were identified, and 118 of them (17.8%) were identified as out-of-range. After the tool was deployed for AY 2014–2015, a total of 545 unique PMIDs were identified, with 22 PMIDs (4.0%) identified as out-of-range. There was a statistically significant reduction in error rates (17.8% versus 4.0%) after the tool was deployed (table; P < .0001). Overall, 17.6% of faculty and 18.6% of resident PubMed submissions were found to be out-of-range in AY 2013–2014, versus 3.2% of faculty and 7.2% of resident submissions in AY 2014–2015. There was no significant difference in error rates found between faculty and residents for AY 2013–2014 (P = .73) versus AY 2014–2015 (P = .06).
Table.
Comparison of Flagged PubMed ID Submissions Before and After Annual Update Scholarly Activity Monitoring Deployment
After manually evaluating all flagged PMIDs, we found no incorrectly flagged results. In addition, we found that AUSAM was time-efficient. More than 1200 unique PMIDs were screened and manually rechecked in less than 20 hours. The tool is intuitive and requires minimal computer literacy. AUSAM is being offered as a free, unrestricted service for the graduate medical education community (http://www.henryfordem.com/ausam). Although AUSAM could be immensely valuable for large academic medical centers, it may have limited utility for individual programs.
In conclusion, use of AUSAM reduces the rate of errors in the submission of scholarly activity. The tool is free and easy to use, and it has the potential to significantly improve the efficiency of annual ADS updates at a national level.