Abstract
Background
Residents' ability to interpret statistics is important for scholarly pursuits and understanding evidence-based medicine. Yet there is limited research assessing residents' statistical literacy and their training in statistics.
Methods
In 2011 we surveyed US obstetrics-gynecology residents participating in the Council for Resident Education in Obstetrics and Gynecology In-Training Examination about their statistical literacy and statistical literacy training.
Results
Our response rate was 95% (4713 of 4961). About two-thirds (2980 of 4713) of the residents rated their statistical literacy training as adequate. Female respondents were more likely to rate their statistical literacy training poorly, with 25% (897 of 3575) indicating inadequate literacy compared with 17% (141 of 806) of the male respondents (P < .001). Respondents performed poorly on 2 statistical literacy questions, with only 26% (1222 of 4713) correctly answering a positive predictive value question and 42% (1989 of 4173) correctly defining a P value. A total of 51% (2391 of 4713) of respondents reported receiving statistical literacy training through a journal club, 29% (1359 of 4713) said they had informal training, 15% (711 of 4713) said that they had statistical literacy training as part of a course, and 11% (527 of 4713) said that they had no training.
Conclusions
The findings suggest that statistical literacy training for residents could still be improved. A total of 37% (1743 of 4713) of obstetrics-gynecology residents have received no formal statistical literacy training in residency. Fewer residents answered the 2 statistical literacy questions correctly compared with previous studies.
What was known
Residents' ability to interpret statistics is important in practice-based learning and improvement, and in understanding key concepts of evidence-based medicine, yet there is little research exploring residents' statistical literacy and training.
What is new
More than one-third of obstetrics-gynecology residents reported receiving no formal statistical literacy training in residency.
Limitations
This is a brief survey study that may not provide a thorough assessment of statistical literacy.
Bottom line
Statistical knowledge and training among residents could be improved.
Introduction
Statistical literacy can be defined as understanding the statistical aspects and terminology associated with the design, analysis, and conclusions of original research. Statistical literacy is essential for appraising and assimilating scientific evidence, which is part of the practice-based learning and improvement competency.1 Despite statistical literacy's important role in applying the principles of evidence-based medicine, it is not a major focus of evidence-based medicine curricula in medical school or residency.2
A previous survey study assessed internal medicine residents' opinions regarding their statistical literacy skills, and although 95% of respondents reported that it was important to understand statistical concepts to understand medical literature, 75% reported that they did not understand all of the statistics they encountered in journals.3 This previous study, which had a response rate of 75%, was conducted in 11 hospitals. Little is known about whether these findings generalize to other specialties or national samples.
Our study is an assessment of obstetrics-gynecology (Ob-Gyn) residents' experiences and opinions regarding their statistical literacy training. We surveyed residents about statistical literacy training during their annual Council for Resident Education in Obstetrics and Gynecology (CREOG) In-Training Examination, which is completed by all Ob-Gyn residents. The study had 3 objectives: (1) to assess performance on 2 statistical literacy questions and compare the results with the performance of residents and practicing physicians in previous studies, (2) to assess the quality and format of statistical training during Ob-Gyn residency, and (3) to examine how residents' statistical literacy training is associated with performance on the 2 statistical literacy questions.
Methods
Questions about statistical literacy were included in a larger survey that was administered to all residents in obstetrics and gynecology (postgraduate years 1–4) in 2011 before their CREOG In-Training Examination. All residents were informed about the purpose of the study and were asked to return the survey by the end of the day or to mail it to the American College of Obstetricians and Gynecologists. Residents could complete the survey at any point throughout or beyond the time limits of the examination day, participation was voluntary, no incentive was provided, and there was no penalty for noncompletion. Performance on the survey questions did not influence their examination scores.
The survey consisted of 4 questions. Two asked about residents' statistical literacy training (including training format) and examinees' rating of the quality of their statistics training as excellent (1), adequate (2), or inadequate (3). An additional 2 questions assessed residents' statistical literacy. Before the questions, the following definition of statistical literacy was provided: “Statistical literacy can be defined as understanding the statistical aspects of and terminology associated with the design, analysis, and results of original research.” A mean total rating score was computed, with higher scores indicating less adequate training (the range of possible scores was 5–15). The first performance question was a positive predictive value calculation that has been used in previous research,4–6: “Ten out of every 1,000 women have breast cancer. Of these 10 women with breast cancer, 9 test positive. Of the 990 women without cancer, about 89 nevertheless test positive. A woman tests positive and wants to know whether she has breast cancer for sure, or at least what the chances are. What is the best answer?” The response options were: (1) The probability that she has breast cancer is about 90%; (2) The probability that she has breast cancer is about 81%; (3) Out of 10 women with a positive mammogram, about 9 have breast cancer; (4) Out of 10 women with a positive mammogram, about 1 has breast cancer; and (5) The probability that she has breast cancer is about 1%. (The correct answer was 4.) The second performance question read, “True or False: The P value is the probability that the null hypothesis is correct.” (The correct answer is false. The P value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.)
Institutional Review Board approval was obtained from the American College of Obstetrics and Gynecology.
Results
The survey response rate was 95% (4713 of 4961). The table shows the demographic variables, the type of training setting, quality of training, and overall performance on the knowledge questions. When asked to identify the positive predictive value of a positive mammography screen, only 26% (1222 of 4713) chose the correct multiple choice option (1 of 10 with a positive mammography have cancer); 26% (1213 of 4713) answered “90%”; 13% (632 of 4713) answered “1%”; 11% (518 of 4713) answered “81%”; 9% (413 of 4713) answered “9 out of 10”; and 15% (715 of 4713) did not answer or indicated more than one response. A total of 42% (1989 of 4713) correctly answered the P value question, 46% (2184 of 4713) incorrectly said true, and 12% (540 of 4713) did not answer. A total of 12% (558 of 4713) answered both the positive predictive value and the P value questions correctly. No differences were found based on residency year or sex of the respondent.
Table.
Demographics and Descriptive Statistics
Abbreviation: PGY, postgraduate year.
aQuestion asked, "Please rate the quality of the formal training you received in your residency program regarding each of the following." Scale is 1 (excellent) to 3 (inadequate).
bQuestion asked, "Ten out of every 1,000 women have breast cancer. Of these 10 women with breast cancer, 9 test positive. Of the 990 women without cancer, about 89 nevertheless test positive. A woman tests positive and wants to know whether she has breast cancer for sure, or at least what the chances are. What is the best answer?"
cQuestion asked, "True or False: The P value is the probability that the null hypothesis is correct."
A total of 51% (2391 of 4713) of respondents reported receiving statistical literacy training through a journal club, 29% (1359 of 4713) said they had informal training, 15% (711 of 4713) said that they had statistical literacy training as part of a course, and 11% (527 of 4713) said that they had no training. Because responders could indicate more than one type of training, the table shows the breakdown by the percentage of respondents who indicated each combination of training.
Although 63% (2980 of 4713) rated their overall statistical literacy training as adequate, 22% (1040 of 4713) rated it as inadequate, and 8% (365 of 4713) rated it as excellent (7% [328 of 4713] did not answer). Women were more likely to rate their overall statistical literacy training poorly, with 25% (897 of 3575) indicating inadequate training compared with 17% (141 of 806) of the men (χ2 = 54.4; P < .001). Mean ratings of statistical training quality are shown in the table. Ratings on all of the different types of statistical training were added for a total training quality score. Total training quality scores ranged from 5 (6%) to 15 (15%). The mode total was 10 (55%).
We assessed whether ability was associated with quality of training. Those who answered the positive predictive value question correctly rated their training significantly less adequate compared with examinees who answered it incorrectly (F6,4354 = 5.7; P < .001). This association was not found with the P value question.
Finally, we assessed whether training setting or ability was associated with residency location. Residents in community programs were more likely to indicate receiving statistical literacy training in a course only (χ2 = 10.6; P = .001), and residents in university programs were more likely to indicate receiving informal statistical literacy training only (χ2 = 6.2; P = .01). Residents in community and university programs were equally likely to receive training through journal clubs and were equally likely to receive no training. There were no differences in ratings of adequacy of training between program types. However, those in university residency programs were more likely to answer both the predictive value and the P value questions correctly (379 of 2588; 15%) than those in community residency programs (173 of 1482; 12%; χ2 = 5.4; P = .02).
Discussion
The results of this study show that statistical knowledge and training among residents could be improved, that 37% of Ob-Gyn residents reported having received no formal statistical literacy training in residency, and that residents most commonly receive training in a journal club settings.
The residents in this sample performed slightly worse than those in previous samples on the statistical literacy knowledge questions. Compared with a previous study of residents where 59% correctly identified the meaning of a P value,3 42% of this sample correctly answered the true/false question about the definition of a P value. With regard to the positive predictive value question, 25% of this resident sample answered the question correctly, compared to 45%5 and 49%6 of practicing Ob-Gyns. Physicians' misunderstanding of screening test results has been shown in other research on human immunodeficiency virus screening.7
It was notable that those who answered the positive predictive question correctly rated their statistical literacy training more poorly than those who answered the question incorrectly. In other words, residents who need additional statistical literacy training may be the least likely to recognize the inadequacy of their training.
Female respondents were more likely than male respondents to rate their training as inadequate, despite the lack of sex differences on the 2 ability questions in our study. With 80% of residents in obstetrics and gynecology being women, it is important to address feelings of inadequate training among female Ob-Gyns.
Similar to our results, previous studies have reported that journal clubs are the prominent format for learning evidence-based medicine skills for both university and community residency programs.2 Although residents in community residency programs were more likely to indicate that they had received statistical literacy training in a course, residents at university programs were more likely to complete the performance questions correctly. Different types of evidence-based medicine education in both university and community residency programs, such as integrated training, which has been found to be more effective than current training methods, could be useful.8
Our study has several limitations. Some terms, such as residency location of “community” versus “university,” were not defined on the survey for participants and may have been interpreted in inconsistent ways. The short survey length limited our ability to assess statistical literacy knowledge thoroughly. Finally, examinees could have experienced test fatigue, and because their responses did not count toward their examination score, they may have spent little time answering the questions.
Conclusion
Our findings show that statistical knowledge and training among residents could still be improved. A total of 37% (1743 of 4713) of Ob-Gyn residents have received no formal statistical literacy training in residency. Future research should assess whether low statistical literacy skills among residents and practicing physicians has an impact on quality of care.
Footnotes
All authors are with the American College of Obstetricians and Gynecologists. Britta L. Anderson, PhD, is a Research Associate in the Practice Division in the Department of Research; Sterling Williams, MD, is Vice President of the Division of Education; and Jay Schulkin, PhD, is Director of Research of the Practice Division in the Department of Research.
Funding: This study was funded in part by grant UA6MC19010 through the US Department of Health and Human Services, Health Resources and Services Administration, Maternal and Child Health Research Programs.
References
- 1.Accreditation Council for Graduate Medical Education. http://www.acgme.org/acwebsite/RRC_280/280_corecomp.asp. Accessed September 17, 2012. [Google Scholar]
- 2.Rao G, Kanter SL. Physician numeracy as the basis for an evidence-based medicine curriculum. Acad Med. 2010;85:1794–1799. doi: 10.1097/ACM.0b013e3181e7218c. [DOI] [PubMed] [Google Scholar]
- 3.Windish DM, Huot SJ, Green ML. Medicine residents' understanding of biostatistics and results in the medical literature. JAMA. 2007;298(9):1010–1022. doi: 10.1001/jama.298.9.1010. [DOI] [PubMed] [Google Scholar]
- 4.Gigerenzer G, Gaissmaier W, Kurz-Mielke E, Schwartz LM, Woloshin S. Helping doctors and patients make sense of health statistics. Psychol Sci Public Interest. 2007;8(2):53–96. doi: 10.1111/j.1539-6053.2008.00033.x. [DOI] [PubMed] [Google Scholar]
- 5.Anderson BL, Leddy M, Schulkin J. Physician numeracy and associations with breast cancer care. Poster presented at 32nd Annual Meeting & Scientific Sessions of the Society of Behavioral Medicine; April 2011; Washington, DC. [Google Scholar]
- 6.Anderson BL, Gigerenzer G, Parker S, Schulkin J. Statistical literacy in physicians: performance on a physician numeracy scale among obstetrician-gynecologists. J Healthc Qual. doi: 10.1111/j.1945-1474.2011.00194.x. [Google Scholar]
- 7.Gigerenzer G, Hoffrage U, Ebert A. AIDS counseling for low-risk clients. AIDS Care. 1998;10(2):197–211. doi: 10.1080/09540129850124451. [DOI] [PubMed] [Google Scholar]
- 8.Coomarasamy A, Khan KS. What is the evidence that postgraduate teaching in evidence based medicine changes anything?: a systematic review. BMJ. 2004;329(30):1017. doi: 10.1136/bmj.329.7473.1017. [DOI] [PMC free article] [PubMed] [Google Scholar]