Primary health care is a complex world and remains challenging to research. Quality research requires valid tools. Physician behaviour—a constant area of inquiry for primary care researchers—is often best measured by auditing medical charts. These chart audits can provide excellent measures of physical examination, prescribing, laboratory procedures, and specialist referrals.1
Although many studies have used chart audits, there is little practical information on how to perform them. This paper addresses that gap, providing practical information on conducting chart audits based on our experience with many large-scale primary health care research projects in Ontario.2–4
In one of our large primary care studies, the Comparison of Models Study in Primary Care (COMP-PC), we wanted to understand how a range of primary care services were delivered in Ontario. We complemented our surveys of patients, practices, and clinicians with a comprehensive chart audit in each of the 137 primary care practices studied. Appendices A to E,* which include parts of the manual from the COMP-PC project, are available for readers to adapt and use in their own practice-based primary health care research projects. For a copy of the complete chart audit manual, please contact the corresponding author.
Staff training
Chart abstractors (who are often nurses) should have a background in health care, so that they are familiar with the medical terminology used in the charts. They need to be comfortable with the information technology being used in the study, such as laptop computers and data entry software. Chart abstractors should have training opportunities to conduct real chart abstractions at practice sites; these abstractions should be compared with those of others to augment their learning.
Training for the COMP-PC project took place over 2 days and comprised in-depth coverage of the chart abstractor instruction manual. This was followed by 1 day in the field with an experienced chart abstractor. Once in the field on their own, chart abstractors had a toll-free telephone number to call for access to experienced chart abstractors or investigators to resolve questions as they arose.
Preparing for data collection
We designed a chart abstraction manual that provided abstractors with detailed information about methods for making initial contact with the practice, including a draft script; the chart abstraction process, including eligibility and data entry information; the data collection validation process; an annotated copy of the chart abstraction form; and a chart abstraction tracking log. It is important to develop a comprehensive training manual, recognizing that the realities of working with diverse practices might require adapting initial plans for data collection. It is necessary to document changes and decision points as the study progresses; this includes noting the reason for each change and the investigators’ approval for these changes.
In the COMP-PC project, the sample size was calculated to detect a 0.5 difference in the standard deviation of the prevention score at the .05 level. Accounting for the clustered structure of the data and allowing for β of .20, we required 30 charts to be reviewed in each of the 40 practices in each model.
Pilot-testing the chart abstraction form is critical. In the COMP-PC project, the chart audit was pilot-tested in 6 practices, and changes were made based on these experiences before data collection began for the study.
Selecting charts
Random selection of charts is important to provide generalizable results. For paper-based charting, we suggest using a “tape measure method,” in which the total length of shelves with charts is measured and then divided into sections that are similar distances apart. Then the chart that is a fixed number (eg, fifth) from the start of each one of these sections is chosen. If offices have electronic medical records, the sample is produced using a random-number generator. In practices with mixed paper and electronic records, the “tape measure method” is usually used; however, once an eligible chart is located for abstraction, the abstractor should verify if more information is available in the corresponding electronic files.
Reliability and validity
Having a plan to assess interrater reliability is critical when conducting chart audits (Liddy et al, unpublished data, 2009). The manual provides details on this method, which compares chart abstractions completed by 2 independent abstractors working with the same charts. Duplicate data entry should be used to ascertain data entry error rates. Chart abstractors should receive feedback based on this exercise and, where necessary, additional training should be provided to improve data quality.2
Budgeting
Costs for chart audits include compensation for time and travel. The time it takes to abstract each chart depends on the number of data elements being abstracted. As part of the development of the research, it is important to assess what data elements are truly necessary. For the COMP-PC study, abstracting 30 charts per practice took an average of 20 hours per practice.
In 2006, the COMP-PC project paid chart abstractors (and re-abstractors) $30 an hour plus benefits. Supervisors overseeing the abstractors earned $34 per hour. Travel costs depended on the locations of the practice sites. Each practice received an honorarium of $2000 to compensate for any disruption suffered during the data collection (which included patient waiting room surveys, provider surveys, and practice administrator surveys, in addition to chart audits).
Conclusion
Chart audits continue to be an important technique in practice-based primary health care research. Further research is needed to continue to develop our understanding and refine these techniques. Sharing research tools might enable the research community to collectively improve them and improve our ability to generate essential knowledge about primary health care.
Hypothesis is a quarterly series in Canadian Family Physician, coordinated by the Section of Researchers of the College of Family Physicians of Canada. The goal is to explore clinically relevant research concepts for all CFP readers. Submissions are invited from researchers and nonresearchers. Ideas or submissions can be submitted online at http://mc.manuscriptcentral.com/cfp or through the CFP website www.cfp.ca under “Authors.”
Footnotes
Appendices A to E are available at www.cfp.ca. Go to the full text of this article online, then click on CFPlus in the menu at the top right-hand side of the page.
Competing interests
None declared
References
- 1.Robitaille A, Hogg W, Kristjansson E. Comparison of methods used for evaluating prevention and health promotion in primary care: a systematic review. Paper presented at: Annual CAHSPR Conference; May 11–13, 2009; Calgary, Alta. [Google Scholar]
- 2.Dahrouge S, Hogg W, Russell G, Geneau R, Kristjansson E, Muldoon L, et al. The Comparison of Models of Primary Care in Ontario (COMP-PC) study: methodology of a multifaceted cross-sectional practice-based study. Open Med. 2009;3(3):149–64. [PMC free article] [PubMed] [Google Scholar]
- 3.Hogg W, Lemelin J, Graham ID, Grimshaw J, Martin C, Moore L, et al. Improving prevention in primary care: evaluating the effectiveness of out-reach facilitation. Fam Pract. 2008;25(1):40–8. doi: 10.1093/fampra/cmm070. Epub 2008 Jan 21. [DOI] [PubMed] [Google Scholar]
- 4.Champlain Cardiovascular Disease Prevention Network [website] Improved Delivery of Cardiovascular Care (IDOCC) program. Ottawa, ON: Champlain Cardiovascular Disease Prevention Network; 2009. Available from: www.idocc.ca/en_IDOCC.php. Accessed 2010 Apr 7. [Google Scholar]