Comparison of Performance on ADHD Quality Indicators: Practitioner Self-Report Versus Chart Review

Abstract

Objective

This study compared practitioner self-report of ADHD quality of care measures with actual performance, as documented by chart review.

Methods

188 practitioners from 50 pediatric practices completed questionnaires in which they self-reported estimates of ADHD quality of care indicators. A total of 1599 charts were reviewed.

Results

The percentage of patients for whom practitioners self-reported that they employed evidence-based care was higher in every performance category when compared with chart review, including higher use of parent and teacher rating scales during assessment and treatment compared to chart review. Self-reported use of DSM-IV criteria during assessment was also higher than by chart review. The actual number of days until the first contact after starting medication was nearly 3 times longer than self-report estimates.

Conclusions

Practitioners over-report performance on quality of care indicators. These differences were large and consistent across ADHD diagnostic and treatment monitoring practices. Practitioner self-report of ADHD guideline adherence should not be considered a valid measure of performance.

The American Academy of Pediatrics (AAP) released consensus guidelines defining evidence-based ADHD care in 2000 and 2001 and updated them in 2011 (American Academy of Pediatrics, 2000, 2001, 2011). These guidelines recommended that pediatricians use both Diagnostic and Statistical Manual of Mental Disorders (DSM) criteria and standardized rating scales during assessments for ADHD and that they systematically monitor response to treatment.

Previous studies conducted using physician self-report suggest modest adherence to these guidelines (Chan, Hopkins, Perrin, Herrerias, & Homer, 2005; Rushton, Fant & Clark, 2004; Gardner, Kelleher, Pajer, & Camp, 2004). For example, physicians reported using parent and teacher rating scales during ADHD assessment for approximately 70% of patients, and 58% of physicians reported using formal DSM criteria to diagnose ADHD (Chan et al., 2005; Rushton et al., 2004). More recently, a physician self-report survey indicated that the majority of physicians were adhering to the AAP guideline for ADHD care, including frequent use of a validated screening tool and appropriate follow-up after initiation of medication (McElligott et al., 2014). However, another study using external chart review suggests that physician adherence to the AAP guidelines is lower than what has been reported in studies using physician self-report. In this study, chart reviews revealed that parent and teacher rating scales were used during assessment with 55% and 52% of patients, respectively, and rating scales were rarely ever used to assess treatment response (Epstein et al., 2008).

Physicians’ tendencies to over-estimate performance is a documented phenomenon that has led some investigators to raise concerns about the validity of data obtained by self-report (Davis et al., 2006; Adams, Soumerai, Lomas, & Ross-Degnan, 1999). For example, Adams et al. (1999) reported that physicians over-estimate performance by nearly 30% on a range of adult-focused quality indicators. In a systematic review of 20 self-to-external comparisons published in 17 articles, Davis et al. (2006) found self-report to be poorly correlated with actual performance and questioned the usefulness and accuracy of self-report as an assessment tool for physicians.

As ADHD is one of the most common pediatric behavioral health conditions (Perou et al., 2013), accurate assessment of practitioner performance is important. Recent studies have raised concerns about the possibility of ADHD over-diagnosis with limited reliance on diagnostic criteria (Bruchmüller et al, 2012). Health care quality has been identified as a key target area by the Institute of Medicine and patients alike (Institute of Medicine, 2001), and quality improvement efforts are a major focus area of the American Board of Pediatrics. An accurate assessment of care quality is necessary to determine if improvement is warranted and if improvement efforts are successful. Similarly, the shift from volume-based to value-based reimbursement (U.S Department of Health and Human Services, n.d.) requires an assessment of actual performance on quality indicators in order to fully realize this approach.

While the available literature would suggest that ADHD guideline adherence is likely higher as assessed by practitioner self-report as compared to chart review, no study has directly compared practitioner self-report to ADHD care documented by comprehensive chart review. Accordingly, the purpose of this study was to utilize a large sample that included both practitioner self-report and external chart review in order to systematically evaluate the validity of practitioner self-report of ADHD quality of care indicators.

Methods

This study was approved by the Institutional Review Boards at Cincinnati Children’s Hospital Medical Center and Nationwide Children’s Hospital. Data were collected from 50 practices and 188 practitioners (184 physicians and 4 nurse practitioners) in central and northern Ohio from August 2010 to December 2012. Practitioner characteristics are presented in Table 1. All physicians were pediatricians.

Table 1.

Practitioner Characteristics

Primary Practitioner Type	Physicians (98%)
Mean Age	43.5 years (SE = 0.7)
Average Time Since Training	12.9 years (SE = 0.7)
Race	White (86%)
Sex	Female (64%)
Practice Location	Suburban (55%) Urban (28%) Rural (9%) Unspecified (8%)
Practice Type	Non-Academically Affiliated (75%) Academically Affiliated (25%)
Previous Participation in ADHD Quality Improvement Activities	8%

Quality indicators used in both the self-report questionnaire and chart reviews were selected based upon key components of the AAP’s most recent ADHD guideline, such that deviation from this guideline would indicate lower-quality care. All practitioners completed the self-report questionnaire in which they estimated the frequency of their ADHD care behaviors consistent with the AAP ADHD guidelines for the assessment and treatment of ADHD. The questionnaire was based upon the survey tool created by Kwasman, Tinsley, and Leper and modified to include self-report of ADHD care behaviors (1995).

A random sample of at least 10 charts per practitioner was selected by billing audit of ADHD cases seen in the last year. In order to review both assessment and treatment behavior, at least 5 charts per practitioner with an ADHD assessment were selected. A total of 1599 charts of children in grades 1–5 were reviewed for the presence of absence of ADHD care behaviors across all 188 practitioners. There was overlap between assessment and treatment charts. Since these chart reviews required a retrospective review of patient charts, a waiver of consent was granted on the condition that we abstracted only information related to quality of ADHD care and no identifying or demographic information from the patient charts be recorded Research assistants were trained to use a standardized chart review procedure using a structured form. Ten percent of charts were double rated to ensure consistency (intra-class correlation = 0.86 for days to first contact; kappa = 0.71 for all other variables). Chart review data included the percentage of patients for whom parent or teacher ADHD rating scales were collected during assessment, the percentage of patients for whom documentation of DSM-IV ADHD criteria was present during assessment, the time to first contact after the initiation of medication, and the percentage of patients for whom parent or teacher ADHD rating scale were used to monitor treatment response. Documentation of DSM-IV criteria was defined as the presence of completed parent and teacher rating scales or the presence of an assessment report from a specialty provider. Time to first contact after the initiation of medication was determined by comparing the date of first prescription (according to chart review) to date of first contact via either an in person encounter, e-mail, or telephone call. For the purposes of the study, the use of ADHD rating scales during treatment was fulfilled if any ADHD rating scale completed after being prescribed medication was in the chart during the first year of treatment.

Chart review data about each practitioner were compared against the practitioner’s self-reports on the same quality domains. To compare practitioner self-reports against chart review results, we calculated the difference between them. We then tested the null hypothesis that the mean of the difference scores equaled zero. These tests were conducted using a linear mixed-effects model that tested the null hypothesis that the mean of the practitioners’ difference scores – that is, the intercept of a linear mixed model – was zero versus a two-sided alternative hypothesis. The physician self-report data were modeled to reflect nesting at the practice level using a random intercept associated with the practice to account for variance attributable to practices. The patient-level chart review data were modeled to reflect nesting at the physician and practice level. Analyses were conducted using the linear mixed-effects model function in the non-linear mixed-effects package of R version 3.0 (www.r-project.org/).

Results

Practitioners over-reported the quality of care in every category of performance as compared to chart review (Table 2). Practitioners overestimated use of parent and teacher rating scales during assessment by 51% and 47% respectively as compared to chart review. Also, practitioners overestimated the use of parent and teacher rating scales during treatment by 241% and 332% respectively as compared to chart review. There was a 12% overestimation in the use of DSM-IV criteria to diagnose ADHD comparing practitioner self-report to chart review. Chart review estimates of days to contact with families were nearly 3 times longer than estimated by self-report. All of the differences between data obtained by practitioner self-report and chart review reached statistical significance at the p < 0.05 level.

Table 2.

Performance as Reported by Practitioner Self-Report and Chart Review

	Self-report			Chart review			Difference between self- report and chart review means	Significance level, p	Effect Size, ES (95th CI)
Performance Measure	Number of Practitioners	Mean	SE	Number of Charts	Mean	SE
Parent Scales, Assessment (%)	176	85.4	1.8	1098	56.7	3.7	28.7	0.000	−0.25 (−0.41 to 0.09)
Teacher Scales, Assessment (%)	178	81.8	2.0	1098	55.5	3.6	26.3	0.000	−.0.24 (−0.40 to 0.07)
DSM Criteria (%)	159	78.7	2.2	708	70.4	3.4	8.3	0.014	−0.10 (−0.27 to 0.07)
Days To Contact (days)	179	26.1	2.2	1405	72.4	5.4	−46.3	0.000	0.24 (0.09 to 0.40)
Parent Scales, Treatment Monitoring (%)	173	36.8	2.7	784	10.8	2.7	26.0	0.000	−0.37 (−0.54 to −0.21)
Teacher Scales, Treatment Monitoring (%)	172	32.4	2.3	784	7.5	1.5	25.5	0.000	−0.62 (−0.79 to −0.45)

Conclusion

Our study design allowed us to directly compare individual practitioner’s self-report of ADHD care to actual care as determined by patient chart review thus highlighting discrepancies between perceived and actual rates of ADHD care. We found that practitioners appeared to estimate some practices more accurately (e.g., use of DSM-IV criteria; 8.3% discrepancy between self-report and chart review) than others (e.g., days to first contact; 46.3% discrepancy). It is possible that certain contextual factors at the physician, practice, or patient level could moderate these differences. However, higher rates of ADHD care were evident based on self-report compared to chart review across all ADHD assessment and treatment monitoring practices. Given the results of this study, as well as similar results found by other investigators in other disease states, we suggest that practitioner self-report of ADHD guideline adherence should not be considered a valid measure of performance.

Accurate documentation of care is essential if we are to improve ADHD care, as we cannot change care unless we can measure it. Practitioners are thus faced with multiple questions: 1) how to efficiently document and assess the care that they provide 2) how to receive feedback about their performance in a timely manner and 3) how to improve suboptimal care. Although chart reviews are more accurate than self-report, they are cumbersome and time consuming for busy practitioners. Manual patient registries and tracking systems can be similarly challenging. Electronic Health Records (EHRs) hold promise but pose particular difficulties with regard to obtaining teacher information, as most EHRs are unable to effectively process information from external sources. Moreover, extracting data from EHRs has proven to be difficult in practice, as it can still be time-intensive and is “dependent on the resources of the organization to pull and analyze the data” (Baum, Epstein, & Kelleher, 2013, p. 371). Finally, quality measures calculated from data extracted from EHRs by computers often correlate poorly with measures calculated from data extracted by human chart reviewers (Gardner et al., 2014).

It is likely that novel technologies will be necessary to meet this need. Web portals allow parents and teachers to document rating scale responses directly, can provide rapid aggregation of quality indicators to allow for practitioner feedback, and can automate certain aspects of care (e.g., e-mailing parent and teacher reminders to complete rating scales) (Epstein et al., 2011). Automated reminder systems in web portals or EHRs may be especially helpful to improve care behaviors that occur infrequently, such as the use of parent and teacher rating scales during treatment. Internet-based applications or “apps” are also becoming a popular tool for both documentation and patient self-management (Mosa, Yoo, & Sheets, 2012). While a clear solution to accurately and efficiently capture care provided has not yet emerged, it is necessary in order for efforts to improve ADHD care to succeed.

Practitioners must also address opportunities for improvement once they are identified. Clinical practice guidelines provide a benchmark against which quality care can be measured, and strategies from the field of quality improvement are increasingly being applied to help practitioners achieve guideline-based care (American Board of Pediatrics, n.d.; Epstein, Langberg, Lichtenstein, Kolb, & Simon, 2013). Pay for performance strategies that reward practitioners for achieving quality indicators may provide additional incentives. Ultimately, however, system change will be necessary in order to help practitioners analyze their practice patterns and institute process changes to support improved care.

Our study contains certain limitations. Participating practitioners were recruited to participate in an ADHD quality improvement effort in Ohio, such that our sample may be biased towards a group of practitioners interested in improving care, and generalization to care provided in other areas may be limited. Data collected by chart reviews may under-represent practitioner performance, as only care documented in the medical record can be captured. Hence, the relatively lower rates of care obtained through chart review may not reflect over-reporting but may reflect under-documentation of care. Analysis of patient, practice and physician characteristics that may affect the magnitude of the discrepancy between physician self-report and chart review was outside the scope of our manuscript but may be explored as a future direction. Lastly, our chart review definition of the use of DSM-IV criteria (i.e., documentation of parent and teacher rating scale or presence of a specialty assessment) may not be fully reflective of actual practice (i.e. reviewing ADHD symptoms during the parent history). This definition may have artificially increased chart review estimates of the use of DSM-IV criteria, leading to a smaller discrepancy between self-report and chart review results.

Regardless of these limitations, the findings in this study suggest that improvements in community based ADHD care are warranted and that practitioner self-report should not be considered a valid measure of performance. Further study will be needed to determine the most effective and efficient ways for practitioners and other stakeholders to monitor and improve performance.