Maintenance of Certification: How Performance in Practice Changes Improve Tobacco Cessation in Addiction Psychiatrists’ Practice

Abstract

Background and Objectives

In 2000, the American Board of Medical Specialties implemented the Maintenance of Certification (MOC), a structured process to help physicians identify and implement a quality improvement project to improve patient care. This study reports on findings from an MOC Performance in Practice (PIP) module designed and evaluated by addiction psychiatrists who are members of the American Academy of Addiction Psychiatry (AAAP).

Method

A 3-phase process was utilized to recruit AAAP members to participate in the study. The current study utilized data from 154 self-selected AAAP members who evaluated the effectiveness of the MOC Tobacco Cessation PIP.

Results

Of the physicians participating, 76% (n 120) completed the Tobacco PIP. A paired t-test analysis revealed that reported changes in clinical measure documentation were significant across all six measures. Targeted improvement efforts focused on a single clinical measure. Results found that simple change projects designed to improve clinical practice led to substantial changes in self-reported chart documentation for the selected measure.

Conclusions

The current findings suggest that addiction psychiatrists can leverage the MOC process to improve clinical care.

INTRODUCTION

Approximately, 80% of physicians licensed to practice in the United States are board certified by 1 of the 24 American Board of Medical Specialties (ABMS) member boards. ABMS board-certified physicians are expected to maintain specialty expertise by participating in a robust continuous professional development program. In 2000, ABMS launched a continuous learning plan called the ABMS Program for (MOC). The MOC process, offered by all 24 ABMS members,¹ involves ongoing measurement of core competencies defined by ABMS and the Accreditation Council for Graduate Medical Education (ACGME). It also provides a board-certified physician with a structured approach to demonstrate commitment to enhancing patient care and improving patient outcomes through focused assessment and quality improvement activities.^2–7 As a result, credentialing organizations, insurers, hospitals, and the federal government recognize MOC.

The MOC process also supports practice-based learning.⁸ Specifically, MOC Part IV: Improvement in Medical Practice, utilizes the Performance in Practice (PIP) model. The PIP model enables physicians to develop and maintain necessary skills and knowledge to respond effectively to patients’ needs. Studies^9–16 indicate that a physician benefits from participating in the MOC process by: (i) gaining more focused learning based on his or her individual clinical practice needs; (ii) increasing knowledge and awareness of educational needs; (iii) increasing the effectiveness and efficiency of care; (iv) reducing malpractice claims; and (v) receiving improved training in evidence-based clinical practices. Alternatively, it is expected that patients treated by physicians who actively participate in MOC will benefit by: (i) fewer medical errors; (ii) improved communication with their physicians; and (iii) better clinical outcomes through Patient-Centered Outcomes Research (PCOR) informed practice.

Across different medical specialties, completion of the MOC process is associated with improved patient outcomes.^17–20 Few studies have explored how completing the MOC process influences quality of care for psychiatrists.^21,22 To address this knowledge gap, the AAAP, in partnership with the University of Wisconsin–Madison, responded to a request for application (RFA) from the Agency for Healthcare Research and Quality (AHRQ) (RFA-HS-12-006). The RFA specifically targeted grant proposals from health professional associations “to develop dissemination programs that integrate patient-centered outcomes research into clinical practice.” Our research study evaluated the effectiveness of traditional versus social media-supported training on PIP completion within the MOC process. This manuscript specifically explores the impact of physician completion of the Tobacco Cessation PIP on self-reported physician practice improvement.

METHODS

The AAAP’s Evidence-Based Practice Committee developed the Tobacco Cessation PIP. Approved by the American Board of Psychiatry and Neurology (ABPN), this PIP focuses on treating patients with substance use disorders who were also tobacco users. The PIP includes three stages (Fig. 1). In Stage A, the physician completes a 20-question clinical knowledge self-assessment on the subject of the PIP. Each physician also conducts a review of five self-selected patient charts to assess documentation of the PIP clinical measures. Stage B focuses on practice improvement. The physician uses the clinical knowledge self-assessment scores to identify where gaps in knowledge about a specific PIP topic may exist. The physician also uses the chart documentation results to target an area for improvement and then implements practice changes to improve clinical measure documentation. In Stage B of this PIP, physician could self-select to participate in social media (eg, using hashtags via Twitter or group messages in LinkedIn) or receive calendar reminders about the PIP completion process. Finally, Stage C of the PIP involves a re-assessment of the physician’s clinical knowledge and another self-selected five chart review to determine if the changes implemented in Stage B were successful in the physician’s own estimation.

FIGURE 1.

Stages and structure of a performance in practice.

Tobacco Cessation PIP Process and Data Collection

The Tobacco Cessation PIP was initially provided to AAAP members who agreed to participate in the study after attending an informative workshop at the AAAP 2013 Annual Meeting. The Tobacco Cessation PIP was then offered to AAAP members who attended the 2014 annual meeting PIP workshop. Finally, it was made available as a no cost as an AAAP member benefit. This study includes Tobacco Cessation PIP data from all three groups.

In completing the Tobacco Cessation PIP, each physician reviewed five charts (Stage A) to assess the degree of chart documentation for six clinical measures. The first two measures focused on whether or not the physician asked about the patients’ tobacco use (Clinical Measure 1) and whether the physician documented the tobacco use in the treatment plan (Clinical Measure 2). The next three clinical measures focused on documentation of the patients’ motivational level and readiness to change (Clinical Measure 3); counseling and the provision of information about local and national community psychosocial resources (Clinical Measure 4); and prescriptions for one of the seven Food and Drug Administration (FDA)-approved tobacco cessation treatment medications (Clinical Measure 5). Clinical Measure 6 focused on providing information to the patient about the appropriate use of the prescribed medication, in particular nicotine replacement therapy administration. After reviewing each chart, the physician provided a “Yes” or “No” response to indicate whether or not the clinical measure was documented in the patient’s chart. A physician could also provide reasons why the chart documentation did not occur for a specific clinical measure.

In Stage B of the Tobacco Cessation PIP, the physician reviewed his or her performance across the six measures and selected one measure as the focus of the practice improvement project. Once identified, the physician documented up to three practice improvement SMART (Specific, Measurable, Agreed Upon, Realistic, and Time-based) objectives and identified changes to implement to improve processes of care within his or her practice. To support the development of the practice change, the PIP encouraged the physician to structure the response around six questions: (i) What change should you implement in your practice to improve your clinical measure? (ii) What change can you get yourself to do? (iii) What support does your system have for you? (iv) When will you start making this change in your practice? (v) What barriers could you encounter? and (vi) At what time period will you assess the outcomes of this change? All question responses were self-reported by the physician. After implementation, the physician re-assessed documentation for the six clinical measures (Stage C). This paper reports on how completing the Tobacco Cessation PIP influenced physician practices, and on the changes implemented to improve the documentation of clinical measures.

Data Analysis

The primary outcome measure was the percent of charts with documentation (a “Yes” response by the physician) for a given clinical measure at baseline and post assessment. A paired t-test examined the change in the percent of charts with documentation from Stage A to C. An ANOVA explored differences in the clinical measure change for those physicians who selected the measure when compared to those physicians who did not identify the specific clinical measure as a target for improvement. A qualitative review of the physician notes about the charting efforts (objectives and actual implementation plans) was conducted to identify and categorize the responses. The study was approved by the University of Wisconsin Health Sciences Institutional Review Board (2012–1120).

RESULTS

Table 1 shows the demographics of individual physicians who started the Tobacco Cessation PIP for each of the three study groups. Of the 120 physicians who started the PIP, 76.7% (n = 92) completed the Tobacco Cessation PIP. Practice setting and areas of practice varied across the groups. The average age slightly differed across the groups with those who completed the Tobacco Cessation PIP. Finally, the percent of physicians who self-selected a social media reminder did not differ between the three groups.

TABLE 1.

Tobacco Cessation PIP respondent demographics

	2013 annual meeting (N=69)	2014 annual meeting (N=36)	Free to AAAP members (N=49)	Total (N= 154)	X2/p-value
#Who started the Tobacco Cessation PIP	36	35	49	120
%Completing the Tobacco Cessation PIP	88.9	68.6	73.5	76.7	4.57 (.102)
Respondent demographics Practice setting (% and N)					27.83 (.015)
Office based	11.1 (4)	25.7 (9)	26.5 (13)	21.7 (26)
VA Hospital	27.8 (10)	8.6 (3)	8.2 (4)	14.2 (17)
Academic medical center	11.1 (4)	20.0 (7)	12.2 (6)	14.2 (17)
Outpatient addiction treatment program or clinic	19.4 (7)	2.9 (1)	8.2 (4)	10.0 (12)
Inpatient addiction treatment program or clinic	11.1 (4)	5.7 (2)	4.1 (2)	6.7 (8)
Inpatient general medical service	8.3 (3)	.0 (0)	20 (1)	3.3 (4)
Outpatient general medical clinic	.0 (0)	2.9 (1)	2.0 (1)	1.7 (2)
Refused/no response due to non-completion	11.1 (4)	34.3 (12)	36.7 (18)	28.3 (34)
Area of practice (% and N)					31.8 (.023)
Clinical only	38.9 (14)	31.4 (11)	51.0 (25)	41.7 (50)
Clinical and education	13.9 (5)	2.9 (1)	4.1 (2)	6.7 (8)
Clinical, education, research and administrative	13.9 (5)	8.6 (3)	.0 (0)	6.7 (8)
Clinical and administrative	8.3 (3)	2.9 (1)	4.1 (2)	5.0 (6)
Clinical, education and research	5.6 (2)	11.4 (4)	.0 (0)	5.0 (6)
Clinical, education and administrative	2.8 (1)	5.7 (2)	4.1 (2)	4.2 (5)
Clinical, research and administrative	2.8 (1)	.0 (0)	.0 (0)	.8 (1)
Education	.0 (0)	2.9 (1)	.0 (0)	.8 (1)
Other	2.8 (1)	.0 (0)	.0 (0)	.8 (1)
Refused/no response due to non-completion	8.3 (3)	34.3 (12)	37.2 (16)	28.3 (34)
Self-selected social media reminders (% and N)a					7.23 (.300)
Yes	47.2 (17)	42.9 (15)	40.8 (20)	43.3 (52)
No	52.3 (19)	57.1 (30)	59.2 (29)	56.7 (59.2)
Respondent age (Mean SD)b	53.8 (10.2), N = 26	55.8 (11.1), N = 27	49.9 (7.4). N = 34	52.9 (9.7). N = 87	F/p-value 3.04 (.053)

^aSocial Media included Twitter and LinkedIn as well as calendar reminders or signing up for a list serv;

^bAge is calculated as of Junuary 22, 2016.

Completing the Tobacco Cessation PIP significantly improved self-reported physician documentation in patients’ charts across the six clinical measures (Table 2). At baseline, physicians asked about tobacco use (92% of the time) and documented tobacco use in the treatment plan (74% of the time). The focus of the efforts for the other four clinical measures was not well documented and ranged between 40% (Clinical Measure 4) and 53% (Clinical Measure 5). After completing the PIP, the net change in the percent of charts with evidence of clinical measure documentation was greatest for the final four clinical measures.

TABLE 2.

Impact of physician practice change on improved clinical measure chart documentation: overall versus measure specific focus^a

	Clinical Measure 1			Clinical Measure 2			Clinical Measure 3			Clinical Measure 4			Clinical Measure 5			Clinical Measure 6
	Ask patient about tobacco use			Documented that patient was asked about tobacco use			Document patient motivational level and readiness to change			Document counseling and provision of information about local and national resources			Document prescribing of an FDA approved medication for tobacco cessation			Document that patient provided with information about appropriate use of the prescribed medication
Overall chart changeb	Pre	Post		Pre	Post		Pre	Post		Pre	Post		Pre	Post		Pre	Post
	.92	.98	**	.74	.93	***	47	.86 •••		40	.80	***	.53	.70	***	.46	.72	***
	Physician focused on measure 1			Physician focused on measure 2			Physician focused on measure 3			Physician focused on measure 4			Physician focused on measure 5			Physician focused on measure 6
Pre to post change forc:	Yes (n=10)	No (n=82)		Yes (n=13)	No (n=79)		Yes (n=30)	No (n=62)		Yet (n=21)	No (n=71)		Yes (n=10)	No (n=82)		Yes (n=8)	No (n=84)
Measure 1	.26	.02		.09	.04		.02	.06		.02	.06		−.02	.06		−.03	.06
Measure 2	.52	.15	***	.38	.15	*	.08	.24	*	.13	.20		.06	.20		.15	.19
Measure 3	.46	.37		.49	.36		.51	.32	*	.23	.43	*	.18	.41		.30	.39
Measure 4	.38	.40		.42	.36		.37	.42		.63	.33	**	.26	.42		.10	.43
Measure 5	.24	.16		.12	.18		.08	.21		.20	.16		.38	.14	*	.15	.17
Measure 6	.46	.24		.15	.28		.13	.32	*	.35	.13		.36	.25		.30	.26

^aSignificance levels:

^***< .001;

^**< .01 and

^*< .05;

^bTotal sample size is 92 Tobacco Cessation PIP completers;

^cThe table section examines the change in the percent of charts where the measure was documented when the physician focused their improvement efforts on the clinical measure.

Fifty-two physicians self-reported a re-assessment time in the improvement plan. For these physicians, the average elapsed time between chart reviews was 2 months. The three most frequent times between reviews were 1 month (N = 18, 34.6%), 3 months (N = 9, 17.3%), and 6 months (N = 6, 11.5%).

In completing the PIP, the physician should implement an improvement project to improve documentation for one of the six clinical measures. In this sample, 30 physicians (32.6%) focused on documenting the patients’ motivational level and readiness to change (Clinical Measure 3). Another 21 physicians (22.8%) implemented changes to better document the provision of counseling and information related to community psychosocial resources (Clinical Measure 4). Thirteen of the 92 physicians (14.1%) sought to improve the documentation of tobacco use in the patient chart (Clinical Measure 2). Ten physicians (10.9%) each addressed asking patients about tobacco use (Clinical Measure 1) or documenting prescriptions to patients for tobacco cessation medications (Clinical Measure 5). Only 8 of the 92 physicians (8.7%) focused the improvement project on providing the patient with information about appropriate medication use (Clinical Measure 6).

The ANOVA results found a significantly higher change in the percent of charts where the clinical measure documentation was completed for those physicians whose PIP improvement project targeted the measure (except for Clinical Measure 6, which was related to information about the use of Tobacco Cessation medications), as compared to those physicians who did not (Table 2). No significant differences were found for Clinical Measure 6 because five of the eight physicians showed no difference in pre and post charting documentation for this measure.

In Stage B, a physician could identify up to three objectives in the performance improvement plan. The qualitative analysis classified the objectives into one of seven categories (Table 3). Within each category, the results indicate that certain objectives were more prevalent than others for a specific clinical measure. For example, a physician focused on improving assessment and documentation of patient readiness (Clinical Measure 3) was more likely to indicate that these efforts would emphasize improving assessment and documentation of patient readiness (56%). On the other hand, objectives for physicians targeting Clinical Measure 4 (provision of resource information) were more likely to concentrate on developing or acquiring treatment resources. These physicians then documented either that patients received resources (40%) or counseling (24%). Approximately, 50% of the objectives for the administration of FDA-approved medications (Clinical Measure 5) and the provision of information about the prescribed medication (Clinical Measure 6) focused on efforts to discuss, offer, or prescribe medications, and document this information in the chart. Table 3 also includes examples of the objectives physicians identified within each of the seven categories.

TABLE 3.

Performance in practice improvement plan objectives

Practice change objective category	Frequency of practice objectives within Clinical Measure Focus (N and % of the objective within Clinical Measure)
	Clinical Measure 1	Clinical Measure 2	Clinical Measure 3	Clinical Measure 4	Clinical Measure 5	Clinical Measure 6	Example of an Objective within Category
Discuss, offer or prescribe medications and document in chart	2 (7%)	2 (5%)	4 (5%)	5 (7%)	13 (38%)	5 (24%)	Provide patient with information about pharmacological interventions Discuss/document tobacco use disorder medications in low motivated tobacco users Educate patients about appropriate use of medication/doses and NRT
Provide and document client counseling	0 (0%)	2 (5%)	9 (10%)	17 (24%)	5 (15%)	4 (19%)	Counsel patients using tobacco products at each visit Identify benefits and problems to patient from nicotine Document counseling and provision of resources in all relevant patient charts
Develop, provide and document use of treatment resources	1 (3%)	0 (0%)	3 (3%)	28 (40%)	4 (12%)	5 (24%)	Collect resources and prepare easy to disseminate packet for patients Develop or obtain patient information handouts about NRT Document NRT handouts provision
Discuss and document patient response to treatment	1 (3%)	3 (8%)	5 (6%)	8 (11%)	1 (3%)	0 (0%)	Use log book to record positive & negative experiences that may impact patient skills Follow-up with patients who decline help Document if patient follow up to tobacco cessation referral if accepted the referral
Discuss and document options in treatment plan	4 (14%)	14 (35%)	11 (13%)	5 (7%)	6 (18%)	2 (10%)	Record & document pattern of tobacco use Increase documentation re: nicotine use in treatment plan even when low motivation for change
Assess and document patient readiness	4 (14%)	1 (3%)	48 (56%)	5 (7%)	1 (3%)	2 (10%)	Use URICA to assess readiness Use URICA results to provide feedback to patients on their motivation to change Assess readiness in 70% of patients admitted to OTP
Implement other practice changes (eg, screening, revise forms)	17 (59%)	18 (45%)	6 (7%)	2 (3%)	4 (12%)	3 (14%)	Diagnosis nicotine abuse using DSM-V Learn about TT model of change Verify diagnosis in assessment/plan is an accurate reflection of treatment intervention
Total responses	29	40	86	70	34	21

The online MOC Tobacco Cessation PIP did not require a specified length of response related to the improvement plan. As such, the details and completeness, in terms of answering the six questions, of the implementation plan differed by physician. An examination of the information shows that 42 physicians (45.6%) answered the six implementation plan questions. Another 31 physicians (33.7%) who completed the Tobacco Cessation PIP chose not to provide any information about the improvement plan. The other physicians (n = 19, 20.7%) provided some information in the improvement plan, but did not appear to structure the response around the six questions.

DISCUSSION

The MOC process, including completion of PIP projects, is designed to improve a physician’s practice and enhance the quality of care provided to his or her patients. The delivery mechanism used to introduce physicians to the MOC process may influence its success in fostering physician practice change.^23,24 This project integrated the MOC into the structure of a quality improvement collaborative (QIC). A structured MOC approach improves physician skills and knowledge.^10,25 In the QIC, physicians were introduced to the MOC process at an initial learning session and then received support throughout the process via ongoing communication efforts, which included social media reminders and educational resources and tools. Unlike a typical QIC, this study did not include directed one-on-one physician coaching because of the interest in understanding the role of social media in improving patient centered care.

This study is the first to explore the impact of completing a PIP focused on practice change using online mechanisms for in the field of addiction psychiatry. The PIP utilizes a clinical activity quality improvement cycle²² to drive the physicians’ quality improvement efforts. The process starts with data collection from patient charts comparing the results to core standards. The physician then identifies opportunities for improvement and implements practice changes to achieve those goals. In this study, the six questions included in Stage B model a similar process quality improvement effort utilized by addiction psychiatrists called the Plan-Do-Study-Act cycle.²⁶ These questions allowed the physician to develop and complete the Plan-Do-Study part of a rapid-cycle change. Physician self-report response to Question 6 in the improvement plan revealed that 10% (5 of 52 respondents) conducted one or more chart documentation re-assessments. Unique to the process is a question that asks the physician to self-identify any barriers that might be encountered and thus influence implementation success. Despite concerns about the accuracy of physician self-assessment,²⁷ successful PIP completion has been associated with improved patient outcomes across medical specialties.^{17–19,28–31} However, it is unknown if the improvements were associated with long-term sustainability of the changes within a physicians’ practice, thereby extending the health benefits to all patients.

Only a few AAAP members reported using social media outside of personal needs. Using social media was helpful in outreach and in providing reminders to physicians to complete the Tobacco Cessation PIP. The comparison of the impact of social media supports versus traditional training was inconclusive. The study indicates that addiction psychiatrists who engaged in and completed the MOC process by implementing a directed quality improvement project that targeted a specific clinical measure were able to significantly improve clinical documentation related to the measure. For example, one addiction psychiatrist who sought to improve care related to providing his or her patients with information about local and national community psychosocial resources (Clinical Measure 4) focused on a multi-step intervention plan. The plan (described below and organized by the six questions) focused on first providing information to the patient about resources including medications, documenting the provision of the information, and then following up with the patient at the next session to determine if the patient followed through with the recommendation.

“1. Discuss a specific intervention with patient (e.g., read brochure, attend cessation session). 2. Write a ’treatment’ note and follow-up at next visit. 3. A nurse on our service does tobacco cessation sessions and instructs on nicotine dosing. System makes nicotine, bupropion, and (with pharmacy oversight) varenicline [available to the patient] 4. I have begun to do this intervention recently, but have not had follow-up feedback as yet. 5. Barrier could be follow-up session, if we become busy with other issues. 6. [Evaluate in] February and March, 2015.”

Another implementation plan example targeting improvements in prescriptions for nicotine dependent treatment medications involved a multi-step process in the physician’s office.

“I plan to implement separate diagnosis of Nicotine Abuse in my patient’s chart. [University of Rhode Island Change Assessment Scale] URICA will be implemented. Patient will see our in-house therapist to treat Opioid Use Disorder AND Nicotine Abuse. My chart will document if patient has called 1 800 QUIT NOW. I will discuss [Nicotine Replacement Therapy] NRT based on a list of options to my patient. I will place Rx used in my Rx record along with my patients Rx for their Opioid Use Disorder.”

While 92% of the physicians who completed the Tobacco Cessation PIP detailed his or her performance objectives and improvement plans, the quality of these plans was not evaluated.

The study had several limitations. The focus of the study was on the ABMS approved MOC process for addiction psychiatrists. It did not examine similar certification processes utilized by the American Osteopathic Association (AOA), and participants in this study may also have AOA board certification. Study recruitment was open to all members of the AAAP through a phased implementation process. The physicians self-selected to participate in the study, and were encouraged but not required to complete the Tobacco Cessation PIP. Given the geographic spread of the participants, it was not feasible for the study to include a process to verify the chart review collection process or explore whether the reported practice changes were actually implemented. Future studies of the MOC process should include a structured in-person or medical record verification process to assess the extent of the implementation of the changes in the physicians’ practices, and study the impact of the changes on patient outcomes.

Regardless of the impact of MOC on patient care, physician opinions about the benefits of MOC are mixed.^32–34 We found similar differences of opinion. Since the start of the project, physicians expressed concerns to the ABPN about the MOC and PIP requirements. As a result, the ABPN changed its MOC program requirements. The first change no longer required that a physician solicit both a peer and a patient feedback module. Instead, the revised MOC process required that the physician use one ABPN approved feedback option. The feedback can come from five patient surveys or an assessment of general competencies. The latter is completed through an evaluation by five peers, five residents, and a supervisor; or through a 360-degree evaluation or institutional peer review with five respondents. Although the MOC process has been streamlined, the feedback may result in practice changes benefiting the patients. The second change reduced the number of PIP units to be completed between certifications. The change was necessary due to the lack of ABPN approved MOC products. Despite these changes, the fundamental core of the PIP clinical feedback module remains—the focus on quality improvement and practice change.

Overall, the current study provides evidence of improved physician self-assessment related to treatment for Tobacco Cessation. However, the evidence is inconclusive if the MOC process is an effective mechanism for improving the quality of Tobacco Cessation practice delivery and care in addiction psychiatry practices. Future studies need to explore how completion of other PIP modules for addiction psychiatrist impacts clinical documentation for other conditions in the practice and examine how a more structured quality improvement collaborative with directed peer physician coaching affects the delivery of care within physician practices as well as the impact on patient outcomes.

Scientific Significance.

The study demonstrates that a quality improvement collaborative can support an MOC process to help addiction psychiatrists eliminate barriers to access to tobacco cessation treatment.