Radiology Quality Measure Compliance Reporting: an Automated Approach

Abstract

As part of its ongoing effort to improve healthcare quality, the Center of Medicare and Medicaid Services (CMS) has transitioned from monetary rewards to reimbursement penalties for noncompliance or nonparticipation with its quality measurement initiatives. More specifically, eligible providers who bill for CMS patient care, such as radiologists, will face a 2 % negative payment adjustment, if they fail to report adequate participation and compliance with sufficient CMS quality measures in 2015. Although several methods exist to report participation and compliance, each method requires the gathering of relevant studies and assessing the reports for compliance. To aid in this data gathering and to prevent reduced reimbursements, radiology groups should consider implementing automated processes to monitor compliance with these quality measure standards. This article describes one method of creating an automated report scanner, utilizing an open source interface engine called Mirth Connect, that may facilitate the data gathering and monitoring related to reporting compliance with CMS standard #195 Stenosis measurement in Ultrasound Carotid Imaging Reports. The process described in this article is currently utilized by a large multi-institutional radiology group to assess for report compliance and offers the user near real time surveillance of compliance with the quality measure.

Background

With the Center of Medicare and Medicaid Services (CMS) transitioning its Physician Quality Reporting System (PQRS) and Value-Based Payment Modifier from monetary incentives to negative adjustments, the importance for compliance with CMS’s quality measure standards has become an increasingly pertinent topic for the radiologist and informatics specialist. Although reporting compliance with PQRS quality measures is voluntary, reimbursement penalties have begun and which will likely incentivize participation. Unfortunate for the unaware, noncompliant or nonparticipating radiologist, these penalties are applied on a retrospective basis. Specifically, radiologists who do not report satisfactory compliance with quality measurements in 2015 will receive a 2 % negative reimbursement adjustment in 2017 as detailed by the CMS PQRS quality measurements program. Additional negative reimbursements may result from the associated Value-Based Payment Modifier resulting from a 2–4 % negative adjustment based on the number of providers in the group—resulting in a negative adjustment possibly up to 6 % [1]. This loss of income has been estimated at over $100 million and has been estimated to affect more than 75 % of radiologists [2].

There are many reasons to implement efficient and flexible data processing methods, such as an automated report scanner, while reporting standard compliance. First, if inefficient methods are utilized to gather compliance data, the cost and time spent compiling necessary data may encroach upon the financial benefit of reporting. Additionally, not only can data mining automation minimize the time and cost of compliance reporting but it also can easily compile data to identify noncompliant personnel or workflow patterns [3]. Third, while only a limited number of the more than 200 quality measures currently apply to radiology, one may expect the number of reporting requirements to continue to increase in the future. Early implantation of streamlined processes now may ease pain associated with future quality measurement standards.

This article describes a large multi-institutional radiology group’s method to automate the data mining process in monitoring compliance with CMS PQRS measurement #195 Radiology: Stenosis measurement in Carotid Imaging Reports. Compliance with CMS PQRS quality measurement #195 requires a direct or indirect reference to the percentage of carotid stenosis being reported with every carotid imaging exam, including neck MR angiography, neck CT angiography, neck duplex ultrasound, or carotid angiogram [1]. The CMS PQRS specification manual suggests referencing the methodology utilized or describing the actual technique used to measure stenosis. As will be described later in this article, to assess compliance with the quality measure, our radiology group adopted a standardized phrase in their report containing a specific measurement description designed to facilitate the automated compliance assessment.

Methods

Our automated report quality scanner process is illustrated in Fig. 1. The process begins with HL7 messages containing the finalized radiology reports being sent to Mirth Connect, which processes the data and writes relevant data to a MS SQL database. This database is linked and summarized in a Microsoft Excel pivot table that provides for easy reporting of compliance with quality measurements. Additional channels can be built to evaluate compliance with a number of report templates.

Fig. 1.

Automated report quality scanner process

Previous approaches have focused on building and mining a complete database of radiology reports which can then be retrospectively mined to generate compliance reports [4]. Our method is event-based rather than retrospective, which could easily lead to real-time alerting.

Effective data mining requires knowledge regarding the underlying data. Our report scanner starts with a finalized radiology report transmitted in a Health Level Seven (HL7) version 2.x message. This message originates within our transcription software and is broadcasted enterprise wide through the radiology information system (RIS) and interface engine.

Configuration

While the HL7 protocol provides general standards regarding the data content of the message, there may be considerable variability in where data is placed within a message. In part, this results from the vague or overlapping description of message segments, such as an observation (OBX) or note (NTE) segment. Consequently, software programs frequently will use the fields differently and a report may be placed by one program within an OBX segment while another program may utilize a NTE segment.

Mirth Connect utilizes a user-determined set of steps called a channel to process data, and our compliance scanner uses a two-channel setup. The first channel receives real-time HL7 messages from the enterprise interface engine and forwards only select HL7 messages to the second channel to be evaluated for compliance. In our example, we only forward messages that contain reports for carotid Doppler ultrasound scans, by filtering for messages containing the corresponding local procedure code. This filtering could be configured based on each site’s compliance needs. We used the same mechanism to evaluate for template compliance for a variety of other use cases including compliance with dose reporting for pediatric CT scans, and usage of a standardized template for reporting ER right upper quadrant ultrasounds.

Compliance Checking

Mirth Connect provides user-defined Javascript steps that allow for complex logic including extensive message reformatting, useful database queries (such as radiologist id to name conversions, CPT/exam id lookups), or calculating time differences (such as turnaround time).

To assess for compliance with reporting standards, important strings in the report can be identified to indicate compliance. For the carotid ultrasound example described in this article, documenting measurement method used to interpret carotid artery velocities was a key to compliance. For that reason, our radiology group adopted a standardized phrase in their report similar to the following:

Measurement of carotid stenosis is based on velocity parameters that correlate the residual internal carotid diameter with North American Symptomatic Carotid Endarterectomy Trial (NASCET)-based stenosis levels.

Our automated report scanner ultilized the substring “NASCET” as a surrogate for compliance. Using a Javascript step in a Mirth channel, a Boolean variable is set to true if the string NASCET was present, inferring compliance with CMS’s carotid stenosis quality measure and referring to the measurement technique utilized in the North American Symptomatic Carotid Endarterectomy Trial [5]. We performed periodic hand audits to verify that the presence of the NASCET substring was a reasonable surrogate for actual compliance.

The binary used_template variable set by the presence or absence of our test string was written alongside other report metadata into our MS SQL tables as demonstrated in Table 1.

Table 1.

Database structure for our MS SQL table

ID	Bigint
accession_number	Varchar (50)
used_nascet	Int
report_text	Ntext
report_date	datetime

The id field is set to increment automatically. The used_nascet field is set to 1 if the appropriate template was used, and 0 if the NASCET phrase was omitted

Reporting

We created a single table within Microsoft SQL server 2003 for each compliance project. The table structure was selected for simplicity and does not reflect the vast wealth of data available from the HL7 stream.

Utilizing its external data function, we connected Microsoft Excel directly to the MS SQL server. We provided our MS SQL server name and credentials through an Excel wizard to import the MS SQL carotid data table to a new sheet in Microsoft Excel. From this sheet, pivot tables were created to aggregate data by radiologist, year and quarter and to display a high level summary (Table 2). Pivot tables also allow users to drill down to specific reports by double-clicking on the radiologist name, which can be helpful in identifying falsely classified reports or obtain additional detail regarding unsuspected summary results.

Table 2.

Summary of results by year

Year	Noncompliant (%)	Compliant (%)
1	393 (97)	11 (3)
2	47 (5)	939 (95)
3 (Through Q3)	20 (4)	514 (96)

Year 1 was baseline, before implementation of our quality improvement project. Overall compliance was quite high and improved slightly in year 3

If you do not have easy access to a database, Mirth Connect can easily be configured to write a comma-separated values (CSV) file with a similar column layout to our database table. This CSV file could then be easily imported by Excel through its external data menu and used to generate the same pivot table report.

Results

Results by year and quarter are summarized in Table 2. Year one was an observational year recording data prior to routinely informing radiologists of their compliance results. When beginning our quality project in year 2, we had 96 % compliance within the first quarter and similar compliance for the remainder of the year. A selection of per-radiologist data from quarter 2 of year 2 (Table 3) indicates that the low number of noncompliant dictations come predominantly from radiologists who infrequently dictate carotid ultrasound exams.

Table 3.

Sample of compliance data from quarter 2 of year 2

Radiologist	Noncompliant (%)	Compliant (%)
A	0 (0)	65 (100)
B	0 (0)	65 (100)
C	0 (0)	39 (100)
D	0 (0)	24 (100)
E	0 (0)	23 (100)
F	0 (0)	18 (100)
G	0 (0)	12 (100)
H	2 (22)	7 (78)
I	2 (40)	3 (60)
J	0 (0)	5 (100)
K	0 (0)	4 (100)
L	3 (100)	0 (0)
M	0 (0)	3 (100)
N	0 (0)	3 (100)
O	2 (100)	0 (0)
P	0 (0)	2 (100)
Q	0 (0)	1 (100)
R	1 (100)	0 (0)
S	1 (100)	0 (0)
T	1 (100)	0 (0)
U	0	1 (100)
Grand total	12 (4)	275 (96)

Discussion

At our institution, reporting to CMS for compliance with quality measures is handled by our billing department rather than radiology. Unfortunately, our radiology practice has little-to-no access to the compliance information that is submitted. For that reason, we elected to build our own compliance tracking mechanism. We do not recommend that this technique be used exclusively for gathering data for CMS. Carotid Doppler ultrasound reports are only a subset of those required by PQRS measure 195, which also includes other cross-sectional imaging exams, such as neck MRA. However, a similar approach could easily be utilized to filter for the procedure codes of those other reports. Reporting PQRS compliance and participation may be done utilizing a variety of methods, such as claims based or registry. Unfortunately, the varying methods of reporting compliance require different data formats and structure. Consequently, the results in this article, summarized in a pivot table, cannot be submitted directly to CMS for reporting purposes. However, the results are useful for internal management and may be used in part during the data gathering prior to reporting to CMS.

Another limitation with the method described in this article is that it is prospective rather than retrospective. This can make it difficult to establish a historical baseline when you want to start a project immediately. In order to address this limitation, we have performed manual chart review, identifying exams based on our full-text report search tool. We use report search to retrieve 20 exams per radiologist and review them by hand to establish a baseline. We have used manual retrieval from our report-search tools to establish baselines. We have also used the report search tool to perform similar template audits. However, this mechanism is typically limited in the number of reports that one given person can review and is very time consuming. This tends to miss the very low numbers of noncompliance that can be detected with a prospective evaluation.

Another limitation of the method described in this article is the difficulty of text matching. In projects that we have performed subsequently to our carotid ultrasound project, we have encountered errors related to capitalization and punctuation that create false negatives. This requires either education of those creating dictations, or continued refinement of our Mirth channel accordingly. These issues are similar to many other natural language processing tasks. One benefit of the Excel pivot table functionality is that it is very easy to drill-down on the offending reports and evaluate whether the scanner is working correctly.

Additionally, we have learned that more important than educating the attending radiologists who are signing reports regarding required templates is educating the residents who are creating reports. Simply reminding residents who are beginning rotations which templates are required for a particular service increases compliance.

The next steps regarding this automated scanner include altering our process such that the signing radiologists are provided with timely alerts and suggestions rather than quarterly reports.

Conclusion

The automated report scanner method described in this article offers a flexible, widely available, and efficient way of data mining radiology reports for assessing compliance with report quality measurement standards. Although, this article describes the process implemented specifically for reporting about CMS quality standard #195, the process could be easily adapted for other standards or template audits.