Development and early implementation of automated audit and feedback monitoring instruments in Belgian primary care: lessons for scalable systems

Abstract

Background

Electronic health records (EHRs) can potentially revolutionize primary care by enhancing data storage, ­communication, and quality measure reporting. The COVID-19 pandemic accelerated the adoption of digital tools in Belgium, highlighting the potential of EHR data in audit and feedback (A&F) strategies. This study aims to outline the design and national implementation process of automated A&F monitoring instruments (locally known as barometers), in Belgian ­primary care.

Methods

Using Clinical Performance Feedback Intervention Theory, we developed three monitoring instruments for primary care: COVID-19 vaccination coverage, type 2 diabetes management, and appropriate antibiotic use. Quality indicators were selected at the national level using a Rand-modified Delphi method and validated by the Flemish Institute for Quality of Health Care (VIKZ). Data were collected from general practitioners’ EHR systems, aggregated at the practice level, and analyzed using the Healthdata.be platform. Feedback was provided through the Healthstat.be interface, incorporating local and regional benchmarks and evidence-based recommendations.

Results

The COVID-19 vaccination monitoring instrument included 5223 GPs from 2269 practices, the type 2 diabetes monitoring instrument involved 9373 GPs from 3596 practices, and the antibiotics monitoring instrument covered 10 486 GPs from 3724 practices. These monitoring instruments collectively covered approximately eight million patients. Feedback reports were designed to be low in cognitive load, frequent, and benchmarked against the best-performing decile of practices. Integration of active, in-EHR delivery and formal evaluation of use are planned for future phases.

Conclusion

The implementation of automated A&F instruments in Belgian primary care demonstrated the feasibility and scalability of such systems. These monitoring instruments can provide valuable insights for quality improvement and support the transition toward a Learning Health System. Future work will focus on expanding the range of monitoring instruments and integrating active feedback mechanisms within EHR systems.

Introduction

Electronic health records (EHRs) have significantly transformed primary care in the past decades as an essential tool for storing information, facilitating communication, and computing and reporting quality measures [1]. Basic health data functionalities of EHRs are currently fully adopted by general practitioners in primary healthcare in Europe and more than half are routinely using clinical decision support functionalities as well [2]. Beyond its obvious uses in information retrieval and communication, data from EHRs can yield secondary benefits when integrated into audit and feedback (A&F) strategies in general practice. These are defined as a summary of clinical performance of health care over a specified period of time aimed at providing information to health professionals to allow them to assess and adjust their performance [3]. Such strategies can lead to insights and quality improvement at the level of individual health care practitioners as well as at the policy level [1]. However, unlike some countries with long-standing national performance frameworks, such as the UK (Quality Outcomes Framework [4]), Belgium historically lacked a unified system for primary care quality measurement. This gap reflects a highly fragmented EHR landscape and decentralized governance.

In this regard, the COVID-19 pandemic was an important accelerator. Not only was there a rapid adoption of digital tools across health care systems, but healthcare professionals, researchers and policymakers turned to EHRs for critical information on patient care, resource allocation and epidemiological trends [5, 6]. In Belgium, the burden of COVID-19 on primary care practices was monitored using structured electronic forms integrated into GPs EHR [7]. This instrument formed an integral part of the data-driven health policy efforts in 2020. It was supplemented by a COVID-19 syndromic illness surveillance tool using daily automatic data extraction from participating GPs’ EHR [8].

Since then, three other automated data extraction instruments have been developed in primary care to automatically audit general practitioners and provide feedback to individual practices and to policy makers: one for COVID-19 vaccination coverage, one for type 2 diabetes and one for appropriate use of antibiotics [9–11]. These instruments, referred to locally as “barometers,” were named during the COVID-19 pandemic to reflect their role in monitoring system “pressure.” Although the term differs from its usual English meaning, it has become established in Belgian health policy. For clarity, we use the term “monitoring instrument” in this manuscript. Their development is driven by several observations. First, aging populations and a rise in multimorbidity severely impact primary care health systems [12]. Second, chronic disease such as heart failure or chronic kidney disease is often registered inaccurately in primary care EHRs. For example, the prevalence of unregistered CKD in Belgian primary care records is 68% [13], for heart failure this is 69.5% [14]. Third, there are significant gaps in the translation of guidelines into daily clinical practice. For example, in Belgian primary care records only 7.5% of the population with type 2 diabetes and associated cardiovascular disease, heart failure or chronic kidney disease are prescribed SGLT2 inhibitors or GLP1 analogues, despite the changes in ESC and KDIGO guidelines in recent years [15, 16]. A&F, as implemented in a monitoring instrument system integrated into GPs’ daily used EHR, has the potential to address these issues through the provision of cost-effective and scalable timely feedback, benchmarking against peers, and enhanced documentation to ensure the integrity of medical records. This project represents the first nationwide effort to standardize quality indicators (QIs) and leverage EHR data for population health management.

The aim of this manuscript is to outline the design and implementation process of such monitoring instruments in a primary care setting.

Materials and methods

Intervention theory

In order to conceptualize the A&F process, we used the Clinical Performance Feedback Intervention Theory (CP-FIT) as a basis [17]. This theory helps explain what factors influence feedback success, and guides approaches to enhance effectiveness. Figure 1 outlines the different steps of the A&F cycle and our translation of these steps into outcomes within the context of monitoring instruments. We used the SQUIRE 2.0 guidelines to report on new knowledge about how to improve healthcare and the TIDieR checklist for reporting on implementation interventions (see Supplementary Files).

Figure 1.

A&F cycle according to the CP-FIT theory, adapted from Brown et al. White circles belong to the audit phase of the cycle, green circles belong to the feedback and change phase, adapted from Jamtvedt et al.

Data platform and registry

Healthdata.be is a platform to safely collect, store and analyze data from various clinical registries in Belgium. The platform and its functionalities are described in more detail elsewhere [18]. Healthdata.be can host data collected by various registries from clinicians during routine care, and one of its intended purposes is therefore to facilitate feedback on quality of care to different levels of health care professionals, from policy makers to individual practitioners. In order to test automated A&F monitoring instruments to general practitioners, and assess their functionality and validity we used a large primary care morbidity registry (Intego) as an A&F laboratory [19]. Intego is a registry currently collecting data from 132 GP practices more than half a million patients. After testing monitoring instruments within the Intego registry, implementation was opened to the broader population of general practitioners. Figure 2 illustrates how these monitoring instruments act as a federated data network through the healthdata.be platform.

Figure 2.

Illustration of the federated data structure of monitoring instruments through the healthdata.be platform. Adapted from source: https://healthdata.sciensano.be/nl/data-collectie.

Audit

Objective—goal setting

Each monitoring instrument starts with the choice of an important health topic after which EHR extractable QIs are developed that are used as key criteria to measure and provide feedback on the quality of care. These QI are designed and validated with a Rand modified Delphi method to ensure a solid evidence-base, EHR extractability and practical usability in a monitoring instrument [20–22]. Each monitoring instrument (except for the time-sensitive pragmatic instruments developed for COVID-19) is underpinned by a RAND-modified Delphi process described in detail in the respective QI publications [21, 22]. These processes included multiple stakeholder groups and consensus rounds to ensure clinical relevance and EHR extractability. While all indicators identified through the Delphi process were clinically relevant, only a subset could be operationalized due to technical and governance constraints. The monitoring instruments collects aggregated data at the practice level to ensure GDPR compliance and interoperability across multiple EHR vendors. This design precludes linking individual patient characteristics to treatments, applying conditional logic, or verifying time-dependent sequences (e.g. repeat tests within a defined interval). Many indicators also rely on nuanced coding (e.g. lifestyle counseling, sexual health inquiry) that is inconsistently structured across systems. Consequently, we prioritize indicators that are (i) clinically relevant, (ii) feasible to compute from aggregated counts, and (iii) implementable across all EHR platforms. This pragmatic filtering process results in a reduced set of indicators for each monitoring instrument.

Audit—data collection and analysis

After validation, QI are transformed into queries and tested within Intego. To consequently integrate queries into all Belgian primary care EHR systems, EHR and healthdata.be engineers conduct monthly meetings to work out a standardized implementation. Agoria, the Belgian technology sectoral employers’ organization, represents Belgian EHR vendors and organizes and presides these meetings [23]. Belgium currently has seven major EHR vendors, Data are collected automatically and periodically through standardized SQL queries embedded in these seven EHR systems where GPs have activated their monitoring instrument. This activation (and data collection) is voluntary, although participation is financially incentivized by the National Health Insurance (NIHDI) (participation in the monitoring instrument counts as one of the eight criteria to receive a full integrated practice stipend of 6000 EUR). Data are collected in aggregate at the individual practice level and not at the patient level to ensure patients’ privacy, with secure ETL (Extract—Transfer—Load) processes transferring outputs to Healthdata.be. With regards to the participating practices, GPs are asked to provide the number of GPs and distribution of age and sex, the practice type and general workload in terms of number of house calls and consultations. General practitioners can consult visualized feedback at the healthstat.be platform which is the front-end environment for the back-end data storage within healthdata.be. Future European Health Data Space (EHDS) regulations may facilitate interoperability and enable cross-country benchmarking.

Feedback and change

Feedback

Presentation of the feedback is based on important features identified in previous research: a low cognitive load, evidence-based (with a link to the evidence), frequent (more than once) and with the use of benchmarks [24, 25]. To achieve a realistic and attainable quality improvement through benchmarking, the Achievable Benchmark of Care (ABC) method is used [26, 27]. We considered alternative benchmarks (median, quartile thresholds); however, the Achievable Benchmark of Care method was selected for its evidence base and feasibility with aggregated data, and its demonstrated motivational effect in prior studies. In this method, the mean of the ten percent best performing practices is calculated and used as a benchmark. After determining the ABC, the QIs and corresponding ABC are validated by the Flemish Institute for Quality of Health Care (VIKZ), a governmental body instructed with improving Flemish health care [28]. Feedback reports use color coded performance bands to enhance interpretability and support quick signal detection. Practices can benchmark their results against peers within their front line zone, at the provincial level, and nationally. An example visualization is provided in Supplementary Fig. S1.

Reception—interaction, perception, and acceptance

The feedback report is consultable through a pull system with a secure access. The report has limited interaction possibilities in the sense that you can select and view the different QI’s. It is unclear to what extent GPs currently interact with the monitoring instruments, since healthdata.be does not collect individual metadata. To gauge end-user perceptions and acceptance, a 30-item questionnaire was recently developed [17, 29]. Although usage metrics are currently not captured given technical and privacy constraints, we intend to introduce privacy-preserving audit trails (e.g. log-ins, page views, filter selections) as part of the forthcoming evaluation framework.

Clinical performance improvement

Because data are collected periodically, with the interval dependent on the monitoring instrument, participating GPs can compare their performance with previous periods.

Ethics

All structures processing data from citizens within the European Union fall within the remit of the General Data Protection Regulation (GDPR) of the European Union [30]. Within the legal entities defined by the GDPR, general practices act as data controllers of data gathered within the patient-physician relationship whereas KU Leuven is data controller regarding data processed to evaluate monitoring instruments. The healthdata.be platform acts as data processor. Data capture is aggregated, and no personal identifiable data are processed in the data flow through healthdata.be. The monitoring instrument project was assessed to comply with GDPR regulations by the Privacy and Ethics Team of KU Leuven, and received a favorable ethical review by the Social and Societal Ethics Committee (SMEC) of the KU Leuven [G-2023-6352-R2(AMD)]. Each participating practice gives informed consent to the project by activation of the monitoring instrument within their respective EHR system.

Results

Timeline

For the COVID-19 monitoring instrument, the selection of indicators (that is, vaccination coverage within target populations) was done pragmatically due to the urgent nature of the pandemic and vaccination effort. The monitoring instrument was implemented into GPs’ EHR (across all acknowledged vendors) in March 2021 and immediately followed by a first data collection. The COVID-19 vaccination monitoring instrument was discontinued after the vaccination campaign ended in 2022; data collection is no longer ongoing. For type 2 diabetes, a selection of indicators from guidelines and a previous Delphi round [21] were translated into the EHR in December 2022. The first data were collected in May 2023. For the appropriate use of antibiotics we started from European QIs published in 2011 [22]. These were reviewed and validated during a 2023 consensus meeting to ensure alignment with current prescribing guidelines, which have remained largely stable for key conditions such as pneumonia or cystitis. The validated indicators were consequently integrated into different EHR packages and the first data were collected in December 2023. For the type 2 diabetes monitoring instrument, data are collected biannually, for the appropriate use of antibiotics each trimester. A list with original indicators is available in the Supplementary Files.

Coverage

Figure 3 demonstrates the level of coverage of these monitoring instruments in Belgium. Differences in the number of participating GPs and practices across monitoring instruments reflect staggered implementation timelines and voluntary participation. For the COVID-19 vaccination monitoring instrument, there were 5223 participating GPs from 2269 practices. For the type 2 diabetes monitoring instrument, there were 9373 participating GPs from 3596 practices as of June 2024. These covered ±8 million registered patients, of whom ±400 000 had a registered type 2 diabetes diagnosis. For the antibiotics monitoring instrument, there were 10 486 GPs from 3724 practices as of September 2024. The map shows geographic variation in participation, with higher uptake in northern regions (Flanders) compared to southern regions (Wallonia). These differences may relate to population density and absolute numbers of general practitioners, as well as possible differences in attitudes toward data sharing and trust, although these factors were not formally assessed. Absolute numbers should be interpreted with caution, as the total number of active GPs per region in Belgium is not precisely known, and therefore participation rates cannot be calculated. For active monitoring instruments, participation continues to increase as practices activate each instrument.

Figure 3.

Number of participating general practices per province per monitoring instrument in Belgium.

Contents

Table 1 details the QIs used to provide GPs with feedback on their current performance (see Supplementary Files for examples). For the COVID-19 vaccination monitoring instrument contents were relatively limited and aimed at informing GPs as to the level of vaccinated patients at risk of serious COVID-19 complications due to comorbidities such as chronic renal disease, cardiovascular disease, obesity or cancer. The indicators of the type 2 diabetes monitoring instrument concern (for now) the relevant clinical process indicators for quality of care. The antibiotics monitoring instrument gives feedback to GPs on the appropriateness of prescribing per trimester. This can be requested per type of infection and age group. For example, for otitis media, the available age groups are <2 years, ≥2 years, 2–18 years, 18–65 years and ≥65 years. All monitoring instruments allowed for visual benchmarking with peers aggregated at the local and supralocal levels (health district, province, region) with the benchmark calculated as the mean of the top 10% of practices.

Table 1.

Quality indicators per barometer and visualization of comparisons with benchmarks.

	Benchmark visualization
COVID-19
Percentage of eligible patient population fully or partly vaccinated for SARS-CoV2	Barchart
Prevalence of at-risk comorbidities (renal disease, cardiovascular disease, cancer…)	Boxplot
Longitudinal evolution of vaccination coverage, by comorbidity or age group (March 2021-June 2021)	Line chart
Type 2 diabetes
Population
Prevalence of registered patients aged 40 years or older with T2D	Barchart
Biochemical follow-up
Proportion of patients with T2D and HbA1c determination in the last year	Barchart
Proportion of patients with T2D and eGFR determination in the last year	Barchart
Proportion of patients with T2D and ACR determination in the last year	Barchart
Proportion of patients with T2D and LDL-cholesterol determination in the last year	Barchart
Clinical follow-up
Proportion of patients with T2D and blood pressure determination in the last year	Barchart
Proportion of patients with T2D and body mass index determination in the last year	Barchart
Proportion of patients with T2D and abdominal circumference determination in the last year	Barchart
Proportion of patients with T2D and smoking behaviour determination in the last year	Barchart
Proportion of patients with T2D and foot examination in the last year	Barchart
Therapeutic
Proportion of patients with T2D and influenza vaccination in the last year	Barchart
Antibiotic appropriateness
Infections: acute otitis media, tonsillitis, upper respiratory infection, sinusitis, bronchitis, pneumonia, cystitis
Incidence of infections in the last three monthys by infection and age group	Barchart
Percentage of infections treated with antibiotics in the last three months by infection and age group	Barchart
Percentage of infections treated with antibiotics where a first choice antibiotic was used by infection and age group	Barchart
Percentage of infections treated with quinolones in the last three months by infection and age group	Barchart

Discussion

Statement of principal findings

This study used CP-FIT to design and implement automated A&F instruments (so-called monitoring instruments) at a national scale, currently operational through an external platform (Healthstat.be). While integration into EHR systems and evaluation of user engagement remain future steps, this represents a significant advance toward a fully embedded feedback system. These monitoring instruments currently form a federated data network that collects data based on QIs extractable from the EHR and provides feedback for quality improvement in primary care. The network is sustainable due to financial support from the Belgian national health insurer NIHDI.

Interpretation within the context of wider literature

To promote the knowledge base and implementation of A&F in primary care, we presented the monitoring instruments as a theory informed methodology and process. The actual development was more organic and illustrates two important points. First, the potential to develop and implement sustainable A&F for primary care at scale. As far as we know, the only equivalent to our monitoring instrument project is the Quality and Outcomes Framework (QOF) in the United Kingdom [4]. This initiative also extracts anonymized data from a variety of different EHR systems and is financially incentivized through the national health insurer, although contingent on the achievement of specific targets rather than mere participation. QOF was founded in 2004, and the lack of adoption of national A&F systems in countries with comparably strong primary care systems highlights the second point: the importance of constant support for research and e-health infrastructure at the national policy level. The monitoring instrument project builds on decades of Belgian stakeholder investment in secure exchanges and integration of health data through the eHealth and healthdata.be platforms [18, 31, 32], as well as in natural implementation laboratories such as the Intego primary care registry to test, refine and scale interventions and a corresponding process evaluation [19].

Implications for policy, practice, and research

This investment is important considering that we conceive our monitoring instruments to fit iteratively within a dynamic transformation of the Belgian primary health care system into a Learning Health System (LHS) through the use of A&F. LHS are defined as health systems in which internal data and experience are systematically integrated with external evidence, and that knowledge is put into practice [33]. In their current state, the monitoring instruments connect GPs clinical experience to external guideline recommendations, complying with a majority of CP-FIT high-level hypotheses for successful feedback, the most important of which are importance, relevance, automation and benchmarking. However, active delivery, in which feedback is pushed to recipients rather than pulled is still not technically possible. This is why the integration of such active delivery within the EHR of GPs remains an important focus of implementation. In addition, several important health problems are not yet covered through the monitoring instrument project. In that regard, we plan to expand and integrate the monitoring instruments horizontally and vertically. First, horizontally, through the creation and integration of novel monitoring instruments. A monitoring instrument for chronic kidney disease (CKI) has already been designed and data collection will begin at the end of 2024. A monitoring instrument for atherosclerotic cardiovascular disease (ASCVD) and adequate use of EHR software are under construction. Second, vertically, through the integration of a single sign-on log-in for the healthstat.be platform launching from the EHR, which is a near-term priority to minimize access barriers. In later phases and subject to governance and vendor development this could lead to piloting in-EHR prompts and ultimately, to the integration of a monitoring instrument dashboard within different EHR systems. These changes are also important from a quality improvement perspective, since these monitoring instruments provide a foundation for systematic benchmarking and iterative learning and are supported by actionable follow-up queries already implemented in the EHR. The aforementioned integration within EHR systems and incorporation of longitudinal visualizations [such as run charts and statistical process control (SPC) charts] could support clinical decision-making and enable practices to track progress over time and apply evidence-based improvement methods, thus aligning the system more closely with Learning Health System principles.

While the development of EHR-based QIs is well established in some health systems such as the aforementioned UK (QOF) or the Netherlands (NIVEL [34]), our experience provides valuable insights for countries with fragmented infrastructures or limited national performance measurement frameworks. First, establishing a federated data platform such as healthdata.be proved essential for enabling privacy-compliant data exchange across multiple EHR vendors, a strategy that may be relevant where centralized data collection is not feasible. Second, early and structured engagement with EHR vendors through a neutral governance body (Agoria) facilitated interoperability and standardized implementation, illustrating the importance of multi-stakeholder governance models. Finally, voluntary participation was supported by financial incentives from the national health insurer (NIHDI), which ensured broad uptake without mandating compliance. These lessons highlight that successful implementation depends not only on technical solutions but also on governance, trust, and sustained investment in digital health infrastructure.

Strengths and limitations

The major strengths of our monitoring instrument project are scope and sustainability, as well as the use of a robust theory to design and inform the implementation process. In addition to the lack of active delivery mentioned earlier, a major current limitation of our project is the lack of process evaluation around the implementation of monitoring instruments. In other words, we have no data yet as to how GPs perceive and use the monitoring instruments. An essential next step is therefore to evaluate how general practitioners and practices engage with the feedback provided. While we have developed a 30-item questionnaire [29] and initiated its distribution, comprehensive evaluation—including usage analytics and qualitative feedback through focus groups—is planned for 2025. These efforts will inform refinements to the feedback design and integration strategy.

Conclusions

This study details the sustainable development of automated A&F monitoring instruments (locally known as barometers), in Belgian primary care for Covid-19 vaccinations, type 2 diabetes and appropriate use of antibiotics. As of 2024, more than 8 million individuals are covered by one or more of these instruments.

Author contributions

W.R., S.V.d.B., and B.V. conceptualized and supervised the research project. W.R. and J.S. were responsible for data curation, and jointly supervised formal analysis of the data. W.R. conceived of the research methodology. W.R. wrote a first draft of this manuscript, to which S.V.d.B., J.S., M.V.d.P., G.V.P., R.D.S., B.A., and B.V. offered substantial feedback and revisions. All authors reviewed and approved the final manuscript.

Supplementary material

Supplementary material is available at IJQHC online.

Conflicts of interest

B.V. is holder of a chair in Population Health Management at the University of Leuven financed by Novartis.

Funding

Development of the monitoring instruments was funded by the Belgian National Institute for Health and Disability Insurance (NIHDI). Intego was funded regularly by the Flemish Government (Ministry of Health and Welfare). We hereby state the independence of the authors from the funders.

Data Availability

De-identified data are available from the authors upon reasonable request.

Ethics approval and consent to participate

All structures processing data from citizens within the European Union fall within the remit of the GDPR of the European Union [30]. Within the legal entities defined by the GDPR, general practices act as data controllers of data gathered within the patient-physician relationship whereas KU Leuven is data controller regarding data processed to evaluate monitoring instruments. The healthdata.be platform acts as data processor. Data capture is aggregated, and no personal identifiable data are processed in the data flow through healthdata.be. The monitoring instrument project was assessed to comply with GDPR regulations by the Privacy and Ethics Team of KU Leuven, and received a favorable ethical review by the Social and Societal Ethics Committee (SMEC) of the KU Leuven [G-2023-6352-R2(AMD)]. Each participating practice gives informed consent to the project by activation of the monitoring instrument within their respective EHR system.