Can we simplify the hospital accreditation process? Predicting accreditation decisions from a reduced dataset of focus priority standards and quality indicators: results of predictive modelling

Abstract

Objectives

Accreditation in France relies on a mandatory 4-year cycle of self-assessment and a peer review of 82 standards, among which 14 focus priority standards (FPS). Hospitals are also required to measure yearly quality indicators (QIs—5 in 2010). On advice given by the accreditation committee of HAS (Haute Autorité en Santé), based on surveyors proposals and relying mostly on compliance to standards, accreditation decisions are taken by the board of HAS. Accreditation is still perceived by hospitals as a burdensome process and a simplification would be welcomed. The hypothesis was that a more limited number of criteria might give sufficient amount of information on hospitals overall quality level, appraised today by accreditation decisions.

Design

The accuracy of predictions of accreditation decisions given by a model, Partial Least Square-2 Discriminant Analysis (PLS2-DA), using only the results of FPS and QIs was measured. Accreditation decisions (full accreditation (A), recommendations or reservation (B), remit decision or non-accreditation (C)), results of FPS and QIs were considered qualitative variables. Stability was assessed by leave one out cross validation (LOOCV).

Setting and participants

All French 489 acute care organisations (ACO) accredited between June 2010 and January 2012 were considered, 304 of them having a rehabilitation care sector (RCS).

Results

Accuracy of prediction of accreditation decisions was good (89% of ACOs and 91% of ACO-RCS well classified). Stability of results appeared satisfactory when using LOOCV (87% of ACOs and 89% of ACO-RCS well classified). Identification of worse hospitals was correct (90% of ACOs and 97% of ACO-RCS predicted C were actually C).

Conclusions

Using PLS2-DA with a limited number of criteria (QIs and FPS) provides an accurate prediction of accreditation decisions, especially for underperforming hospitals. This could support accreditation committees which give advices on accreditation decisions, and allow fast-track handling of ‘safe’ reports.

Article summary.

Article focus

• Can we simplify hospitals accreditation's process?

• Can we reduce accreditation's burden?

• Can we use a small number of criteria to predict accreditation's results?

Key messages

• A prediction model was shown 90% accurate in predicting accreditation decisions.

• Detecting high-risk quality hospitals was effective (97%).

• If focus priority standards (FPS) were to be assessed annually, we could have a regular estimation of the level of quality of hospitals.

Strengths and limitations of this study

• Leave one out cross validation showed excellent stability of the prediction model.

• The choice of the criteria (FPS and quality indicators) can be discussed.

• Only acute care organisations with an eventual additional rehabilitation sector were considered.

Introduction

In many countries, hospitals’ quality is assessed through accreditation reports and quality indicators (QIs).^1–3 Accreditation has been developed to improve the quality of care and patient safety.^4–11 Moreover, quality indicators have been introduced in the accreditation process in order to give a quantitative assessment of hospitals’ quality. For instance, the Joint Commission has integrated accountability measures into the accreditation process¹; the Australian Council on Healthcare Standards (ACHS) clinical indicator (CI) programme was established to facilitate the measurement of important aspects of health services for the members of the ACHS EQuIP accreditation programme.¹²

The French accreditation programme, set since 1999, nowadays in France called certification, is a government-sponsored initiative, patient centred and orientated to continuous quality improvement (CQI).^{13 14} Hospitals’ accreditation procedure is a mandatory 4-year cycle organised in a few steps. First, a hospital assesses itself on 82 standards defined by the National Authority for Health (‘Haute Autorité de Santé’—HAS), 14 of which being Focus Priority Standards (FPS). Hospitals evaluate their compliance to each standard with a score from 0% to 100%, which leads to four classes (A, B, C or D) from best to worst. FPS are weighted according to achieved steps in continuous improvement programmes concerning these specific standards. Five months later, surveyors conduct an external evaluation of several of the same standards during a structured visit where hospitals’ assessments are confirmed or modified. They assess all FPS, other standards where hospitals are low and a couple of standards randomly selected. A report is then submitted to the hospital, which has an opportunity to question it within a month, and then to an independent review panel—HAS accreditation committee—which then prepares an advice, according to standards quotations, on the level of accreditation.¹⁵ Unlike other countries where accreditation decisions are built entirely on computed algorithms,¹⁶ accreditation decisions are finally taken by the board of HAS and reports are made public and transmitted to regulatory agencies.

In the healthcare accreditation context, a standard is defined as “a desired and achievable level of performance against which actual performance is measured.”¹⁷ Standards are generally considered as important means by which to improve clinical practices and organisational performance. They were also reported to improve organisational and staff efficiency.¹⁸

Several countries have developed the use of FPS considered as essential or priority for accredited centres. Such essential standards are used within the national accreditation programme for breast centres in the USA.^{19 20} The Joint Commission has also developed a set of essential standards focused ‘on important patient, individual or resident care and organisation functions that are essential to providing safe, high quality care’.²¹ In the Canadian accreditation programme, nine Required Organisational Practices were integrated within the standards.²² In Australia, the EQuIP accreditation programme contains 13 essential standards that are assessed by the ACHS surveyors during the accreditation process.²³ In France, among the 82 standards available in the accreditation manual, 14 FPS have been defined.¹⁵ These 2010 FPS linked to critical activities, on which HAS expects a high level of compliance. They are especially focused on the quality and safety of care in hospitals and are systematically assessed during accreditation visits.

In addition, since 2008, a mandatory collection of standardised QIs has been set in France. They have been publicly reported since 2010, and partially integrated in the accreditation decision process. In other countries, QIs have also been used in the accreditation programme. For instance, since 2012, the Joint Commission has integrated performance expectations on accountability measures into accreditation standards when it introduced the following performance improvement requirement: the hospital improves its performance on ORYX accountability measures; the hospital achieves a composite performance rate of at least 85% on the ORYX accountability measures.²⁴ In Australia, the ACHS Clinical Indicator programme has been set to facilitate the measurement of important aspects of health services during the EQuIP accreditation programme. Indeed, performance monitoring provides evidence for use in accreditation as indicators are collected, monitored and benchmarked and as participation in the CI programme is included in all ACHS healthcare accreditation.²⁵ Implementing QIs has been proved useful by promoting development of quality measurement, answering to the public demand for transparency and stimulating quality improvement when publicly reported.²⁶

Even though measuring quality and safety of care in hospitals seem an essential step, the accreditation process was found bureaucratic and time consuming.⁷ The French procedure has already been revised in 2010¹⁵ but hospitals as well as surveyors may still resent its burden. Simplification of the accreditation procedure may be useful and welcome.

Our hypothesis was that a limited number of criteria may give a sufficient amount of information on hospital performance concerning quality improvement, which could help surveyors to better organise the visits and provide decision support to HAS accreditation committee. To test this hypothesis we assessed the accuracy of a prediction model of accreditation decisions based only on the results of FPS and QIs.

Methods

Population and dataset

The study was based on a dataset of all 489 acute care hospitals accredited between June 2010 and January 2012, including 304 hospitals with an additional rehabilitation care sector (RCS).

Accreditation decisions were divided in three classes: full accreditation (A); recommendation or reservation (B) and respite of accreditation decision or non-accreditation (C).

Among the 82 standards of the 2010 accreditation procedure, 14 FPS concerned the acute care sector and 7 additional FPS were specific to the RCS. FPS were standards that the HAS considers as essential. No selection of FPS was undertaken (table 1). Each FPS was assessed by surveyors with four possible decisions: ‘Compliance’, ‘Recommendation’, ‘Reservation’ and ‘Major Reservation’. For hospitals without RCS, FPS concerning the RCS were scored ‘Not Applicable’ (NA).

Table 1.

List of FPS and QIs

FPS	QIs
1f—continuous professional development and organisation	PR—quality of patient record
8a—quality and safety improvement programme	DSI—delay in sending information at hospital discharge
8f—management of adverse events	PS—pain status evaluation
8g—control of infection risk	NS—nutritional status evaluation
9a—system for managing claims and complaints	PAN—quality of patient anaesthetic record
12a—pain management
13a—care and rights of patients at the end of life
14a—management of the patient record
14b—patients’ access to their medical record
15a—patient identification at all stages of treatment
20a—drug management (hospital level)
20abis—drug management (patient level)
25a—management of emergencies and unscheduled care
26a—organisation of the operating room

A detailed description of FPS (criteria and modality of assessment) can be found in: http://www.has-sante.fr/portail/jcms/r_1439924/fr/manuel-de-certification-des-etablissements-de-sante-v2010-revise-2011

Information on quality indicators may be obtained from: http://www.has-sante.fr/portail/jcms/fc_1249986/fr/indicateurs-de-qualite-et-de-securite.

FPS, focus priority standards; QI, quality indicator.

In addition, the values of all five mandatory QIs collected in each hospital in 2010 and 2011 were considered, plus four QIs specific to RCS (table 1). In France, values of QIs could not be extracted automatically, as, in 2011, only 6% of hospitals had fully computerised medical records. QIs were thus computed from a random sample of 80 medical records in each hospital.²⁵ Hospitals were classified depending on the QI value and its normal CI into four classes: hospital's score was: (1) significantly higher (t test) than the target of 80% demanded in France (class ‘1’); (2) not significantly different from 80% (class ‘2’); (3) lower than 80% (class ‘3’); and hospital did not respond to its obligation to collect data (class ‘4’). Hospitals with less than 30 patient records were scored ‘Not Applicable’ (NA).

A dataset was obtained for each hospital, including its initial accreditation level coded into three classes; the surveyors’ decisions for the 14 FPS (21 FPS for ACO-RCS) coded into five classes; and the class level for the five mandatory QIs (9 for ACO-RCS) coded into five classes.

We conducted a univariate analysis of the variables in the dataset in order to describe the frequency of hospitals in the different classes for each FPS and QI. We also investigated the link between the accreditation classes and both FPS and QIs by running χ² tests. We then performed a multivariate analysis in order to predict initial accreditation decisions from FPS’ assessments and QIs’ classes. A statistical method was considered: Partial Least Square-2 Discriminant analysis (PLS2-DA).^26–28

PLS2-DA model

PLS2-DA is an efficient statistical technique that is especially suited to deal with a large number of predictors with multicollinearity,²⁹ which is the case in our study. Considering that the response variable (accreditation decision) is multivariate, PLS components explain all the responses simultaneously and the method is called PLS2 in contrast to the univariate PLS regression. The PLS2 model was run on the complete disjunctive table in order to produce a set of PLS2-components summarising the explanatory variables and taking into account the categorical response variable. Discriminant analysis was then run on the PLS2 components to predict an accreditation class for each hospital.^{30 31} For example, if a hospital has a posteriori probabilities of being respectively, A, B, C: 0.30, 0.60, 0.10 then the given hospital will be ranged in the B accreditation class.

Validation of the model

Agreement levels between predicted and actual accreditation classes were determined by Cohen-weighted κ coefficients and a percentage of well-classified hospitals.³² With the weighted form of κ, different kinds of disagreement are not equivalent (eg, a top vs bottom disagreement is worse than a top vs neutral disagreement).³³

In addition, leave one out cross validation (LOOCV) was computed in order to assess the stability of the model:³⁴ PLS2-DA was run on all hospitals but one to construct the model. The hospital left apart had its actual class compared with its predicted class. This process was repeated for each hospital. Predicted and observed responses were then compared to assess the stability of the method: external confusion matrices were developed. A Wald proportion statistical test was carried out to compare the proportion of hospitals well classified with LOOCV and without LOOCV.

Distribution of a posteriori probabilities was given for each class.

Results

Most hospitals belonged to the medium accreditation class (77%), 12% of hospitals were ranged A and 11% C.

Distribution of FPS and QIs in classes is given by table 2. Some FPS, had a high level (‘Compliance’) for almost all hospitals such as ‘Pain Management’; ‘Patient needs at the end of life’; ‘Patient access to medical record’; ‘Patient identification’; ‘Operating room’; ‘Adverse event’; ‘Infectious risk’ and ‘Claims and complaints’. For some other FPS, such as ‘Drug management—hospital level’, the distribution of hospitals across the classes was more balanced. For example, for FPS_20a, 41% of hospitals were ranged ‘Compliance’, 31% had a recommendation of improving the quality of care, 25% a reservation and 3% a major reservation.

Table 2.

Distribution of variables (focus priority standards – FPS and quality indicators-QIs) in classes and χ² tests (p value)

FPS	Compliance	Recommandation	Reservation	Major reservation	NA	p Value
1f—evaluation of clinical practices policy	74%	17%	8%	1%		<0.0001
8a—quality and safety improvement programme	78%	17%	5%			0.0331
8f—adverse events	81%	12%	7%			<0.0001
8g—infectious risk	95%	4%		1%		0.0107
9a—claims and complaints	98%	2%				0.0002
12a—pain management	85%	14%	1%			<0.0001
12a—(RCS)-pain management	50%	8%	0%		42%	<0.0001
13a—patient needs at the end of life	81%	10%	1%		8%	0.0003
13a—(RCS)-patient needs at the end of life	45%	6%	0%		49%	0.0004
14a—patient record	62%	30%	8%			<0.0001
14a—(RCS)-patient record	34%	20%	4%		42%	<0.0001
14b—patient access to medical record	96%	4%				0.0007
14b—(RCS)-patient access to medical record	54%	5%			42%	<0.0001
15a—patient identification	80%	15%	5%			<0.0001
15a—(RCS)-patient identification	45%	9%	4%		42%	<0.0001
20a—drug management (hospital level)	41%	31%	25%	3%		<0.0001
20a—(RCS)-drug management (hospital level)	28%	18%	12%	0%	42%	<0.0001
20abis—drug management (patient level)	53%	5%	2%	0%	40%	0.0449
20abis—(RCS)-drug management (patient level)	35%	4%	0%	0%	61%	0.1130
25a—emergency room	53%	6%	2%	0%	39%	0.0005
26a—operating room	81%	4%	6%	1%	9%	<0.0001
QIs	‘1’	‘2’	‘3’	‘4’	‘NA’	p Value
PR—quality of patient record	11%	16%	73%			0.0155
PR—quality of patient record RCS	3%	5%	46%	3%	43%	0.0418
DSI—delay in sending information at hospital discharge	2%	4%	94%			0.5492
DSI—(RCS)-delay in sending information at hospital discharge	14%	15%	26%	3%	43%	0.0092
PS—pain status evaluation	29%	12%	58%			0.0004
PS—(RCS)-pain status evaluation	13%	4%	38%	3%	43%	0.0006
NS—nutritional status evaluation	39%	29%	32%			0.0274
NS—(RCS)-nutritional status evaluation	8%	11%	36%	3%	43%	0.0437
PAN—quality of patient anaesthetic record	35%	14%	27%			0.0022

QI, quality indicator; RCS, rehabilitation care sector.

Concerning QIs’ distributions, a high proportion of hospitals belonged to a low class (26–94% in class ‘3’ or ‘4’) and a low percentage to the high class (2–39% in class ‘1’ or ‘2’).

Significant associations were shown between accreditation decisions and the results of each FPS and each QI (χ² tests, p values in table 2), except for the delay in sending information QI.

The PLS2-DA model was used to predict accreditation decisions. Contingency tables were produced: table 3 shows the concordance between actual and predicted accreditation classes, and the same results given by LOOCV, first for acute care organisations (ACO) without an RCS, second for hospitals with an RCS (ACO-RCS). Substantial agreement was observed between actual versus predicted accreditation decisions (κ=0.70 for ACOs, 0.73 for ACO-RCS). Prediction was found consistent: for ACOs, 89% of hospitals were well classified (respectively, 91% for ACO-RCS). Prediction was accurate in 80% of the 35 ACO ranged A (72% of 25 ACO-RCS). Likewise, 9 of 10 ACO (97% of 33 ACO-RCS) ranged C were actually C. In addition, on the 35 ACOs actually A, 80% were ranged A (72% of the 25 ACO-RCS). Seventy-five per cent of ACOs actually ranged C were predicted C for (80% of ACO-RCS).

Table 3.

Actual versus predicted accreditation level without and with LOOCV for (1) acute care organisations; (2) acute care organisations with a rehabilitation care sector (number of hospitals)

ACO	1. PLS2-DA Prediction				ACO	1. PLS2-DA Prediction with LOOCV
		A	B	C			A	B	C
Actual	A	28	7	0	Actual	A	26	9	0
	B	10	127	1		B	10	127	1
	C	0	3	9		C	0	4	8
ACO-RCS	2. PLS2-DA Prediction				ACO-RCS	2. PLS2-DA Prediction with LOOCV
		A	B	C			A	B	C
Actual	A	18	7	0	Actual	A	15	10	0
	B	12	226	1		B	13	224	2
	C	0	8	32		C	0	8	32

ACO: acute care organization; LOOCV: leave one out cross-validation; PLS2-DA: Partial Least Square-2 Discriminant Analysis; RCS: rehabilitation care sector.

When computing LOOCV, we observed no difference concerning stability: the percentage of well-classified hospitals remained at 87% for ACOs; 89% for ACO-RCS. A Wald test showed no statistical difference between the proportions of well-classified hospitals with and without LOOCV (p=0.647 for ACO; p=0.65 for ACO-RCS).

For each hospital, a posteriori probabilities of belonging to each of the three accreditation classes were computed, the sum being of 1 for each hospital. Figure 1 shows a posteriori probabilities distributions for ACO-RCS (similar results were observed for ACOs). A hospital was ranged in a class according to its higher probability. We observed that hospitals predicted A and actually A had a posteriori probabilities of actually being A always close to 1, which means that we can be confident in the result given by the model. On the contrary, when a posteriori probabilities were close to 0.5, there were misclassifications (for instance for hospitals actually ranged A and predicted B). To resume, we highlighted the fact that when a posteriori probabilities were close to 1, hospitals were well classified. Nevertheless, when the model made a mistake in predicting hospitals’ class, a posteriori probabilities were close to 0.5 in most hospitals. This is true but for three ACOs and eight ACO-RCS whose actual class was C and which were given a high a posteriori probability of belonging to the class B.

Figure 1.

Distribution of a posteriori probabilities in each class, actual and predicted, computed from Partial Least Square-2 Discriminant Analysis, in hospitals without Rehabilitation Care Sector.

Discussion

In France, accreditation is a mandatory 4-year procedure, built on the same method than those used in many countries.^{35 36} Even though the accreditation procedure supports the quality of care, it has been criticised, particularly owing to the important workload of hospitals’ self-assessment. The self-assessment validation process by surveyors has also been considered time consuming. At last, the accreditation procedure represents an important financial expenditure considering that surveyors are made available for hospitals and compensated by HAS. Simplification and priorisation efforts are unanimously requested by stakeholders, all the more considering the obligation for French hospitals to collect regularly mandatory QIs. Moreover, they are only partly taken into account in the accreditation process. Our aim was to test if a restricted number of all essential standards and all 2010 French QIs were sufficient to assess the level of quality reached by hospitals, currently measured by its accreditation decision, and to detect underperforming hospitals.

The most important result of this study was that a model based on these few variables, using the PLS2-DA method, reproduced satisfying accreditation decisions (89% of ACOs were well classified, this number is 91% for ACO-RCS) taken by the French National Authority for Health after the surveyors’ visit validating 82 standards. Even though some hospitals were misclassified by the model, the results are stimulating. This is true especially for hospitals predicted C by the model, which actually had a C level in 97% of cases for ACO-RCS (90% for ACOs). Results were reproducible and trustworthy as LOOCV showed stability of the model: percentage of well-classified hospitals is a good tool to appraise validity of statistical models.³⁷ LOOCV has been considered as a standard procedure for an almost unbiased assessment of the error rate of classification models.^38–40 In our study, 89% of ACO-RCS (87% of ACOs) were shown well classified with this method.

A posteriori probabilities could be used to increase confidence in the model predictions: when hospitals were correctly predicted, a posteriori probabilities were always close to 1, whereas when hospitals were misclassified, a posteriori probabilities were mostly closer to 0.5. Consequently, with the PLS2-DA method, a predicted accreditation class should be given with its a posteriori probability. PLS2-DA method can be used to detect underperforming hospitals with a very small risk of error (3%). However, 11 hospitals (3 ACOs, 8 ACO-RCS) were predicted B when their actual accreditation class was C. It appeared regretful that not all worse hospitals were detected. These 11 hospitals were more deeply examined. Their QI scores and FPS decisions were close to those of the actual class B and their a posteriori probability of being classified B was high. However, in these hospitals surveyors gave a major reservation on other standards than FPS, concerning the proper use of antibiotics, safety of information system or management of endoscopy, not taken into account in the model (because non FPS).

The PLS2-DA model was chosen as there were constraints regarding the dataset: qualitative variables, multicollinearity issue and not applicable data. We can regret that these models do not take into account the ordinal characteristic of the variables (‘Compliance’ is better than ‘Major Reservation’) and thus some information could have been lost. However, the choice was made to consider all variables as qualitative in order to use models best suited to data and to avoid both the multicollinearity and the missing data issues.

The PLS2-DA method was stable as most hospitals were ranged in the actual accreditation class when using LOOCV. The method has been used in several studies to analyse genomic data. For instance, it was used to predict tumour classification from cDNA microarray gene expression data,^41–43 and in the fields of bioinformatics, genomics and proteomics or chemometrics sciences.^{29 43–46} The technique is suited to deal with a large number of predictors with multicollinearity.²⁹ It was reported to lead to high prediction accuracy.^{47 48} In our study it gave a strong answer to the multicollinearity issue and did take the accreditation variable into account when constructing PLS2 components. In addition, no selection of initial variables was needed.

There are limits to this study. First, the choice of the variables taken into account in this study can be discussed. Only FPS were kept among the 82 standards assessed by hospitals and validated by surveyors. However, all FPS were considered as they are unavoidable standards for the accreditation visit. This choice seemed relevant as FPS explore issues most relevant to safety and quality of care. A good compliance to these standards implied that the hospital had been through a complete cycle of quality improvement and achieved a sufficient level of quality. However, the list of FPS could be updated since compliance to some of them is already very high and since other standards than FPS can lead to major reservations. Considering the choice of QIs, all 2010 mandatory QIs were included in the study. QIs are only partly taken into account in the accreditation process, thus there was a slight duplication issue by considering them in the model. The number of mandatory QIs has recently increased in France; in 2014, 26 QIs will be mandatory, including, for example, appropriate care of acute myocardial infarction patients at discharge and of postpartum haemorrhage. The hospitals’ involvement in the collection of patients’ experiences will also be assessed by accreditation. The study will therefore have to be reproduced with all mandatory QIs and with an updated list of FPS at this time. Thus, the ideal set of variables is still to be found in order to detect underperforming hospitals and the model might be improved.

Second, a performance assessment based on a reduced set of variables could lead to bias. Hospitals could be tempted to manipulate data. A quality control of QIs is already organised in randomly selected patients’ files and the survey can detect improper assessment of FPS compliance. Hospitals may also concentrate their efforts on these topics, which could lead to undermine the benefits of a hospital-wide programme. However the aim of our study was not to propose a substitute to the accreditation process. The 2014 to come accreditation version will recentre hospitals towards continuous improvement activity.⁴⁹ It will ask hospitals to provide, annually or every 2 years, statements on the achieved level and planned actions concerning CQI of processes, particularly of those assessed by FPS. These informations will be used to prepare more targeted, customised visits. We believe that a model based on FPS and QIs results, by highlighting poor performers, could allow fast-track handling of ‘safe’ reports, and therefore help accreditation committee, and help surveyors to prepare customised visits.

Third, only ACO with an eventual additional rehabilitation sector were considered. This study should thus be completed with all hospital sectors, including psychiatric and home care sectors.

As a conclusion, even though there are limits to this study, we succeeded in predicting accreditation decisions using only FPS and QIs. As most hospitals were ranged in the actual accreditation class with the PLS2-DA method, the prediction of accreditation decisions from a reduced set of FPS and QIs was found accurate enough to conceive a simple assessment of hospital performance concerning quality improvement. Detecting high-risk quality hospitals was effective. The results given by the model at an early phase of the visit could help surveyors in organising their on-site time more effectively and centre a valuable survey time on priority processes and systems, enhancing the value of accreditation to the hospital. This could also support the HAS accreditation committee, when proposing final accreditation decisions according to surveyors reports, by highlighting poor performers and thus allowing fast-track handling of ‘safe’ reports. Finally, if FPS were to be assessed on an annual or biannual basis, which is relevant if one assumes that a CQI programme is necessary in these priority topics, we could have a regular estimation of the level of quality of hospitals. However, it would not replace the whole visit process, which offers additional advantages.