Abstract
Introduction
The second Slovenian national healthcare-associated infections (HAIs) prevalence survey (SNHPS) was conducted in acute-care hospitals in 2011. The objective was to assess the sensitivity and specificity of the method used for the ascertainment of six types of HAIs (bloodstream infections, catheter-associated infections, lower respiratory tract infections, pneumoniae, surgical site infections, and urinary tract infections) in the University Medical Centre Ljubljana (UMCL).
Methods
A cross-sectional study was conducted in patients surveyed in the SNHPS in the UMCL using a retrospective medical chart review (RMCR) and European HAIs surveillance definitions. Sensitivity and specificity of the method used in the SNHPS using RMCR as a reference was computed for ascertainment of patients with any of the six selected types of HAIs and for individual types of HAIs. Agreement between the SNHPS and RMCR results was analyzed using Cohen’s kappa coefficient.
Results
1474 of 1742 (84.6%) patients surveyed in the SNHPS were included in RMCR. The sensitivity of the SNHPS method for detecting any of six HAIs was 90% (95% confidence interval (CI): 81%-95%) and specificity 99% (95% CI: 98%-99%). The sensitivity by type of HAI ranged from 63% (lower respiratory tract infections) to 92% (bloodstream infections). Specificity was at least 99% for all types of HAIs. Agreement between the two data collection approaches for HAIs overall was very good (κ=0.83).
Conclusions
The overall sensitivity of SNHPS collection method for ascertaining HAIs overall was high and the specificity was very high. This suggests that the estimated prevalence of HAIs in the SNHPS was credible.
Keywords: healthcare-associated infections, prevalence, retrospective medical chart reviews, sensitivity, specificity, cross-sectional study, Slovenia
Izvleček
Uvod
Druga slovenska nacionalna presečna raziskava bolnišničnih okužb (SNPRBO) je potekala leta 2011 v slovenskih bolnišnicah za akutno oskrbo v okviru evropske presečne raziskave okužb, povezanih z zdravstvom, in uporabe protimikrobnih zdravil v bolnišnicah za akutno oskrbo. Cilj naše raziskave je bil oceniti občutljivost in specifičnost metode za prepoznavanje šestih pomembnih vrst bolnišničnih okužb (BO): okužb kirurške rane, okužb krvi, okužb, povezanih z žilnimi katetri, okužb sečil, okužb spodnjih dihal brez pljučnic in pljučnic (ki predstavljajo približno tri četrtine vseh BO) v Univerzitetnem kliničnem centru Ljubljana (UKCL).
Metode
Izvedli smo presečno raziskavo med bolniki UKCL, ki so bili vključeni v SNPRBO. Uporabili smo metodo retrospektivnega pregleda medicinske dokumentacije (RPMD) in evropske standardne definicije za namene epidemiološkega spremljanja BO. Izračunali smo občutljivost in specifičnost metode za prepoznavanje bolnikov z vsaj eno izmed šestih izbranih BO v SNPRBO v primerjavi z referenčno metodo RPMD in za posamezne vrste BO. Skladnost rezultatov SNHPS in RMCR smo ugotavljali s koeficientom kappa po Cohenu.
Rezultati
Od 1742 bolnikov, ki so bili vključeni v SNPRBO, smo v RPMD vključili 1474 (84,6%) bolnikov. Občutljivost SNPRBO metode za prepoznavanje bolnikov z vsaj eno od šestih izbranih vrst BO je bila 90% (95-odstotni interval zaupanja: 81%-95%). Specifičnost je bila 99% (95-odstotni interval zaupanja: 98%-99%). Ocenjena občutljivost za posamezne vrste BO je bila najnižja za okužbe spodnjih dihal brez pljučnic (63%) in najvišja za okužbe krvi (92%). Specifičnost za vse vrste BO je bila 99% ali višja. Skladnost rezultatov SNPRBO in RPMD glede prepoznanih BO je bila zelo dobra (κ=0,83). Najpogostejši vzrok za neprepoznavanje BO v SNPRBO je bil nepravilna uporaba definicij BO za namene epidemiološkega spremljanja oziroma njihovo slabo poznavanje. Pomanjkljiva medicinska dokumentacija v času RPMD pa bi bila lahko vzrok za neprepoznavanje nekaterih BO z RPMD.
Zaključki
Občutljivost metode, uporabljene za prepoznavanje BO v SNPRBO, je bila visoka in specifičnost zelo visoka. Skladnost pristopov prepoznavanja BO v SNPRBO in RPMD je bila zelo dobra. Zaključimo lahko, da je bila ocena prevalence BO v SNPRBO verodostojna. Temeljito usposabljanje zbiralcev podatkov za pravilno uporabo definicij BO za namene epidemiološkega spremljanja in zagotavljanje čim bolj popolnega beleženja podatkov, ki so pomembni za prepoznavanje BO v medicinski dokumentaciji, sta pomembni za izboljšanje občutljivosti metod prepoznavanja vseh vrst BO v bodočih nacionalnih presečnih raziskavah.
1. Introduction
Surveillance is an essential part of effective infection control programs (1). A cost-effective alternative to more resource-demanding prospective healthcare-associated infection (HAI) surveillance systems are point prevalence surveys of HAIs (2, 3).
To estimate the prevalence of all types of HAIs in all acute-care hospitals in Slovenia, the first national prevalence survey was conducted in 2001 (4). On the day of the survey, 4.6% patients had at least one HAI. Among the limitations of the survey, the researchers noted no piloting or validation of the data collection methods and a possibility that the sensitivity and specificity of approaches to ascertain HAIs in some participating hospitals were less than optimal. Ten years later, in 2011, the second Slovenian national HAIs prevalence survey (SNHPS) was conducted (5). The agreed standard methodology for the European point prevalence survey of HAIs and antimicrobial use in European acute-care hospitals (EUPPS) coordinated by the European Centre for Disease Prevention and Control (ECDC) was used (6). The estimated proportion of patients with at least one HAI on the day of the survey or still treated for any HAI on the day of the survey was 6.4% (5).
Data validity is a major issue in the surveillance of HAIs and the accuracy with which HAIs are ascertained varies considerably due to differences in experience, qualifications, training and awareness of surveillance staff, and consistency in the application of surveillance definitions for HAIs (7, 8). In order to contribute to the accuracy of results of future surveys, sensitivity and specificity of methods used should be determined regularly (9, 10). Recently, the results of the ECDC pilot validation of the EUPPS in 10 European Union (EU) Member States conducted in 2011 were published (11). The overall sensitivity and specificity of the method used for ascertainment of HAIs were estimated to be 83% and 98%, respectively. The sensitivity by type of HAIs ranged from 83% for bloodstream infections to 100% for lower respiratory tract infections, and specificity was higher than 99% for all types of HAIs. Due to limited resources at the time of the SNHPS, Slovenia did not participate.
The objective of our study was to estimate the sensitivity and specificity of the method used for the ascertainment of patients with any of the selected six types of HAIs (excluding neonatal infections) overall and for individual types of HAIs in the SNHPS conducted in 2011 in the largest Slovenian teaching hospital (the University Medical Centre Ljubljana - UMCL), in comparison to the reference method based on retrospective medical chart review (RMCR). The six selected HAIs were bloodstream infections (including microbiologically proven catheter related infections), catheter-related infections without bloodstream infections, lower respiratory tract infections (other than pneumonia), pneumoniae, surgical site infections, and urinary tract infections (asymptomatic bacteriuria excluded). These account for approximately three quarters of all HAIs. Our secondary objectives were to determine the level of agreement between the two HAI ascertainment methods, the SNHPS and RCMR, and to explore the reasons for discrepancies.
2. Methods
2.1. The Methods Used in the Slovenian National HAI Prevalence Survey
The methods were described elsewhere (5). In brief, a one-day cross-sectional study was conducted in all Slovenian acute-care hospitals in 2011. Before the start of the data collection, the national SNHPS coordination expert team trained all SNHPS co-ordinators for data collection in individual hospitals, who, except for one, were infection prevention and control physicians. The national SNHPS coordination expert team, together with the SNHPS hospital coordinators, trained hospital teams of SNHPS data collectors. Special attention was dedicated to good understanding of the method used for ascertainment of HAIs and European standard surveillance definitions for different types of HAIs (6). During the period of three weeks in October 2011, SNHPS data collectors collected standard information for all patients according to the SNHPS protocol. The presence of different types of HAIs, or ongoing treatment for these HAIs on the day of the SNHPS, was ascertained by reviewing all medical records available at the time of the survey and through consultations with attending physicians and nurses. The SNHPS data collectors ascertained HAIs by judging whether the criteria were fulfilled according to the SNHPS protocol and the surveillance definitions. They recorded the following information about each ascertained HAI: the code of the type of HAI; the exposure to relevant indwelling devices for pneumonia and bloodstream infection within 48 hours before the onset, and for urinary tract infection within seven days before the onset; the source for bloodstream infection (catheter related, secondary to another site (e.g. surgical site infection), other/unknown); the date of onset; presence at admission; and the source of HAI (current hospital, other hospital, other/unknown). The information about the presence of individual criteria for fulfilment of the European HAI surveillance definitions was not recorded.
2.2. Data Collection with Retrospective Medical Chart Review
We conducted a cross-sectional study. All patients enrolled into the SNHPS in the UMCL in 2011, with the exception of neonates, were eligible for RMCR. The RMCR team consisted of the primary data collector (microbiologist), infectious disease specialist (infection control physician) and epidemiologist, all with expertise in HAI surveillance. During the period from December 2012 to July 2013, more than a year after the SNHPS data collection, the primary data collector, blinded with respect to the HAI status of patients as ascertained in the SNHPS, reviewed all available medical documentation of all eligible patients. The data sources used, either paper forms or different electronic hospital information systems, included clinical information (medical charts, nursing care reports, antibiotics use records, temperature lists), laboratory and radiology reports. Extracted information was recorded on RMCR data collection forms. The data that had also been collected during the SNHPS included: patient’s hospital registration number, age, sex, hospital admission date, and SNHPS date. Patients were classified according to the McCabe severity of illness index, at the time of admission, into three categories: non-fatal diseases, ultimately fatal diseases (expected survival between one and five years), and rapidly fatal diseases (expected survival less than one year) (12). Exposures to indwelling devices (central vascular catheter, peripheral vascular catheter, urinary catheter), intubation during hospitalisation, and surgical procedures during a month preceding the survey, or, for insertion of implants, during 12 months preceding the survey, were recorded. Additional information not collected during the SNHPS included all individual criteria for the ascertainment of selected six types of HAIs according to the SNHPS protocol and European HAI surveillance definitions (6). For example, to be able to ascertain lower respiratory tract infection (other than pneumonia), recorded information included the presence of clinical or radiographic evidence of pneumonia and following signs or symptoms (with no other recognized cause): fever (>38 C), cough, new or increased sputum production, rhonchi, wheezing. In addition, the following information was recorded: positive culture obtained by deep tracheal aspirate or bronchoscopy; positive antigen test on respiratory secretions; organisms seen on smear or cultured from lung tissue or fluid, including pleural fluid; a lung abscess or empyema seen during a surgical operation or histopathologic examination; an abscess cavity seen on radiographic examination of lung.
2.3. Data Management and Analysis
The primary data collector checked the RMCR completed data collection forms for errors, missing information and internal inconsistencies. The data were double entered using Epi Info (Epi Info, version 7, CDC, Atlanta, GA, USA). Code range, filter, and some internal consistency checks were built-in. Discrepancies due to entry mistakes were checked against information recorded on RMCR data collection forms, and corrected. The data collected during the SNHPS for all patients surveyed by RMCR, including the data about the six types of HAIs, was added to the merged RMCR and SNHPS dataset. The records of individual patients were matched by using patients’ hospital registration numbers.
Data analysis was performed using SPSS (Statistical Package for the Social Sciences, version 21.0, Chicago, IL, USA). The overall proportion of patients with at least one of the six types of HAIs, or still treated for any of these on the day of the SNHPS, as ascertained by the RMCR among all patients included into RMCR (RMCR HAIs prevalence) and of patients with at least one of the six types of HAIs, or still being treated for any of these on the day of the SNHPS, as ascertained during the SNHPS among all patients included into the RMCR (SNHPS HAIs prevalence), was calculated. Respective prevalence estimates for individual types of HAIs were also calculated. The extrapolation of the RMCR results to the 1655 eligible patients (UMCL patients included into the SNHPS without neonates and duplicates) was performed. We used the same approach as described in Reilly et al. (11). The positive predictive value (PPV) and negative predictive value (NPV) were calculated from combined RMCR and SNHPS datasets. PPV was calculated as the percentage of patients with true HAI (as ascertained by RMCR - the “reference”) among all positive patients with at least one of the respective six types of HAIs as ascertained in the SNHPS, and NPV as the percentage of the true negative cases (as ascertained by RMCR - the “reference”) among all patients identified as negative as ascertained in SNHPS. We estimated the number of true positive patients with at least one of the six types of HAIs by multiplying the number of all positive cases in SNHPS with the PPV. The same procedure was performed for negative cases with the NPV (11). Using the RMCR ascertainment of HAIs as the ‘’reference’’, the sensitivity and specificity of the method used during the SNHPS (SNHPS method) for the ascertainment of patients with any of the selected six types of HAIs overall, and for individual types of HAIs were determined. The sensitivity of the SNHPS method refers to the ability of the method to correctly identify patients with HAIs. It was estimated by dividing the number of patients with any of the selected six types of HAIs detected with both methods by the number of patients with any of the selected six types of HAIs detected by RMCR, together with its 95% confidence interval (CI) (13). The specificity of the SNHPS method refers to the ability of the method to correctly identify those patients without HAIs. It was estimated by dividing the number of cases with no HAIs, as ascertained by both methods, by the number of cases with no HAIs by RCMR and its 95% CI (13). The same procedure was performed for calculating of the sensitivity and specificity of SNHPS method for the identification of individual types of HAIs. We used kappa (κ) statistics to analyse agreement between the two different HAIs ascertainment approaches. Kappa coefficient values between 0.81-1.00 were interpreted as very good agreement, values 0.61-0.80 as good, values 0.41-0.60 as moderate, values 0.21-0.40 as fair/marginal, and values below 0.2 as poor agreement. Negative values are possible and also denote poor agreement) (14, 15). 95% CIs were calculated. All discrepant cases with respect to HAIs, as ascertained during the SNHPS and by the primary RMCR data collector, were reviewed by the RMCR team, in order to explore the reason for discordance.
3. Results
1742 patients were surveyed in the UMCL, in the SNHPS. After excluding 9 duplicates and 78 neonates, 1655 patients remained eligible for RMCR. Of those, records were not available for 162 (9.3%) patients during the RMCR period and were insufficient for the ascertainment of possible HAIs for additional 19 (1.1%) patients. Thus, 1474 patients (84.6% of all surveyed in the SNHPS and 89.1% of all eligible for RMCR) were included in the RMCR. Characteristics of these 1474 patients and of 1655 patients surveyed during the SNHPS and eligible for RMCR are presented in Table 1. Both groups are very similar with respect to sex, age and McCabe index at admission, length of hospital stay and proportions with operation during the preceding month.
Table 1.
Characteristics of patients surveyed in the University Medical Centre Ljubljana, during the Slovenian national healthcare-associated infections survey (SNHPS) in 2011, eligible for the retrospective medical chart review (RMCR), and of those included into the RMCR.
| SNHPS population eligible for RMCR | RMCR population | ||
|---|---|---|---|
| The number of patients | 1655 | 1474 | |
| Sex | |||
| Women | 809 (48.9%) | 738 (50.1%) | |
| Men | 846 (51.1%) | 736 (49.9%) | |
| Age (at the time of admission) | |||
| Median | 60 | 60 | |
| Range | 0-98 | 0-98 | |
| Mean | 55 | 55 | |
| ≤49 years | 589 (35.6%) | 508 (34.5%) | |
| 50-79 years | 807 (48.8%) | 715 (48.5%) | |
| ≥80 years | 259 (15.6%) | 251 (17.0%) | |
| McCabe index (at the time of admission) | |||
| nonfatal disease | 1298 (78.4%) | 1183 (80.3%) | |
| ultimately fatal disease | 280 (16.9%) | 239 (16.2%) | |
| rapidly fatal disease | 66 (4.0)% | 52 (3.5%) | |
| unknown | 11 (0.7%) | 0 (0.00%) | |
| The length of hospital stay (until SNHPS day) | |||
| Median | 6 | 6 | |
| Range | 0 - 367 | 0 - 323 | |
| Mean | 13 | 12 | |
| 0-3 days | 594 (35.9%) | 533 (36.1%) | |
| 4-7 days | 396 (23.9%) | 369 (25.0%) | |
| 8-14 days | 267 (16.1%) | 230 (15.6%) | |
| ≥15 days | 398 (24.0%) | 342 (23.2%) | |
| Operations last month (until SNHPS day) | |||
| Surgery | 571 (34.5%) | 574 (38.9%) | |
| No surgery | 1084 (65.5%) | 900 (61.1%) | |
ain the case of implant, operations during 12 months, preceding the SNHPS
A primary prevalence, the proportion of the 1655 eligible patients with at least one of the six types of HAIs estimated in the UMCL in SNHPS, was 5.8% (95% CI: 4.8%-7.0%). The estimated prevalence of patients with at least one of the six types of HAIs ascertained during the SNHPS (SNHPS HAIs prevalence), among 1474 patients included into RMCR, was 5.4% (82 HAIs in 79 patients). The estimated prevalence of patients with at least one of the six types of HAIs on the day of SNHPS, ascertained by RMCR (RMCR HAIs prevalence) among these 1474 patients, was 4.8% (75 HAIs in 71 patients).
In comparison to the RMCR, the overall sensitivity of the data collection method used during the SNHPS for ascertaining at least one of the six types of HAIs among 1474 UMCL patients included into RMCR was 88.7% (95% CI: 79.0%-95.0%) and specificity 98.9% (95% CI: 98.2%-99.4%) (Table 2). Extrapolating these RMCR results to the 1655 patients eligible for RMCR, the overall sensitivity of the data collection method used during the SNHPS for ascertaining of at least one of the six types of HAIs among UMCL patients was 89.5% (95% CI: 81.1%-95.1%) and specificity 98.8% (95% CI: 98.1%-99.2%) (Table 3). The respective sensitivity by type of HAI ranged from 62.5% for lower respiratory tract infections (excluding pneumoniae) to 91.7% for bloodstream infections, while specificity was higher than 99.0% for all types of HAIs (Table 4).
Table 2.
Sensitivity, specificity, positive predictive value and negative predictive value of the method used for ascertaining any of the six types of healthcare-associated infections (HAIs) among 1474 patients enrolled at the University Medical Centre Ljubljana in the Slovenian national HAI prevalence survey in 2011, in comparison to the retrospective medical chart review.
| Retrospective medical chart review | ||||
|---|---|---|---|---|
| HAI | No HAI | Total | ||
| Slovenian national HAI prevalence survey | HAI | 63 | 16 | 79 |
| No HAI | 8 | 1387 | 1395 | |
| Total | 71 | 1403 | 1474 | |
Sensitivity: 63/71x100=88.7% (95% CI: 79.0%-95.0%) Specificity: 1387/1403x100=98.9% (95% CI: 98.2%-99.4%) Positive predictive value (PPV): 63/79x100=79.7% (95% CI: 64.9%-84.4%)
Negative predictive value (NPV): 1387/1395x100=99.4% (95% CI: 98.4%-99.5%)
Table 3.
Sensitivity, specificity of the method used for ascertaining any of the six types of healthcare-associated infections (HAIs) among 1655 eligible patients enrolled at the University Medical Centre Ljubljana in the Slovenian national HAI prevalence survey in 2011, in comparison to the retrospective medical chart review estimated by extrapolation of the results of the retrospective medical chart review conducted among 1474 patients.
| Retrospective medical chart review | ||||
|---|---|---|---|---|
| HAI | No HAI | Total | ||
| Slovenian national HAI prevalence survey | HAI | 77a | 19 | 96 |
| No HAI | 9 | 1550b | 1559 | |
| Total | 86 | 1569 | 1655 | |
77=PPVx96
1550=NPVx1559
Sensitivity: 77/86x100=89.5% (95% CI: 81.1%-95.1%)
Specificity: 1550/1569x100=98.8% (95% CI: 98.1%-99.2%)
Table 4.
Sensitivity and specificity of the method used for ascertaining six different types of healthcare-associated infections (HAIs) among 1474 patients enrolled at the University Medical Centre Ljubljana in the Slovenian national HAI prevalence survey in 2011, in comparison to the retrospective medical chart review and respective kappa coefficients by type of HAIs.
| Surveillance method | SSI | UTI | PN | LRI | BSI | CRI |
|---|---|---|---|---|---|---|
| RMCR (reference) | ||||||
| Number of HAIs | 15 | 21 | 19 | 8 | 12 | 0 |
| Prevalence of HAI episodes* (%) 95% CI |
1.0 (0.6-1.7) |
1.4 (0.9-2.2) |
1.3 (0.8-2.0) |
0.5 (0.3-1.1) |
0.8 (0.5-1.4) |
0 (0.0-0.3) |
| SNHPS | ||||||
| Number of HAIs | 15 | 20 | 26 | 7 | 13 | 0 |
| Prevalence of HAI episodes* (%) 95% CI |
1.0 (0.6-1.7) |
1.4 (0.9-2.1) |
1.8 (1.2-2.6) |
0.5 (0.2-1.0) |
0.9 (0.5-1.5) |
0 (0.0-0.3) |
| Sensitivity (%) 95% CI |
86.7 (59.5-98.3) |
76.2 (52.8-91.8) |
89.5 (66.9-98.7) |
62.5 (24.5-91.5) |
91.7 (61.5-99.8) |
/ |
| Specificity (%) 95% CI |
99.9 (99.5-100) |
99.7 (99.3-99.9) |
99.4 (98.8-99.7) |
99.9 (99.5-100) |
99.9 (99.4-100) |
100 (99.8-100) |
| Kappa coefficient 95% CI |
0.87 (0.74-1.00) |
0.78 (0.64-0.92) |
0.75 (0.61-0.89) |
0.67 (0.39-0.94) |
0.88 (0.74-1.00) |
/ |
on the day of SNHPS
SSI: surgical site infections. UTI: urinary tract infections. PN: pneumoniae. LRI: lower respiratory tract infections, excluding pneumoniae. BSI: bloodstream infections (including microbiologically proven catheter related infections). CRI: catheter-related infections without bloodstream infections. 95% CI: 95% confidence interval
The sensitivity by type of HAI ranged from 62.5% for lower respiratory tract infections (excluding pneumoniae) to 91.7% for bloodstream infections, while specificity was higher than 99.0% for all types of HAIs (Table 4).
The agreement between HAI ascertainment during the SNHPS and the RMCR for any of the six HAIs was very good (κ=0.83). The level of agreement across different types of HAIs was ranging from good for lower respiratory infections (excluding pneumoniae) (κ=0.67) to very good for bloodstream infections (κ=0.88).
13 episodes of HAIs (five urinary tract infection, three lower respiratory infections (other than pneumonia), two pneumoniae, two surgical site infections and one bloodstream infection) ascertained during the RMCR had not been ascertained during SNHPS, although all criteria for respective HAI surveillance definitions were fulfilled at the time of SNHPS data collection. This presumably resulted from difficulties with application of HAI surveillance definitions by SNHPS data collection teams.
20 episodes of HAIs had been ascertained during the SNHPS, but not during the RMCR. Among these, there were 9 pneumoniae, four urinary tract infections, three bloodstream infections, two surgical site infections and two lower respiratory tract infections. For most of these episodes, it was clear that they were false positive HAIs, since the criteria for ascertainment of HAIs had not been fulfilled. For example, four HAIs (two pneumoniae, one bloodstream infection, and one surgical site infection) occurred during current hospitalisation; however, sings and/or symptoms were no longer present and patients were no longer treated for them on the day of the SNHPS. One example involved a community-acquired urinary tract infection that was present on admission. Another example involved a bloodstream infection for which there was no evidence (none of the criteria for bloodstream infection surveillance definition fulfilled) in the medical documentation during RMCR. In contrast, in some cases of discrepancies between the SNHPS and RMCR results, HAIs might have been accurately ascertained during the SNHPS, while missed by the RMCR. It seemed possible that the documentation available to the SNHPS data collection teams (on signs and symptoms of HAIs and/or results of examinations) at the time of the SNHPS was no longer available to the RMCR team. For example, among 9 cases of pneumonia ascertained during the SNHPS, there were six cases in which all other criteria according to the surveillance definition were fulfilled during the RMCR, except for radiology evidence. Since in 2011 not all wards at the UMCL had started with electronic archiving of radiology reports, it is possible that radiology reports had been available to the SNHPS data collection teams at the time of the SNHPS, while they were no longer available to the RMCR team.
4. Discussion
The overall sensitivity of SNHPS collection method for ascertaining HAIs in the UMCL in 2011 was high, and specificity was very high. The level of agreement between the two data collection methods for ascertainment of these HAIs was very good overall. Although these results were obtained in only one hospital, where conditions may be different to those in other Slovenian hospitals, this may indicate that data collection methods used in the SHNPS were reliable in identifying HAIs, which is also reassuring with respect to credibility of overall SNHPS results (5). Sensitivity of the SNHPS method varied according to the type of HAI, which indicated greater difficulties in application of some surveillance definitions, and suggests that reliability of HAIs prevalence surveys’ data can be improved by better training data collectors in accurate implementation of HAI surveillance definitions. Some under-ascertainment of HAIs during the SNHPS, as well as during the RMCR, may have resulted from insufficient medical documentation. Good quality and completeness of medical documentation are crucial for accurate ascertainment of HAIs for surveillance purposes.
Slovenia did not participate in the ECDC pilot validation study of the EUPPS that enrolled 1950 patients from 20 acute-care hospitals from 10 EU Member States (11). The overall sensitivity and specificity of the method used for the ascertainment of patients with any of the selected six types of HAIs overall, in the SNHPS in the UMCL, as estimated by our RMCR (90%; 95% CI: 81%-95% and 99%; 95% CI: 98%-99%), were higher than the corresponding estimates in the EUPPS validation study (83%; 95% CI: 79%-87% and 98%; 95% CI: 98%-99%). In addition, the levels of agreement between our SNHPS and RMCR results and the EUPPS and EUPPS validation study results for HAIs overall were very similar (κ=0.83 and κ=0.81). Finally, in both validation studies, estimates for the sensitivity of primary data collection methods (SNHPS and EUPPS) varied by the type of HAI. In the SNHPS, it ranged from 63% for lower respiratory tract infections (excluding pneumoniae) to 92% for bloodstream infections, and in the EUPPS, from 83% for bloodstream infections to 100% for lower respiratory tract infections. Reilly et al. also emphasized the importance of the training of data collectors for accurate implementation of HAI surveillance definitions (11).
We should be cautious in comparing these results, since we used a blind RMCR data collection approach, while the countries participating in the ECDC pilot validation study of the EUPPS used a variety of methodological approaches, which included retrospective, simultaneous same day, simultaneous same time, blind and un-blind data collection. It should also be noted that the EUPPS validation study was conducted on the same day as the EUPPS. Thus, the availability of the data to both data collecting teams was very similar, while in our case, it is possible that some data available to the SNHPS team were no longer available to the RMCR team. Reilly et al. also emphasized the importance of good quality of medical documentation for accurate ascertainment of HAIs in prevalence surveys (11).
Relatively high estimated sensitivity for bloodstream infections (92%) and pneumoniae (90%) in the SNHPS may have resulted from a good knowledge and comprehension of respective surveillance definitions. Ascertainment of bloodstream infection requires a positive blood culture, and ascertainment of pneumonia a positive radiology result, in addition to the evidence of signs and symptoms, which is relatively straightforward. Good sensitivity for surgical site infections (87%) may have resulted from a good system for ascertainment of these infections in the UMCL and a high likelihood that surgeons note surgical site infections in the medical documentation. Our relatively low sensitivity for lower respiratory tract infections (63%) and urinary tract infections (76%) may have resulted from the non-recording of signs and symptoms of these infections in the medical documentation.
Our study allowed for a comprehensive RMCR review and ascertainment of the selected six types of HAIs when complete medical documentation was available. The strengths of our study included the high proportion of eligible individuals surveyed during the SNHPS enrolled into our RMCR (89%), and very similar characteristics of eligible patients to those surveyed during the RMCR. We tried to limit the measurement bias by blinding the primary data collector to the HAI ascertainment status of patients during the SNHPS. The major limitation of our RMCR may have been occasional non-availability and poor quality and incompleteness of some of the medical documentation available to the RMCR team more than a year after the SNHPS had been conducted. If the RMCR was conducted shortly after the data collection during the SNHPS, the availability of the data to both data collection teams would be more similar. However, it was clear, that signs and symptoms of patients, diagnostic procedures and results, treatment and care procedures are not always documented in such a way that it would be possible to ascertain all HAIs. To better estimate the sensitivity and specificity of the SNHPS method used during any future SNHPS, the RMCR should be conducted in several hospitals for better representativeness, and on the same day as SNHPS or shortly after, so as to avoid the unavailability of some of the data available during the SNHPS to the RMCR team. The results of our RMCR would be also more reliable if data were collected by two separate investigator teams, and if any discrepant result was examined before the final ascertainment of HAIs.
Finally, recent increase in the use of electronic healthcare information systems in Slovenian acute-care hospitals may make it possible to electronically harvest data for the purpose of HAI surveillance. Structured recording of information with respect to the criteria for the ascertainment of at least the most important types of HAIs according to the European surveillance definitions should be incorporated into healthcare information systems.
5. Conclusions
The overall sensitivity of SNHPS collection method for ascertaining HAIs in the UMCL in 2011 was high, and specificity was very high. Although these results were obtained in the UMCL, this indicates that the data collection methods used in the SHNPS are reliable in identifying HAIs, which is reassuring with respect to credibility of SNHPS published results (5). Reliability of HAIs prevalence surveys’ data can be improved by a better training of data collectors in accurate implementation of HAI surveillance definitions. Good quality and completeness of medical documentation are crucial in the accurate ascertainment of HAIs for surveillance purposes. Development and increasing use of electronic healthcare data systems is an opportunity for the development of less work-intensive electronic surveillance of HAIs, as an alternative to the traditional surveillance of HAIs.
Conflicts of Interest
The authors declare that no conflicts of interest exist.
Funding
The coordination of the implementation of the second Slovenian national point prevalence survey of HAI was co-financed by the Slovenian Research Agency and the Ministry of Health as applied research project (L3-2345). The work of the first author was co-financed by the Slovenian Research Agency (Contract Number 1000-09310192).
Ethical Aproval
The Republic Slovenia Medical Ethics Committee consented to the SNHPS protocol (consent number: 68/04/08). A study protocol of RMCR was approved by Republic of Slovenia National Medical Ethics Committee (consent number: 195/10/12).
References
- 1.Haley RW, Culver DH, White JW, Morgan WM, Emori TG, Munn VP. et al. The efficacy of infection surveillance and control programs in preventing nosocomial infections in US hospitals. Am J Epidemiol. 1985;121:182–205. doi: 10.1093/oxfordjournals.aje.a113990. [DOI] [PubMed] [Google Scholar]
- 2.Llata E, Gaynes RP, Fridkin S.. Measuring the scope and magnitude of hospital-associated infection in the United States: the value of prevalence surveys. Clin Infect Dis. 2009;48:1434–40. doi: 10.1086/598328. [DOI] [PubMed] [Google Scholar]
- 3.Humphreys H, Smyth ET.. Prevalence surveys of healthcare-associated infections: what do they tell us, if anything? Clin Microbiol Infect. 2006;12:2–4. doi: 10.1111/j.1469-0691.2005.01273.x. [DOI] [PubMed] [Google Scholar]
- 4.Klavs I, Bufon Lužnik T, Škerl M, Grgič-Vitek M, Lejko Zupanc T, Dolinšek M. et al. Prevalance of and risk factors for hospital-acquired infections in Slovenia-results of the first national survey, 2001. J Hosp Infect. 2003;54:149–57. doi: 10.1016/s0195-6701(03)00112-9. [DOI] [PubMed] [Google Scholar]
- 5.Klavs I, Kolman J, Lejko Zupanc T, Kotnik-Kevorkijan B, Korosec A, Serdt M. et al. Prevalence of and risk factors for healthcare-associated infections in Slovenia - results of the second national survey. Zdrav Var. 2016;55(4) doi: 10.1515/sjph-2016-0033. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.European Centre for Disease Control (ECDC) Point prevalence survey of healthcare - associated infections and antimicrobial use in European acute care hospitals - protocol version 4. 3. Stockholm: ECDC; 2012. http://ecdc.europa.eu/en/publications/publications/0512-ted-pps-hai-antimicrobial-use-protocol.pdf Available Dec 22, 2015 from. [Google Scholar]
- 7.Gastmeier P, Kampf G, Hauer T, Schlingmann J, Schumacher M, Daschner F. et al. Experience with two validation methods in a prevalence survey on nosocomial infections. Infect Control Hosp Epidemiol. 1998;19:668–73. doi: 10.1086/647896. [DOI] [PubMed] [Google Scholar]
- 8.McCoubrey J, Reilly J, Pollock KG, Johnston F.. Validation of surgical site infection surveillance data in Scotland. J Hosp Infect. 2005;61:194–200. doi: 10.1016/j.jhin.2005.02.014. [DOI] [PubMed] [Google Scholar]
- 9.Mannien J, van der Zeeuw AE, Wille JC, van den Hof S.. Validation of surgical site infection surveillance in the Netherlands. Infect Control Hosp Epidemiol. 2007;28:36–41. doi: 10.1086/509847. [DOI] [PubMed] [Google Scholar]
- 10.Fabry J, Morales I, Metzger MH, Russell I, Gastmeier P.. Quality of information: a European challenge. J Hosp Infect. 2007;65(2):155–8. doi: 10.1016/S0195-6701(07)60035-8. [DOI] [PubMed] [Google Scholar]
- 11.Reilly JS, Price L, Godwin J, Cairns S, Hopkins S, Cookson B. et al. A pilot validation in 10 European Union Member States of a point prevalence survey of healthcare-associated infections and antimicrobial use in acute hospitals in Europe, 2011. Euro Surveill. 2015;20 doi: 10.2807/1560-7917.es2015.20.8.21045. pii=21045. [DOI] [PubMed] [Google Scholar]
- 12.McCabe WR, Jackson GG.. Gram-negative bacteremia: I. Etiology and ecology. Arch Intern Med. 1963;110:847–55. [Google Scholar]
- 13.European Centre for Disease Control (ECDC) PPS Validation Protocol Version 2.1. Stockholm: ECDC; 2014. http://ecdc.europa.eu/en/publications/Publications/HAI-PPS-Validation-protocol-Sept-2014.pdf Available Dec 23, 2015 from. [Google Scholar]
- 14.Banerjee M, Capozzoli M, McSweeney L, Sinha D.. Beyond kappa: a review of interrater agreement measures. Can J Stat. 1999;27:3–23. [Google Scholar]
- 15.Landis JR, Koch GG.. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74. [PubMed] [Google Scholar]