Reliability and feasibility of registered nurses conducting web‐based surgical site infection surveillance in the community: A prospective cohort study

Abstract

Surgical site infections increase health care costs, morbidity, and mortality in 2% to 5% of surgical patients. Standardised post‐surgical surveillance is rare in community settings, causing under‐reporting and under‐serving of the documented 60% of surgical site infections occurring following hospital discharge. This study evaluated feasibility and concordance (inter‐rater reliability) of paired registered nurses using a web‐based surveillance tool (how2trakSSI, based on validated guidelines) to detect surgical site infections for up to 30 days after surgery in a cohort of 101 patients referred to Calea Home Care Clinics in Toronto, Canada, March 2015 to July 2016. After paired registered nurse assessors used the tool‐less than 10 minutes apart to measure concordance 5 to 7 days postoperatively, they provided feedback on its usefulness at two teleconference discussion groups September 6 to 7, 2016. Overall concordance between assessors was 0.822, remaining consistently above 0.65 across assessor education level and experience, patient age and weight, and wound area. Assessors documented 39.6% surgical site infection prevalence 5 to 7 days after surgery, confirming clinical need, relevance, reliability, and feasibility of using this web‐based tool to standardise community surgical site infection surveillance, noting that it was user‐friendly, more efficient to use than traditional paper‐based tools and useful as a registry for tracking progress.

Abbreviations

CCAC

continuing care centres

CDC

Centers for Disease Control and Prevention

CI

confidence Interval

H2TSSI

how2trakSSI© tool

OR

odds ratio

PDS

post‐discharge surveillance

PI

principal Investigator

RN

registered nurse

SSI

surgical site infection

WHO

World Health Organisation

1. INTRODUCTION

Surgical site infections (SSI) are the most common and costly hospital‐acquired infection, ¹ complicating up to 33 ² percent of the estimated 312.9 million surgical procedures performed globally each year. ³ An SSI reduces patient quality of life while increasing the likelihood of hospital re‐admission, patient mortality, and length of hospital stay by 9.7 to 11 days. ⁴ , ⁵ In Europe ⁶ and the United States, ⁷ estimated SSI incidence varies with type and location of surgery from 1% for uncomplicated cholecystectomy to 33% for complicated abdominal surgery. Higher rates of SSI occur in lower‐income countries. True SSI incidence is likely greater than these estimates, based on evidence that 60.1% of SSIs occur after standardised SSI surveillance ends when patients are discharged from the hospital. ⁵

Recognised as a costly, preventable, pervasive surgical complication, each SSI adds over $20 000 US per admission to hospitalisation costs in the United States. ⁴ It has been estimated that evidence‐based 30‐day surveillance and practice could prevent up to 60% of SSIs acquired in acute care. ⁸ As a result, standardised acute care SSI surveillance ⁹ now supports clinical practice quality improvement programmes and pay‐for‐performance metrics in most acute care settings. ⁴ Post‐discharge SSI surveillance is rarely conducted for the full 30 days after surgery as recommended in evidence‐based surgical SSl guidelines. ⁴ Most patients do not receive standardised SSI surveillance once they are discharged to home care within a week after surgery. Up to 50% of Canadian community patient care on any day is for wound management, including surgical wounds, (McIsaac C, Sibbald G, Woo K. Central West Community Care Access Centre: Prevalence Study. Personal communication of unpublished raw data analysis, 2009.) with wound infections accounting for 14% of all harmful wound incidents. ¹⁰ Studies implementing post‐discharge community SSI surveillance in Brazil, Canada, the United Kingdom, and the United States have all reported increased SSI prevalence as compared with monitoring SSIs in acute care alone. ¹¹ , ¹² , ¹³ , ¹⁴ More complete SSI surveillance across settings, documenting factors contributing to SSI development would help optimise resource use and SSI‐related clinical, patient‐centred and economic outcomes.

After literature searches exploring clinical literature on current SSI surveillance practices revealed no validated web‐based tools to track post‐discharge SSIs in Canada, the first author adapted the previously validated web‐based home care wound monitoring tool, How2Trak©¹ to include CDC SSI surveillance indicators. This adaptation, the H2TSSI was designed to facilitate regular, validated post‐discharge SSI surveillance and engage home care patients and their caregivers in reliable, valid SSI documentation and management. This study evaluated the feasibility, clinical relevance, and inter‐rater reliability (concordance) of using the web‐based H2TSSI tool to conduct standardised postoperative SSI surveillance in three home care clinics in Toronto.

2. METHODS

2.1. Study design

A prospective observational study evaluated inter‐rater reliability (primary outcome) and clinical feasibility (secondary outcome) of using the H2TSSI to identify and determine the prevalence of SSIs in patients referred by the Continuing Care Access Care Centre (CCAC) of Greater Toronto to the Calea Home Care Clinics for postoperative incisional care. Standardised video discussion group interviews evaluated RN assessor appraisals of H2TSSI clinical relevance and ease of use compared with prior paper SSI charting.

2.1.1. Setting

After appropriate review and approval by the Dalhousie Research and Toronto CCAC and Calea Clinics Ethics Boards, the study was carried out from April 2014 to March 2015 within three downtown Toronto, Ontario, Canada, Calea home care clinics experienced in using how2trak ^* on‐line applications to assess chronic wounds. These clinics were selected based on clinic leadership interest in addressing the unmet challenge of optimising community‐based SSI surveillance, ease of geographic access, and volume of patients receiving postoperative incision care: N = 1446 patients or 36% of the 4017 Calea Clinic referrals during that time.

2.2. Participants

After giving informed consent, 101 qualifying patients referred within 5 to 7 days after surgery by their CCAC Case Managers for surgical incision follow‐up at one of the three participating Calea Clinics were enrolled in the study for SSI surveillance for up to 30 days after surgery.

Clinical staff for the study was the principal investigator (PI), three research assistants (RAs), and a representative sample of 15 registered nurses (RNs) selected from the 18 RNs working at the Calea Clinics to balance nursing experience, willingness to sign consent forms, and availability to participate in the study. The PI trained all RN assessors attending one of two 60‐minute slide sessions describing study purpose, methods, and H2TSSI tool use.

Interested patients were recruited after Calea Clinic nursing or reception staff mentioned the study and directed them to the corresponding RA, who explained the study purpose, time commitment, and assured patients they would receive the same standard of care regardless of whether or not they participated. The PI screened each patient for inclusion and exclusion criteria, (Table 1) then the PI or RA obtained informed patient consent using a standardised consent form conforming to the Declaration of Helsinki principles. This included permission for patient withdrawal up to 30 days after participation, and consent to have the PI/RA review their Calea medical charts to collect study demographic and clinical data for H2TSSI data (Table 2).

TABLE 1.

Patient inclusion and exclusion criteria

Inclusion criteria	Exclusion criteria
Patients who had undergone surgery at one of the Toronto area hospitals	Patients without a surgical wound
Patients who were referred to the Calea Home Care Clinics within 5 to 7 d after their surgery through the CCAC for postoperative incisional care between March 2015 and July 2016	Patients with a surgical wound who had been discharged for more than 7 d after surgery
Aged ≥18 y old	Patients on service at Calea Home Care Clinic prior to initiation of study data collection
Signed an informed consent form to participate in the study	Patients who had undergone any surgery in which an implant was left in place
Patients who were willing to be seen at the Calea Clinic at least once during the first week after surgery and 30 d after surgery	Inability to understand the study procedure.
Able to converse and follow instructions in English

TABLE 2.

Electronic fields in the how2trak SSI data collection tool used by registered nurses evaluating all qualifying home care patients

Date of birth
Gender
Client ID code
Caseload ID code and geographic ID code
Ethnicity
Diabetes
Comorbidity factors
Date assessed
Hospital admission date
Surgery date
Surgical classification according to the CDC National Healthcare Safety Network 9 Please check only one below:
Class I/Clean: An uninfected operative wound in which no inflammation is encountered and the respiratory, alimentary, genital, or uninfected urinary tract are not entered. In addition, clean wounds are primarily closed, and, if necessary, drained with closed drainage. Operative incisional wounds that follow non–penetrating, blunt trauma should be included in this category if they meet the criteria. Class II/Clean‐Contaminated: An operative wound in which the respiratory, alimentary, genital, or urinary tracts are entered, under controlled conditions and without unusual contamination. Specifically, operations involving the biliary tract, appendix, vagina, and oropharynx are included in this category, provided no evidence of infection or major break in technique is encountered. Class III/Contaminated: Open, fresh, accidental wounds. This category also includes incisions from operations with major breaks in sterile technique (e.g., open cardiac massage) or gross spillage from the gastrointestinal tract, and incisions in which acute, non–purulent inflammation is encountered. Class IV/Dirty‐Infected: Old traumatic wounds with retained, devitalised tissue, and those that involve existing clinical infection or perforated viscera. This definition suggests that the organisms causing postoperative infection were present in the operative field before the operation.
Surgery site/location
Wound measurements: Two fields: (a) longest length in cm (b) longest perpendicular width in cm
CDC Signs and symptoms of surgical site infection 9 Please check all below relevant to this patient's incision:
Superficial surgical site infection Purulent drainage from the incision Organisms isolated from the culture of fluid or tissue from the incision At least one of the following and the superficial incision is deliberately opened by the surgeon Pain or tenderness Localised swelling Redness or heat Diagnosis of a surgical site infection by a physician	Deep surgical site infection Purulent drainage from the incision A deep incision spontaneously dehisces or is deliberately opened by a surgeon and has at least one of the following: Fever present, greater than 38° Celsius Localised pain or tenderness Abscess or other evidence of infection involving the incision is found by examination or radiological examination Diagnosis of a deep incision infection by a physician
Home care admission date

2.3. Development of the H2TSSI tool

To identify a valid conceptual framework for this research, explore prior post‐discharge SSI surveillance practices and use validated operational definitions and risk factors for SSI, the author searched PubMed, Cochrane Library Database, Science Direct, Wiley Web of Science, and Google Scholar databases between October 2000 and October 2017 using MeSH search terms for SSI, post‐discharge surveillance programmes, antibiotic prophylaxis, nosocomial infections, SSI in‐home care, patient safety and patient safety incidents in‐home care, harmful incidents, risk factors associated with the development of SSIs, and feasibility of electronic data collection tools.

SSI surveillance was conducted per CDC definitions ⁹ using the conceptual framework of the validated World Health Organisation International Classification for Patient Safety ¹⁵ reporting SSI outcomes in the WHO incident category of “healthcare‐associated infection.”

The operational definitions of surgical classification and of superficial or deep incisional SSI based on CDC Guidelines for SSI surveillance ⁴ , ⁵ , ⁶ , ⁹ were incorporated into the H2TSSI tool, plus risk factors for developing an SSI from the content validated International Consolidated Wound Infection Guideline, ¹⁶ listed with variables measured in this study in Table 2. Organ space SSI, which would require re‐admission to acute care to evaluate tissue manipulated during surgery, were not monitored in the home care setting of this study.

2.4. Protected health information and data privacy

All data collected during this study, including consent forms, discussion group data, and information were uploaded to, and contained in, the H2TSSI triple encrypted software storage database and protected by quality assessment procedures. Before clinical use, the secure H2TSSI system underwent security testing and evaluation by independent authorities who confirmed its compliance with the Personal Information Protection Electronic Documents Act (http://laws-lois.justice.gc.ca/eng/acts/P-8.6/index.html) and the US Health Insurance Portability and Accountability Act (https://www.hhs.gov/hipaa/for-professionals/privacy/laws-regulations/index.html) as well as a Privacy Impact Assessment (PIA) completed in 2017 by Grant Thornton privacy experts. These test results and all H2TSSI data were stored and maintained on a data server in Montreal, Canada, and backed up daily on a separate server, as required by the PIA. Any identifying data collected during this study were accessible only to the PI, RAs, and PI's dissertation committee and will be deleted, 5 years after study completion per ethics committee requirements.

2.5. SSI assessment

Two paired RN assessors independently assessed each patient's SSI 5 to 10 minutes apart, using the H2TSSI tool at the patient's clinic visit starting within 3 to 7 days of providing informed consent. Assessments were scanned and attached to the corresponding patient's records using H2TSSI. Each examination room had a desktop computer with on‐line access to the how2trak system. The participant met with the first RN, who logged into H2TSSI and entered patient information, antibiotic use, the number of dressing changes and a comprehensive standardised wound assessment, including measured wound length, width and depth, determination of wound area, pain, odour, granulation, exudate level, peri‐wound skin colour and oedema, undermining, and wound infection.

Each RN used an iPad ^† provided by the Calea Clinic to take a digital photograph of the surgical incision with the patient's name, identification number, date of birth, location, and brief clinical history. A white drape was used as background and a ruler was placed beside the wound to indicate wound size. A close‐up photo was taken with the camera body parallel to the subject. The RN uploaded the photograph to the how2trak system.

Next, the RN proceeded with the SSI assessment using the H2TSSI tool, which took approximately 3 to 5 minutes. If at least one CDC criterion was met, an SSI was determined to be present. The RN assessed the incision sites of the wound as having a superficial or deep infection. Then the first RN exited the room and a different RN performed a second independent H2TSSI assessment of the same site within 5 to 10 minutes. The second RN then dressed the wound, provided the patient with care instructions, then sent the patient home or for more follow‐up tests, or to their physician, or to the emergency room, as appropriate if SSI treatment was needed.

The two independent RN H2TSSI assessments populating two separate H2TSSI records for each unique patient were used to calculate the inter‐rater reliability of the H2TSSI tool. This measured concordance between RN pairs, assessing whether or not they agreed that the incision had an SSI (simple concordance) or agreement on both SSI presence and depth (exact concordance).

The PI returned to the study clinics regularly at least twice each month during SSI assessments and data collection, to assess each H2TSSI data field and ensure completeness of the dataset. RAs attended the clinics 3 days per week and remotely assessed H2TSSI weekly to review data collection. If there were portions of the tool that were not filled in, the PI and/or RA contacted the RN assessor to determine why there were missing data.

2.6. Follow‐up visit

For patients without an SSI on the initial visit, one trained RN assessor conducted a second H2TSSI assessment 30 days after surgery using the same SSI criteria listed in Table 2 plus an added query to patients whether or not they received treatment for an SSI. No inter‐rater reliability data were collected at these follow‐up visits as this goal of the study was met during the first visit. Data collected during the first 30 postoperative days were used to calculate the SSI prevalence rate for the study sample based on CDC SSI surveillance recommendations.

Some patients were assessed and treated more frequently than others depending on the frequency of care required for their postoperative incisional wound, but SSI information was only to be collected at the initial study assessment and at the 30‐day visit where applicable. If a patient with a follow‐up visit scheduled did not return to the clinic, the PI and/or RA conducted a follow‐up phone call and the patient self‐reported whether or not they had been treated for an SSI by a healthcare professional.

2.7. Discussion group RN assessor feedback

Upon completion of the SSI assessments, data collection and follow‐up visits, RN assessors participated in one of two thirty‐minute RA‐led discussion groups conducted between September 6 and 7, 2016 to identify workflow and process issues regarding the use of the H2TSSI tool and compare RNs' perspective on the use of the web‐based tool compared with the current practice of using non‐standardised paper‐based SSI documentation. Responses collected during the discussion groups were recorded and de‐identified and tabulated to reflect RN opinions and RN feedback on the H2TSSI tool.

To facilitate the discussion the RA asked the RN assessors the following open‐ended questions:

1. How would you describe your overall experience of using the H2TSSI tool?

2. How do you think the H2TSSI tool compares to the clinic's current method of assessing surgical patients?

3. How would you compare the H2TSSI tool with the clinic's current method of assessing surgical patients in terms of efficiency?

4. How would you compare the H2TSSI tool with the clinic's current method of assessing surgical patients in terms of user‐friendliness?

5. Overall, what effect, if any, do you think using the H2TSSI tool had on your delivery of care?

2.8. Statistical analysis

Patient sample size was originally estimated as 300 for β = 0.80 and α = 0.01 but reduced to 100 based on logistic feasibility, meeting significance criteria for β = 0.80 and α = 0.05. Descriptive tables reported the sample demographic and clinical characteristics of the patient and RN participants. Continuous data were summarised as a mean or median and standard deviation (SD). Reported SSI frequencies were summarised as proportions of the intent‐to‐treat patient sample.

The 202 SSI assessments by paired RN assessors were analysed using Chi‐square analyses with α error set at P ≤ .05 to test the effects of RN‐related, patient‐related, and wound‐related parameters on SSI reporting, using breakpoints based on data distribution patterns with sufficient data in each category for analysis. Generally, RNs were considered experienced if they had five to 10 years in a specific area. ¹⁷ , ¹⁸

Inter‐rater reliability correlations quantified concordance, defined as the degree of agreement between two or more coders who made independent ratings about the SSI in this set of subjects. ¹⁹ For this study, simple and exact concordance was determined for the RN pairs based on the three possibilities identified by SSI assessment coded and analysed separately according to CDC standardised criteria: no infection, superficial infection, or deep infection. All entries in the statistical database file in which there were duplicate entries for a patient‐ or wound‐ or RN‐related data were deleted leaving one concordance value for each patient.

During the SSI assessment, each RN assessor rated the patient's wound according to the eight H2TSSI criteria in Table 2 by clicking the relevant criteria applicable to each patient, determining the presence and depth of the patient's SSI. A simple agreement resulted if the two RN assessors agreed that there was an SSI. The exact agreement meant that the two agreed on both the presence and depth of the SSI.

Two methods were used to construct logistic regression models with multiple independent variables that affected the inter‐rater reliability of SSI surveillance. For the first multivariate model, all variables with marginal P values (<.1) were identified from the initial univariate analyses with the most significant variable entered in the first block, followed by the next most significant variable in the next block, until all variables had been added. This process was also done for other orders of the identified variables to ensure that other viable model combinations were not accidentally dismissed. The dispersion was calculated as Pearson deviance divided by degrees of freedom.

For the second multivariate model, all variables were entered in one block and the model was refined by removing the least significant variables one‐by‐one. Odds ratios (OR) and the 95% confidence intervals (CIs) for effects of each parameter on inter‐rater reliability were also calculated.

The PI summarised additional feasibility measures identified during the discussion groups, including practical benefits of using H2TSSI, its ease in use, efficiency, the effect on the delivery of care, clinical work‐flow strengths and limitations, and the overall experience of using the tool as compared with paper documentation. The discussion group notes were transcribed by the PI who summarised key points articulated by the RN assessors and identified patterns or themes in the data.

RN assessments were used to determine the prevalence of SSIs among study patients. An SSI was noted if it occurred during the initial SSI assessment or at any point up to, and including clinic visit or phone follow‐up on day 30 following the surgery.

For SSIs identified during the initial assessments, each superficial or deep SSI identified by one RN in an RN pair was counted as 0.5 SSI so that SSI prevalence was reported as one per‐patient if both RNs agreed. To include SSI identified by only one member of an RN pair, all 0.5 counts were totalled for each category of depth to determine prevalence. For SSIs reported at follow‐up each SSI was counted as one per patient.

3. RESULTS

3.1. Patient population

Of 1432 patients with postoperative incisions referred to Calea Clinics during this study; 316 (22.1%) interested in participating in the study were screened for study eligibility (Figure 1). Among 315 who signed consent forms, scheduling issues were the main reason for excluding all but 109 patients who presented for their initial SSI assessment. After excluding three duplicate‐listed patients and five who received only one RN SSI assessment, 101 patients assessed by two paired RNs 5 to 7 days after surgery using H2TSSI were evaluated in the analyses.

FIGURE 1.

Patient flow chart

All 101 patients were called for 30‐day follow‐up, but by the time they returned to be assessed using H2TSSI, the 30‐day period had expired for all but 61 patients qualifying for follow‐up assessment.

Table 3 presents the baseline included patient and wound characteristics. Most patients were overweight Caucasian non‐smokers, younger than 60 years of age, who abstained from alcohol use, and did not have diabetes mellitus. Most wounds were less than one square centimetre (1 cm²) in area, with no wound‐related pain reported.

TABLE 3.

Patient and wound baseline characteristics of the 101 intent‐to‐treat patients

Characteristic	Patient N (%) or mean (SD/range)
Patient gender
Male	55 (54.5%)
Female	46 (45.5%)
Race/ethnicity
African	3 (3%)
Asian	7 (6.9%)
Arab	4 (4%)
White	38 (37.6%)
Hispanic/Latino	2 (1.9%)
Unknown	47 (46.5%)
Patient age
Sample mean (SD)	46.9 (16.8)
20 to 49 y	60 (58.4%)
50 to 79 y	39 (38.6%)
≥ 80 y	2 (2%)
Body mass index
Sample (range)	27.2 (18.9 to 42.5)
18.5 to 25.0	29 (28.7%)
25.1 to 29.9	30 (29.7%)
≥30	22 (21.8%)
Unknown	21 (20.8%)
Current smoker
Yes	16 (15.8%)
No	77 (76.2%)
Unknown	8 (7.9%)
Patient alcohol use
Yes	26 (25.7%)
No	60 (59.4%)
Unknown	15 (15.9%)
Patient diabetes
Yes (HbA1c > 7)	8 (7.9%)
No (HbA1c ≤ 7)	93 (92.1%)
Wound areas
Less than 1 cm2	63 (62.4%)
At least 1 cm2	38 (37.6%)
Wound‐related pain
None (VAS = 0)	70 (69.3%)
Pain (VAS ≥ 1)	31 (30.7%)

The 25 patients over 60 years of age were three times as likely to have a reported superficial SSI as compared to a deep SSI (P = .02), while this likelihood of SSI depth did not differ for younger patients (Table 3). SSI was more likely for the 38 patients with a wound larger than 1 cm². Superficial SSI were more likely than deep SSI (P = .04) for these larger wounds. The presence of wound‐related pain was strongly associated with the likelihood of an SSI (P = .0009). A superficial SSI was more likely to be reported than a deep SSI in those experiencing wound‐related pain. Body mass index (BMI) did not affect the likelihood of superficial as compared with deep SSI reporting.

3.2. RN assessor ratings of SSI categories

Each of the 15 RN assessors assessed a mean of 13 patients (SD 8.1, range 4‐28). Figure 2 displays the percent of the 202 paired RN assessor diagnoses in each SSI category. Most (122) paired observations agreed that the surgical site was not infected. Forty‐four ratings agreed on a rating of “Superficial SSI.” Eight agreed on a rating of “Deep SSI.” For surgical sites receiving different ratings by the paired RN assessors, 15 and 13, respectively, rated the SSI as “Superficial” or “Deep.”

FIGURE 2.

Surgical site infections (SSI) identified by paired RN assessors rating 101 home care patients' surgical sites 5 to 7 days after surgery

3.3. SSI prevalence in‐home care within 30 days after surgery

Estimating SSI prevalence conservatively as a percent of the 101 patients reported by both RN assessors as having an SSI at the time of the initial observation 5 to 7 days after surgery, the prevalence of SSIs among the 101 patients with postoperative incisions evaluated at Calea Home Care Clinic was 25.7% (N = 26). Of these, 22 patients (21.7%) had a superficial SSI and 4 (3.9%) had a deep SSI (Figure 3).

FIGURE 3.

SSI prevalence during the first week of community clinic care following surgery on 101 patients by paired RN assessors using how2trakSSI or by clinical observation or single RN assessor report 30 days after surgery per SSI surveillance recommendations. Each RN assessor observation counted 0.5 for each patient so each patient's ratings totalled to one SSI if they agreed

If the 15 superficial and 13 deep SSIs reported by only one RN of the pair were added to prevalence calculations, using 0.5 to report each RN assessor rating made the superficial and deep SSIs identified by a single RN assessor, respectively, 7.5 and 6.5, resulting in a total SSI prevalence of 39.6%, counterbalancing the 60.4% of patients diagnosed by both RN assessors as without an SSI.

Of the 61 patients without a reported SSI on days 5 to 7 after surgery, 30‐day follow‐up assessments were performed on 25 using H2TSSI (n = 8), telephone reporting (n = 6), or patient‐reported treatment for an SSI (n = 11). During this optional follow‐up assessment performed by one RN assessor, three more patients reported one SSI each: two superficial and one deep SSI. This brought the total 30‐day postoperative SSI prevalence to 43 SSI experienced by the 101 patients during post‐discharge home care or 42.6%.

3.4. RN assessor population

Fifteen RNs out of 18 Calea nursing staff (83.3%) consented to participate in this study and were trained as RN assessors. Table 4 presents the planned Chi‐Square analyses of associations of RN assessor demographics, patient, and wound variables with reporting of a deep or superficial SSI during the first week after surgery. The mean RN age was 39.3 years (SD: 11.7), with 7 RNs (46.7%) being at least 40 years old. Most RN assessors had a degree (n = 10, 66.7%). Each of the five RNs who did not have a degree had a nursing diploma. The majority of RNs were experienced and had at least 4 years of experience at Calea Clinic. Only three (20%) had ten or more years of surgical experience.

TABLE 4.

Registered nurse (RN) home care assessor (n = 15) or patient and wound demographics associated with surgical site infection (SSI) reporting within 5 to 7 days after surgery

Characteristic (N of assessments)	Mean (SD) or number ofpatients (%)	Reported not an SSI (N, %)	Reported superficial SSI (N, %)	Reported deep SSI (N, %)	Effect on SSI reporting (P value) a
RN assessors characteristics
Age, y (N = 198)	39.3 (11.7)				.038
<40 y	8 (53.3%)	53 (73%)	6 (8%)	14 (19%)
≥40 y	7 (46.7%)	78 (62%)	28 (22%)	19 (15%)
Degree (N = 198)					.076
Yes	10 (66.7%)	42 (58%)	18 (25%)	13 (18%)
No	5 (33.3%)	89 (71%)	16 (13%)	20 (16%)
Nursing experience (N = 198)	13.7 (9.1)				.066
<10 y	7 (46.7%)	51 (74%)	6 (9%)	12 (17%)
≥10 y	8 (53.5%)	80 (62%)	28 (22%)	21 (16%)
Surgical experience (N = 198)	4.2 (5.5)				.047
<10 y	12 (80%)	104 (70%)	20 (14%)	24 (16%)
≥10 y	3 (20%)	27 (54%)	14 (28%)	9 (18%)
Calea clinic experience (N = 198)	3.9 (3.0)				.141
<4 y	5 (33.3%)	67 (63%)	17 (16%)	23 (22%)
≥4 y	10 (66.7%)	64 (70%)	17 (19%)	10 (11%)
Wound and patient variables
Wound area (N = 202)					.0497
≥1 cm2	63 (62.4%)	88 (70%)	16 (13%)	22 (18%)
<1 cm2	38 (37.6%)	45 (59%)	20 (26%)	11(15%)
Patient age (N = 202)					.0234
≥60 y	76 (75.2%)	103 (68%)	21 (14%)	28 (18%)
<60 y	25 (24.7%)	30 (60%)	15 (30%)	5 (10%)
Body Mass Index (N = 160)	27.2 (18.9 to 42.5)				.4899
BMI ≤ 25	29 (28.7%)	35 (63%)	11 (20%)	10 (18%)
BMI > 25	51 (50.5%)	74 (71%)	14 (14%)	16 (15%)
Wound‐related pain (N = 202)					.0009
VAS no pain	70 (69.3%)	102 (73%)	16 (11%)	22 (16%)
VAS pain ≥1	31 (30.7%)	31 (51%)	20 (32%)	11 (18%)

^a P values less than .05 are in bold font.

3.5. Effects on SSI reporting

RN assessors over 40 years of age were more likely to report no or superficial SSI than younger RNs (Table 4; P = .038). There were generally fewer SSI diagnoses by RNs with at least 10 years of surgical experience compared with those with less surgical experience (P = .047). RNs with no degree or with more than 10 years of nursing experience were marginally less likely to report SSI, respectively, compared to those with a degree (P = .076) or those with over 10 years of nursing experience (P = .066).

Home care clients reporting wound pain above zero were more likely to be diagnosed with any depth SSI (P = .0009). Among the 62 patients reporting surgical site pain, half (31) were assessed as having an SSI. Those over 60 years of age (P = .023) or those with wounds over 1 cm² in the area (P = .0497) were more likely to be diagnosed with a deep SSI.

3.6. Effects of variables studied on paired RN SSI surveillance concordance

The overall simple concordance among 99 RN assessor pairs (n = 198 assessments) was high: 0.822 [83/101; 95% CI: 0.73‐0.89] with no significant SSI concordance differences reported between RNs of different ages, clinical or surgical experience, or education levels. The overall exact concordance for 202 RN assessments on 101 surgical sites for the presence of superficial infections was 0.782 (79/101; 95% CI: 0.69‐0.86), or 0.819 for deep infections (68/83; 95% CI: 0.72‐0.90).

No significant effects were found of the RN‐centric, variables analysed on the odds ratios and CI of simple or exact concordance in univariate analyses. Patient‐reported wound pain was the sole variable significantly affecting exact concordance on SSI presence and depth. Paired RN assessors were less likely to agree on SSI presence and depth if the VAS pain rating was one or two (OR 0.24, CI 0.064‐0.91; P = .036) or if the VAS pain rating was over two (OR 0.25; CI 0.076‐0.80; P = .020). Paired RN assessors were marginally more likely to agree if the patient's BMI exceeded 0.25 (OR 3.07, CI 0.99‐9.51; P = .052).

Multiple logistic regression analyses, including all parameters (Table 5) found only BMI > 25 significantly increasing simple concordance of paired RN assessors (OR 7.85, CI 1.08‐56.97; P = .042). Exact concordance of the paired RNs on SSI presence and depth increased if both RNs had a degree (OR 26.75, CI 1.31‐54.86; P = .033) or if patient BMI exceeded 25 (OR 7.45, CI 1.34‐42.27; P = .022). the likelihood of exact concordance marginally increased (P = .082) if both paired assessor RNs had over 10 years of surgical experience.

TABLE 5.

Variables affecting simple or exact concordance of SSI reporting by pairs of registered nurse (RN) home care assessors or client or wound characteristics based on multiple regression

Concordance type	Simple (% reported similarly)	Exact (% reported similarly)	Deep (% reported similarly)	P value for effect on simple concordance	P value for effect on exact concordance
characteristic (number of surgical sites)
RN age (97)				1.0	.420
Both <40 y	68%	63%	71%
Both ≥40 y	84%	82%	86%
One <40, one ≥40	88%	82%	83%
RN degree (97)				1.0	.033
Both yes	82%	74%	77%
Both no	100%	89%	80%
One yes one no	80%	80%	85%
Years of nursing experience (97)	13.7 (9.1)			1.0	.400
Both <10 y	79%	79%	83%
Both ≥10 y	83%	78%	83%
One <10, one ≥10	84%	78%	79%
Years of surgical experience (97)	4.2 (5.5)			.12	.082
Both <10 y	78%	73%	77%
Both ≥10 y	100%	100%	100%
One <10, one ≥10	86%	84%	87%
Years at calea home care (97)				.45	.38
Both <4 y	79%	72%	75%
Both ≥4 y	85%	90%	93%
One <4, one ≥4	83%	77%	81%
Client or wound variables (101)
Wound area (101)				.87	.350
At least 1 cm2	81%	78%	83%
Less than 1 cm2	84%	79%	79%
Patient age (101)				.35	.36
At least 60 y	84%	80%	78%
Less than 60 y	82%	78%	83%
BMI (80)				.042	.022
More than 25	87%	87%	91%
At least 25	82%	68%	67%
Client‐reported wound pain (101)
VAS pain 0	85%	85%	87%
VAS pain > 0	76%	62%	65%
VAS pain = 1 to 2				.48	.011
VAS pain > 2				.16	.037

For simple or exact SSI reporting, the only concordance values below 0.7, the recognised lower limit for good concordance, were observed if both RN assessors were younger than 40 years of age (Table 5). Exact or deep SSI concordance was also <0.7 for patients with a BMI of 25 or less, or for patients reporting any wound pain.

All types of concordance tended to increase with years of clinical, surgical, or Calea experience and were perfect (1.0) for the 3 RN pairs who both had over 10 years of surgical experience. However, limited sample sizes of pairs with greater experience prevented these differences from reaching statistical significance compared with concordance values of 0.73 to 0.78 for RN pairs with no surgical experience.

3.7. Feasibility of web‐based SSI surveillance per RN discussion groups

Eight of the 15 RN assessors participated in a group discussion (53.3%): 6 RN assessors participated in the first discussion group and two participated in the second discussion group. Seven RNs were unable to participate because of lack of availability (n = 4), being on maternity leave (n = 1), is no longer employed with Calea (n = 1), or have had a change of position (n = 1). The RN assessors provided feedback on their overall experience using H2TSSI, how it is compared with paper charting (including its efficiency and user‐friendliness), and the effect that the H2TSSI tool may have had on their delivery of care.

3.7.1. Overall experience using the H2TSSI tool

The RNs generally agreed that the tool was user‐friendly, simple, and easy to use, with one RN also commenting that “it's fairly easy for patients to use as well.” The RNs reported that because they already had experience using the how2trak tool for general wound care assessment, it was easier for them to use.

3.7.2. How the H2TSSI tool compared to paper charting

The RN assessors noted that an important benefit of H2TSSI was as a centralised registry of all of each patient's data and wound photographs in one place. This facilitated monitoring each patient's progress as compared with traditional paper charting. RN assessors also reported that H2TSSI improved ease of photo uploading and storage as compared with paper charting. Given the option to choose one method over the other, they agreed that they would prefer to use H2TSSI compared with paper charting. However, the RNs commented that for the purposes of this study, using H2TSSI resulted in a “duplication” of their work, as they were still required to paper chart SSIs in addition to the web‐based tool.

The RN assessors also discussed the feasibility of a potential patient‐oriented how2trak application, which was under development but not tested during the study. A patient application would allow patients to sign into the how2trak system, track their wound's progress with regards to SSI, and upload photos of their wounds. One RN expressed concern that patients would not be able to tell the difference between a general wound infection and an SSI.

One RN reported that the SSI criteria tested via H2TSSI were appropriate for the clinical setting. However, some RNs also commented on the confusion when checking off antibiotic use in the past 30 days on H2TSSI. They noted that many patients spoke of having intravenous antibiotic therapy that the RNs would never have used for surgical wounds and questioned the value of having to check off general antibiotic use for SSI assessment.

3.7.3. The efficiency of H2TSSI compared to paper charting

The RNs reported that ease and efficiency of taking, storing, and retrieving incision pictures at each H2TSSI evaluation enabled users and their patients to efficiently monitor wound progress and/or deterioration. “This process is not available when using traditional paper charting and is a major advantage,” one RN explained, “…because there are many RNs working in the clinic, so if…one of the RNs goes and sees a patient for the first time we can go back to see the pictures taken before and compare how the wound is looking.” That same RN also found added efficiency in being able to motivate patients with their wound pictures, “so they can see the difference between the initial appointment and then after the next assessment.”

3.7.4. Ease of H2TSSI Use compared to paper charting

With its photo storage capabilities and consolidation of data, the RN assessors agreed that H2TSSI was overall more user‐friendly than traditional paper charting. They suggested ways in which this tool could be further optimised to suit the needs of its user(s) and be better tailored to the specific needs of each clinic's experience. For example, the RNs explained that it would be helpful if (a) the number of dressing change visits were counted by the application; (b) they could specify which dressing product they were applying to each patient; (c) the application was mobile so they did not have to upload the digital photographs to the application on cumbersome desktop computers.

3.7.5. Effect of H2TSSI on delivery of care

A major theme that emerged regarding the delivery of care was that regardless of which method was used to assess SSIs, RNs required education to provide and consistently deliver a good quality standard of care to optimise outcomes. The RNs agreed that H2TSSI assisted them in consistently delivering that standard of care. One RN assessor remarked, “…if we compare this one [H2TSSI] to paper, I would say, yes, it would help with the patient outcomes.”

4. DISCUSSION

4.1. Major findings

Trained RN assessors using standardised web‐based H2TSSI tracking reliably reported a 39.6% incidence of SSI in patients discharged to home care during the first week after surgery. These findings suggest that standardised home care SSI surveillance is crucial, especially during the first week after surgery and confirms earlier findings and validated guidelines recommending continued SSI surveillance for up to 30 days after surgery. (4,6, 9, 16).

A practical application of these findings is that the how2trak surgical site surveillance application provides a reliable, convenient method for trained home care RNs to conduct surgical site surveillance in the home care setting, which is becoming the norm, rather than clinic‐based care.

Concordance, measured as mean inter‐rater reliability of trained RN assessors using the H2TSSI web‐based tool for home care SSI surveillance was high (82%) with no simple SSI concordance differences reported between RNs of different ages, clinical or surgical experience, or education levels. Exact concordance on SSI presence and depth decreased only if both RN assessors had a university degree or if patients had a BMI of 25 or less or reported wound pain. Further research is needed to determine how these variables obscure or mask clues to signs of SSI depth. Despite these significant trends in exact concordance, overall paired RN concordance was high.

Among the 202 assessments, the 28 wherein paired RN assessors disagreed on SSI presence or depth suggest the value of confirming SSI evaluation by a second trained health care professional, for example to ascertain causes of patient‐reported pain (e.g. trauma, pressure, foreign matter) at sites of potentially deep SSIs.

Standardised interviews of RN assessor experiences engaged in the reliability testing reported that this method of SSI surveillance was preferable to paper charting because it was easier to use and to monitor the patient and wound progress on the H2TSSI registry. The tool was described as useful for providing motivating feedback to patients and it was suggested that patients be trained to recognise SSI signs and symptoms as well. Participants also emphasised the importance of training assessors on how to recognise and categorise SSIs as part of a home care surveillance programme.

4.2. Clinical implications

This work provided compelling evidence that trained RNs can reliably conduct standardised SSI surveillance using the H2TSSI tool in community settings. The high incidence of SSI supports CDC and other evidence‐based guidelines ⁴ , ⁶ , ⁹ , ¹⁵ recommending SSI surveillance during the first 30 days postoperatively across settings to identify and manage SSI early enough to optimise patient outcomes. The fact that RN assessors agreed on reporting SSI status of 87 (86.1%) of the 101 intent‐to‐treat patients observed supports the high reliability and feasibility of H2TSSI‐trained RN assessor surveillance in‐home care during the clinically important first week after surgery. Despite significant trends in exact concordance on SSI depth, overall paired RN concordance on SSI presence was high, indicating that a wound observation by one trained RN would suffice to identify the presence of an SSI. Two trained RNs were needed to assess each wound for an SSI only to measure inter‐rater reliability in the study. This would not be required during routine clinical practice.

4.3. Limitations

The literature review conducted for this research preceded the use of newer strategies, including telehealth, mobile health (mHealth), or smartphones to provide real‐time patient‐provider image sharing and communication to conduct SSI surveillance in the community.. ²⁰ , ²¹

The current updated version of the how2trak SSI surveillance application allows for both synchronous and asynchronous patient‐provider connections. After each patient photographs his/her incision the resulting assessment report is shared with the physician and team who may then either (a) review and advise the patient through secure messaging or (b) schedule a virtual visit if needed. Surveyed current H2TSSI users expressed a very high level of satisfaction with the application.

Counting single observer SSI reports as 0.5 avoided inflation of SSI prevalence numbers, but it did not resolve the clinical question of which of these 28 patients had a genuine SSI that needed treatment. This suggests the merit of clinical practice, including a blinded second opinion by an H2TSSI‐trained medical professional to confirm an SSI assessment before treatment is initiated in order to limit unneeded antibiotic use in clinical practice. Further research is needed to explore the effects of such practice on clinical and economic outcomes.

One limitation of this research was that it excluded organ space SSI. Home wound care RNs were not in a position to evaluate organ space SSI, which would have required re‐admission to acute care. Further research is needed on organ space SSI assessment.

It is important to note that of the 61 patients qualifying for follow‐up in this study, only 25 actually reported SSI outcomes 30 days after surgery. The three new SSI observed at follow‐up may represent an under‐estimate of SSI at 30 days. This aspect of SSI surveillance merits further study.

4.4. Opportunities for research

These findings suggest the value of further research exploring the clinical and economic outcomes of using this valid, reliable web‐based tool to standardise post‐discharge SSI surveillance across settings extending through the full 30 days recommended by infection control authorities.

4.5. Opportunities for education

There is a critical need for a standardised methodology for postoperative SSI surveillance following hospital discharge. Education is needed to train health care personnel and patients on rigorous standardised post‐discharge SSI surveillance across the health care continuum. Web‐based tools such as the H2TSSI can help facilitate these educational programmes while populating wound and SSI registries to monitor and manage related outcomes.

5. CONCLUSION

This research confirms the clinical need, relevance, reliability, and feasibility of RNs using the web‐based H2TSSI tool to standardise SSI community surveillance after surgical patients are discharged from the hospital to home care.

CONFLICT OF INTEREST

The first author discloses that she is President and Change Agent, Health Outcomes Worldwide, Sydney, Nova Scotia, the source for how2trak web‐based wound monitoring tools, which she founded to empower home health care professionals to engage in standardised, evidence‐based outcomes measurement.

McIsaac C, Bolton LL. Reliability and feasibility of registered nurses conducting web‐based surgical site infection surveillance in the community: A prospective cohort study. Int Wound J. 2020;17:1750–1763. 10.1111/iwj.13464