Inter-rater Agreement and Checklist Validation for Postoperative Wound Assessment Using Smartphone Images in Vascular Surgery

Abstract

Objective

Surgical site infection (SSI) is the most common nosocomial infection, particularly in vascular surgery patients who experience a high rate of readmission. Facilitating transition from hospital to outpatient care with digital image-based wound monitoring has the potential to detect and enable treatment of SSI at an early stage. In this study we evaluate whether smartphone digital images can supplant in person evaluation of postoperative vascular surgery wounds.

Methods

We developed a wound assessment checklist using previously validated criteria. We recruited adults who underwent a vascular surgical procedure between 2014 and 2015, involving an incision of at least 3cm in size from a high-volume academic vascular surgery service. Vascular surgery care providers evaluated wounds in person using the assessment checklist; a different group of providers evaluated wounds via a Smartphone digital image. Inter-rater agreement coefficients (AC) for wound characteristics and treatment plan were calculated within and between 1) the in-person group and 2) the digital image group; the sensitivity and specificity of digital images relative to in person evaluation were determined.

Results

We assessed a total of 80 wounds. Regardless of modality, inter rater agreement was poor to when evaluating wounds for the presence of ecchymosis and redness, moderate for cellulitis and high for the presence of a drain, necrosis or dehiscence. As expected, the presence of drainage was more readily observed in person. Inter rater agreement was high for both in-person and image-based assessment with respect to course of treatment, with near-perfect agreement for treatments ranging from antibiotics to surgical debridement to hospital readmission. No difference in agreement emerged when raters evaluated poor-quality compared to high-quality images. For most parameters, specificity was higher than sensitivity for image-based compared to gold-standard in-person assessment.

Conclusions

Using Smartphone digital images is a valid method for evaluating postoperative vascular surgery wounds and is comparable to in-person evaluation with regard to most wound characteristics. The inter-rater reliability for determining treatment recommendations was universally high. Remote wound monitoring and assessment may play an integral role in future transitional care models to decrease readmission for SSI in vascular or other surgical patients. These findings will inform smartphone implementation in the clinical care setting as wound images transition from informal clinical communication to becoming part of the care standard.

Introduction

Providers are increasingly recognizing the utility of technology-based remote monitoring for patients who are discharged home after surgery.¹ Ensuring normal postoperative recovery at home has traditionally required phone calls by hospital staff or home health nurse visits. However, as payers hold providers accountable for post-discharge outcomes and postoperative lengths of stay decrease, implementing efficient and effective wound monitoring protocols via telemedicine is timely and feasible. Because the majority of American adults now own a smartphone or tablet with photograph and video capability,² post-discharge monitoring via digital photos and video conferencing presents an opportunity to detect early wound complications without patients' traveling to clinic or nursing staff investment.

Surgical site infection (SSI) represents a unique opportunity for telemedicine image-based monitoring given the visual component to its development. In addition, SSI is the most common nosocomial infection in surgical patients, accounting for 38% of post-operative complications.^3,4 The majority of SSIs develop after hospital discharge³ and are often not discovered until the routine post-operative visit 2-3 weeks later⁵ because patients rarely recognize the early stages of a SSI.^6,7 Post-operative wound monitoring via digital images would permit traditional wound checks to be performed by an experienced clinician earlier than current convention. These patient-generated digital images could facilitate earlier diagnosis and thus less invasive wound care with potential avoidance of readmission and reintervention.

Telemedicine-based transitional care programs that focus on surgical patients are currently in development.^1,8,9 However, two fundamental gaps require investigation prior to widespread implementation of such programs. First, a simple, validated, and comprehensive tool to evaluate post-operative wound characteristics via digital images is necessary to relieve the clinical documentation burden of writing a note for each image submitted to a surgical service by a patient after discharge. Second, no evidence exists to support the use of smartphone photographs in lieu of standard-of-care in-person wound examination; this is particularly salient for vascular surgical wounds, in which the appearance of normal healing is commonly compromised by hemosiderin staining, shiny skin that reflects light in images, or stasis dermatitis. To address these gaps, we develop a comprehensive post-operative wound evaluation checklist, assess the reliability of the checklist, and determine whether surgeons and other health care providers can differentiate between complicated wounds and normally healing wounds using a smartphone digital photograph. This validity and reliability study focuses on vascular surgery wounds because this population has the highest readmission rate among surgical specialties, largely for SSI treatment.^10–12

Methods

Development of the Checklist

The authors drafted a preliminary version of the wound evaluation checklist using an iterative process modeled after that used for the World Health Organization (WHO) Surgical Safety Checklist.¹³ We used previously described wound infection diagnostic criteria defined by the Centers for Disease Control and measures compiled by Cutting and colleagues.^3,14–16 The checklist is comprised of two sections: Wound characteristics and treatment recommendations. We composed the wound assessment section by retaining only those diagnostic measures that could be visualized using a photograph, resulting in exclusion of pain/tenderness, abnormal smell, heat, delayed healing, and positive culture. We composed a second set of questions pertaining to subsequent diagnosis, treatment, and disposition.

Revision of the Checklist

We presented the preliminary checklist to an expert panel (two vascular surgeons, two residents, a health services researcher, and one biostatistician) for feedback and made revisions until we reached consensus. The result was presented in a local clinical vascular surgery meeting and a local surgical outcomes research meeting, and the first author (JTW) made adjustments based on feedback.

Six vascular surgery care providers including attending surgeons, surgical residents, nurse practitioners and registered nurses (RN) evaluated normal and infected/complicated vascular surgery patients' post-operative wounds using the checklist on the inpatient service. None of the raters was certified in wound care. Feedback informed additional revisions, ensuring that the final checklist encompassed the full scope of wound assessment and treatment plans. Upon mutual consensus among evaluators and research staff, we used the finalized checklist for data collection (summarized in Table I; full checklist in supplementary Figure 1, online only).

Table I. Domain Summary for Wound Evaluation Checklist.

Wound characteristics	Treatment Factors
Ecchymosis	Wound complication
Width/Length	Antibiotics
Redness	IV
Width/Length	PO
Cellulitis	Drainage
Drainage	Debridement
Type of drainage	Emergent in-person exam
Wound drain	Need for Hospitalization
Dehiscence
Width/Length/Depth
Necrosis
Width/Length
Drainage
Type of drainage
Wound drain
Dehiscence
Width/Length/Depth
Necrosis
Width/Length

Subjects

Eligible patients were age 18 and older and had a vascular surgery procedure involving an incision at least 3 cm size between May 2014 and February 2015. The level of analysis was the wound such that, for the small number of patients with multiple wounds (e.g., leg bypass graft using arm vein), each wound was included separately.

We recruited vascular surgery attending surgeons, surgical residents, physician assistants, nurse practitioners, and RNs, to evaluate post-operative wounds, representing the breadth of individuals who provide wound care to vascular patients in everyday practice. We obtained consent prior to data collection for both patients and evaluators; the University of Wisconsin Health Sciences Institutional Review Board approved the study protocol.

In-person and Remote Wound Evaluation Protocol

Using the finalized checklist, between 1 and 4 providers examined in-person 80 post-operative wounds. A non-clinical researcher concurrently captured digital images of patients' wounds without flash. Following in-person evaluations, 9 healthcare professionals used digital images to examine all 80 post-operative wounds (also using the finalized checklist). We presented each evaluator with an image or series of images for the wound on a computer screen. For each wound, each evaluator saw the same set of images as the other evaluators. Each wound had a total of 1 to 4 total images depending on the size of the wound (e.g., a single image for a carotid incision; three to four images for a lower extremity bypass). All evaluators were instructed to evaluate the wound under the assumption that the patient was discharged home within the last 1-2 weeks.

Image Capture Protocol

We captured the incision only with a white centimeter ruler in the frame for measurement and assessment of lighting. We held the camera 6 to 18 inches away from the incision; the overhead exam light was on rendering all images comparable in terms of environmental lighting. Photographers were instructed to “fill the frame” with the wound, to angle the camera to be in line with the incision, and to take 3 images of large wounds such as lower extremity bypass and thoracoabdominal incisions. The first 30 digital images were taken with three different smartphone cameras to assess a breadth of available technologies: a Samsung S4 (13 megapixel camera), Samsung S5 (16 megapixel camera), and Apple iPhone 5c (8 megapixel camera). After reviewing the quality of the images and obtaining better quality images on the Apple devices, the remaining 50 images were captured with an Apple iPod Touch (5 megapixel camera) to validate a relatively low cost option that is available on the current market. The images were uploaded to a secure server using a hardline connection to avoid the problem of HIPAA compliance in wirelessly transmitting the images.

Photo Quality Assessment

A professional photographer judged image quality on a scale from 0 to 3 along the following dimensions: clarity/focus, lighting, scope, and color quality.^17,18 Based upon the image quality, photos were categorized into two categories: 1) high quality, and 2) suboptimal. We used Spearman's correlation to evaluate associations between the components of wound image quality.¹⁹ All correlations were high and significant, varying from 0.35 for scope and clarity/focus to 0.89 for light and color. As a result, we summed the ratings for each image to create a composite image quality score ranging from 0 to 8; the distribution of scores was skewed left, indicating that photos were largely of high quality. We thus designated an image as being of high quality versus suboptimal if it had a summary score of 7 or 8.

Statistical Analysis

Considering in-person wound evaluation as the standard of care, our analysis plan proceeds in three phases. First, we establish the standard of care for in-person wound characteristics; we examine the percent of wounds with an abnormality according to in-person assessment and the associated interrater agreement. This analysis establishes a baseline for the frequency and variability in clinical assessment, serving as a reference point for image-based assessment. We then evaluate frequency and interrater agreement for image-based assessment of wound characteristics. This analysis determines whether agreement between image-based evaluations differs from in-person agreement. Second, we follow a similar program of evaluation in-person – followed by image-based – wound treatment recommendations. Third, we explicitly analyze agreement between in-person and image-based wound evaluations and treatment recommendations. Specifically, we compute sensitivity and specificity for each image-based wound characteristic and treatment recommendation, treating in-person assessment as the gold standard. We then calculate between-modality interrater agreement. This analysis determines whether detecting an abnormality or recommending treatment varies based on the modality of assessment.

We quantified interrater agreement and reliability as follows: For each wound that had at least two raters, we formed all possible rating pairs and took the proportion of those pairs that agreed on each characteristic. The observed agreement is calculated as average proportion of agreement for all wounds. We measure inter-rater reliability for the checklist wound assessment and treatment plan using Gwet's agreement coefficient (AC). Gwet's AC is a statistical measure of inter-rater agreement for qualitative and categorical items when there are 2 or more raters; for the same 2 raters, the observed agreement in the more familiar Cohen's kappa is equal to that in Gwet's AC.²⁰ Extending to the setting with multiple different raters per wound, we chose Gwet's AC over the more commonly used suite of kappa statistics to overcome the kappa paradox,²² wherein agreement appears low owing to overcorrection for chance agreement, and to address our data structure, which involved distinct, multiple raters for each arm (in-person and image-based). AC values are interpreted as follows: < 0 indicates no agreement, 0–0.20 is slight, 0.21–0.40 is fair, 0.41–0.60 is moderate, 0.61–0.80 is substantial, and 0.81–1 is almost perfect agreement.²¹

In addition, to compare image-based evaluation to in-person standard of care, we present the sensitivity and specificity of the image-based evaluation using the aforementioned in-person consensus as the gold standard. Sensitivity measures the rate of true positives and indicates the proportion of in-person wound abnormalities correctly identified by image-based evaluation. Specificity measures the rate of true negatives and indicates the proportion of normal in-person wounds correctly identified as normal by the image-based raters. To assess agreement between the in-person and image-based evaluations, we reshaped the data such that, for each wound, every response for a given question from an in-person rater was paired with every response from a remote rater for the same question. This was done for all the questions in the survey. Observed agreement and Gwet's AC were calculated using this in-person vs. remote paired dataset. We used 1000 bootstrap samples with replacement to calculate 95% CI of Gwet's AC for in-person vs. remote raters; the samples were drawn by wound.

Results

Wound types and evaluators

The distribution of clinician types performing ratings is summarized in Table II. The majority of in-person ratings were provided by vascular mid-level providers (n=82), followed by surgery residents (n=68), attending vascular surgeons (n=21), and RNs (n=9). For remote evaluations, the majority were performed by vascular mid-level providers (n=240). Table III summarizes the wound types included in this analysis. Eighty in-person wounds were evaluated with a median of 2 to 3 raters per wound. For remote evaluations, 9 providers rated every wound. The majority of wounds were found in the lower extremity (n=23), followed by groin incisions (n=20), thoracic/abdominal wounds (n=18), carotid neck incisions (n=10), upper extremity wounds (n=5), major amputation stumps (n=3), and one toe amputation.

Table II. Number of Ratings by Clinician Type.

Clinician Type	In-person evaluation (N=180)	Image-based evaluation (N=720)
Attending surgeon	21 (12%)	160 (22%)
Mid-level Provider	82 (45%)	240 (34%)
General Surgery Resident	68 (38%)	160 (22%)
Registered Nurse	9 (5%)	160 (22%)

Mid-level providers include nurse practitioners and physician assistants.

Table III. Median Number of Ratings by Surgical Site.

Surgical Site	# of Wounds (N=80)	Median # of In-person Ratings/Wound	Median # of Image-based Ratings/Wound
Lower Extremity	23 (29%)	2	9
Upper Extremity	5 (6%)	2	9
Abdomen/Thorax	18 (23%)	2	9
Amputation Stump	3 (4%)	2	9
Carotid	10 (12%)	3	9
Groin	20 (25%)	2.5	9
Toe Amputation	1 (1%)	2	9

For in-person ratings, each wound is rated by a different number of raters between 1 and 4; raters may also differ from one wound to the next. Each image was evaluated by 9 raters.

Wound complications: In-Person Evaluation

We regarded as the standard of care the in-person evaluation of wound complications (Table IVa). A wound was considered to have either an abnormality or complication or an indication for further treatment if at least 50% of in-person raters indicated its presence; this represents the most conservative majority-based approach to indicating the presence of a wound abnormality given that wound checks are usually performed by only one provider. Using this rule for in-person evaluations of wounds, 43% were found to have ecchymosis, 45% had redness, and only 26% met the threshold for cellulitis. Drainage was present in 28% of wounds and dehiscence was identified in 21% of wounds. Less commonly, a drain (13%) or necrosis (8%) was visible. Observed pairwise agreement between in-person raters was lowest for redness (0.70) and highest for necrosis (0.94). Interrater agreement (adjusted for chance agreement) was moderate for ecchymosis (AC=0.56). Although agreement for redness was only fair (AC=0.44), the in-person agreement for cellulitis was markedly better (AC=0.68). Inter-rater agreement was close to perfect for more severe abnormalities: drainage (0.76), dehiscence (0.86), and necrosis (0.93).

Table IVa. Standard of Care In-Person Assessment of Wound Abnormalities.

	% of Ratings with abnormality (N=180)	% of Wounds with abnormality (N=80)	Observed agreement between Raters	Gwet's AC (95% CI)
Ecchymosis	38	43	0.77	0.56 (0.37,0.76)
Redness	35	45	0.70	0.44 (0.23,0.65)
Cellulitis	25	26	0.79	0.68 (0.26,1.00)
Drainage	27	28	0.85	0.76 (0.60,0.92)
Presence of drain	10	13	0.92	0.91 (0.78,1.00)
Dehiscence	19	21	0.91	0.86 (0.72,1.00)
Necrosis	7	8	0.94	0.93 (0.81,1.00)

Note: Wound is characterized as having an abnormality if 50% or more of raters for that wound indicate the abnormality is present.

Wound complications: Image-Based Evaluation

Table IVb summarizes the image-based evaluation of wounds. Abnormalities were less common in image-based evaluations, including redness (40%) and cellulitis (19%). Necrosis (10%) and drainage (29%) were the only abnormalities more commonly detected in images. Interrater agreement for image-based wound evaluations was comparable to in-person agreement with some variability depending on image quality. Image-based raters achieved fair to moderate agreement for ecchymosis (AC=∼0.5), drainage (AC=0.41-0.58), and cellulitis (AC=0.67-0.83); agreement for redness was lower (AC=0.27-0.34). Similar to in-person agreement, image-based agreement was nearly perfect (AC=0.9-1.0) for presence of a drain, dehiscence, and necrosis. Notably, image quality did not meaningfully affect interrater agreement.

Table IVb. Remote Care Image-based Assessment of Wound Abnormalities.

	% of Ratings with abnormality (N=720)	% of Wounds with abnormality (N=80)	Observed Agreement between Raters		Gwet's AC (95% CI)
			Poor photos	Good photos	Poor photos	Good photos
Ecchymosis	35	31	0.75	0.71	0.52 (0.37,0.67)	0.49 (0.36,0.63)
Redness	48	40	0.63	0.67	0.27 (0.11,0.42)	0.34 (0.21,0.46)
Cellulitis	26	19	0.87	0.78	0.83 (0.67,0.98)	0.67 (0.49,0.85)
Drainage	32	29	0.73	0.69	0.58 (0.41,0.75)	0.41 (0.26,0.56)
Presence of drain	7	8	0.95	0.98	0.94 (0.88,1.00)	0.97 (0.94,1.00)
Dehiscence	21	19	0.92	0.93	0.90 (0.83,0.97)	0.88 (0.79,0.96)
Necrosis	12	10	0.90	0.88	0.88 (0.78,0.97)	0.85 (0.76,0.93)

N=34 poor images; 46 good images; Wound is characterized as having an abnormality if 50% or more of raters for that wound indicate the abnormality is present.

Treatment Indicated: In-person

Twenty-five percent of in-person raters indicated a complication severe enough for additional treatment, amounting to 29% of wounds overall (Table Va). In-person agreement for whether a wound needed additional treatment was substantial (AC=0.72). Additional treatment consisted of antibiotics (18%), drainage (13%), debridement (13%), a visit to the emergency department (ED) (10%) or readmission (13%). Interrater agreement (AC) was universally high for recommended treatment based on in-person evaluations, ranging from 0.82 for antibiotics to 0.90 for ED visit and readmission.

Table Va. Standard of Care In-Person Assessment of Treatment.

	% of Ratings indicating treatment (N=180)	% of Wounds indicating treatment (N=80)	Observed Agreement between Raters	Gwet's AC (95% CI)
Any complication	25	29	0.82	0.72 (0.54,0.90)
Antibiotics	15	18	0.86	0.82 (0.69,0.96)
Drainage Needed	9	13	0.89	0.87 (0.72,1.00)
Debridement needed	9	13	0.91	0.89 (0.75,1.00)
ED visit	10	10	0.92	0.90 (0.78,1.00)
Readmission	11	13	0.92	0.90 (0.77,1.00)

Treatment Indicated: Image-Based Evaluations

Agreement for treatment via image-based evaluations was comparable to in-person evaluations (Table Vb). Inter-rater agreement (AC) for image-based treatment recommendations was universally high (range: 0.66-0.92) and comparable to in-person agreement. Highest agreement via images occurred for antibiotics (0.92), drainage (0.91), debridement (0.92), and readmission (0.88) in poor images. Again, image quality did not appear to factor into interrater agreement for additional treatment recommendations.

Table Vb. Remote Care Image-based Assessment of Treatment.

	% of Ratings indicating treatment (N=720)	% of Wounds indicating treatment (N=80)	Observed Agreement between Raters		Gwet's AC (95% CI)
			Poor image	Good image	Poor image	Good image
Any complication	30	29	0.79	0.87	0.66 (0.50,0.82)	0.77 (0.66,0.88)
Antibiotics	14	16	0.93	0.88	0.92 (0.85,0.99)	0.83 (0.73,0.94)
Drainage Needed	11	9	0.92	0.87	0.91 (0.85,0.97)	0.83 (0.73,0.93)
Debridement needed	12	11	0.93	0.89	0.92 (0.87,0.98)	0.84 (0.75,0.94)
ED visit	21	19	0.80	0.88	0.72 (0.58,0.86)	0.81 (0.71,0.91)
Readmission	11	11	0.90	0.88	0.88 (0.80,0.96)	0.84 (0.73,0.94)

N=34 poor images, 46 good images; Wound is characterized as having an abnormality if 50% or more of raters for that wound indicate the abnormality is present.

Comparison and Agreement of In-Person and Image-based Wound Evaluations

Table VI summarizes how image-based wound ratings and in-person ratings compare. For low quality images, sensitivity for wound characteristics was highest for ecchymosis (86%) and necrosis (100%). Specificity for low quality images was universally high, ranging from 81% for redness to 100% for dehiscence and the presence of a drain. For high quality images, sensitivity was highest for abnormalities dehiscence (79%) and drainage (79%). Specificity for high quality images was high for ecchymosis (92%), presence of a drain (98%), dehiscence (92%) and necrosis (98%). Interrater agreement indicates that agreement between in-person and remote raters was fair (redness: 0.27; cellulitis: 0.32) to moderate for ecchymosis and drainage (AC Range: 0.49-0.53). Agreement improved to excellent for the presence of a drain, dehiscence, and necrosis (AC=∼0.9).

Table VI. Evaluation of Image-based Care vs. In-person Care.

	Sensitivity: Poor Images	Specificity: Poor Images	Sensitivity: Good Images	Specificity: Good Images	Gwet's AC (95% CI)
Ecchymosis	86	95	50	92	0.49 (0.37,0.59)
Redness	61	81	61	75	0.27 (0.17,0.36)
Cellulitis	50	83	63	63	0.32 (0.08,0.55)
Drainage	50	96	79	78	0.53 (0.42,0.64)
Presence of drain	33	100	75	98	0.91 (0.86,0.97)
Dehiscence	67	100	79	94	0.86 (0.79,0.92)
Necrosis	100	91	50	98	0.89 (0.83,0.94)
Any complication	50	82	71	89	0.66 (0.55,0.78)
Antibiotics	100	94	50	91	0.76 (0.65,0.86)
Drainage Needed	0	97	44	95	0.88 (0.81,0.94)
Debridement needed	NA	97	60	94	0.89 (0.83,0.95)
ED visit	100	91	100	90	0.79 (0.71,0.88)
Readmission	50	100	75	95	0.88 (0.82,0.94)

N=34 poor images; 46 good images

Determining whether a wound required additional treatment depended on image quality to some extent; sensitivity was only 50% for the presence of any complication in low quality images, but increased to 71% for better images. Specificity for the presence of a complication was slightly lower for poor images (82%) than good images (89%). For recommended treatment, sensitivity was highest for antibiotics and ED visits with poor images and ED visit with good images (100%) and lowest for readmission based on poor images (50%) and needing drainage for wounds with good images (44%). Interrater agreement between in-person and image-based raters was substantial for any complication needing treatment (AC=0.66) and antibiotics (AC=0.76) and was almost perfect for recommending drainage, debridement, or readmission (AC=∼0.9).

Figure 1 shows agreement (AC) among in-person raters relative to image-based and between-modality agreement, permitting comparison across evaluation modalities. For redness, image-based and between-modality agreement was slightly lower than standard-of-care agreement, but within the 95% confidence interval. Agreement for the presence of a drain, dehiscence, and necrosis was universally high with tighter confidence intervals; this was true regardless of the modality of evaluation. Overall, agreement for treatment was universally high regardless of evaluation modality.

Figure 1.

The shape represents the value for Gwet's agreement coefficient (AC); the error bar represents the bootstrapped 95% confidence interval for the AC.

Discussion

The role of telemedicine in the care of surgical patients is rapidly expanding. As care transitions and reducing hospital readmissions have attracted attention from payers and providers, the opportunities for telemedicine to aid patients in their homes following hospital discharge have aligned with financial incentives and quality improvement initiatives. Remote monitoring of surgical sites using smartphone images may preclude an unnecessary office or home visit by a nurse care manager and may promote patient mindfulness of the wound and participation in postoperative care. Moreover, the U.S. is one of six countries worldwide in which more than half of the adult population owns a smartphone. Although adults over age 65 had the lowest rates of smartphone ownership (27%), they accrued 8 percentage points compared to earlier that year². Furthermore, our survey of vascular surgery inpatients found that 92% would be willing to send images of their post-operative wounds after discharge.²⁴

Nevertheless, the implementation of telemedicine focused on post-operative wound monitoring remains relatively limited. In addition to reimbursement and legal barriers, this is owed in part to a lack of tools for wound assessment and unproven remote wound monitoring applications. In this study, we created a wound evaluation checklist and demonstrated that digital images taken by a smartphone camera allow providers to make reliable decisions about diagnosis and treatment of postoperative vascular surgery wounds.

We first created a checklist to standardize and streamline remote wound assessment. Checklists have been shown to help improve clinical outcomes in surgery, obstetrics, anesthesia care, emergency departments and in intensive care units.^25–30 They have proven most effective in performance improvement and error prevention and management.^28,31 By using a checklist for remote wound monitoring, assessments done by different providers at different times may be easily compared, and the recovery process for an individual is thus more easily tracked over time. In addition, our checklist reduces the burden of provider documentation by replacing the written note with an efficient and comprehensive instrument. This increases the likelihood that a surgical service can meet the image review burden associated with a full patient panel without additional staff.

Using this checklist, we sought to establish a gold standard of in-person wound assessment. We found a surprising amount of variability among in-person raters when asked to assess whether abnormalities were present, with agreement coefficients as low as 0.4 for redness and 0.6 for ecchymosis. However, agreement was higher for more severe complications of dehiscence or necrosis (0.9). One possible explanation for this finding is that preexisting skin changes (e.g., dependent rubor, venous stasis changes, etc.) commonly seen in vascular patients make it difficult to assess for the presence of truly pathologic redness. Despite disagreement about the specific characteristics of each wound, in-person raters achieved substantial agreement when asked to make treatment recommendations, with AC of 0.8-0.9. This is important because, while raters may disagree about whether a wound is red or ecchymotic, they do agree on whether intervention is necessary and what that intervention should be.

We found similar results when comparing assessments made via image to assessments made in person. Again, agreement was lower for specific wound characteristics, particularly ecchymosis, redness, and cellulitis. However, if raters looking at the same wound in person cannot come to agreement about whether a wound is red or ecchymotic, low agreement when looking at the wound via image is consistent with the standard of care and does not represent a compromise owed to a change in evaluation modality. Interestingly, although in-person raters had moderate agreement about whether a wound was spontaneously draining, this was a much more difficult assessment to make via image; this indicates that a supplementary question regarding the presence of drainage may be necessary for faithful image-based wound assessment. Notwithstanding, we are encouraged by clinicians' ability to make comparable treatment decisions when evaluating wounds via image.

High-quality images conferred surprisingly little benefit over low-quality ones. In anticipation of patients' sending wound images, the fact that they do not need to take professional-grade images is encouraging, particularly in a patient population likely to have limited prior experience with smartphones. However, both low- and high-quality images had high specificity and relatively low sensitivity across almost all parameters. These results are the opposite of what one would hope and indicate a high prevalence of false negatives in image-based assessment. This underscores the limitations of making assessments of a wound based solely on a single image. Therefore, an image-based wound evaluation protocol will require side-by-side comparison of daily wound images starting at discharge. If a rater could compare images across the recovery period and against a baseline image taken at the time of discharge, sensitivity to a burgeoning complication might increase. The collection of supplementary symptom information, such as fever or uncontrolled pain, may also improve the sensitivity of a telemedicine wound monitoring program.

Our results are consistent with existing research on the feasibility of remote monitoring of other types of wounds using digital photographs.^32,33 In a study of surgical reconstructions with free tissue transfer, remote monitoring of vascularization and healing of flaps was comparable to in-person assessment.³⁴ A preliminary 1998 evaluation of vascular wounds also obtained reasonable agreement between in-person post-operative evaluation and digital photographs of surgical wounds.¹⁷ Interestingly, cameras used in this study had only 756 × 504 pixels per square inch; we use current Smartphone digital photos with image resolution greater ≥ 5 megapixels. We presume that the increased resolution improves detection of infection and other anomalies, such as staple line erythema, hematoma and dehiscence.

This study was limited to the vascular patients' wounds, which may potentially restrict generalization to all surgical fields. However, it may also best represent the surgical population as a whole as this population frequently develops SSI,³⁵ and has the highest readmission rate among surgical specialties.^10–12 In addition, we did not undertake any post hoc adjustment of the images' color balance, although the presence of the ruler in each image enables this. Image adjustment may have improved the detection of some abnormalities, such as erythema; when multiple images of the same wound over time are considered in future trials of this approach, balancing the image color will help to ensure appropriate comparisons over time. A final limitation involves having evaluated the wound photos absent the patient's history and clinical context. When reviewing future photos in usual practice, we will ultimately interpret them in the context of the patient's medical record and previous images.

The results of this study will inform a smartphone application and protocol to allow patients to take digital images of their wound and fill out a brief survey from the day of hospital discharge until their scheduled follow-up appointment 2-3weeks after discharge. We are testing this protocol as part of a feasibility study to create and test a toolkit for widespread implementation of image-based wound monitoring. Based on the results of the current study, we believe that digital images provide sufficient information to make meaningful, first-line treatment decisions. We also anticipate that tracking a wound over time will increase the detection of abnormalities that a single static image did not provide, and we plan to do this in upcoming feasibility trials. Supplementary survey questions to accompany the image will provide important information that cannot be ascertained from an image, such as fever, pain, and wound drainage, which was the characteristic that remote raters had the most difficulty accurately detecting. This supplementary information may increase the sensitivity of the protocol to detect wound complications.

Conclusion

This study is the first to (1) create and internally validate a wound evaluation checklist and (2) assess smartphone technology for vascular wound surveillance. Given the increasing prevalence of smartphone-based telemedicine and our finding of high reliability for remote wound assessment, these technologies offer a viable and currently underutilized platform for implementing a wound surveillance program. Our findings lend some nuance and elucidate barriers to using smartphone-generated images to evaluate vascular surgical wounds after hospital discharge. As providers increasingly use these images to care for patients, understanding their clinical utility will ensure that they are incorporated in a way that is consistent with their limitations and advantages.

Supplementary Material

Supplementary Figure. Checklist used to evaluate a) wound characteristics and b) treatment recommendations for all wounds, both in person and via images.