Enhancement of Decision Rules to Increase Generalizability and Performance of the Rule-Based System Assessing Risk for Pressure Ulcer

Abstract

Background

A rule-based system, the Braden Scale based Automated Risk Assessment Tool (BART), was developed to assess risk for pressure ulcer in a previous study. However, the BART illustrated two major areas in need of improvement, which were: 1) the enhancement of decision rules and 2) validation of generalizability to increase performance of BART.

Objectives

To enhance decision rules and validate generalizability of the enhanced BART.

Method

Two layers of decision rule enhancement were performed: 1) finding additional data items with the experts and 2) validating logics of decision rules utilizing a guideline modeling language. To refine the decision rules of the BART further, a survey study was conducted to ascertain the operational level of patient status description of the Braden Scale. The enhanced BART (BART2) was designed to assess levels of pressure ulcer risk of patients (N = 99) whose data were collected by the nurses. The patients’ level of pressure ulcer risk was assessed by the nurses using a Braden Scale, by an expert using a Braden Scale, and by the automatic BART2 electronic risk assessment. SPSS statistical software version 20 (IBM, 2011) was used to test the agreement between the three different risk assessments performed on each patient.

Results

The level of agreement between the BART2 and the expert pressure ulcer assessments was “very good (0.83)”. The sensitivity and the specificity of the BART2 were 86.8% and 90.3% respectively.

Conclusion

This study illustrated successful enhancement of decision rules and increased generalizability and performance of the BART2. Although the BART2 showed a “very good” level of agreement (kappa = 0.83) with an expert, the data reveal a need to improve the moisture parameter of the Braden Scale. Once the moisture parameter has been improved, BART2 will improve the quality of care, while accurately identifying the patients at risk for pressure ulcers.

1. Introduction

Many informatics-based automated clinical tools have been developed in nursing domains using a theory-based implementation model or practice-based research to apply evidence to practice [1]. Informatics intervention tools that were developed based on either method reveal a considerable gap between knowledge and practice [2-4]. There are several reasons for this gap. One possible reason is a lack of detailed or sufficient data that allow for a developed tool being implemented in any health care setting. The study by Kim and colleagues, in which these investigators attempted to draw automated patient acuity decisions based on the detailed data documented in an electronic health record, showed that insufficient documentation of patient data and the lack of patient data structure that can be shared among hospitals prohibited successful implementation of the decision algorithms [5]. Another possible reason may be the use of Health Information System (HIS) developed without considering users’ workflow, tasks, and preferences [29-31]. When the HIS is implemented in healthcare institutions without considering usability, it decreases users’ satisfaction, acceptance, job performance, efficiency, and quality of work [31, 32].

Pressure ulcers are a common health problem, and they remain as one of the most serious safety concerns (http://www.jointcomission.org/) when dealing with patients who are admitted to hospitals or confined to bed, chair or wheelchair. They impact morbidity and mortality, and cause serious human discomfort [18, 19, 27]. Pressure ulcers are considered as an indicator of quality of care, and pressure ulcer treatment has been a large financial burden on health care institutions [7]. As an attempt to improve assessing skin risk for pressure ulcer, Kim and colleagues developed a rule-based prototype decision support tool based on the Braden Scale with the idea to derive pressure ulcer risk scores of the Braden Scale using documented clinical patient data automatically [8].

Although this prototype system, Braden Scale based Automated Risk Assessment Tool (BART), improved workflow efficiency and showed the feasibility of automatic pressure ulcer assessment utilizing data re-use, it identified two major and two minor areas that required further investigation [8]. The first area was enhancing decision rules, especially on moisture status and nutritional status parameters. When the BART was created, developing rules for moisture status assessment was difficult due to the lack of data items directly related to this parameter at the study site. For example, the BART determines the level of Moisture Exposure based on the frequencies of changing diapers, incontinence pads, bed linen, absorbent pads, or wound dressings. Those data were not sufficiently documented [8]. Assessing the nutritional status parameter was another issue. This parameter is commonly identified as the most difficult parameter to assess in other studies [6] because it requires understanding of chronic nutritional status of the patient. Such information was not completely documented in the patient record at the study site. The second area was validating generalizability of the tool, since it was tested with a small sample size (N = 39).

A guideline modeling language, Guideline Interchange Format (GLIF) was utilized to validate the logic of enhanced decision rules of the BART. GLIF3 was applied to convert complex decision rules of the BART at three levels of GLIF. The purpose of this study was to:

• 1.enhance the decision rules of BART to increase performance; and
• 2.validate generalizability (i.e. external validity) of BART to increase possibility of its implementation in any health care settings.

2. Background

2.1 Generalizability of Information System

Generalizability is important because it allows informatics intervention tools to not only be validated but also applied in health care settings that are different than the setting in which the tools were developed and tested. Historically, informatics intervention tools have often failed to achieve what they intended to do when they are implemented in various health care settings [9-11] because of their lack of generalizability. Thus, even though many clinical informatics intervention tools are developed, only a few are implemented and used in practice because there are no sufficient contextual details or information for them to adapt into different settings [5, 33]. The difference in organizational cultures (e.g. attitudes toward innovation, users’ judgment towards technology, etc.) could be a cause for the failures in implementation [9, 12]. To maximize the efficiency and usefulness, the generalizability of these tools should be validated to offer services in various health care settings [1].

2.2 Braden Scale

The Braden Scale [13, 25, 26] comprised of 6 parameters: sensory perception status; skin moisture status; activity level; mobility level; nutritional status; and the level of significance that friction and shear contribute to skin breakdown. Each parameter is assigned an ordinal score of 1-4, except the friction and shear parameter, which was assigned a score of 1-3; and the sum of the 6 scores indicates the level of risk a patient may have. The Braden Scale is a widely used instrument for pressure ulcer risk assessment [20, 21] and is adopted across Partners Healthcare System. Across Partners Healthcare System, it is included as an initial nursing assessment item and from it the nurses derive the initial nursing care plan. After the initial assessment, nurses regularly assess and document the Braden Scale in the patient’s flowsheet over the course of hospitalization, and the nursing care plan is updated as necessary as a result of changes in the Braden Scale’s pressure ulcer risk score.

2.3 Braden Scale based Automated Risk Assessment Tool (BART)

BART, a prototype decision support tool for automatically assessing pressure ulcer risk using documented patient data based on the Braden Scale, was developed in an attempt to address many issues in current practice of pressure ulcer risk assessment of patients who are hospitalized. Among others, redundant documentation and difficulty in aggregating necessary data were the two most significant issues [8]. In the previous study, 56 discrete data items were identified and used to develop decision rules in using the BART [8].

Although the BART showed promising results, it was developed in an acute care hospital and tested with convenience samples of 39 patients. Therefore, testing generalizability in different hospitals with a large sample size is important and necessary before the BART is advanced to a production level. Besides the two major challenges mentioned earlier, two minor challenges were raised during the previous study. One was operationalizing many vaguely described patient status descriptions on the Braden Scale. Examples are underlined terms in activity parameter: “Walks occasionally during day, but for very short distances, with or without assistance”. “Spends majority of each shift in bed or chair.” Such vague descriptions make it hard to assess pressure ulcer risk more accurately and consistently with the Braden Scale. The other minor challenge was identifying a more reliable gold standard. The unofficial blind review of the disagreed cases by a nurse with expertise in pressure ulcer care indicated that expert nurses agreed more with the BART than with the nurses’ assessments.

2.4 Guideline Interchange Format (GLIF)

GLIF is a modeling programming language that was developed to represent sharable computer-interpretable guidelines. Its various classes and their attributes are used to describe and illustrate complex clinical knowledge. The current version of GLIF (GLIF3) converts clinical knowledge at three levels:

• 1.a conceptual flowchart;
• 2.a computable specification for validating logical consistency and completeness; and
• 3.and an implementable specification that can be incorporated into an information system of an institution.

GLIF3 uses Health Level 7 standards, which permit patient data to be exchanged among different healthcare settings, to allow integration of computer-interpretable guidelines into a clinical information system. Its specification is structured based on the Resources Description Framework, which allows extending a computer-interpretable guideline’s specification [14]. GLIF3 has been used to encode different types of clinical knowledge. A dditional technical details of the GLIF3 can be found elsewhere [14-17].

Since the BART was in a computer-interpretable skin assessment guideline format already, GLIF3 was utilized only to validate logics while evaluating three levels of converted decision rules in the BART.

3. Methodology

3.1 Setting

Convenience Spaulding hospital, one of the Partners network hospitals where a prototype system had not been developed, was selected. This Spaulding hospital is an academic acute rehabilitation hospital with 190 certified beds and a 160-average daily census (2011 statistics), where the Braden Scale has been used to assess pressure ulcer risks for several years. It offers six different rehabilitation programs: five for adult patients and one for pediatric patients. The adult programs encompass various types of injuries including cardiac, musculoskeletal, stroke, brain injury, and spinal cord injuries.

3.2. Study Phases

This study was conducted in four phases. In the first phase, data items identified and used to build decision rules in the previous study were re-validated by expert nurses. In the second phase, a smallscale survey was conducted to examine how consistently the nurses interpreted the vaguely described patient characteristics in the Braden Scale. In third phase, decision rules were enhanced and validated. In the last phase, the performance of the enhanced BART (BART2) was evaluated in Spaulding hospital to validate generalizability (i.e. external validity).

3.2.1 Phase I: Re-validate Data Items Identified in the Previous Study.

An expert panel was formed with four expert nurses who have specialized in pressure ulcer care. These nurses were certified in wound, ostomy, and continence care. They were recruited from Spaulding and Massachusetts General Hospitals. Since only one skin expert was available from the Spaulding hospital, recruiting more experts from Massachusetts General Hospital where the Braden Scale had also been used was necessary.

The members of the expert nursing panel also participated in validating data items from the previous study and identifying additional data items. All four experts, three from Massachusetts General Hospital and one from Spaulding hospital, were staff educators and each was credentialed at the master’s degree level. They were 45 to 58 years old and had 10 to 35 years of skin care experience.

Fifty-six data items such as “level of consciousness”, “mode of bowel/urinary elimination”, and “frequency of walking” that were identified in the previous study were presented to the four experts for re-validation. These experts were asked to determine whether those data items are valid to assess a patient’s risk for pressure ulcer congruent with current clinical practices. Three meetings were held, and consensus building technique was utilized. At the first meeting, the investigator (JC) explained the whole process and the rules for communication. In the second meeting, the investigator prepared a spread sheet of 56 data items and presented it to the experts. During the second group discussion, experts exchanged opinions about data item validation and suggested fourteen additional data items that were not included in the original 56 data items [8]. In the third meeting, all data items, including the validated 56 and additional fourteen data items in the spread-sheet, were presented to the four experts. At the end of the third meeting, the four experts were able to validate all of the 56 original and the additional fourteen data items (► Table 3).

Table 3.

Fourteen data items identified by an expert nursing panel and seven data items used in decision rules of parameters of the Braden Scale

Category	Data items	Usage in decision rules of an associated parameter of the Braden Scale
Current/past medical history
	peripheral vascular dis	Sensory Perception
	peripheral neuropathy	Sensory Perception
	seizure	Sensory Perception
	diabetes	Sensory Perception
	dialysis	Not used
	steroid use	Not used
Circulation
	compressive wrap	Friction & Shears
	Infiltrated IV	Not used
	restraints use	Friction & Shears
	pedal pulse	Not used
	tissue perfusion	Not used
Nutrition
	albumin	Nutrition
	WBC	Not used
	Zinc	Not used

Throughout the three meetings, four expert nurses identified and agreed that assessing the skin condition of significant skin folds should be added to the moisture parameter of the Braden Scale. They identified three major body areas to assess: under the breast, the lower abdominal, and groin area. The left diagram of ► Figure 1 shows how this skin assessment was incorporated into the decision rules of moisture parameter. Albumin level of a patient was also found to affect the nutrition parameter. Identifying albumin was important, since many investigators have shown that lower albumin levels were associated with pressure ulcer [22-24, 28].

Fig. 1.

Created new decision rule (left) for the moisture parameter. Newly identified data items were added into decision rule of friction and shear parameter (right). (with permission of: ©copyright Partners HealthCare System, Inc with funding provided by Partners Siemens Research Council)

Phase II: Identify Operational Description of the Braden Scale.

A survey study was conducted to identify a clear operational description of the vaguely described patient status description on the Braden Scale. The goal of this survey was to elicit an operational level description of patient status used by, nurses in daily practice and one that might be used to further refine the decision rules of the BART.

A questionnaire was developed to elicit information on how the vaguely described patient status of the Braden Scale was interpreted and operationalized in practice. Two wound care nurses (not involved in other parts of this study) identified five vague descriptions from four parameters of the Braden Scale and participated in developing the questionnaire. The questionnaire was composed of two parts: one part elicits information on job demographics and the other consists of questions on interpreting five vague descriptions as identified by the two wound care nurses. The following example illustrated how the question on “over most of body” was created;

”The Sensory Perception parameter of the Braden Scale is scored from 1 to 4. Score 1 equals to Completely Limited, meaning “limited ability to feel pain over most of body.”

Please type in the % of body surface affected that you might consider as “over most of body” (e.g., over 80%, over 75%, etc.).”

In case there were different responses between nurses in Spaulding and Massachusetts General Hospitals, the survey study was expanded to Massachusetts General Hospital where the Braden Scale was also being used. After obtaining IRB approval, flyers announcing a survey study were posted on units of Spaulding and Massachusetts General Hospitals. Units of Spaulding hospital were Cardio- General, Musculoskeletal, Stroke, Brain Injury, and Spinal Cord Injury. Units of Massachusetts General Hospital were; Neuromedical, Neurosurgical, General Surgery, Cardiac Surgery Step Down, and Oncology. Survey data were analyzed using SPSS to measure levels of education, length of experience, specialties, and interpretation of the Braden Scale parameters.

Phase III: Enhance Decision Rules.

Previously developed decision rules were refined further while adding identified additional data items and creating new rules by the investigator (JC). ► Figure 1 is an example of how newly identified items were inserted into the decision rule diagram developed in the previous study. The decision rules were then converted into Guideline Interchange Format (GLIF). Six flowcharts in the form of GLIF were presented to the expert from Spaulding for easy review. The expert verified that the decision algorithm and logics were correct.

A standalone BART version 2 (BART2) was re-programmed based on logics of the BART2 represented by GLIF. BART2 was installed on a Windows 7 operating system computer. A web developmental tool, Ruby on Rails was used to develop the system:

• 1.Sqlite as a database;
• 2.WEBrick as a simple web server; and
• 3.Internet Explore/Firefox as a client (http://rubyonrails.org/).

BART2 was tested with ten test scenarios, and the software debugging process was conducted iteratively. ► Figure 2 shows data entry screen of the BART2 and a report screen that shows individualized risk level after processing entered patient data.

Fig. 2.

A partial view of a patient data entry screen (left) and a final report screen shows a result of pressure ulcer assessment (right). (with permission of: ©copyright Partners HealthCare System, Inc with funding provided by Partners Siemens Research Council)

Phase IV: Evaluate the Performance in Spaulding hospital to validate generalizability (i.e. external validity).

In this phase, performance of the BART2 was measured. The representative sample size (N = 100) was calculated using an online sample size calculator (http://www.macorr.com/sample-size-calcula tor.htm) based on 95% confidence level, 5% margin of error, and Spaulding hospital adult unit’s daily census (135 patients) at the time of study.

A paper-based patient data collection package, which contained a Braden Scale form and a series of patient data intake forms developed in the previous study [8], was used and modified by adding newly identified seven data items during this study. The data items were grouped based on their functional and physical areas, not by the related Braden Scale parameters, to minimize the risk of introducing bias in data collection. Recruiting flyers were posted on four adult units of Spaulding hospital after obtaining IRB approval. When a nurse volunteered to collect data, a five minute instructional session was given with the patient data collection package. Each nurse was instructed to fill a Braden Scale before collecting patient data to reduce bias and informed that all data items were mandatory.

Each nurse collected five to ten patients’ data by either copying from electronic medical record(s), if the patient information was assessed and documented less than 4 hours prior to the data collection time, or by reassessing the patient.

Data from data collection package were transcribed into the BART2. Then, the risks for pressure ulcer assessed by the BART2, by the nurses, and by the expert were put into a data table to test the agreement and analyzed using the SPSS statistical tool. A total of twelve nurses were recruited from Cardio, Musculoskeletal, Stroke, Brain Injury and Spinal Cord Injury units. All nurses had more than one year of clinical experience.

The expert from Spaulding hospital assessed the risk for pressure ulcer of the patients whose data were available in data collection packages. The risk levels assessed by this expert were set as gold standards. The performance of the BART2 was measured based on these gold standards. Agreement and Kappa were calculated using SPSS: nurses vs. BART2 and the expert vs. BART2. In order to evaluate how well the BART2 assesses patients’ risk for pressure ulcer, the sensitivity and specificity of the BART2 was calculated.

4. Results

4.1 Re-validated and Added Data Items

All 56 data items identified from the previous study were re-validated. Fourteen new data items were identified by the expert nursing panel. Seven out of fourteen newly identified data items were used to enhance the decision rules (► Table 3). Items such as “restraint use” and “compressive wrap” were added to friction and shear parameter decision rules (Right diagram of ► Figure 1). “Pre-albumin or Serum Albumin level” was added to nutrition parameter decision rules. “Peripheral vascular disease,” “Peripheral Neuropathy,” “Seizure,” and “Diabetes” were added to decision rules of the sensory perception parameter. The remaining seven items were not used because they were either not feasible to assess or they could be replaced with other data items. For example, “steroid use” was not feasible and “pedal pulse” and “tissue perfusion” could be replaced by circulation assessment in existing data items. In addition to this, assessing skin condition of significant skin folds was created and incorporated into moisture parameter decision rules as a new rule. A decision rule of assessing albumin level was created and incorporated into nutrition parameter decision rules.

4.2 Survey Results

A total of 102 survey responses from nurses were analyzed. Specialties, education level, and years of work experience of respondents are described in ► Table 4. ► Figure 3 shows nurses’ responses to each of the five vague descriptions in the Braden Scale as percentages.

Table 4.

Survey respondent characteristics

	Total (n = 102)
	n	%
Specialty areas
Medical inpatient	28	27.5
Surgical inpatient	25	24.5
Rehabilitation	27	26.5
Neurology	17	16.6
Staff education	4	3.9
No response	1	1.0
Education level
Associate	12	11.7
Baccalaureate	15	14.7
RN-BSN	70	68.7
Master’s	5	4.9
Work experience in years
<2	29	28.4
2–5	23	22.6
5–10	32	31.4
>10	18	17.6

Fig. 3.

Percent of nurses’ responses in each of the five vague descriptions in the Braden Scale

4.3 Enhancing Decision Rules and Validating Logical Consistency with GLIF

Sensory perception, moisture, nutrition, and friction and shear parameters, which had new data items in its decision rules, were depicted in decision rule diagram (► Figure 1). Then, all six enhanced decision rules of the parameters were converted into six flowchart forms of GLIF using the GLIF authoring tool implemented as a plug-in application in Protégé 3.4 (http://protege.stanford. edu). All enhanced decision rules of six parameters were converted successfully into six flowchart forms of GLIF with computable and implementable specifications. For example, a decision step – “Use any orthopedic device?” – in friction and shear parameter flowchart form of GLIF can be computed with implementable specifications, such as the P-Boots, TEDS, Sling, Compressive wrap, and Restraint use. For example, if a patient uses one of the P-Boots, TEDS, Sling, Compressive wrap, and Restraint use, then decision outcome of this decision step is “yes” (“true” in Boolean logic) and the next step, which is “Does edema exist?”, is triggered to be executed (► Figure 5). Another example is the “What is albumin level?” decision step in nutrition parameter flowchart. This can be computed and directed to a specific next step with a cutoff value (albumin level = 3.5). ► Figure 4 shows a partial view of enhanced friction and shear parameter enhanced decision rules, converted into a flowchart form that is easy to comprehend, a specific path between two decision steps (i.e. “Use any orthopedic devices?”; “Does edema exist?”) with implementable specifications (i.e. P-Boots, TEDS, Sling, Compressive wrap, and Restraint use).

Fig. 5.

A partial view of decision rule of friction and shear parameter (with permission of: ©Copyright Partners HealthCare System, Inc. with funding provided by Partners Siemens Research Council)

Fig. 4.

A partial view of friction & shear parameter in flowchart form of GLIF: Action_Step: The work that is to be performed by the DSS, the care provider, or external agents (i.e. laboratory tests). Branch_Step: It defines branching points of multiple paths through the guideline. Case_Step: It is an automated decision point where a decision is made by the DSS. Synchronization_Step: It defines synchronization points of multiple concurrent paths through the guideline.

4.4 Performance of the BART

Data from 100 patients collected and from these specific units: CardioGeneral unit (n = 20), Musculoskeletal unit (n = 20), Stroke unit (n = 30), Brain Injury unit (n = 10) and Spinal Cord Injury unit (n = 20). One patient’s data from Spinal Cord Injury unit had to be excluded due to missing critical data items. The verage age of patients in the sample was 55.3 (SD = 19.8) and the average BMI was 28.5 (SD = 16.3). Sixty four percent of the sample was male and 9% of the patients had history of immobility longer than 4 hours. ► Table 2 shows description of samples.

Table 2.

Sample characteristics

Age	Male	Female
≤ 35	17	6
36–50	11	3
51–65	21	12
< 65	14	15
Immobile >4 hrs
Yes	6	3
No	57	33
	Age	BMI
Mean	55.3	28.5
Standard Deviation	19.8	16.3
Median	59	25.5
Mode	61	34.6

For this study, a patient’s pressure ulcer risk is a sum of 6 scores from six parameters of the Braden Scale. When the total score is less than or equal to 18, patients are considered at risk for a pressure ulcer and preventive nursing interventions should be initiated immediately. Detailed Braden Scale pressure ulcer risks are:

• a)at risk (15-18);
• b)moderate risk (13-14);
• c)high risk (10-12); and
• d)very high risk (≤ 9).

Level of agreement between nurses and BART2 and agreement between the expert and the BART2 were calculated (► Table 1). Cohen’s kappa between nurses and BART2 ranged from moderate (0.54) to good (0.75) among parameters. The nutrition parameter had the lowest kappa value and the mobility parameter the highest kappa value. Pressure ulcer risk had a moderate kappa value (0.54). Cohen’s kappa between the expert and the BART2 ranged from moderate (0.58) to very good (0.95) among parameters. The moisture parameter had the lowest level of kappa value. Activity and nutrition parameters had the highest kappa value between the expert and BART2. The pressure ulcer risk was in very good agreement (0.83). Sensitivity and specificity of the BART2 were calculated.

Table 1.

Agreements between the nurses (RNs) and BART2 and between the expert and BART2

		RNs vs. BART2		Expert vs. BART2
Parameters	n	% Agree	Weighted Kappa	% Agree	Weighted Kappa
Sensory Perception	99	60	0.68	81	0.81
Moisture	99	65	0.68	68	0.58
Activity	99	78	0.70	95	0.95
Mobility	99	69	0.75	73	0.81
Nutrition	99	65	0.54	96	0.95
Friction & Shear	99	64	0.73	70	0.77
Pressure Ulcer Risk	99	67	0.54*	88	0.83*

*Simple Kappa

Sensitivity = (59/68)*100 = 86.8

*Sensitivity = Total # of cases that both BART2 and Expert scored as at Risk/ Total # of cases that Expert scored as at Risk

Specificity = (28/31)*100 = 90.3

*Specificity = Total # of cases that both BART2 and Expert scored as not at Risk/ Total # of cases that Expert scored as not at Risk

5. Discussion

We enhanced decision rules of the Braden Scale based Automated Risk Assessment Tool (BART) and increased generalizability and performance of the BART developed in the previous study. BART2 (enhanced BART) showed “very good” agreement with the expert in identifying the patient at risk for pressure ulcer, indicating successful enhancement of decision rules. High value of sensitivity and specificity of the BART2 in a health care setting other than which the BART was developed also indicated increased generalizability and performance.

The survey results of nurses’ interpretation of parameters of the Braden Scale explain the “moderate” level of agreement between nurses and BART2 and narrow down the area that needs further improvement. Nurses’ interpretations of parameters of the Braden Scale varied. Also, no significant difference was observed in the responses between Spaulding and Massachusetts General Hospitals. Therefore, survey results couldn’t be used to enhance the decision rules since these didn’t show consistent values. These results were aligned with identification of poor inter-rater reliability of the Braden Scale sub-scores due to misinterpretation of instrument wording [34]. However, it still indicates further improvement of the BART2.

Two layers of decision rule enhancement were performed in this study. One enhancement made was finding additional data items that could enhance the decision rules further with the experts from two different health care settings. Fourteen data items were identified and only seven out of fourteen were incorporated into the decision rules. This finding implies that adding more data items may not enhance the decision rules any more than they already are. The other enhancement was to validate logics of decision rules using Guideline Interchange Format (GLIF) at the detailed level systematically. Successful translation of decision rules in each of the six parameters of the Braden Scale produced six flowchart forms that were easy to comprehend and revealed necessary computable and implementable specifications. This translation using the protégé authoring tool was more useful and effective to validate logics of decision rules than using ordinary diagrams to represent decision rules.

One expert assessed the risk for pressure ulcers of patients whose data in data collection packages might weaken establishing gold standards. Although this wasn’t the best method, it was necessary and practical for the BART2 evaluation. This expert from Spaulding hospital has been assessing patients’ pressure ulcer risk for 35 years at Spaulding hospital setting and educated nurses about pressure ulcer risk assessment and prevention at the study site.

To enhance the rules of moisture, assessing “skin condition of significant skin folds” was integrated into the decision rules of the moisture parameter. However, it still shows “moderate” level of agreement (kappa = 0.58) between the expert and the BART2 and a “good” level of agreement (kappa = 0.68) between nurses and BART2. This implies that more sophisticated decision rules are needed for the moisture parameter. After adding a new rule of assessing albumin level, a “very good” level of agreement (kappa = 0.95) was found between the expert and BART2 and a “moderate” level of agreement (kappa = 0.54) between nurses and BART2. This suggests that nurses often do not include albumin level as a key factor when assessing the nutrition parameter of the Braden Scale compared with the assessment of the expert. Nurses should be made aware of, educated about, and emphasized to patients the importance of the nutrition and its relationship to the albumin level. This finding was similar to the reports of Stechmiller and colleagues’ from their research, emphasizing that the Braden Scale does not account for current pressure ulcer risk factors such as age, specific medical conditions affecting tissue perfusion, and physiological status changes [35].

This study produced one useful latent finding as a byproduct. The survey findings can be used as references when orienting nurses who score the Braden Scale regularly and update the risk of pressure ulcer in the nursing care plan over the course of hospitalization. The findings emphasize the areas where nurse educators can pay more attention to when they educate nurses with valid and reliable operational definitions of terms in the parameters in the Braden Scale, thereby making the scale more reliable and valid. This allows the associated interventions to mitigate the risk more effectively, ultimately to improve patient safety.

6. Limitations

Even though we demonstrate a good performance with a sufficient sample size, we validated the BART2 in only one acute rehabilitation hospital. Generalizability confirmed through this study is limited to rehabilitation hospitals that are similar in their settings. Also, only one expert assessed the risk for pressure ulcer of patients. If another expert who has experience in rehabilitation hospital settings assessed the same patients, the outcomes might be different. If the BART2 were validated in acute care hospitals and with a sample larger than N = 99, the outcomes might be different.

Another validation study of BART2 in acute health care settings is strongly recommended to further increase generalizability.

7. Conclusion

The study demonstrated successful enhancement of decision rules and showed increased generalizability and performance of the Braden Scale based Automated Risk Assessment Tool version 2 (BART2) in a sufficient sample size (N = 99). Results of this study also support high and effective performance of the BART2. However, our findings still illustrate the fact that more sophisticated decision rules are needed to improve moisture parameter performance. Once the moisture parameter is improved, BART2 will be able to perform at a level as high as that of the expert. Eventually, the BART2 will improve quality of care while accurately identifying the patients at risk for pressure ulcers and providing patient-specific care based on identified risks.

Clinical Relevance Statement

This study will facilitate developing automated pressure ulcer risk assessment tools that ultimately will improve quality care and patient outcomes.

Conflict of Interest

We, authors, declare that we have no conflicts of interest of this research.

Protection of Human and Animal Subjects

Human and animal subjects were not included in this research.