Building a Computer Program to Support Children, Parents, and Distraction during Healthcare Procedures

Abstract

This secondary data analysis used data mining methods to develop predictive models of child risk for distress during a healthcare procedure. Data used came from a study that predicted factors associated with children’s responses to an intravenous catheter insertion while parents provided distraction coaching. From the 255 items used in the primary study, 44 predictive items were identified through automatic feature selection and used to build support vector machine regression models. Models were validated using multiple cross-validation tests and by comparing variables identified as explanatory in the traditional versus support vector machine regression. Rule-based approaches were applied to the model outputs to identify overall risk for distress. A decision tree was then applied to evidence-based instructions for tailoring distraction to characteristics and preferences of the parent and child. The resulting decision support computer application, the Children, Parents and Distraction (CPaD), is being used in research. Future use will support practitioners in deciding the level and type of distraction intervention needed by a child undergoing a healthcare procedure.

INTRODUCTION

Researchers often rely on complex measures to predict complex human behaviors. Children’s distress responses to common healthcare procedures, such as immunizations and needle sticks, may be affected by multiple factors such as child temperament, state and trait anxiety, coping style, parents’ anxiety and parenting style, and other variables.¹ Adding further to the complexity, it may require dozens of questions to quantify complicated factors such as temperament or coping style, and it is not practical for busy clinicians to use extensive research measures to predict a child’s distress response to a procedure. Clinicians need brief, reliable screening tools that can be used to predict a response so that interventions can be tailored to the individual child in the clinical setting.

Pain and distress related to healthcare procedures is a significant problem for most children. Children who experience inadequate pain control during medical procedures can suffer immediate and long term, negative sequelae.^2,3 Research has shown that the immediate effects of minimizing child pain and distress during a procedure include a decrease in child and parent emotional stress, a higher rate of successful intravenous (IV) catheter insertions on the first attempt⁴ and decreased child physiological stress response.^5,6 Distraction, that is changing the individual’s focus of attention, is an evidence-based intervention that can be used to decrease child distress during painful medical procedures.^7,8 Parents can be instructed to coach their child in distraction using non-procedural talk, books, bubbles and other novelty toys to capture the child’s attention. Children differ in the level of distress experienced, and in their response to interventions designed to decrease their distress. To ensure that appropriate interventions are initiated clinicians need practical decision support systems that: (1) predict a child’s response to a painful procedure when parents provide distraction coaching; (2) recommend the level of support and resources needed to facilitate distraction; and (3) provide evidence-based tailored instructions for the parent and child.

The purpose of this paper is to describe the development of a decision support system, Children, Parents, and Distraction (CPaD). This computer based program predicts the level of a child’s distress when parent distraction is provided during a healthcare procedure and identifies tailored instructions for parents. The first step in developing CPaD was the secondary data analysis of an existing data set using data mining methodology to develop predictive models that identify risk for distress in children during a healthcare procedure and parent ability to provide distraction coaching to their child. Data mining is a method designed to search for consistent patterns and relationships among items in a large data set. These patterns are put into a ‘model’ that can be validated and refined by other test cases and then applied to predict responses for new cases.⁹ Once the predictive models were identified, the second step in building the CPaD decision support was to develop a rule-based, clinically derived approach to assign children to groups stratified by overall risk for distress in the clinical setting. The final step was to apply a decision tree for evidence-based instructions for tailoring distraction to characteristics and preferences of the parent and child. These three components were combined and programed for the computer based decision support application, CPaD. Predictive validity was evaluated in a clinically-based research study.

STEP 1: DATA MINING AND PREDICTIVE MODEL BUILDING

In this secondary, data mining analysis, data were used from a randomized clinical trial carried out to identify child, parent and procedural factors that explain children’s responses to parent provided distraction coaching during a healthcare procedure.¹⁰ In the study, parents were randomly assigned to either an intervention group, in which they were provided with distraction training (DT), or to a control group, in which they had no training on using distraction (No DT). Data were obtained on variables identified in the literature as potentially explaining a child’s response to parent provided distraction coaching.¹ Outcome measures of the child’s distress response to an IV insertion included: (a) behavioral distress using the Observational Scale of Behavioral Distress-Revised (OSBD-R)¹¹; (b) physiological response using salivary cortisol levels^12,13; (c) parent perception of the child’s distress using one item from the Perception of Procedures Questionnaire¹⁴; and (d) child report of pain using the Oucher.^15,16 In addition, the level of distraction provided by a parents was measured by a behavioral observation scale, the Distraction Coaching Index (DCI).¹⁷ Figure 1 and other reports^10,18 describe the study methods and results from the primary analyses. The Institutional Review Boards at each data collection site approved this study. All data was de-identified for the secondary analysis and therefore no further review for the protection of human subjects was indicated.

Figure 1.

Synopsis of study and primary analysis.

In order to develop predictive models, knowledge discovery in databases (KDD) or data mining techniques^19,20 were used. Table 1 includes a list of data mining terms (underlined in this article text) and definitions. Predictive data mining uses specific cases with known outcomes to construct models that use independent variables (such as demographics and survey responses) to predict outcomes for future cases. The predictive models were built using support vector machine (SVM) regression as implemented in the Weka [University of Waikato, Hamilton, New Zealand] data mining package.^21,22 Generalization, that is the performance of the model on unseen data, was further improved by an automatic feature selection method. Using this method, individual items are automatically selected from the data to build the best models for predicting outcomes. Together the set of predictive models maps child and parent characteristics to three predicted outcomes.

Table 1.

Data mining terms and definitions

Term	Definition
Automatic feature selection method	Evaluates the worth of a subset of items by considering the individual predictive ability of each item along with the degree of redundancy between them. Performs a combination of forward or backward search through the space of item subsets.
Cross-validation	A repeated hold-out evaluation method in which models are built using 90% of the cases and tested on the remaining, held-out, 10%. This is repeated ten times, such that every subject is used once as a test case.
Data mining techniques	The process of extracting patterns from large data sets to determine relationships between variables.
ε (Epsilon)	The Gaussian kernel parameter that controls the amount up to which deviations are tolerated in the regression.
Gaussian kernels	Statistical functions for normal curves.
Knowledge discovery in databases (KDD)	Process of discovering useful knowledge from data. Includes methods for data mining, but also encompasses data preparation, data selection, data cleaning, applying previous knowledge and interpreting data to assure that useful knowledge come from the data.
Mean Square Error (MSE)	A standard performance measure for regression problems and was used to compare the performance of our predictors with linear least-squares regression models.
Support Vector Machine (SVM) regression	Construct highly nonlinear regression models, and tolerates small fitting errors to improve the generalization of the resulting predictor on unseen data.
Weka	Software package for data mining [University of Waikato, Hamilton, New Zealand].
γ (Gamma)	The Gaussian kernel parameter that controls the complexity of the model. Greater values lead to models that more closely resemble a linear fit to the data, while smaller values capture nonlinear relationships between the predictive variables and the outcome.

System Design

Data from the study were used in a secondary analysis to test variables for predicting outcomes with new parents and children. Four models using a subset of the parent, child, and procedural variables predict three outcomes; coachability (DCI), behavioral distress (OSBD-R), and physiological distress (cortisol responsivity). The ability of the parent to provide distraction (coachability) was predicted from two sub-models that used data from the No DT and DT groups to predict DCI scores. Two other outcomes, the Oucher and parent perception of child distress, did not increase the overall predictive ability, and therefore were not included. The predictive models (Figure 2) are as follows:

Figure 2.

Development of the predictive models and steps for building CPaD decision support program

DCI Predictor: No DT

The DCI Predictor (model #1) for parents with no distraction training was developed using cases from the study’s No DT group. In this model, parent variables are used to predict what a parent’s DCI scores would be if no training in how to provide distraction coaching was given. The intent of this predictor is to generate a baseline expected score for a parent’s ability to provide distraction.

DCI Predictor: DT

The DCI Predictor for parents receiving distraction training (model #2) uses data from the study DT group to predict a parent’s ability to be a distraction coach (DCI scores) when training is provided. For a given set of parental characteristics, the difference between the two DCI Predictor components can produce an estimate of “coachability,” or the predicted improvement of the parent’s ability to provide distraction coaching to the child if training is provided. The DCI differential or coachability is the first risk output. High coachability predicts that the parent will improve in distraction ability if training in distraction is given, while low coachability predicts that training is not likely to change the parent’s ability to provide distraction.

Behavioral Distress Predictor

The OSBD-R Predictor (model #3) predicts child behavioral distress in future cases. Predicted OSBD-R is obtained using parent and child variables as inputs and the predicted DCI score from the study DT group. The second output for the model is a predicted OSBD-R score or risk for child behavioral distress.

Physiological Distress Predictor

The Cortisol Predictor (model #4) predicts child physiological response for future cases. The child’s cortisol level is predicted using the parent and child variables, predicted DCI, and one procedural variable, use of topical analgesic. This leads to the third output, predicted cortisol response or risk for physiological distress.

Model Construction

The process for constructing each model follows the same basic form. The first step was to apply an automatic feature selection method.²¹ Each model was constructed using support vector machine (SVM) regression with Gaussian kernels.²³ Data were collected on 255 items in the study, and automatic feature selection identified 44 items to be used as inputs in one or more of the four predictive models (Table 2). For optimization of the SVM regression performance we utilized two parameters, γ and ε. Optimal parameters for each model were chosen by a greedy search of the parameter space (Table 3).

Table 2.

Examples of items included in the predictive models of distress

Model	Example Items*
#1 DCI Predictor, No Intervention	Have you ever used distraction to help your child cope with a painful event? I am tense. Our family is organized and “together.” Father’s ethnicity.
#2 DCI Predictor, Basic Intervention	Once I decide how to deal with misbehavior of my child, I follow through on it. Do you think you will be effective in using distraction during today’s IV insertion? Mother’s education
#3 OSBD Predictor	Has your child ever had an IV? How do you expect your child to behave during today’s IV insertion? How does your child typically cope with pain? Does your child never seem to stop moving? I often notice how my body feels.**
#4 Cortisol Predictor	When thinking about the IV insertion today, how actively involved do you like to be in helping your child? Is your child generally fearful, anxious or worried? Expected dwell time for topical anesthetic.

Note:

^*Parent report items except where noted,

^**child report items.

Table 3.

Optimal model parameters

Model	Γ	ε
Control DCI	0.2	0.001
Intervention DCI	0.006	0.002
OSBD	0.04	0.0001
Cortisol	1	0.02

Note: γ is the Gaussian kernel parameter that controls the level of non-linearity of the model; ε is the parameter that controls the amount up to which deviations are tolerated in the regression.

Validation Process

Each model was extensively evaluated using multiple cross-validation tests. By averaging the results, we achieve an accurate and nearly unbiased estimate of model performance on unseen data. Mean Square Error (MSE) and standard deviation of MSE on holdout cases are shown in Table 4, estimated by averaging the ten separate runs of cross-validation. Performance on held-out cases was significantly better than linear regression in every case (p < 0.01), meaning predictive accuracy of the SVM regression models was superior to linear regression.

Table 4.

Test Mean Squared Error and (standard deviation) for predictive models

Data Set	Linear Regression Test MSE	SVM Regression Test MSE
Control DCI	8.330 (0.224)	7.446 (0.124)
Intervention DCI	12.059 (0.314)	10.278 (0.056)
OSBD	4.464 (0.121)	3.421 (0.012)
Cortisol	12.305 (0.298)	6.387 (0.115)

The model was also validated by comparing variables identified as explanatory in the traditional multiple regression (primary) analysis with variables identified through the vector SVM regression. Multiple regression analyses identified 30 items that explained the responses of the children in the primary study while data mining identified 44 items for predicting child behavior in future cases. Item by item comparisons showed many of the variables identified as being statistically significant by the explanatory linear regression analyses for OSBD-R and cortisol were also identified by the feature subset selection method and included in the predictive models.

STEP 2: RISK GROUP ASSIGNMENT

The next step in developing the decision support system was to translate the data mining outputs into meaningful clinical information. A rule-based approach was applied to the three SVM regression outputs (i.e., coachability, behavioral distress and physiological distress) in order to assign new individual parent-child dyads to groups based on risk of distress. Scores from each output were categorized into three approximately equal-sized groups of high, medium, and low risk. For DCI Differential, the high-risk group (that is, parents with low coachability scores) was defined by expected changes in the small range 0–10.15, the low-risk group had expected changes above 14, and the medium-risk group consisted of those in between. The intervals for predicted OSBD-R were: low risk, 0–1.075; medium-risk, >1.075–3.19; high-risk, above 3.19. Actual OSBD-R scores were from 0-. Predicted cortisol ranges were: low-risk, 0–2.2; medium-risk, >2.2–3.83; high-risk, above 3.83. Validity for the cutoffs was supported by a panel of clinical experts that reviewed videotapes of subjects at cutoff points.

Next, scores from the model’s three outputs needed to be combined into an overall risk of distress score for the future parent-child dyads. Again, a rule-based system was applied to combine the risk assessments (high, medium or low) of the three outcomes into an overall assessment of risk for distress for each possible combination of outputs. Rules for decision-making may be based on empirical data, clinical judgment, or a need to manipulate group size. In this case, a combination of these strategies was used. A numerical value of 3, 2 or 1 was assigned, respectively, to high, medium and low risks for DCI and cortisol. Because behavioral distress was the primary indicator of total child distress in the study, OSBD-R was given greater weight and assigned values of 5 for high, 3 for medium and 1 for low risk. Total risk for child distress was computed by summing the three outputs. Total scores were considered high risk for distress if they were >8, medium if 8–5, and low if less than 5. For example, a case with low predicted coachability (corresponding to the high-risk group, score = 3), medium predicted OSBD-R (score = 3), and high predicted cortisol level (score = 3) would be placed in the high-risk group (total score = 9). There are a total of 27 combinations of the three output models (DCI, OSBD-R, and cortisol) with the three possible distress levels (high, medium, low) for risk group assignment. When the rules were retrospectively applied to the study sample of 542 families, the distribution of children in the predicted high, medium, and low distress groups was 22%, 65%, and 13%, respectively. Levels of intervention to support distraction, based on the availability of local resources, could be then be allocated according to the individual child’s risk for distress.

STEP 3: TAILORING THE INTERVENTION

The final step in developing the decision support system was to build the capacity to tailor evidence-based instructions, and intervention, based on characteristics and preferences of the parent and child. Some of the items used to assess risk could also be used to tailor the intervention. An evidence-based guideline for tailoring the distraction intervention was developed by the investigators; from this guideline, a decision tree was used to tailor teaching points to parent and child characteristics and preferences. For example, if a parent responds that the child prefers to use books for distraction and prefers to look away during the procedure, specific directions are included for how to use books to distract the child and shield the child’s view of the procedure at the same time. Individualized outputs for each child are created from the decision tree based on answers to the questions.

PROGRAMMING FOR THE COMPUTER BASED APPLICATION

These three components were then combined by the research team into a web-based decision support system, CPaD, available at any location where there is internet access. The primary functions of the application are to (a) predict child risk for distress in response to healthcare procedure; (b) identify the level of support the parents need in order to provide distraction to the child, based on risk for distress; and (c) provide tailored educational material for the family. Additional functions to facilitate randomization and data management for research purposes were also included. The completed CPaD decision support system includes 44 predictive items plus 40 additional items to support tailoring and further research, for a total of 84 items. Computer programming and interfacing including screens with pull down menus, foils, toggles, and graphics to prompt responses and guide the user were developed to provide a simple user friendly interface for the clinical setting (Figure 3). Information is provided to the family both electronically and as an individualized handout that they can take into a treatment room and have available for future procedures. To access a prototype of the CPaD, follow the link and instructions provided.

Figure 3.

Screen shot of CPaD decision support program

CLINICAL VALIDATION

Initial validation to predict child distress in a clinical setting is supported by data from a recently completed follow-up study of child distress during an IV insertion. Of the 159 parents receiving DT (as described in the primary study), CPaD classified 45 children as low risk for distress and 114 children as high risk for distress. Analysis of the distress behaviors (OSBD-R scores) showed that 40% of children had an OSBD score that was within 1 of the predicted OSBD from CPaD, with an additional 36% differing from predicted OSBD by no more than 1 to 3 points. There was a significant difference (p = 0.006) between dyads predicted to be low vs. high (i.e., median OSBD-R scores of 0.67 vs. 1.32) indicating that the CPaD was effective in identifying risk category. The follow-up study took place in an active pediatric outpatient clinic, where parents and children completed the questions on the website in approximately 10 minutes. Both clinic staff and families found CPaD engaging and useful. Parents and children report they prefer the computer application to traditional methods for giving and receiving information.

DISCUSSION

Many of the clinical problems confronting nurses have multiple etiologies or contributing factors. As illustrated in this paper, child response to an invasive healthcare procedure and the effectiveness of parent distraction coaching to reduce child distress are influenced by numerous factors. These include: child age, anxiety, temperament, and coping strategy; parent anxiety, expectations, distraction coaching behavior and parenting style; and procedural difficulty and use of local anesthetics at the IV insertion site. Traditional regression procedures are of some utility for explaining factors accounting for responses, whereas data mining predicts responses for new cases. In order to develop a screening tool to identify those patients most in need of intervention, predictive modeling was needed. Data mining does not replace traditional statistical procedures, but offers subtle pattern recognition that may not be intuitive.²⁴ Using data mining techniques to systematically examining each of the 255 items in the original data set; we reduced the salient predictive items to just 44.

The KDD process demonstrated significantly better predictive accuracy than the traditional statistical modeling approach. The flexibility of the SVM model allows highly nonlinear predictors, which can automatically incorporate interactions among the independent variables. This flexibility can be necessary when modeling a complex, poorly understood phenomenon like reaction to pain. With properly chosen parameters, the SVM model achieves this flexibility while still generalizing well to new cases.

The accuracy and flexibility of the SVM method comes at the expense of interpretability. The constructed models cannot be directly interpreted in the same sense that a linear regression model or a decision tree can be. Hence, a sensitivity analysis would be required to tease out the specific effects of the various independent variables. The approach is also affected by the number of training cases; in general, more cases lead to better predictions, and the relatively small number of cases in this study limits the predictive accuracy. Further, the addition of new cases requires occasional retraining of the models. Since families in the follow-up study all receive some level of distraction based on CPaD, all future cases will have received an intervention of some sort and the "DCI: No Intervention" model will receive no new training cases. Moreover, different levels of intervention explored in the follow-up study are not explicitly modeled. Thus, in order to stay current, the model will need to be retrained and the architecture adjusted.

Clinical judgment is necessary to turn the results of the data mining process into clinically useful decisions. Decision rules are based not only on empirical data, but also on the fit of data to risk groups. In this study, the different combinations of low, medium and high risk for child behavioral and physiologic distress and parent coachability were weighted and assigned to overall risk for child distress. Expert clinicians on the research team decided that the child’s predicted behavioral distress level should carry more weight in assigning overall risk than predicted parent coachability or child physiologic response. This decision was based on the team’s years of clinical experience with children undergoing painful procedures. Team members observed how difficult it could be to change children’s distress responses after a pattern of behavior is established over time. Children who have repeated bad experiences with healthcare procedures and who respond with high distress are more likely to continue with high behavioral distress.^2,3 This observation was important for assigning risk for future distress.

Recently completed research used the CPaD computer application to assign dyads into risk groups in order to test three levels of distraction interventions. The web-based application also serves as a database for research purposes, eliminating the need for data entry and double entry thereby decreasing entry errors. Now that this research is completed, the models will be re-trained. Further validation in other settings is needed.

In summary, this paper provides an example of using data mining to investigate a complex issue related to children’s healthcare. Data mining results and clinical expertise were combined to develop a computer application to support practitioners in deciding the level of distraction intervention or care needed by a child undergoing a medical procedure and tailor instructions based on characteristics and preferences of the parent and child. Collaborative research teams consisting of nurse researchers, traditional statisticians and scientists with data mining skills can help to transform information from large data sets into clinically meaningful knowledge to support decision making.

CPad Prototype Link and Instructions.

• Enter the CPaD website at: CPaD

• Login ID: demo

• Subject ID: for example of child predicted to have high distress enter 4000, for low distress enter 5000 and for medium distress enter 6000

• Password: qwe123