Feasibility of Using Algorithm-Based Clinical Decision Support for Symptom Assessment and Management in Lung Cancer

Abstract

Context

Distressing symptoms interfere with quality of life in patients with lung cancer. Algorithm-based clinical decision support (CDS) to improve evidence-based management of isolated symptoms appears promising but no reports yet address multiple symptoms.

Objectives

This study examined the feasibility of CDS for a Symptom Assessment and Management Intervention targeting common symptoms in patients with lung cancer (SAMI-L) in ambulatory oncology. The study objectives were to evaluate completion and delivery rates of the SAMI-L report and clinician adherence to the algorithm-based recommendations.

Methods

Patients completed a Web-based symptom-assessment, and SAMI-L created tailored recommendations for symptom management. Completion of assessments and delivery of reports were recorded. Medical record review assessed clinician adherence to recommendations. Feasibility was defined as ≥ 75% report completion and delivery rates and ≥ 80% clinician adherence to recommendations. Descriptive statistics and generalized estimating equations were used for data analyses.

Results

Symptom assessment completion was 84% (95% CI: 81–87%). Delivery of completed reports was 90% (95% CI: 86–93%). Depression (36%), pain (30%) and fatigue (18%) occurred most frequently, followed by anxiety (11%) and dyspnea (6%). On average, overall recommendation adherence was 57% (95% CI: 52–62%) and was not dependent on the number of recommendations (P = 0.45). Adherence was higher for anxiety (66%; 95% CI: 55–77%), depression (64%; 95% CI: 56–71%), pain (62%; 95% CI: 52–72%), and dyspnea (51%; 95% CI: 38–64%) than for fatigue (38%; 95% CI: 28–47%).

Conclusion

CDS systems, such as SAMI-L, have the potential to fill a gap in promoting evidence-based care.

Introduction

The majority of patients with lung cancer have multiple symptoms and high degrees of distress at presentation, and these symptoms change with the burden of disease and the cancer treatments themselves.^1–5 To date, most studies have addressed treatment of single symptoms, but oncology clinicians may benefit from assistance in assessing and managing multiple symptoms, which are common in their patients.^6–9 Palliative care clinicians have the expertise but either may not be present in significant numbers or are not consulted early enough in the course of a patient’s disease.¹⁰ Innovative ways are needed to integrate palliative care into oncology care.¹¹ Patients with lung cancer are an ideal group to test new approaches to aiding clinicians in their efforts to manage multiple symptoms.

The use of computerized questionnaires to gather symptom and quality of life (SQL) information in the outpatient setting has been established. Previous studies identified that these systems are associated with accurate assessment, improving communication, and decreasing symptom distress;^12–17 however, no change was noted in clinical management of symptoms.¹⁸ Clinical studies and review articles have identified targets for expanding the impact of SQL questionnaire use on care processes and health outcomes, including: using salient assessment instruments,^19,20 equipping clinicians to interpret SQL reports,^{19, 21} and providing specific recommendations for clinical management of SQL problems.²²

Clinical practice guidelines (CPGs) have been developed as tools to assist clinicians in the management of cancer-related symptoms.^23–31 However, such guidelines are not applied consistently in care delivery.³² A variety of barriers exist for implementation of these guidelines including the lack of: 1) a belief that guidelines will lead to better care, 2) time, 3) a system that reports symptoms over time, and 4) access to guideline-based recommendations that are sufficiently specific to guide patient care.^33–36

A few studies have examined the impact of applying CPGs for pain or depression as part of cancer care.^{6, 7, 37, 38} The results from these studies appear promising but further research is needed. Our study extends the literature by examining the feasibility of an algorithm-based Symptom Assessment and Management Intervention clinical decision support (CDS) system for the assessment and management of the most common Lung cancer (SAMI-L) symptoms (fatigue, pain, dyspnea, and depression/anxiety).

The use of CDS systems may help in the dissemination and adherence to CPGs. CDS is defined as computerized programs providing clinicians with person-specific information that is intelligently filtered and presented at the appropriate time to enhance health care. A variety of tools are available to provide CDS and enhance decision making in real-time: computerized alerts, reminders, condition-specific order sets, documentation templates, and clinical guidelines.³⁹ Features of CDS that improve clinical practice include providing a) CDS as part of the workflow, b) recommendations rather than assessment alone, c) CDS at the time and location of decision making, and d) ongoing computer-based CDS.^{40, 41}

Computer technology exists that can provide individually tailored, guideline-based recommendations at the time of each patient visit.^{33, 42} We reported previously on the modified ADAPTE process and nominal group technique we used to develop and approve locally adapted computable algorithms for the management of multiple symptoms in patients with lung cancer and on the pre-clinical testing of the SAMI-L system.⁴³ The objectives of this phase of the study were to evaluate: 1) patient completion rates for the symptom assessment tool, 2) delivery rates for the tailored report, and 3) clinician adherence to guideline-based recommendations tailored to individual patients and generated by computable algorithms for the management of fatigue, pain, dyspnea, and depression/anxiety.

Methods

Study Design and Participants

The primary aim of this study was to examine the feasibility of using an algorithm-based CDS system to generate symptom management recommendations in an outpatient setting. A single-arm study design was used for this aspect of the study. The study was conducted between December, 2010 and August, 2012 in two settings, Dana-Farber Cancer Institute, a comprehensive cancer center (CCC), and Boston Medical Center, an urban safety net hospital, with approval by the respective Institutional Review Boards.

Eligible clinicians were thoracic oncologists and nurse practitioners working in these sites. Eligible patients were 21 years of age or older; English-speaking; with a diagnosis of Stage III or IV non-small cell lung cancer (NSCLC), recurrence of NSCLC, or any stage small cell lung cancer; and receiving chemotherapy or targeted therapy (+/− any additional modality of therapy) with clinic visits at least once per month. Research assistants (RAs) obtained written consent from interested clinicians and patients.

Prior to using SAMI-L, clinicians met with the principal investigator (M.E.C.) and palliative care co-investigator (J.L.A.) for orientation to the validated patient symptom assessment tool,^44–48 computable algorithms that generated care recommendations and clinician summary reports (described below). Fig. 1 illustrates the summary reports that were used to orient clinicians to the intervention.

Fig. 1.

Annotated sample SAMI-L clinician summary report, used to orient clinicians to intervention.

Patients and their caregivers received instruction in how to use a touch screen notebook computer to answer the computerized questionnaires. Patients and their clinicians used the CDS system at each visit over six months after patient enrollment. Patients also received a written evidence-based symptom management “toolkit” ⁴⁹ that included behavioral suggestions to enhance symptom management.

Description of the Clinical Decision Support Intervention

Data Collection

RAs met patients prior to each clinic visit (maximum of once per week) to conduct the assessment. Patients self-reported their symptom severity, comorbidities, and alcohol use. RAs interviewed patients on their use of prescribed and over-the-counter supportive care medications. These data, along with receipt of cancer therapy and selected laboratory values (creatinine, platelet count, and hemoglobin level), on the visit date were entered by the RAs into SAMI-L. Table 1 describes the instruments and data collected at each visit. Shortened assessment forms, collecting only the essential information needed by algorithms, were available for patients who were feeling ill and unable to complete the regular length questionnaire.

Table 1.

Patient-Reported and Clinical Data Collected at Each Visit

	Patient-Reported Data	Clinical Data from Medical Record
Symptoms	Pain intensity (Pain Numeric Rating Scale26), quality, and pattern (investigator developed items)
	Dyspnea, fatigue, insomnia, anorexia, and overall symptom distress (Symptom Distress Scale44)
	Depression (Personal Health Questionnaire-945)
	Anxiety (Hospital Anxiety and Depression Scale46)
	Opioid-induced constipation (Constipation severity measure47 and Patient Reported Outcomes version of the Common Terminology Criteria for Adverse Events – PRO-CTCAE v373, 74)
Alcohol risk and drug allergiesa	Risk for hazardous alcohol use (Alcohol Use Disorders Identification Test48)
		Drug allergies
Comorbidities a	History of peptic ulcer disease
Cancer therapy		Start and end dates of all regimens of chemotherapy, radiation, or targeted therapy
Lab values for visit date		Platelet count, serum creatinine, hemoglobin
Patient characteristics		Age, weight on visit date
Supportive care medications		Prescribed medications with dose and frequency (opioid and non-opioid pain medications, antidepressants, anxiolytics, hypnotics, psychostimulants, laxatives)
	Actual use of prescribed and over-the-counter supportive care medications

^aData collected once at the beginning of the study and used at each visit.

SAMI-L Intervention

SAMI-L is a Web-based system that collects, processes and then presents the data to clinicians at the point of care. First, data are collected as described above. The de-identified patient data are transmitted outside the study settings across the Internet to a remote CDS system, SEBASTIAN (System for Evidence-Based Advice through Simultaneous Transaction with an Intelligent Agent across a Network),⁵⁰ in which the symptom management algorithms were encoded in computer language. Processing the data through the algorithms yielded tailored recommendations for symptom management for each patient at each visit.

Subsequently, a graphical clinician summary report (Fig. 1) was generated at each visit. It showed the patient’s symptom severity longitudinally and at each visit for five targeted common symptoms: pain, dyspnea, depression, anxiety and fatigue; receipt of cancer treatment over the course of the entire study; supportive care medication use; alcohol use and drug allergy warnings; and tailored recommendations to enhance symptom management. Recommendations included specific medications from appropriate drug classes to introduce, or dose adjustments calculated by the system based on current use of medications, and referrals to supportive services with reasons for referral. RAs printed and delivered reports to clinicians prior to each visit and collected reports after the visit.

Measures

Feasibility of the intervention was pre-defined as ≥75% completion rate for the symptom assessment tool across all visits, ≥75% pre-visit delivery rate of completed reports to clinicians and ≥ 80% clinician adherence to the guideline-based recommendations provided through the computerized algorithms. Feasibility levels for completion and delivery rates were based on a study by Wright et al. that demonstrated a 70% completion rate for longitudinal patient-reported symptom assessment in an outpatient oncology setting.⁵¹ Feasibility levels for clinician adherence were chosen based on the voluminous patient-related adherence literature in which 80% adherence is considered the standard.⁵² There is scant literature and no accepted standard for clinician adherence to guideline-based care at the current time because it is recognized that variation in care is influenced by patient characteristics, patient preferences and clinician judgment.^{35, 53, 54}

Completion rates for the symptom assessment tool were derived from documentation at each clinic visit of patient survey completion or reasons for non-completion. Delivery rates of the summary report were derived from documentation at each clinic visit of whether the report was generated and delivered to the clinician on time (before the exam visit), or reasons it was not.

Clinician-adherence was defined as following the guideline-based recommendations. Adherence was measured by medical record review and written comments provided on the report by clinicians. Each of 40 possible recommendations was coded as present or absent, and clinician responses were initially categorized into one of 16 adherence categories, which were then combined to a binary measure of complete/partial or no adherence (Table 2). The coding scheme and operational definitions for each code category were developed and pilot-tested to ensure the scheme was clear and easy to implement. Three coders were trained over two months through practice coding and discussion. Coding occurred over 10 weeks, with weekly discussions. A random selection of cases (n=17, 20%) were double-coded for reliability. Three sets of coding forms were not assessed because they were inadvertently recycled prior to calculation of inter-rater reliability. In reviewing records of 101 visits by 14 (17-3) patients, coders reached a mean of 98.9% (SD 2.5) and a median of 100% agreement per visit (range: 85.4–100%) in coding which recommendations were present. Percent agreement in coding adherence outcomes per recommendation had a mean of 80.6% (SD 26.9) and a median of 100% (range: 0–100%) per visit in 51 visits for which recommendations were made.

Table 2.

Outcomes Coded as Adherent or Non-Adherent, and Reasons for Non-Adherence

Adherent (fully or partially)	Non-Adherent
1. The recommendation was followed exactly	13. The health care provider did not follow any of the above strategies to manage the symptom
2. The recommended medication was given at a lower dose	Reasons for non-adherence
3. An alternative or additional medication was given	13a. The recommendation was not appropriate or was contraindicated
4. An alternative or additional referral to a specialist was made	13b. The patient’s report of the symptom to the health care provider did not match their self-report in the computerized assessment
5. The recommended prescription or referral was already in place for the symptom	13c. No reason given
6. The recommended prescription was already in place for another indication
7. Another specified health care provider or supportive care service was managing the symptom
8. Cancer therapy was held or dose reduced due to severity of the symptom
9. A behavioral strategy was advised
10. The health care provider offered the recommended medication, referral, or behavioral strategy; it was refused by the patient
11. The patient was sent to the emergency department or hospitalized in response to the symptom
12. “Other” responses categorized as adherent: The recommended medication was given at a higher dose

Statistical Analyses

Completion and delivery rates were analyzed using generalized estimating equations (GEE) clustering on the patient (possible repeated measurements for each patient). One-sided tests were performed to assess the probability that the proportion of completed (or delivered) reports was greater than 0.75.

Definitions for overall adherence were pre-determined prior to analyses. An adherence score was computed for each visit and defined as the proportion of recommendations to which the clinician fully or partially adhered. For example, if 10 recommendations were made for a visit and the clinician fully/partially adhered to eight, then the adherence score for that visit would be equal to 0.8. Adherence was analyzed with two methods to ensure consistent results regardless of the analysis method. First, the mean adherence score was estimated using a linear mixed model that accounted for the clinician and potential repeated measurements from a patient. The response variable was the adherence score, which ranged from 0 to 1. The second method fit the mean proportion of recommendations adhered to using a GEE model with a binomial probability distribution and logit link function to estimate the parameters. The response variable in the GEE method was the ratio of number of recommendations adhered to, to the total number of recommendations given in a single visit. Each method allowed for an adjustment for the total number of recommendations.

Once it was determined that there were consistent results across analysis methods, only the GEE model was used to explore further specific symptoms of interest (pain, depression, anxiety, fatigue, dyspnea) and specific recommendations of interest within a symptom (specifically, all fatigue recommendations). All analyses were conducted using SAS v. 9.2 and R; all statistical tests were at a significance level of 0.05.

Results

Sample

Tables 3 and 4 provide characteristics of the clinician (n=14) and patient (n=88) participants receiving the SAMI-L intervention. The majority of clinicians were white, male, and practiced in the cancer center. The typical patient was 61 years, white, female, had NSCLC and received combination chemotherapy.

Table 3.

Characteristics of Clinician Participants in the SAMI-L Intervention (N=14)

Characteristics	N (%)
Clinical Site
Boston Medical Center	3 (21)
Dana-Farber Cancer Institute	11 (79)
Role
Physician	7 (50)
Nurse Practitioner	7 (50)
Gender
Female	6 (43)
Male	8 (57)
Race
White	11 (79%)
Other	3 (21)
Age, yrs, median (range)	41 (28–61)
Years in oncology, median (range)	11 (4–30)

Table 4.

Characteristics of Patient Participants in the SAMI-L Intervention (N=88)

Characteristics	N (%)
Clinical Site
Boston Medical Center	3 (3)
Dana-Farber Cancer Institute	85 (97)
Gender
Male	33 (38)
Female	55 (62)
Age, yrs, median (range)	61 (39–81)
Ethnicity
Hispanic/Latino	5 (6)
Non-Hispanic/Latino	83 (94)
Race
White/Caucasian	76 (86)
Other	9 (10)
Missing	3 (3)
Marital Status
Single, never married	15 (17)
Married/living together	48 (55)
Separated, divorced, widowed	24 (27)
Missing	1 (1)
Computer Use
Never/Rare	14 (16)
Sometimes	10 (11)
Often/Very Often	61 (69)
Missing	3 (3)
Education
≤ HS	22 (25)
> HS	64 (73)
Missing	2 (2)
Disease type, and stage
SCLC	9 (10.2)
Limited stage	1 (11.1)
Advanced stage	8 (88.9)
NSCLC, adenocarcinoma	50 (56.8)
NSCLC, favor adenocarcinoma	4 (4.6)
NSCLC, squamous	16 (18.2)
NSCLC, favor squamous	0
NSCLC, not otherwise Specified	7 (8.0)
Other (e.g., carcinoid)	2 (2.3)
Stage I (including 0, IA, IB) a	2 (2.3)
Stage II (including IIA, IIB) a	2 (2.3)
Stage IIIA	9 (10.2)
Stage IIIB	7 (7.9)
Stage IV	59 (67.0)
Treatment types received b
Chemotherapy only	26 (29.5)
Investigational therapy only	14 (16.0)
Combined therapy	48 (54.5)

SCLC=small cell lung cancer; NSCLC=non-small cell lung cancer.

^aRecurrent disease.

^bTreatments received beginning one month prior to study entry until end of study.

Completion and Delivery Rates for SAMI-L

Eighty-eight patient participants had a total of 777 visits. As Table 5 indicates, the mean rate of patient completion of the symptom assessment tool overall was 84% (95% CI: 81–87%) with little fluctuation in the completion of the reports (82–87% in every two-month period) over the course of the study. Of 123 assessments not completed, reasons included staff missing the patient’s appointment because of schedule changes (n= 48, 39%), not enough time before the appointment (n= 27, 22%), patient refused (n=26, 21%) or was too sick (n=16, 13%), and technical problems (n=3, 2%). Only one patient on one visit requested the short-form assessment.

Table 5.

Reports Completed by Patients and Delivered to Clinicians, Clustering on Patient

	Total N (%)	Completed N (%)	95% CI for Completed	P-value a	Delivered N (%)	95% CI for Delivered	P-value a
Overall time frame	774	651 (84)	(81, 87)%	< 0.0001	580 (76)	(71, 80)%	0.39
0–2 months	333	275 (82)	(78, 86)%	0.0001	243 (73)	(67, 79)%	0.76
2–4 months	251	214 (85)	(80, 90)%	<0.0001	191 (77)	(71, 84)%	0.24
4–6 months	190	162 (87)	(82, 92)%	< 0.0001	146 (80)	(73, 86)%	0.10

CI = confidence interval.

^aThe P-value indicates the probability that the proportion of completed or delivered reports is greater than the predefined benchmark of 0.75.

Once patients completed the symptom assessment tool and the clinical information required by the algorithms was entered into the SAMI-L system, data could be submitted to the local server, sent through the Web service to the SEBASTIAN decision engine, and returned to the client server for printing in 0.4–2.1 (median 0.9) seconds. It took another one to two minutes for RAs to walk to the printing station, login, and print the report. There were 580 SAMI-L reports printed and delivered. The mean rate of timely delivery of the clinician report was 76% (95% CI: 71–80%), with consistent delivery rates (73–80% in every two-month period) over the six months of the study. The most common reasons that the SAMI-L reports were not delivered included: questionnaires not completed (n=123, 63%), no time before appointment (n= 20, 10%), and technical problems (n=37, 19%). Overall, almost all of the completed reports were delivered (90%; 95% CI: 86–93%) (Table 6).

Table 6.

Completed Reports Delivered to Clinicians, Clustering on Patient

	Completed N (%)	Delivered N (%)	95% CI for Delivered	P-valuea
Overall timeframe	651	580 (90)	(86, 93)	< 0.0001
0–2 months	275	243 (88)	(83, 93)	< 0.0001
2–4 months	214	191 (90)	(85, 95)	< 0.0001
4–6 months	162	146 (90)	(85, 96)	< 0.0001

CI = confidence interval.

^aThe P-value indicates the probability that the proportion of completed reports delivered to clinicians is greater than 0.75.

Clinician Adherence to Algorithm-Generated Recommendations

The median number of SAMI-L outpatient visits/patient was 7 (range, 1–14). There were 580 SAMI-L reports delivered representing 83 patients and 872 total recommendations. Of the 872 recommendations, clinicians completely or partially adhered to 494 and did not adhere to 378. Of the 580 reports, 270 (47%) included at least one recommendation and 214 (37%) included two or more recommendations. Using either the linear mixed model or GEE method, the adjustment for the total number of recommendations was not statistically significant (P=0.61 and 0.45, respectively). The intercept-only models for the average adherence score yielded similar results; 56% (95% CI: 50–63%) for the linear mixed model and 57% (95% CI: 52%–62%) for the GEE. Ninety-five percent of the time clinicians did not indicate a reason for failing to adhere to the recommendations; recommendations were perceived as not appropriate or contraindicated in 4% (n=14) of cases, and the clinician judged that the patient’s report of symptom severity during the visit did not match the electronic assessment in 1% (n=4) of cases where recommendations were made.

Recommendations were given at the following unadjusted rates: depression (n=315, 36%), pain (n=258, 30%) and fatigue (n=153, 18%), followed by anxiety (n= 93, 11%) and dyspnea (n=55, 6%). Adherence score estimates varied by individual symptoms as seen in Fig. 2. Adherence also varied by type of recommendation. For example, among recommendations for fatigue, adherence to recommendations for stimulant (44%, 95% CI: 27–61%) or hypnotic (44%, 95% CI: 26–62%) medications was higher than for referral to an exercise specialist/physical therapist (PT) (16%, 95% CI: 3–29%).

Fig. 2.

Estimates and 95% CIs for the average adherence to recommendations scores across all visits by symptom (GEE model).

Discussion

Our study extends the literature by providing information on the feasibility of using longitudinal assessment of symptoms and generating real-time reports that include tailored, evidence-based recommendations for common symptoms experienced by patients with lung cancer. We also provide data on adherence to those recommendations by clinicians in the ambulatory setting.

Patient completion rates for the symptom assessment tool were high, exceeding the criteria set to establish feasibility, and remained consistent over six months. These rates are consistent with or higher than other studies reporting longitudinal (three or more times) computerized SQL assessment in clinical trials.^{17, 51, 55–57} Wright et al.,⁵¹ comparing a prospective cohort trial to a study offering assessment routinely to all comers, concluded that overall compliance with assessment completion was higher when assessment was integrated into routine care. This result is cogent to our findings of non-completion reasons. Seventy-five (61%) of the times patients did not complete assessments were because of appointment changes of which RAs were not aware or patients not having enough time from arrival in clinic to the exam visit, two problems that would be largely obviated if the assessment were part of the clinical patient flow.

This is the first study to deliver CDS recommendations to enhance symptom management in “real-time” for more than one symptom. Although overall delivery rates of the summary reports were 76%, delivery rates for completed reports were 90% and remained consistent over time. The main reason that reports, including completed reports, were not delivered before the visit was lack of time, usually related to the patient arriving in clinic with insufficient time for the whole intervention process to be completed. This finding is not surprising, as the RAs worked outside the routine flow of the clinic and were trained to minimize any disruption to patient visits with their clinicians. Despite these limitations, in the majority of cases, we were able to gather complete, accurate and timely information needed for the SAMI-L algorithms and to integrate the summary reports successfully into the workflow of clinicians.

Although overall adherence to the algorithm-generated recommendations was lower than estimated for the study, we found that the overall adherence score was not dependent on the number of recommendations that were delivered. This observation is an important finding, suggesting that clinicians considered each recommendation separately and that addressing multiple symptoms does not adversely affect adherence. Thus, assessment and CDS for multiple symptoms appears to be a feasible approach for future interventions in patients who often experience more than one troublesome symptom.^{5, 8, 9}

The rates of clinician adherence to recommendations for individual symptoms were below our targeted level of 80%. Adherence to guideline-based care often depends on a variety of factors related to the guidelines themselves, the users and the organizational environment.^33,35,58 Further evaluation of the reasons for non-adherence will be informative to determine whether issues contributing to lower adherence can be addressed to improve future iterations of the algorithms. In addition, understanding whether certain patient, clinician and organizational characteristics are related to increased adherence to the recommendations would

Adherence to the recommendations for the fatigue algorithm was lower than for other symptoms. To our knowledge, only one other study⁶² evaluated adherence to guideline-based symptom management for fatigue among patients with cancer. Borneman and colleagues⁶² described usual care as compared with National Comprehensive Cancer Network (NCNN) cancer-related fatigue guidelines in 69 patients with heterogeneous types of cancer in an outpatient comprehensive cancer center. A medical chart review was conducted assessing adherence to guideline-based care among those who had scores ≥ 4 (0–10 Likert-type scale). Percent adherence was assessed for individual NCCN cancer-related fatigue recommendations, some of which were similar to the recommendations provided to clinicians in our study. Our post-hoc analysis of adherence to fatigue recommendations revealed a higher rate of adherence for specific interventions: referral to exercise/PT (0% in the Borneman et al. study⁶² vs. 14% in our study) and use of pharmacotherapy to treat fatigue (3% in the Borneman et al. study⁶² vs. 36% in our study). It is possible clinicians were unfamiliar with the exercise/PT referral service.

Further study assessing clinician adherence to guideline-based care for various symptoms is warranted using more rigorous methods. It is important to note that as advances are made in integrating guidelines into clinical practice, it is imperative to standardize measurement for adherence to guideline-based care so that findings can be compared across studies.^{63, 64}

Limitations

There were several limitations to our study. First, we assessed feasibility of only a subset of factors relevant to using real-time CDS for guideline-based symptom management, namely the willingness and ability of patients and clinicians to use the system at each clinic visit. This approach was necessary because of the complexity of implementing this novel approach in the clinical setting. We relied on RAs, where a similar role would be taken by clinic assistants or clinicians if the intervention were routinely implemented. Therefore, results cannot be generalized outside a research setting to clinical practice. Similarly, integration of the SAMI-L system into the electronic medical record was not possible for this phase of the study because the main purpose was to establish feasibility for implementing the intervention in real-time. Second, the overwhelming majority of participants were recruited and data collected in a single, National Cancer Institute-designated comprehensive cancer center, which has a different patient population, clinical volume, and supportive care resource structure than other cancer centers or many community cancer hospitals. Although we aimed to enroll equal numbers of participants at a community hospital, we were limited by its clinical volume and the trial timeline. Further study would be required to determine if findings would be comparable outside the setting of this study. Third, we did not require clinicians to specify reasons for non-adherence to clinical management suggestions, because participants in pre-clinical testing stated this would be too burdensome. As a consequence, we have limited data on why specific recommendations were not followed.

Conclusion

Innovative ways of integrating evidence-based palliative care into the front-line care of cancer patients are urgently needed.⁶⁵ Computerized point-of-care palliative care recommendations might be an important component of a broad approach aimed at overcoming barriers at the levels of patient, clinician and healthcare system (e.g., as a part of patient and health professional education and clinical audits⁶⁶). Voluntary, practice-based, quality improvement programs, using performance measurement and benchmarking among oncology practices across the U.S., are underway^67–70 and palliative care-focused quality improvement models also have been proposed.^{71, 72} The use of CDS systems, such as SAMI-L, in combination with these initiatives has the potential to fill an important gap in promoting evidence-based care. Further research is needed to assess the impact of the SAMI-L system on improving such clinical outcomes as decreased symptom severity and decreased utilization of services, for example, emergency room visits.