Abstract
Background:
Predictive cancer tools focus on survival; none predict severe symptoms.
Aim:
To develop and validate a model that predicts the risk for having low performance status and severe symptoms in cancer patients.
Design:
Retrospective, population-based, predictive study
Setting/Participants:
We linked administrative data from cancer patients from 2008 to 2015 in Ontario, Canada. Patients were randomly selected for model derivation (60%) and validation (40%). Using the derivation cohort, we developed a multivariable logistic regression model to predict the risk of an outcome at 6 months following diagnosis and recalculated after each of four annual survivor marks. Model performance was assessed using discrimination and calibration plots. Outcomes included low performance status (i.e. 10–30 on Palliative Performance Scale), severe pain, dyspnea, well-being, and depression (i.e. 7–10 on Edmonton Symptom Assessment System).
Results:
We identified 255,494 cancer patients (57% female; median age of 64; common cancers were breast (24%); and lung (13%)). At diagnosis, the predicted risk of having low performance status, severe pain, well-being, dyspnea, and depression in 6-months is 1%, 3%, 6%, 13%, and 4%, respectively for the reference case (i.e. male, lung cancer, stage I, no symptoms); the corresponding discrimination for each outcome model had high AUCs of 0.807, 0.713, 0.709, 0.790, and 0.723, respectively. Generally these covariates increased the outcome risk by >10% across all models: lung disease, dementia, diabetes; radiation treatment; hospital admission; pain; depression; transitional performance status; issues with appetite; or homecare.
Conclusions:
The model accurately predicted changing cancer risk for low performance status and severe symptoms over time.
Keywords: Cancer, prognosis, palliative care, logistic model, ADL, depression, dyspnea, pain
What is already known about the topic?
There are numerous predictive cancer tools that focus on survival. However, no tools predict risk of low performance status or severe symptoms, which are important for patient decision-making and early integration of palliative care.
What this paper adds
Our cancer study validated a model showing certain covariates (i.e. lung disease, dementia, diabetes, radiation treatment, hospital admission, pain, depression, transitional performance status, issues with appetite, and receipt of homecare) increase one’s risk by >10% of having low performance status, severe pain, well-being, dyspnea, and depression in the subsequent 6 months respectively. Generally these covariates were consistently associated with these outcomes even years after diagnosis.
Implications for practice, theory, or policy
Providing accurate predictions of future performance status and symptom severity can support decision-making and earlier initiation of palliative care, even alongside disease modifying therapies.
Introduction
There is evidence from several randomized trials showing the benefits of palliative care integration at time of diagnosis for cancer patients.1–3 A clinical practice guideline from the American Society of Clinical Oncology also supports the early integration of palliative care concurrently with standard oncologic care. 4 Yet at the population level, palliative care is often provided very late in the disease trajectory or not at all. In the US, palliative care was accessed in 45% of all deaths at a median of 17 days before death. 5 An enabler to integrate timely palliative care is the use of prognostic tools, particularly online tools, that can inform discussions about survival and support patient decision-making. A systematic review identified 22 online prognostic tools addressing 89 different cancers. 6
The systematic review however, also identified several major challenges in using prognostic tools to integrate palliative care earlier in the disease trajectory. First, tools focus on predicting mortality, but “no tool used quality-of-life as one of its outcomes. . .yet quality-of-life outcomes are most meaningful and important to patients when making treatment decisions.” 6 Clinicians and patients may be more inclined to discuss performance status and symptoms which affect their quality of life, rather than an estimated date of death. Second, the tools neither account for patient-reported outcomes, such as their current symptoms, nor prior health services use, which are clinically relevant, and predictive variables that vary over time.7,8 This limits their utility since decision-making often centers around potential trade-offs between quality-of-life now, in the future, and survival expectancy, which changes as the disease progresses. Third, the majority of tools used biological and laboratory variables (e.g. cancer antigen levels, elevated C-reactive protein, leukocytosis, etc.)9,10 which are not typically known by patients. This prevents patients from obtaining individualized prognostic information that could help them initiate discussions about palliative care.
Our team sought to address these limitations. Previously we developed and validated a model, including patient-reported outcomes to predict survival across the disease trajectory for patients with any cancer type. 11 In this study, we validated the model to also predict the risk for having low performance status and several severe symptoms. We call the final model PROVIEW+. By providing information about survival in the context of outcomes related to quality-of-life, such as anticipated symptom severity, PROVIEW+ can support decision-making and initiating palliative care earlier, even alongside disease modifying therapies.
Methods
Study design and population
We performed a population-based, retrospective cohort study of adults diagnosed with cancer, as confirmed by the provincial cancer registry in Ontario, Canada, from January 1 2008, to December 31, 2015. The study was reviewed by Hamilton Integrated Research Ethics Board and deemed exempt because it used de-identified secondary administrative data. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline. 12
Data sources
We used the following linked administrative databases [and corresponding covariates]: (1) Ontario Cancer Registry [cancer type, stage]; (2) Vital Statistics [age, sex, date of death]; (3) Statistics Canada [distance from cancer center]; (4) Activity Level Reporting [chemotherapy regime, radiation treatment]; (5) Discharge Abstract Database [hospitalization dates, diagnoses, cancer surgery, comorbidity]; (6) National Acute Care Registry System [Emergency Department (ED) visits]; (7) physician billing [physician home visits for palliative care, rostered patient]; (8) Home care database [end of life home care service use]; 9) Ontario Drug Benefit [long-term care status]; (10) Symptom Management Dataset [symptoms, performance status]; (11) interRAI database [performance status, symptoms].
The Symptom Management database contains data from a province-wide systematic screening program where oncology outpatients 13 at each outpatient visit voluntarily complete valid tools, namely the Edmonton Symptom Assessment System (ESAS) for symptoms and the Palliative Performance Scale (PPS) for performance status.14,15 The ESAS asks patients to self-report the severity of nine symptoms (i.e. pain, depression, well-being, shortness of breath, anxiety, nausea, tiredness, drowsiness, and appetite) on a scale of 0 (symptom absent) to 10 (most severe). The PPS summarizes a patient’s performance status based on a patient’s level of ambulation, activity, and self-care. It is scored from 80 to 100 indicating stable, 40–70 indicating transitional, 10–30 indicating end of life, and 0 indicating dead. The PPS was completed by a clinician starting in 2007. In 2013, Ontario switched to collecting functional scores using a patient-completed Eastern Cooperative Oncology Group score. Research has shown that the patient-reported PPS is comparable to and highly correlated with clinician-reported PPS.16–18
The interRAI database contains data from the Resident Assessment Instrument for Home Care (RAI-HC), a standardized tool for patients receiving publicly-funded home care services. The assessment is akin to the Minimum Data Set used internationally and has strong validity and reliability.19–21 Seventy percent of cancer patients use home care in the last year of life. 22 The assessment collects various quality-of-life related items, such as the presence of pain or symptoms of depression. The assessment is completed by the case manager at intake and reassessed approximately every 6–12 months. All above mentioned databases were linked using unique encoded identifiers and analyzed at ICES.
Outcomes
The model predicts the below five dichotomous (Yes/No) outcomes:
Low performance status: Defined as a score of 30–10 on clinician- or patient-reported PPS; or high health instability as indicated by a score of 4 or 5 on the CHESS Scale (Changes in Health, End-stage Disease, Signs and Symptoms) embedded within the RAI-HC (the scale includes items related to change in Activities of Daily Living status, evidence of end stage disease, etc.). 23 Both tools are validated to be highly predictive of mortality, including in hospital and community-based settings, and include physical functioning to complete activities of daily living as a main predictor of survival.23–26
Severe pain: Defined as: a score of 7–10 (severe) for pain on ESAS; or a score of 3 (severe or excruciating) for pain intensity from an item on the RAI-HC.
Severe dyspnea: Defined as: a score of 7–10 (severe) for shortness of breath on ESAS; or Yes for the presence of “shortness of breath” item on the RAI-HC.
Poor well-being: Defined as a score of 7–10 (poor) well-being on ESAS; or Yes for “client feels he/she has poor health when asked” from the RAI-HC.
Moderate-severe depression: Defined as a score of 4 to 10 (moderate to severe) for depression on ESAS; or a score of three or more on the Depression Rating Scale from the RAI-HC (e.g. made negative statements, expressions of unrealistic fears, repetitive anxious complaints, crying/tearfulness). 27
Index date for the models
For each of the five outcomes, prediction methods for logistic regression models were implemented independently starting at diagnosis in Year 0 (abbreviated as Y0) and re-implemented at Y1–Y4 after diagnosis. Thus, we have 25 unique models. Doing this means that to be included in each model, one must be alive at the start of each year and have an outcome measurement during the 6 month follow-up window, which aims to address loss to follow-up or death over time. In the Y0 model, all baseline covariates are measured from index to 3 months (after index), and the outcome window is from month 3 to month 9. In the Y1 model (and subsequent yearly models), baseline covariates are measured from ± 3 months from the new 12-month index point and the outcome is measured from month 3 to month 9 from the new index. Where multiple assessments were available, we used the assessment closest to the 6 month end point (for outcome) and closest to the index date (for baseline covariate).
Covariates
Each model included the following covariates: demographic characteristics (age at diagnosis, sex, caregiver living with patient (yes/no), lives within 50 km of a cancer center (yes/no)); clinical data (cancer type, cancer stage, presence of 1 of 13 other chronic diseases as determined by validated algorithms,28,29 type of chemotherapy (publicly funded oral drugs, immunotherapy, and systemic agents), receipt of radiation treatment (yes/no) and/or cancer surgery (yes/no) in the past (from diagnosis up to 3 months previously), and ongoing (within the past 3 months)); patient-reported outcomes (Performance status and nine symptom scores within 3 months of index date); and health care use within 3 months of index date (prior hospitalization, hospitalizations for palliative care (including palliative care consultations), living in long-term care, receipt of end-of-life homecare services, having a regular family physician, and received physician home-visit). 30
Statistical analysis for each model
Development
We randomly selected 60% of eligible patients for model derivation and used the other 40% for validation. To ensure random sampling, we assessed and compared the distribution of baseline characteristics between the derivation and validation cohorts. Each outcome was examined separately. Using the derivation cohort, we used a multivariable logistic regression model with baseline (time-fixed) characteristics to predict the risk of an outcome at 6 months from index. A priori, we created a multivariable model consisting of all potential predictors mentioned above. We then used a backward stepwise selection procedure for variable selection with a liberal 2-sided p-value <0.15 as the retention criterion. 31 We centered continuous covariates such as age and explored both linear and quadratic terms. Missing information from patient-reported categorical variables were handled by creating an additional missing category for that variable. Most of the missing data arose due to patients not completing an ESAS. As there was no obvious missing pattern, we elected to create a missing category rather than to impute or remove these patients from the analysis. Interactions between cancer type and stage were also incorporated to achieve maximal discrimination.32,33
Validation
For each outcome, the predictive performance of the derived model was assessed and compared using the validation cohort. The regression model estimates were applied to every individual in the validation cohort to obtain their corresponding estimated risk probability. The predicted number of outcomes was then compared to the actual number of outcomes in the validation cohort by composing a confusion matrix; we calculated sensitivity (true positive fraction), specificity (true negative fraction), accuracy (true positive or negative fraction), and discrimination. Discrimination was measured using the area under the ROC curve (AUC). 34
Calibration plots were also constructed using the validation cohort. This was done by grouping patients into deciles based on their predicted risk, and then plotting the observed outcome risk within a decile against the corresponding mean predicted risk within that decile.7,35 Points closer to the 45° line indicate better calibration. Characteristics of individuals belonging to the highest predicted risk decile were also examined. As a sensitivity analysis, we assessed model performance using complete case data (i.e. excluding patients with missing covariate values) and determined that the concordance statistics were very similar to our model using all patients (including missing covariate values). All analyses were conducted using the statistical software R version 2.15 and SAS version 9.3. 36
Results
Our population-based cohort identified 255,494 patients diagnosed with cancer between 2008 and 2015 in Y0. Each total cohort was then randomly split into derivation (60%) and validation cohorts (40%). As we repeated the derivation and validation process each year up to 4 years after diagnosis, conditional on survival and having an assessment in the outcome window, the total cohort reduced each year. For instance, the validation cohort in dyspnea model reduced over time from 101,696 (Y0) to 61,511 (Y1), 43,759 (Y2), 30,383 (Y3), and 20,672 (Y4).
The demographics for each of the derivation models in Y0 are presented in Table 1. (Supplemental Appendix e Table 1 includes all variables across all years). Across the five models in Y0 generally, the median age at diagnosis was 64 years old, 57% were female, and the most common cancers were breast (24%), lung (13%), prostate (9%), and colorectal (12%). Approximately 34% of the cohorts had Stage 3 or 4 disease, 42% had Stage 1 or 2, and 24% had unknown stage in the cancer registry. Within the first 3 months of diagnosis, 49% had cancer-related surgery, 34% received chemotherapy, and 26% received radiation therapy. In the cohort, 5% had high pain, while 33% had no pain, and 32% had missing values (e.g. did not complete an ESAS). In Y0, the prevalence of outcomes among the derivation and validation cohorts were very similar, which ranged from 2.4% (low performance status) to 10.5% (poor well-being). Across all years and models, the validation cohorts were very similar in their distribution of patient profiles to the derivation cohorts.
Table 1.
Baseline characteristics of cohort at time of first diagnosis.
| Variable | Value | Y0 |
||||
|---|---|---|---|---|---|---|
| Functional status |
Pain |
Dyspnea |
Depression |
Wellbeing |
||
| Cohort A (analysis) N = 75,287 | Cohort A (analysis) N = 100,578 | Cohort A (analysis) N = 101,696 | Cohort A (analysis) N = 99,915 | Cohort A (analysis) N = 100,828 | ||
| Prevalence of outcome | 1829 (2.4%) | 7354 (7.3%) | 9867 (9.7%) | 9277 (9.3%) | 10,577 (10.5%) | |
| Age at diagnosis | Median (IQR) | 64 (55–73) | 64 (55–73) | 64 (55–73) | 64 (55–73) | 64 (55–73) |
| Sex | Female | 42,950 (57.0%) | 57,377 (57.0%) | 57,810 (56.8%) | 57,258 (57.3%) | 57,305 (56.8%) |
| Cancer type** | Breast | 18,255 (24.2%) | 24,210 (24.1%) | 24,665 (24.3%) | 24,439 (24.5%) | 24,494 (24.3%) |
| Colorectal | 9251 (12.3%) | 11,863 (11.8%) | 11,962 (11.8%) | 11,785 (11.8%) | 11,941 (11.8%) | |
| Lung | 9398 (12.5%) | 12,525 (12.5%) | 12,464 (12.3%) | 12,162 (12.2%) | 12,303 (12.2%) | |
| Prostate | 6089 (8.1%) | 8827 (8.8%) | 9157 (9.0%) | 8805 (8.8%) | 9175 (9.1%) | |
| Cancer stage | Stage 1 | 13,824 (18.4%) | 20,319 (20.2%) | 20,569 (20.2%) | 20,211 (20.2%) | 20,165 (20.0%) |
| Stage 2 | 16,719 (22.2%) | 22,861 (22.7%) | 23,390 (23.0%) | 22,917 (22.9%) | 23,301 (23.1%) | |
| Stage 3 | 14,302 (19.0%) | 18,684 (18.6%) | 18,937 (18.6%) | 18,535 (18.6%) | 18,765 (18.6%) | |
| Stage 4 | 11,563 (15.4%) | 15,132 (15.0%) | 15,044 (14.8%) | 14,877 (14.9%) | 14,980 (14.9%) | |
| Uknown | 18,879 (25.1%) | 23,582 (23.4%) | 23,756 (23.4%) | 23,375 (23.4%) | 23,617 (23.4%) | |
| Radiation (within 3 months) | Yes | 19,445 (25.8%) | 26,548 (26.4%) | 26,697 (26.3%) | 26,269 (26.3%) | 26,486 (26.3%) |
| Chemotherapy (within 3 months) | Yes | 26,398 (35.1%) | 33,600 (33.4%) | 33,944 (33.4%) | 33,741 (33.8%) | 33,821 (33.5%) |
| Cancer surgery (within 3 months) | Yes | 36,576 (48.6%) | 49,667 (49.4%) | 50,076 (49.2%) | 49,670 (49.7%) | 49,878 (49.5%) |
| Chronic diseases † | CHF | 4263 (5.7%) | 5527 (5.5%) | 5577 (5.5%) | 5444 (5.4%) | 5465 (5.4%) |
| COPD | 6808 (9.0%) | 8874 (8.8%) | 8885 (8.7%) | 8708 (8.7%) | 8872 (8.8%) | |
| Dementia | 1317 (1.7%) | 1634 (1.6%) | 1582 (1.6%) | 1566 (1.6%) | 1578 (1.6%) | |
| Diabetes | 16,927 (22.5%) | 22,066 (21.9%) | 22,086 (21.7%) | 21,532 (21.6%) | 21,936 (21.8%) | |
| Renal disease | 3583 (4.8%) | 4523 (4.5%) | 4511 (4.4%) | 4423 (4.4%) | 4483 (4.4%) | |
| Distance from regional cancer center | < = 50 km | 60,737 (80.7%) | 79,976 (79.5%) | 80,708 (79.4%) | 79,390 (79.5%) | 79,964 (79.3%) |
| Was pt hospitalized in the past 3 months? | Yes | 4757 (6.3%) | 6286 (6.2%) | 6384 (6.3%) | 6163 (6.2%) | 6283 (6.2%) |
| Functional score at index (+3 months) ‡ | 0 = 100 | 24,314 (32.3%) | 26,944 (26.8%) | 27,386 (26.9%) | 26,692 (26.7%) | 27,186 (27.0%) |
| 1 = 90–80 | 15,895 (21.1%) | 16,756 (16.7%) | 16,868 (16.6%) | 16,740 (16.8%) | 16,857 (16.7%) | |
| 2 = 70–60 | 5304 (7.0%) | 5786 (5.8%) | 5734 (5.6%) | 5610 (5.6%) | 5585 (5.5%) | |
| 3 = 50–40 | 2795 (3.7%) | 2921 (2.9%) | 3006 (3.0%) | 2917 (2.9%) | 2949 (2.9%) | |
| 4 = 30–10 | 869 (1.2%) | 975 (1.0%) | 945 (0.9%) | 905 (0.9%) | 931 (0.9%) | |
| Missing | 26,110 (34.7%) | 47,196 (46.9%) | 47,757 (47.0%) | 47,051 (47.1%) | 47,320 (46.9%) | |
| Pain score at index (+3 months) | None | 25,071 (33.3%) | 33,236 (33.0%) | 33,727 (33.2%) | 33,030 (33.1%) | 33,172 (32.9%) |
| Low | 13,969 (18.6%) | 18,530 (18.4%) | 18,597 (18.3%) | 18,397 (18.4%) | 18,538 (18.4%) | |
| Moderate | 8953 (11.9%) | 11,265 (11.2%) | 11,356 (11.2%) | 11,241 (11.3%) | 11,275 (11.2%) | |
| High | 4386 (5.8%) | 5404 (5.4%) | 5314 (5.2%) | 5300 (5.3%) | 5304 (5.3%) | |
| Missing | 22,908 (30.4%) | 32,143 (32.0%) | 32,702 (32.2%) | 31,947 (32.0%) | 32,539 (32.3%) | |
| Wellbeing score at index (+3 months) | 0 = Best | 13,383 (17.8%) | 18,053 (17.9%) | 18,292 (18.0%) | 17,999 (18.0%) | 18,021 (17.9%) |
| 1–3 | 21,626 (28.7%) | 27,462 (27.3%) | 27,739 (27.3%) | 27,501 (27.5%) | 27,671 (27.4%) | |
| 4–6 | 13,850 (18.4%) | 17,824 (17.7%) | 18,035 (17.7%) | 17,649 (17.7%) | 17,786 (17.6%) | |
| 7–10 = Worst | 5240 (7.0%) | 6928 (6.9%) | 6880 (6.8%) | 6756 (6.8%) | 6776 (6.7%) | |
| Missing | 21,188 (28.1%) | 30,311 (30.1%) | 30,750 (30.2%) | 30,010 (30.0%) | 30,574 (30.3%) | |
| Dyspnea score at index (+3 months) | Yes | 4659 (6.2%) | 5814 (5.8%) | 5803 (5.7%) | 5696 (5.7%) | 5804 (5.8%) |
| No | 50,065 (66.5%) | 65,135 (64.8%) | 65,833 (64.7%) | 64,882 (64.9%) | 65,114 (64.6%) | |
| Missing | 20,563 (27.3%) | 29,629 (29.5%) | 30,060 (29.6%) | 29,337 (29.4%) | 29,910 (29.7%) | |
| Depression score at index (+3 months) | Yes | 10,059 (13.4%) | 13,136 (13.1%) | 13,080 (12.9%) | 12,898 (12.9%) | 12,960 (12.9%) |
| No | 41,688 (55.4%) | 54,624 (54.3%) | 55,221 (54.3%) | 54,413 (54.5%) | 54,667 (54.2%) | |
| Missing | 23,540 (31.3%) | 32,818 (32.6%) | 33,395 (32.8%) | 32,604 (32.6%) | 33,201 (32.9%) | |
| Pt has had palliative home care (nursing/personal support) | 3233 (4.3%) | 4383 (4.4%) | 4366 (4.3%) | 4266 (4.3%) | 4308 (4.3%) | |
CHF: congestive heart failure; COPD: chronic obstructive pulmonary disease; IQR: interquartile range.
Other cancer disease sites were other genitourinary, other gastrointestinal, hematologic, head and neck, gynecologic, and other sites.
Other chronic diseases measured but not reported were acutemyocardial infarction, arrythmia, asthma, coronary heart disease, diabetes, hypertension, inflammatory bowel disease, mood disorder, osteoarthritis, osteoporosis, renal disease, rheumatoid arthritis, and stroke; mental health hospital admission was also measured but not reported.
Functional score ranges from 0 to 100 (in 10-point increments), with 80 to 100 indicating stable, 40 to 70 indicating transitional, 10–30 indicating end of life, and 0 indicating dead.
After backward stepwise selection, each outcome model had a slightly different set of variables included in the final prediction model. We present the results of our models for Year 0 in Table 2. Covariates that increased the risk of having a low PPS in 6 months by >10% were: Chronic Obstructive Pulmonary Disease, dementia, diabetes; radiation treatment; a hospital admission in the prior 3 months; high pain; symptoms of depression; a current PPS score of 70–10; any issues with appetite; or received end-of-life homecare. Having an existing poor score on a symptom at index was one of the biggest predictors of having a poor score on the same symptom in 6 months’ time. Generally, these variables were also usually associated with a >10% increased risk of having other high symptoms in Y0. Moreover, these variables were also usually highly predictive in the other models, though this varied by year and by symptom model (Supplemental Appendix eTable 2).
Table 2.
Fully adjusted main effects model associations fo having low performance status or severe symptoms at 6 months from diagnosis across cohorts in Year 0 † .
| Parameter at index (diagnosis) | Y0 |
||||
|---|---|---|---|---|---|
| Low PPS at 6 months |
Severe pain at 6 months |
Severe dyspnea at 6 months |
Moderate-severe depression at 6 months |
Worst eellbeing at 6 months |
|
| OR (95% CI) | OR (95% CI) | OR (95% CI) | OR (95% CI) | OR (95% CI) | |
| Age at index date | 1.03 (1.02, 1.03) | 1 (0.99, 1) | 1.02 (1.02, 1.02) | N/A | 1 (1, 1) |
| Female (ref = male) | N/A | 1.15 (1.08, 1.22) | N/A | 1.18 (1.12, 1.25) | 1.11 (1.05, 1.17) |
| Chronic diseases | |||||
| CHF | N/A | 1.11 (1, 1.23) | 1.48 (1.36, 1.6) | N/A | 1.11 (1.02, 1.21) |
| COPD | 1.17 (1.03, 1.35) | 1.31 (1.21, 1.42) | 2.21 (2.08, 2.36) | 1.19 (1.1, 1.28) | 1.22 (1.14, 1.31) |
| Dementia | 1.52 (1.2, 1.93) | N/A | N/A | 1.36 (1.18, 1.58) | N/A |
| Diabetes | 1.17 (1.05, 1.31) | 1.2 (1.14, 1.28) | 1.1 (1.05, 1.16) | 1.16 (1.1, 1.22) | 1.17 (1.11, 1.23) |
| Cancer type (ref = lung)** | |||||
| Prostate | 0.39 (0.28, 0.52) | 0.61 (0.53, 0.69) | 0.22 (0.19, 0.25) | 0.41 (0.36, 0.47) | 0.43 (0.38, 0.48) |
| Colorectal | 0.78 (0.65, 0.95) | 0.84 (0.75, 0.93) | 0.34 (0.31, 0.37) | 0.85 (0.78, 0.94) | 0.72 (0.66, 0.78) |
| Breast | 0.57 (0.46, 0.71) | 0.76 (0.69, 0.84) | 0.33 (0.3, 0.36) | 0.76 (0.7, 0.84) | 0.67 (0.61, 0.73) |
| Cancer stage (ref = stage 1)** | |||||
| 4 | 2.24 (1.83, 2.74) | 1.71 (1.55, 1.88) | 1.59 (1.46, 1.73) | 1.62 (1.48, 1.76) | 1.57 (1.45, 1.7) |
| 3 | 1.41 (1.14, 1.74) | 1.37 (1.25, 1.5) | 1.33 (1.23, 1.44) | 1.41 (1.3, 1.53) | 1.22 (1.13, 1.31) |
| 2 | 1.03 (0.82, 1.3) | 1.15 (1.05, 1.26) | 1.17 (1.08, 1.27) | 1.22 (1.13, 1.32) | 1.09 (1.01, 1.17) |
| Radiation in last 3 months (yes) | 1.59 (1.43, 1.78) | 1.31 (1.23, 1.38) | N/A | 1.08 (1.03, 1.14) | 1.19 (1.13, 1.25) |
| Chemotherapy in last 3 months (yes) | N/A | 0.87 (0.82, 0.92) | N/A | 0.88 (0.84, 0.93) | 0.88 (0.83, 0.92) |
| Surgery in last 3 months (yes) | 0.78 (0.69, 0.88) | 0.75 (0.71, 0.8) | 0.79 (0.75, 0.84) | 0.85 (0.81, 0.9) | 0.89 (0.84, 0.93) |
| Within 50 km from cancer centre | 0.84 (0.75, 0.94) | N/A | 0.87 (0.82, 0.92) | 0.94 (0.89, 1) | 1.07 (1.01, 1.13) |
| Admitted to hospital in last 3 months (yes) | 1.24 (1.06, 1.45) | 1.12 (1.02, 1.22) | 1.13 (1.04, 1.22) | 1.18 (1.09, 1.28) | 1.18 (1.09, 1.28) |
| Pain score at index (ref = Level 0)* | |||||
| 3 | 1.19 (1, 1.42) | 3.92 (3.56, 4.31) | 1.26 (1.15, 1.39) | 1.26 (1.15, 1.39) | 1.32 (1.21, 1.45) |
| 2 | 1.08 (0.93, 1.25) | 2.29 (2.1, 2.49) | 1.1 (1.02, 1.19) | 1.26 (1.17, 1.36) | 1.34 (1.25, 1.44) |
| 1 | 0.97 (0.84, 1.12) | 1.47 (1.35, 1.6) | 1.13 (1.06, 1.21) | 1.12 (1.05, 1.21) | 1.18 (1.1, 1.26) |
| Wellbeing score at index (ref = Level 0)* | |||||
| 3 | N/A | N/A | 1.14 (1.02, 1.27) | 1.49 (1.33, 1.66) | 1.99 (1.79, 2.2) |
| 2 | N/A | N/A | 1.14 (1.05, 1.25) | 1.41 (1.28, 1.55) | 1.39 (1.27, 1.52) |
| 1 | N/A | N/A | 1.06 (0.98, 1.15) | 1.29 (1.18, 1.41) | 1.27 (1.17, 1.38) |
| Dyspnea at index (yes)* | N/A | N/A | 3.01 (2.8, 3.23) | N/A | 1.09 (1.01, 1.18) |
| Depression at index (yes) | 1.16 (1.01, 1.33) | 1.18 (1.1, 1.27) | 1.11 (1.04, 1.19) | 2.26 (2.12, 2.42) | 1.31 (1.23, 1.4) |
| Functional score at index (ref = Level 0)* ‡ | |||||
| 4 | 4.16 (3.32, 5.23) | 0.84 (0.67, 1.05) | 1.3 (1.08, 1.56) | 1.47 (1.23, 1.76) | 1.28 (1.08, 1.53) |
| 3 | 1.65 (1.36, 1.99) | 1.04 (0.91, 1.18) | 1.25 (1.12, 1.4) | 1.3 (1.17, 1.46) | 1.21 (1.09, 1.35) |
| 2 | 1.09 (0.92, 1.31) | 1.21 (1.09, 1.33) | 1.26 (1.15, 1.38) | 1.18 (1.08, 1.3) | 1.13 (1.03, 1.23) |
| 1 | 1.04 (0.9, 1.21) | 1.09 (1, 1.18) | 1.13 (1.05, 1.21) | 1.09 (1.01, 1.17) | 1.02 (0.95, 1.1) |
| Appetite at index (ref = Level 0)* | |||||
| 3 | 1.58 (1.31, 1.92) | 1.09 (0.97, 1.21) | 1.09 (0.98, 1.21) | N/A | 1.13 (1.03, 1.25) |
| 2 | 1.28 (1.08, 1.51) | 1.05 (0.96, 1.15) | 1.12 (1.03, 1.22) | N/A | 1.13 (1.05, 1.22) |
| 1 | 1.12 (0.94, 1.32) | 1 (0.91, 1.09) | 1.09 (1.01, 1.18) | N/A | 1.05 (0.97, 1.13) |
| Palliative homecare (yes) (ref = no homecare) | 1.33 (1.14, 1.55) | 0.82 (0.74, 0.91) | 0.82 (0.75, 0.9) | 0.77 (0.7, 0.85) | 1.09 (1, 1.19) |
CHF: congestive heart failure; COPD: chronic obstructive pulmonary disease; HR: hazard ratio; NA: not applicable (indicating covariate was not significant in the final model).
A full list of covariates for each model is given in eTable 2 in the Appendix.
Missing category is not shown.
The HR estimates are from the main effects–only model (without the interaction between cancer type and cancer stage).
Functional score ranges from 0 to 100 (in 10-point increments), with 80 to 100 indicating stable, 40 to 70 indicating transitional, 10 to 30 indicating end of life, and 0 indicating dead.
Calibration plots for all models in Year 0 in our validation cohorts show the observed values plotted along the predicted values closely falling along the 45° line (Figure 1). Model discrimination in our validation cohorts is very high. The AUC for all our models are shown in Table 3, an average AUC across all 25 models is 0.7676 (ranging from 0.8202 (Y3-Dyspnea) to 0.6630 (Y4-Well-Being)) (Supplemental Appendix e Figure 1).
Figure 1.
Calibration plots by deciles of predicted probability in Y0.
Table 3.
Area under the curve (AUC) scores, positive predictive values (PPV) and negative predictive values (NPV) by year.
| Y0 | Y1 | Y2 | Y3 | Y4 | ||
|---|---|---|---|---|---|---|
| PPS | AUC | 0.807 | 0.804 | 0.818 | 0.802 | 0.792 |
| PPV | 0.061 | 0.074 | 0.073 | 0.062 | 0.062 | |
| NPV | 0.992 | 0.990 | 0.992 | 0.991 | 0.991 | |
| Pain | AUC | 0.713 | 0.762 | 0.761 | 0.754 | 0.768 |
| PPV | 0.129 | 0.155 | 0.149 | 0.143 | 0.164 | |
| NPV | 0.962 | 0.967 | 0.967 | 0.967 | 0.966 | |
| Dyspnea | AUC | 0.790 | 0.820 | 0.820 | 0.820 | 0.819 |
| PPV | 0.215 | 0.230 | 0.239 | 0.249 | 0.255 | |
| NPV | 0.961 | 0.965 | 0.963 | 0.963 | 0.962 | |
| Depression | AUC | 0.723 | 0.786 | 0.787 | 0.779 | 0.789 |
| PPV | 0.173 | 0.184 | 0.179 | 0.196 | 0.183 | |
| NPV | 0.948 | 0.966 | 0.966 | 0.961 | 0.969 | |
| Well-being | AUC | 0.709 | 0.743 | 0.684 | 0.677 | 0.663 |
| PPV | 0.182 | 0.187 | 0.196 | 0.196 | 0.179 | |
| NPV | 0.942 | 0.952 | 0.953 | 0.947 | 0.947 |
Application of the model
The model was developed into an online tool, called PROVIEW+. To exemplify how the model could be used, we consider the following hypothetical scenario: a 70-year old male who was diagnosed with stage III lung cancer 3 years ago (i.e. the calculator would use the Year 3 model). His baseline characteristics at Year 3 are: while he received chemotherapy and radiation since diagnosis, he only continues to receive chemotherapy in the past 3 months; he has diabetes and hypertension; no symptoms except moderate dyspnea (score of 4); and has a PPS of 70. For someone with these baseline characteristics in our model, the probability in the next 6 months of having a poor PPS is 4.6% (95%CI: 3.0–7.2), severe pain is 2.1% (95%CI: 1.6–2.9), dyspnea is 10.1% (95%CI: 7.3–14.0), depression is 4.3% (95%CI: 3.0–6.1), and worst well-being is 9.3% (95%CI: 6.7–12.7). If the man was hospitalized shortly thereafter, and now has a PPS of 30 and severe scores of “8 out of 10” for all nine ESAS symptoms, his probability in the next 6 months of having a poor PPS is 28.2% (95%CI: 16.0–44.6), severe pain is 35.1% (95%CI: 22.9–49.5), dyspnea is 71.7% (95%CI: 57.9–82.3), depression is 78.6% (95%CI: 69.4–85.6) and worst well-being is 71.3% (95%CI: 59.3–80.9). These statistics coupled with his severe symptom burden and low PPS may trigger the man to discuss with his doctor additional surveillance and care planning to address potentially severe symptoms. They could also discuss uncertainty around the predictions in the context of where he is on his illness trajectory and his goals of care.
Discussion
In this study, we developed and validated several predictive models for the risk of poor performance status and severe symptoms for all cancer types over time. The models achieved high calibration and discrimination. The model shows how risk for various patient-reported outcomes changes due to changes in a patient’s condition, such as hospitalization, a decline in PPS, or a prolonged symptom exacerbation. Moreover, because this work advances a previously validated survival model, users can examine the trade-offs between future healthcare use, survival, and performance status and symptoms.
There are a few features of our model that are novel. Compared with other online tools that predict cancer survival,37,38 to our knowledge, our model is the only one that uses symptoms and performance status as covariates and outcomes. This is important because other tools might account for treatments, 39 but they do not differentiate among individuals who had the same treatments but have different performance status or symptom profiles. Our model was re-developed each year post-diagnosis (up to first 5 years) based on updated covariate information at each new anchor point. Thus it can be used at any time within the first 5 years after diagnosis, while accounting for changes in a patient’s condition or treatment over time.
Patients and families often make decisions that try to achieve both longer survival, but also improve the quality of life remaining. The model can support discussions about a palliative approach to care and shared decision-making, particularly for patients, families or clinicians who are hesitant to discuss a specific timeframe for death. For instance, a high risk of poor performance status has implications for the quality-of-life of both patient and family, and options for managing this potential transition can be explored without discussing risk of death. Further, by providing risks for short-term outcomes related to quality of life, such as severe pain or dyspnea in the next 6 months, the tool can then trigger discussions on how to manage those risks, such as initiation of palliative care services. Finally, because the tool can be completed by patients and families directly, pre-contemplative discussions could occur before visits with the oncologists, so that the clinic time is used most productively and the discussions do not necessarily need to be initiated by clinicians.
Limitations
Our study does not incorporate genetic biomarkers and specific targeted therapies, which were not available for this project. These variables, along with other patient-reported covariates (e.g. preferences, ethnicity, etc.) could be pursued in another iteration. Because patient-reported outcomes were either voluntary or among those receiving a homecare assessment, we do not have these data for all eligible patients at every time point which is a risk for selection bias, though we have among the largest, longitudinal databases with this information. Although the model was validated and the initial online calculator is available, future research is needed to test the model’s usefulness for shared decision-making, early palliative care integration, and improved outcomes. Testing and refining the online tool with patients and family users, as well as clinicians, is a planned subsequent step in the research program.
Conclusion
Our models showed that changing cancer risk for poor performance status and severe symptoms can be accurately predicted using administrative clinical and patient-reported outcomes data. Combining these risks with survival risk can help patients and families to understand how transition points (e.g. hospitalization or performance status decline) and treatment decisions (e.g. continuing treatment or initiating homecare) might affect different aspects of their disease journey ahead. In this way, the model can support the initiation of conversations about palliative care supports earlier in the disease trajectory.
Supplemental Material
Supplemental material, sj-xlsx-1-pmj-10.1177_02692163211019302 for Development and validation of a prediction model of poor performance status and severe symptoms over time in cancer patients (PROVIEW+) by Hsien Seow, Peter Tanuseputro, Lisa Barbera, Craig C Earle, Dawn M Guthrie, Sarina R Isenberg, Rosalyn A Juergens, Jeffrey Myers, Melissa Brouwers, Semra Tibebu and Rinku Sutradhar in Palliative Medicine
Acknowledgments
The authors would like to acknowledge the following people for analytic and coordination support: Julia Ma, Ana Gayowsky, Erin O’Leary, Lesley Moody, and Amina Benmessaoud.
Footnotes
Authorship: HS and RS conceived the study, acquired the data, and designed the analysis plan. HS drafted the manuscript. All authors interpreted the data, critically revised the manuscript for important intellectual content, and approved the final manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Declaration of conflicting interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the Canadian Institutes for Health Research (Grant #379009 and #383402). The study was supported by ICES, formerly known as the Institute for Clinical Evaluative Sciences, which receives funding from the Ontario Ministry of Health and Long Term Care. It also uses data from Cancer Care Ontario, Canadian Institutes of Health Information, IQVIA Solutions Canada Inc, and the Ontario Association of Community Care Access Centers. The opinions, results, and conclusions reported in this paper are those of the authors solely and the funders and data sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Data management and sharing: The de-identified administrative data are not publicly available and may be obtained from a third party, ICES (formerly the Institute for Clinical Evaluative Sciences) for researchers who meet the criteria for permissible access. These data represent secondary data analysis and are not owned or collected by the study authors. A data request can be send here: https://www.ices.on.ca/About-ICES/ICES-Contacts-and-Locations/contact-form
Research ethics and patient consent: The study was reviewed by Hamilton Integrated Research Ethics Board and deemed exempt because it used de-identified secondary data.
Patients and public involvement: Patients and the public were not involved in this research. It used de-identified, secondary administrative data analysis, (which is allowed to be used for research purposes), and thus patient consent was not obtained.
ORCID iDs: Hsien Seow 
https://orcid.org/0000-0001-6701-1714
Peter Tanuseputro
https://orcid.org/0000-0002-4409-0795
Sarina R Isenberg
https://orcid.org/0000-0001-6059-5366
Supplemental material: Supplemental material for this article is available online.
References
- 1. Bakitas M, Lyons KD, Hegel MT, et al. Effects of a palliative care intervention on clinical outcomes in patients with advanced cancer: the Project ENABLE II randomized controlled trial. JAMA 2009; 302: 741–749. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Zimmermann C, Swami N, Krzyzanowska M, et al. Early palliative care for patients with advanced cancer: a cluster-randomised controlled trial. Lancet 2014; 383(9930): 1721–1730. [DOI] [PubMed] [Google Scholar]
- 3. Temel JS, Greer JA, Muzikansky A, et al. Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med 2010; 363(8): 733–742. [DOI] [PubMed] [Google Scholar]
- 4. Ferrell BR, Temel JS, Temin S, et al. Integration of palliative care into standard oncology care: ASCO clinical practice guideline update summary. J Oncol Pract 2017; 13(2): 119–121. [DOI] [PubMed] [Google Scholar]
- 5. National Hospice and Palliative Care Organization. NHPCO facts and figures: hospice care in America. Alexandria, V: National Hospice and Palliative Care Organization, 2016. [Google Scholar]
- 6. Rabin BA, Gaglio B, Sanders T, et al. Predicting cancer prognosis using interactive online tools: a systematic review and implications for cancer care providers. Cancer Epidemiol Biomarkers Prev 2013; 22(10): 1645–1656. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Sutradhar R, Rostami M, Barbera L. Patient-reported symptoms improve performance of risk prediction models for emergency department visits among patients with cancer: a population-wide study in Ontario using administrative data. J Pain Symptom Manage 2019; 58(5): 745–755. [DOI] [PubMed] [Google Scholar]
- 8. Seow H, Barbera L, Sutradhar R, et al. Trajectory of performance status and symptom scores for patients with cancer during the last six months of life. J Clin Oncol 2011; 29(9): 1151–1158. [DOI] [PubMed] [Google Scholar]
- 9. Maltoni M, Caraceni A, Brunelli C, et al. Prognostic factors in advanced cancer patients: evidence-based clinical recommendations—a study by the Steering Committee of the European Association for Palliative Care. J Clin Oncol 2005; 23(25): 6240–6248. [DOI] [PubMed] [Google Scholar]
- 10. Stillwell AP, Ho YH, Veitch C. Systematic review of prognostic factors related to overall survival in patients with stage IV colorectal cancer and unresectable metastases. World J Surg 2011; 35(3): 684–692. [DOI] [PubMed] [Google Scholar]
- 11. Seow H, Tanuseputro P, Barbera L, et al. Development and validation of a prognostic survival model with patient-reported outcomes for patients with cancer. JAMA Netw Open 2020; 3(4): e201768. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015; 350: g7594. [DOI] [PubMed] [Google Scholar]
- 13. Barbera L, Seow H, Howell D, et al. Symptom burden and performance status in a population-based cohort of ambulatory cancer patients. Cancer 2010; 116(24): 5767–5776. [DOI] [PubMed] [Google Scholar]
- 14. Anderson F, Downing GM, Hill J, et al. Palliative performance scale (PPS): a new tool. J Palliat Care 1996; 12(1): 5–11. [PubMed] [Google Scholar]
- 15. Chang VT, Hwang SS, Feuerman M. Validation of the edmonton symptom assessment scale. Cancer 2000; 88(9): 2164–2171. [DOI] [PubMed] [Google Scholar]
- 16. Popovic G, Harhara T, Pope A, et al. Patient-reported functional status in outpatients with advanced cancer: correlation with physician-reported scores and survival. J Pain Symptom Manage 2018; 55(6): 1500–1508. [DOI] [PubMed] [Google Scholar]
- 17. Martin L, Watanabe S, Fainsinger R, et al. Prognostic factors in patients with advanced cancer: use of the patient-generated subjective global assessment in survival prediction. J Clin Oncol 2010; 28(28): 4376–4383. [DOI] [PubMed] [Google Scholar]
- 18. Al-Rashdan A, Sutradhar R, Nazeri-Rad N, et al. Comparing the ability of physician-reported versus patient-reported performance status to predict survival in a population-based cohort of newly diagnosed cancer patients. Clin Oncol (R Coll Radiol). Epub ahead of print 3 February 2021. DOI: 10.1016/j.clon.2021.01.008. [DOI] [PubMed] [Google Scholar]
- 19. Hirdes JP, Ljunggren G, Morris JN, et al. Reliability of the interRAI suite of assessment instruments: a 12-country study of an integrated health information system. BMC Health Serv Res 2008; 8: 277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Morris JN, Fries BE, Steel K, et al. Comprehensive clinical assessment in community setting: applicability of the MDS-HC. J Am Geriatr Soc 1997; 45(8): 1017–1024. [DOI] [PubMed] [Google Scholar]
- 21. Poss JW, Jutan NM, Hirdes JP, et al. A review of evidence on the reliability and validity of minimum data set data. Healthc Manage Forum 2008; 21(1): 33–39. [DOI] [PubMed] [Google Scholar]
- 22. Barbera L, Sussman J, Viola R, et al. Factors associated with end-of-life health service use in patients dying of cancer. Healthc Policy 2010; 5(3): e125–e143. [PMC free article] [PubMed] [Google Scholar]
- 23. Hirdes JP, Frijters DH, Teare GF. The MDS-CHESS scale: a new measure to predict mortality in institutionalized older people. J Am Geriatr Soc 2003; 51(1): 96–100. [DOI] [PubMed] [Google Scholar]
- 24. Hirdes JP, Poss JW, Mitchell L, et al. Use of the interRAI CHESS scale to predict mortality among persons with neurological conditions in three care settings. PLoS One 2014; 9(6): e99066. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Olajide O, Hanson L, Usher BM, et al. Validation of the palliative performance scale in the acute tertiary care hospital setting. J Palliat Med 2007; 10(1): 111–117. [DOI] [PubMed] [Google Scholar]
- 26. Downing M, Lau F, Lesperance M, et al. Meta-analysis of survival prediction with Palliative Performance Scale. J Palliat Care 2007; 23(4): 245–252. [PubMed] [Google Scholar]
- 27. Martin L, Poss JW, Hirdes JP, et al. Predictors of a new depression diagnosis among older adults admitted to complex continuing care: implications for the depression rating scale (DRS). Age Ageing 2008; 37(1): 51–56. [DOI] [PubMed] [Google Scholar]
- 28. Muggah E, Graves E, Bennett C, et al. The impact of multiple chronic diseases on ambulatory care use; a population based study in Ontario, Canada. BMC Health Serv Res 2012; 12: 452. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Muggah E, Graves E, Bennett C, et al. Ascertainment of chronic diseases using population health data: a comparison of health administrative data and patient self-report. BMC Public Health 2013; 13: 16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Tanuseputro P, Budhwani S, Bai YQ, et al. Palliative care delivery across health sectors: a population-level observational study. Palliat Med. 2017; 31(3): 247–257. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Agopian VG, Harlander-Locke M, Zarrinpar A, et al. A novel prognostic nomogram accurately predicts hepatocellular carcinoma recurrence after liver transplantation: analysis of 865 consecutive liver transplant recipients. J Am Coll Surg 2015; 220(4): 416–427. [DOI] [PubMed] [Google Scholar]
- 32. Kawai K, Ishihara S, Yamaguchi H, et al. Nomograms for predicting the prognosis of stage IV colorectal cancer after curative resection: a multicenter retrospective study. Eur J Surg Oncol 2015; 41(4): 457–465. [DOI] [PubMed] [Google Scholar]
- 33. Harrell F. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. New York, NY: Springer-Verlag, 2001. [Google Scholar]
- 34. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21(1): 128–138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Yi M, Meric-Bernstam F, Kuerer HM, et al. Evaluation of a breast cancer nomogram for predicting risk of ipsilateral breast tumor recurrences in patients with ductal carcinoma in situ after local excision. J Clin Oncol 2012; 30(6): 600–607. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Team. Rdc. R Development Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing, 2009. http://www.R-project.org [Google Scholar]
- 37. Wishart GC, Bajdik CD, Dicks E, et al. PREDICT Plus: development and validation of a prognostic model for early breast cancer that includes HER2. Br J Cancer 2012; 107(5): 800–807. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38. Kattan MW, Cuzick J, Fisher G, et al. Nomogram incorporating PSA level to predict cancer-specific survival for men with clinically localized prostate cancer managed without curative intent. Cancer 2008; 112(1): 69–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Stephenson AJ, Kattan MW, Eastham JA, et al. Prostate cancer-specific mortality after radical prostatectomy for patients treated in the prostate-specific antigen era. J Clin Oncol 2009; 27(26): 4300–4305. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental material, sj-xlsx-1-pmj-10.1177_02692163211019302 for Development and validation of a prediction model of poor performance status and severe symptoms over time in cancer patients (PROVIEW+) by Hsien Seow, Peter Tanuseputro, Lisa Barbera, Craig C Earle, Dawn M Guthrie, Sarina R Isenberg, Rosalyn A Juergens, Jeffrey Myers, Melissa Brouwers, Semra Tibebu and Rinku Sutradhar in Palliative Medicine