Abstract
Objectives There is limited research on suicide risk screening (SRS) among head and neck cancer (HNC) patients, a population at increased risk for suicide. To address this gap, this single-site mixed methods study assessed oncology professionals' perspectives about the feasibility, acceptability, and appropriateness of an electronic SRS program that was implemented as a part of routine care for HNC patients.
Methods Staff who assisted with SRS implementation completed (e.g., nurses, medical assistants, advanced practice providers, physicians, social workers) a one-time survey ( N = 29) and interview ( N = 25). Quantitative outcomes were assessed using previously validated feasibility, acceptability, and appropriateness measures. Additional qualitative data were collected to provide context for interpreting the scores.
Results Nurses and medical assistants, who were directly responsible for implementing SRS, reported low feasibility, acceptability, and appropriateness, compared with other team members (e.g., physicians, social workers, advanced practice providers). Team members identified potential improvements needed to optimize SRS, such as hiring additional staff, improving staff training, providing different modalities for screening completion among individuals with disabilities, and revising the patient-reported outcomes to improve suicide risk prediction.
Conclusion Staff perspectives about implementing SRS as a part of routine cancer care for HNC patients varied widely. Before screening can be implemented on a larger scale for HNC and other cancer patients, additional implementation strategies may be needed that optimize workflow and reduce staff burden, such as staff training, multiple modalities for completion, and refined tools for identifying which patients are at greatest risk for suicide.
Keywords: suicide prevention, patient-reported outcomes, EHR, ePROs
Background and Significance
Suicide is a leading cause of death in the United States and disproportionately affects cancer patients. 1 2 3 4 5 Individuals with cancer are four times more likely to commit suicide than the general population. 5 Compared with other cancer types, individuals with head and neck cancer (HNC) are at elevated risk for suicide. 6 7 Prior research suggests that HNC cancer patients are twice as likely to die from suicide compared with other cancer patients. 6 HNC is the seventh most common cancer globally and is partially attributed to alcohol and tobacco use, factors associated with suicide risk. 8 9 10 Additionally, due to the location of HNC and the aggressive treatment, HNC patients experience high levels of pain, financial hardship, emotional distress, and functional impairment (e.g., difficulty with speech and swallowing—HNC patients, there is limited research on suicide prevention interventions, such as suicide risk screening (SRS).
Health care systems are increasingly using electronic patient-reported outcomes (ePROs) to screen patients for suicide risk as a part of routine care. Prior studies have tested interventions that integrate ePROs into the electronic health record (EHR) to standardize screening for suicide risk in primary care and emergency department settings. 14 15 16 17 18 These interventions have demonstrated feasibility and improved access to behavioral health services. 14 17 18 ePRO systems for SRS are also being tested in oncology. For example, the Princess Margaret Cancer Centre in Canada integrated the nine-item Patient Health Questionnaire (PHQ-9) into the EHR to screen for suicide risk and found that completion of risk screening was associated with reduced suicide mortality among cancer patients. 19 Additional studies are needed to test EHR-based SRS in other oncology settings and to assess implementation outcomes, such as feasibility and acceptability. While there is strong evidence to suggest that integrating ePROs into routine care delivery can improve patient outcomes, the reach of ePROs has been limited due to implementation barriers. 20 21 22 23 24 For example, routine screening for psychosocial distress among cancer patients is recommended by clinical guidelines, but implementation varies across settings due to barriers, such clinician time and workflow integration. 25 26 27 28 To ensure SRS can be reliably implemented as a part of routine cancer care, additional studies are needed to assess SRS implementation.
To address this gap, this study assessed oncology professionals' perspectives about the feasibility, acceptability, and appropriateness of implementing an EHR-based SRS program as a part of routine care for HNC patients. The study was conducted at Moffitt Cancer Center (Moffitt), a National Cancer Institute (NCI)-designated Comprehensive Cancer Center. Study findings can be used to design future SRS programs for oncology settings.
Methods
Study Design
The study used a sequential explanatory mixed methods design. A survey was administered among HNC oncology professionals who assisted with SRS implementation and followed by qualitative interviews to expand upon the survey findings and develop a qualitative description of participants' experience with SRS implementation. 29 30 Data were collected during the initial implementation of the SRS intervention (first 3 months) from February to April 2021 so information gathered could be used to refine the intervention early on during implementation.
Suicide Risk Screening Intervention
Moffitt Cancer Center has implemented ePROs as a part of routine care since 2015 in the Departments of Radiation Oncology and Supportive Care Medicine. 31 32 33 34 In 2021, Moffitt's selected the Department of Head and Neck Oncology (HNC) to be the next clinic for ePRO integration. After gathering feedback from key stakeholders, the Patient Reported Information and Outcomes committee decided to include SRS as a part of the ePRO assessment given the high burden of suicide risk among HNC patients. 6 The final tool, named the patient-reported symptom assessment (PRSA), included physical (e.g., pain, nausea) and psychosocial (e.g., distress) symptoms that were already being collected in other clinics and added ePROs to assess suicidal ideation. The PRSA included 16 items from the PHQ-9 (new addition), the National Comprehensive Cancer Network (NCCN) Distress Thermometer, and the revised Edmonton Symptom Assessment System (ESAS). 7 8 9 These tools were selected based on input from the PRIO committee, which includes a diverse group of stakeholders (e.g., oncology, supportive care, social work, research). The ESAS has been validated in cancer patients in general and HNC patients specifically. 35 36 The NCCN Distress Thermometer has been validated in cancer patients in general. 37 The PHQ-9 has not been validated in cancer patients as a screening tool for suicidal ideation but has been paired with validated SRS tools to determine which patients may benefit from additional screening. Similar to the Princess Margaret Cancer Centre, our Cancer Center implemented a two-stage process: (1) patients first completed the PHQ-9 as a brief screener and (2) patients with concerning scores were prompted to complete a more comprehensive SRS screening (described below). The PRSA was made available in English and in Spanish. All patients with in-person visits were eligible to complete the PRSA during the check in process. Patients were given an iPad by a patient access representative (PAR) and provided with technical assistance (e.g., how to use tablet keyboard). A cloud-based application developed by the study site's application development and patient portal team was installed on the iPad to facilitate PRSA administration and EHR data integration. Multiple presentations at various faculty meetings (e.g., surgical team, medical oncology team) were held to make key stakeholders aware of the change and trainings were held for team members that would support implementation (e.g., nurses, medical assistants, PARs).
Once patients completed the PRSA, clinicians could review the responses in close to real time in the patient's medical record. Patients who reported any response other than “not at all” to the PHQ-9 question, “How often have you been bothered by thoughts that you would be better off dead or thoughts of hurting yourself in some way over the past 2 weeks?” were automatically referred to social work to be screened for active suicidal ideation (e.g., thoughts and plans for self-harm). The social work team assessed patients using the Columbia-Suicide Severity Rating Scale, which has been validated in clinical settings for assessing suicidal ideation. 10 11 12 13 If a patient was identified as having active suicidal ideation, the social work team implemented additional interventions (e.g., suicide safety plan). Nurses were responsible for reviewing patients' PRSA scores with the patient and determining if additional intervention was needed (e.g., symptom management support). For example, patients requiring additional symptom management support could be referred to Moffitt's Supportive Care Medicine Department. Patients reporting distress were offered the option to meet with the social work team.
During the pilot, the PRSA was completed by 746 patients. Among the 746 patients who completed the PRSA, 34 patients reported passive suicidal ideation based on PHQ-9 score and were referred to social work for SRS. Of the 34 patients screened for active suicidal ideation by social work, 3/34 (8.8%) patients were identified as having immediate suicide risk and received additional intervention (e.g., suicide safety plan). The remaining patients not at immediate risk for suicide ( n = 31) received additional services including supportive care ( n = 28), referral to social service agencies (e.g., financial counseling) ( n = 22), and facilitation of goals of care conversations ( n = 7).
Data Collection
Participating oncology health care professionals completed a one-time survey and one-time interview. The survey was administered electronically via Qualtrics (Qualtrics, Provo, Utah, United States). All staff who assisted with PRSA implementation (e.g., nurses, medical assistants, PARs, advanced practice providers [APPs], social workers, physicians) received an email inviting them to participate in a survey. Informed consent was provided at the time of the survey. The survey response rate was 65.9% (29/44 individuals). The survey assessed perceptions about implementation outcomes including feasibility, acceptability, and appropriateness. 38 39 Outcomes were assessed using the Feasibility of the Intervention Measure, the Acceptability of the Intervention Measure, and the Intervention Appropriateness Measure. 38 The measures contain four items with response options ranging from 0 (strongly disagree) to 5 (strongly agree). Items for each measure were summed to create a total score (range: 0–20). A total score of 20 represents the best implementation outcome (e.g., highest level of participant satisfaction). A validated cutoff for these measures has not been established; therefore, a score of 16 or higher was defined as a cutoff for establishing feasibility, acceptability, and appropriateness based on a prior study among oncology professionals (a score of 16 denotes an average response of agree or strongly agree for each item within the measure). 40
All staff who assisted with PRSA implementation or support (e.g., clinical informatics staff) received an email inviting them to participate in an interview. We did not include informatics staff in the survey given they may not be able to answer key questions (e.g., how appropriate the intervention is for this patient population). We chose, however, to include them in the interview process given their key role in supporting the intervention. Informed consent was provided at the time of the interview. The interview participation rate was 51.0% (25/49 individuals). Individuals who declined to participate cited lack of time and competing priorities as the primary reasons. The interviews were conducted by two qualitative specialists from Moffitt's Participation Research, Interventions, and Measurement Core through videoconference, recorded, and transcribed verbatim. The interviews were facilitated using a semi-structured interview guide that assessed perceptions about implementation outcomes, barriers, and facilitators guided by the Consolidated Framework for Implementation Research 41 ( Supplementary Table S1 , available in the online version). The interviews were approximately 30 minutes in length (mean time: 31.1 minutes; standard deviation [SD]: 12.3 minutes). Participants did not receive any incentives for study participation.
Data Analysis
For the survey data, we calculated descriptive statistics (e.g., mean, SD, percentages) to summarize staff perceptions about feasibility, acceptability, and appropriateness using Stata version 17.0 (StataCorp, College Station, Texas, United States). For the interview data, we used a hybrid approach by developing a codebook that included codes based on concepts from the interview guide (implementation outcomes, barriers, and facilitators) and themes that emerged from the data. 42 43 Two qualitative research specialists from Moffitt's Participant, Research, Intervention, and Measurement Core coded all transcripts and discussed and resolved any coding discrepancies using NVivo 12 Plus (Burlington, Massachusetts, United States). The individuals set a threshold for determining when data saturation was achieved, which was reached at 25 interviews. 44 Therefore, no additional interviews were conducted. For study reporting, the study team adhered to the Consolidated Criteria for Reporting Qualitative Research guidelines. 45 Moffitt's Institutional Review Board of Record, Advarra, reviewed the study protocol and determined the study to be exempt.
Results
Sample Characteristics
Survey participants ( N = 29) included nurses (20.7%), medical assistants (20.7% ) , PARs (17.2%), APPs (13.8%), physicians (10.3%), and social workers (17.2%) ( Table 1 ). About a quarter (27.6%) of survey participants had experience with implementing PROs as a part of clinical care previously. The average job tenure of survey participants was 8.2 years (SD: 6.7).
Table 1. Survey and interview participant sample characteristics.
| Characteristic | Survey participants, N = 29 | Interview participants, N = 25 |
|---|---|---|
| Occupation, N (%) | ||
| Nurse | 6 (20.7) | 5 (20.0) |
| Medical assistant | 6 (20.7) | 4 (16.0) |
| Patient access representative | 5 (17.2) | 3 (12.0) |
| Advanced practice provider | 4 (13.8) | 3 (12.0) |
| Physician | 3 (10.3) | 2 (8.0) |
| Social worker | 5 (17.2) | 4 (16.0) |
| Informatics staff | NA | 4 (16.0) |
| Job tenure in years, mean (standard deviation) | 8.2 (6.7) | 7.9 (6.1) |
| Prior experience with patient reported outcomes, N (%) | ||
| Yes | 8 (27.6) | 10 (40.0) |
| No | 21 (72.4) | 15 (60.0) |
| Sex, N (%) | ||
| Female | 17 (58.6) | 16 (64.0) |
| Male | 12 (41.4) | 9 (36.0) |
| Race/ethnicity, N (%) | ||
| Non-Hispanic Black/African American | 3 (10.3) | 3 (12.0) |
| Hispanic/Latinx | 5 (17.2) | 4 (16.0) |
| Asian | 5 (17.2) | 5 (20.0) |
| Non-Hispanic White | 16 (55.2) | 13 (52.0) |
Abbreviation: NA, not applicable.
Interview participants ( N = 25) included nurses (20.0%), medical assistants (16.0%), PARs (12.0%), APPs (12.0%), physicians (8.0%), social workers (16.0%), and informatics staff (16.0%; Table 1 ). Almost half (40.0%) of interview participants had experience with implementing PROs in clinical care previously. The average job tenure of interview participants was 7.9 years (SD: 6.1).
Perceptions about Feasibility
Overall, participants rated the SRS assessment as having low feasibility (mean total score: 12.6; SD: 4.4 points; Table 2 ). The ratings varied across staff roles ( Table 3 ). Feasibility scores were lower among nurses (mean: 9.6; SD: 4.2) and medical assistants (mean: 10.4; SD: 5.0) compared with APPs (mean: 12.8; SD: 5.9), PARs (mean: 14.6, SD: 3.0), physicians (mean: 16.7; SD: 31), and social workers (mean: 17.5; SD: 0.7).
Table 2. Perceptions about feasibility, acceptability, and appropriateness of patient-reported symptom assessment, N = 29 .
| Item | Strongly disagree N (%) |
Disagree N (%) |
Neutral N (%) |
Agree N (%) |
Strongly agree N (%) |
Average score mean (SD) |
|---|---|---|---|---|---|---|
| Feasibility | ||||||
| PRSA seems implementable |
4 (13.8) | 6 (20.7) | 7 (24.1) | 10 (34.5) | 2 (6.9) | 3.1 (1.2) |
| PRSA seems possible |
4 (13.8) | 3 (10.3) | 6 (20.7) | 13 (44.8) | 3 (10.3) | 3.3 (1.2) |
| PRSA seems doable | 4 (13.8) | 3 (10.3) | 6 (20.7) | 13 (44.8) | 3 (10.3) | 3.3 (1.2) |
| PRSA seems easy to use |
4 (13.8) | 7 (24.1) | 8 (27.6) | 8 (27.6) | 2 (6.9) | 2.9 (1.2) |
| Total score | – | – | – | – | – | 12.6 (4.4) |
| Acceptability | ||||||
| PRSA meets my approval |
5 (17.2) | 11 (37.9) | 8 (27.6) | 2 (6.9) | 3 (10.3) | 2.6 (1.2) |
| PRSA is appealing to me |
7 (24.1) | 8 (27.6) | 7 (24.1) | 2 (6.9) | 5 (17.2) | 2.7 (1.4) |
| I like PRSA | 6 (20.7) | 7 (24.1) | 10 (34.5) | 2 (6.9) | 4 (13.8) | 2.8 (1.3) |
| I welcome PRSA | 5 (17.2) | 2 (6.9) | 8 (27.6) | 9 (31.0) | 5 (17.2) | 3.3 (1.3) |
| Total score | – | – | – | – | – | 11.5 (4.8) |
| Appropriateness | ||||||
| PRSA seems fitting | 4 (13.8) | 7 (24.1) | 8 (27.6) | 7 (24.1) | 3 (10.3) | 3.0 (1.3) |
| PRSA seems suitable |
4 (13.8) | 5 (17.2) | 9 (31.0) | 7 (24.1) | 4 (13.8) | 3.2 (1.3) |
| PRSA seems applicable |
5 (17.2) | 5 (17.2) | 9 (31.0) | 7 (24.1) | 3 (10.3) | 3.0 (1.3) |
| PRSA seems like a good match |
5 (17.2) | 8 (27.6) | 9 (31.0) | 4 (13.8) | 3 (10.3) | 2.8 (1.3) |
| Total score | – | – | – | – | – | 12.1 (4.8) |
Abbreviations: PRSA, patient-reported symptom assessment; SD, standard deviation.
Note: The Feasibility, Acceptability, and Appropriateness scores range from 0 to 20 with higher scores indicating better implementation.
Table 3. Perceptions about feasibility, acceptability, and appropriateness based on staff role.
| Characteristic | Feasibility score, mean (SD) |
Acceptability, mean (SD) |
Appropriateness, mean (SD) |
|---|---|---|---|
| Occupation | |||
| Nurse | 9.6 (4.2) | 8.2 (3.9) | 8.0 (4.0) |
| Medical assistant | 10.4 (5.0) | 7.8 (4.3) | 9.8 (4.2) |
| Patient access representative | 14.6 (3.0) | 12.0 (5.7) | 12.5 (5.3) |
| Advanced practice provider | 12.8 (5.9) | 12.0 (4.8) | 13.2 (5.3) |
| Physician | 16.7 (3.1) | 18.3 (1.5) | 16.7 (3.1) |
| Social worker | 17.5 (0.7) | 19.5 (0.7) | 20.0 (0.0) |
Abbreviation: SD, standard deviation.
During the interviews, participants discussed several factors that may affect feasibility, which varied across staff roles ( Supplementary Table S1 , available in the online version). Nurses and medical assistants, for example, described there was insufficient time before and during the visit to screen for suicide risk and review other symptoms (e.g., pain). Prior to the visit, medical assistants did not feel there was enough time to read through and complete the SRS, leading patients to rush when completing the questionnaire and not fully reading the questions. During the visit, nurses described having many priorities to address related to other initiatives (e.g., medicine reconciliation) and not having enough time to address suicide risk.
In addition to lack of time, medical assistants, nurses, and APPs described that there was insufficient staffing to reliably implement SRS. APPs described how nursing staff availability varies across providers and that if an APP does not have nursing staff support for a given patient, review of the patient-reported symptoms may get missed or delayed until the end of the visit. Medical assistants and nurses expressed concerns about social work staff availability. Participants noted that there was only one main social worker assigned for the HNC clinic. When the assigned social worker was out of the office, some team members felt it was challenging to find another social worker to assist with SRS in a timely manner.
Participants also discussed how SRS fits within their current workflow. Certain care team members, including PARs and social workers, felt that the initiative fit well within their workflow. The social work team discussed how they are well prepared for screening patients for suicide risk and are already engaged in this work. PARs also felt that SRS fit within their workflow because they already have a process for assisting patients with new questionnaires. Other staff, however, including nurses, APPs, and physicians described not knowing how to fit the review of patient-reported outcomes into their workflow. Some physicians, for example, described not knowing what to do with the information and ignoring it.
Perceptions about Acceptability
Overall, participants rated SRS as having low acceptability intervention (mean: 11.5, SD: 4.8) ( Table 2 ). Acceptability scores were lower among nurses (mean: 8.2; SD: 3.9) and medical assistants (mean: 7.8, SD: 4.3) and higher among physicians (mean: 18.3; SD: 1.5) and social workers (mean: 19.5; SD: 0.7; Table 3 ).
All health care team members noted several factors that affected staff and patient satisfaction with the SRS. Participants were satisfied, for example, with the symptom discussions that resulted from the screening. Staff members noted being able to identify unmet mental health care needs, facilitate advanced care planning and goals of care conversations, and develop a more holistic view of the patient. Participants also appreciated that the screening collects symptoms that can be difficult for patients to discuss, such as mental health concerns and certain physical symptoms (e.g., constipation, diarrhea). A few health care team members noted that screening improves data collection on quality of life, information that could be useful for researchers to identify gaps in care delivery.
Nurses, medical assistants, PARs, and clinical informatics staff noted some components of the screening process that they were dissatisfied with. Clinical informatics and nursing staff described challenges with EHR data integration. For example, if the Wi-Fi connection was lost in the HNC clinic, data could be lost if a patient had not completed their screening. Clinical informatics staff described instances where patient data were lost and the SRS had to be recompleted, creating frustration for staff and patients. Nurses were also concerned patient data that may not display in real time due to Wi-Fi connectivity issues, which could result in missing a patient who may be at-risk for suicide (e.g., in the instance that positive screening results do not display in EHR before the visit ends). Nurses also noted that the screening could be burdensome for patients who had frequent visits (e.g., more than one visit in a week), resulting in patient frustration. Medical assistants and PARs expressed concerns about only providing one modality to complete the screening (e.g., tablet completion). They noted that some patients had difficulty using the tablet (e.g., vision impairment, dexterity issues). Nurses also expressed fears about risk management. Nurses were concerned that they are primarily responsible for the intervention and felt that there should be another level of data review (e.g., by APP or physician) so that if the nurse misses a concerning score, another member of the care team will have the opportunity to catch it. Nurses felt that secondary review of the data by an APP or physician highly varied across providers.
Perceptions about Appropriateness
Participants rated suicide screening as having low appropriateness for the HNC patient population and the oncology clinic setting (mean: 12.1; SD: 4.8; Table 2 ). Appropriateness scores were lower among nurses (mean: 8.0; SD: 4.0) and medical assistants (mean: 9.8; SD: 4.2) and higher among physicians (mean: 16.7; SD: 3.1) and social workers (mean: 20.0; SD: 0.0; Table 3 ).
During the interviews, participants described factors that may improve the appropriateness of the PRSA, such as changing the suicidal ideation items, the target population, and the timeframe for symptom recall. For example, health care team members expressed divergent opinions about the appropriateness of the PHQ-9 as a measure of suicidal ideation among cancer patients. Some members of the clinical team indicated that the item was too broad and should be separated into two questions, one that measures passive wish for death and one item that measures intention for self-harm. Other team members felt that the PHQ-9 was an appropriate tool for identifying patients with complex mental health needs who needed further intervention. For example, one social worker mentioned that the PHQ-9 helps identify two groups—patients who may have suicidal ideation and patients who have other unmet mental health care needs (e.g., high distress). Both groups are likely to benefit from social work intervention. Participants also recommended adding disease-specific symptoms that are relevant to HNC (e.g., swallowing difficulty) and social determinants of health correlated with suicide risk (e.g., financial hardship).
Team members also noted that the PRSA may not be relevant for all patients served by the HNC clinic. For example, one medical assistant mentioned that the PRSA captures symptoms (e.g., diarrhea) that may be more relevant for patients who are undergoing active treatment rather than patients who are in the surveillance phase of their cancer care journey. A member of the clinical informatics staff described how the PRSA is currently targeted to all patients with an in-person visit in the HNC clinic and that it may be more valuable to target the form based on patient characteristics (e.g., type of cancer, phase of cancer care continuum). Participants were also concerned about the relevance of the recall period. For example, the PHQ-9 recall period is 2 weeks, which may be confusing for patients with multiple visits within a 2-week timeframe. Additionally, participants felt that the PRSA may lack relevance for patients who have not had a change in symptoms since the last PRSA completion.
Recommendations
During the interviews, participants provided several recommendations for improving implementation including (1) increasing staffing availability (e.g., nursing, social work) to support SRS, (2) improving SRS staff training (e.g., providing ongoing rather than one-time training, guidance on patient communication about suicide risk), (3) additional modalities for SRS completion to give patients more time and accommodate patients with disabilities (e.g., vision impairment), (4) educating patients and staff on the purpose of SRS and how it fits within broader institutional goals to improve buy-in, (5) reducing the number of times patients are required to complete the SRS (e.g., once a month versus every in-person visit), and (6) ensuring the screening captures PROs important for HNC patients (e.g., swallowing difficulty).
Discussion
In this study, we assessed oncology professionals' perceptions about the feasibility, accessibility, and appropriateness of an electronic SRS for individuals with HNC. Overall, perceptions about feasibility, acceptability, and appropriateness varied widely depending on the team member's role within implementation. Oncology professionals who were responsible for administering the SRS and reviewing the scores, including medical assistants and nurses, rated SRS as having low feasibility, acceptability, and appropriateness. Other members of the care team, including social workers, PARs, APPs, and physicians rated SRS as having higher feasibility, acceptability, and appropriateness. Participants recommended several strategies that could be tested in the future to support successful implementation of SRS, such as strengthening staff training, increasing accessibility for individuals with disabilities, and improving risk stratification.
Numerous studies have documented staff training as a barrier to PRO implementation in general and SRS implementation specifically. 46 47 48 49 Several training programs have been developed that focus on how to identify and monitor symptoms based on PRO data and how to use PRO data as a part of clinical care. 50 51 Recommended elements of training include having talking points that clinicians can use to facilitate communication about PROs, information about the evidence base for PROs (e.g., how, when used effectively, PRO collection can affect patient outcomes), and information about implementation (e.g., rationale for PRO selection, how to interpret, and document PRO scores). There is a need for additional research to rigorously evaluate clinician training programs to determine which models may be most effective at supporting integration of PROs into routine care in the context of SRS. One model that could be tested is implementation coaching, which has been used to support implementation of other evidence-based interventions (e.g., human papillomavirus vaccination). 52 53
In addition to clinician training, study participants recommended improving PRO accessibility for patients with disabilities through multimodal screening. Completing SRS on a tablet proved difficult for patients with visual impairment, cognitive impairment, or loss of dexterity. Previous research by Bennett et al comparing patient-reported data collection by tablet and voice response found that both modalities can be used to effectively collect PROs, an option that may be preferable for adults with vision impairment and older adults compared with tablet screening. 54 55 Further research is needed on optimizing access to PROs for individuals with cancer and disabilities that may affect PRO completion (e.g., vision impairment, cognitive impairment, dexterity-related conditions).
Participants in our study reported mixed perspectives about whether SRS is feasible to implement given current staffing and workload. Therefore, strategies should be tested that optimize the workflow of SRS and reduce burden on staff, particularly nurses. For example, researchers have tested the validity of using data already available within the EHR to predict which patients may be at risk for suicide in primary care and emergency department settings. 56 57 58 59 60 One study found that an EHR-data model (area under the curve [AUC] = 0.775) had similar performance at classifying suicide risk among patients receiving care in a pediatric emergency department ( N = 13,420) compared with a patient-reported SRS model (AUC = 0.754). 59 Future studies could test a similar approach in additional care settings including oncology. Other strategies may include improving risk stratification. Like prior studies, our study found that 8.8% of patients flagged for additional follow-up based on PHQ-9 scores were classified as having active suicidal ideation. 12 61 Research has recommended approaches such as combining the use of multiple PROs to improve classification of suicide risk. 62 Additional testing is needed to evaluate what data types and which PROs may be most effective at identifying suicide risk in HNC and other cancer patients.
Participants were divided about whether the included ePROs were appropriate for HNC patients. The PRSA was designed to capture symptoms that are common among all cancer patients (e.g., nausea) and could be scaled across oncology clinics. Participants focused on HNC care delivery (e.g., nurses) rated the tool as lower in appropriateness than staff who work across clinics (e.g., social work), which may account for these differences. The tool did not capture common symptoms for HNC patients (e.g., swallowing difficulty). Further, some staff had concerns that SRS may be less relevant for individuals who are no longer undergoing active treatment and that assessing SRS and other ePROs at every clinic visit was burdensome for patients and staff. Additional research is needed to come to a consensus on what are best measures for SRS, when individuals with cancer should be screened, and how often.
Limitations
Our study has a few limitations. First, our study was conducted at an NCI-designated Comprehensive Cancer Center, and therefore, the findings may not be generalizable to other cancer care settings. Second, our study captured the perspectives of health care professionals about SRS implementation and does not capture patients' perspectives. Future studies are needed to capture the patient perspective. Additionally, while our survey (65.9%) and interview (51.0%) response rates were moderately high; there are likely important perspectives that were missed. This highlights the need for additional studies to assess implementation of SRS in oncology settings. Further, SRS was implemented alongside other ePROs (e.g., distress), and therefore, our study identified implementation issues regarding ePRO implementation broadly and SRS implementation specifically. We also did not measure cost or cost-effectiveness as a part of this pilot study. Future studies should assess the cost of SRS implementation. Finally, our interviews were conducted early-on during implementation so that information from the surveys and information could be used to refine the intervention. Therefore, perspectives regarding implementation may have changed over time, as the intervention was refined, and staff grew more accustomed to SRS. We are unable to evaluate this currently because SRS was removed from the PRSA tool due to the implementation barriers documented in this study. Our team is currently working on refining our approach through usability testing of the tool, development of additional training tools, and selection of additional instruments to assess suicide risk prior to a second pilot.
Conclusion
Overall, cancer care team members had mixed perspectives about the feasibility of implementing SRS as a part of routine care and identified important areas for future research to improve future SRS implementation. Before SRS screening can be implemented on a larger scale for HNC and other cancer patients, additional implementation strategies may be needed, such as staff training, multiple modalities for completion, and refined tools for identifying which patients are at greatest risk for suicide.
Clinical Relevance Statement
Individuals with cancer are at increased risk for suicide. ePRO monitoring programs can identify patients at-risk but are challenging to implement due to limited staffing, insufficient training, and limited information technology capacity among health care systems. Strategies are needed that improve implementation of ePRO systems to reduce burden among already taxed clinic staff.
Multiple Choice Questions
-
Why are patient-reported outcome (PRO) monitoring programs challenging to implement in clinical practice?
Patient-level barriers (e.g., patient engagement)
Clinician-level barriers (e.g., clinician self-efficacy)
System-level barriers (e.g., information technology [IT] capacity)
All of the above
Correct Answer: The correct answer is option d. PRO monitoring programs can be challenging to implement due to patient-level barriers, such as patient engagement with PROs, clinician-level barriers, such as clinician self-efficacy for using PROs in clinical practice, and system-level barriers, such as limited IT capacity to support implementation.
-
How can PRO systems be modified to improve accessibility among individuals with disabilities?
Offering multiple modalities for completion
Increasing the font size of PRO assessments
Providing technical assistance with PRO assessments
All of the above
Correct Answer: The correct answer is option d. Research has recommended several strategies for improving PRO accessibility among individuals with disabilities including offering multiple modalities for completion (e.g., paper, electronic), increasing the font size of assessments, and providing technical assistance with PRO assessments for individuals who may have difficulty using electronic PRO systems.
Funding Statement
Funding This research was supported in part by the Participant Research, Interventions, and Measurements Core at the Moffitt Cancer Center, a National Cancer Institute-designated Comprehensive Cancer Center (P30-CA076292).
Conflict of Interest D.E.R. is on the Board of Directors for NanoString Technologies, Inc.
Availability of the Data
To protect the privacy of the individuals that participated in this study, the individual-level data underlying this article cannot be shared. Summary-level data may be requested.
Author Contributions
B.K.: Conceptualization; Writing—original draft; Writing—review & editing; Methodology; A.B.: Writing—review & editing; M.M.: Writing—review & editing; R.P.: Writing—review & editing; P.R.: Writing—review & editing; J.H.J.: Writing—review & editing; D.E.R.: Writing—review & editing; O.T.N.: Writing—review & editing; S.P.: Writing—review & editing; S.M.G.: Writing—review & editing; K.T.: Conceptualization; Methodology; Project administration.
Protection of Human and Animal Subjects
Moffitt Cancer Center Institutional Review Board of Record, Advarra, reviewed the study protocol and determined the study to be exempt.
Supplementary Material
Supplementary Material
Supplementary Material
References
- 1.Fang F, Fall K, Mittleman M A et al. Suicide and cardiovascular death after a cancer diagnosis. N Engl J Med. 2012;366(14):1310–1318. doi: 10.1056/NEJMoa1110307. [DOI] [PubMed] [Google Scholar]
- 2.Henson K E, Brock R, Charnock J, Wickramasinghe B, Will O, Pitman A. Risk of suicide after cancer diagnosis in England. JAMA Psychiatry. 2019;76(01):51–60. doi: 10.1001/jamapsychiatry.2018.3181. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Misono S, Weiss N S, Fann J R, Redman M, Yueh B. Incidence of suicide in persons with cancer. J Clin Oncol. 2008;26(29):4731–4738. doi: 10.1200/JCO.2007.13.8941. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ravaioli A, Crocetti E, Mancini S et al. Suicide death among cancer patients: new data from northern Italy, systematic review of the last 22 years and meta-analysis. Eur J Cancer. 2020;125:104–113. doi: 10.1016/j.ejca.2019.08.019. [DOI] [PubMed] [Google Scholar]
- 5.Zaorsky N G, Zhang Y, Tuanquin L, Bluethmann S M, Park H S, Chinchilli V M. Suicide among cancer patients. Nat Commun. 2019;10(01):207. doi: 10.1038/s41467-018-08170-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Osazuwa-Peters N, Simpson M C, Zhao L et al. Suicide risk among cancer survivors: head and neck versus other cancers. Cancer. 2018;124(20):4072–4079. doi: 10.1002/cncr.31675. [DOI] [PubMed] [Google Scholar]
- 7.Kam D, Salib A, Gorgy G et al. Incidence of suicide in patients with head and neck cancer. JAMA Otolaryngol Head Neck Surg. 2015;141(12):1075–1081. doi: 10.1001/jamaoto.2015.2480. [DOI] [PubMed] [Google Scholar]
- 8.Sung H, Ferlay J, Siegel R L et al. Global Cancer Statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021;71(03):209–249. doi: 10.3322/caac.21660. [DOI] [PubMed] [Google Scholar]
- 9.Echeverria I, Cotaina M, Jovani A, Mora R, Haro G, Benito A. Proposal for the inclusion of tobacco use in suicide risk scales: results of a meta-analysis. Int J Environ Res Public Health. 2021;18(11):6103. doi: 10.3390/ijerph18116103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Isaacs J Y, Smith M M, Sherry S B, Seno M, Moore M L, Stewart S H. Alcohol use and death by suicide: a meta-analysis of 33 studies. Suicide Life Threat Behav. 2022;52(04):600–614. doi: 10.1111/sltb.12846. [DOI] [PubMed] [Google Scholar]
- 11.Chua K S, Reddy S K, Lee M C, Patt R B. Pain and loss of function in head and neck cancer survivors. J Pain Symptom Manage. 1999;18(03):193–202. doi: 10.1016/s0885-3924(99)00070-6. [DOI] [PubMed] [Google Scholar]
- 12.Chang D C, Chen A W, Lo Y S, Chuang Y C, Chen M K. Factors associated with suicidal ideation risk in head and neck cancer: a longitudinal study. Laryngoscope. 2019;129(11):2491–2495. doi: 10.1002/lary.27843. [DOI] [PubMed] [Google Scholar]
- 13.Massa S T, Osazuwa-Peters N, Adjei Boakye E, Walker R J, Ward G M. Comparison of the financial burden of survivors of head and neck cancer with other cancer survivors. JAMA Otolaryngol Head Neck Surg. 2019;145(03):239–249. doi: 10.1001/jamaoto.2018.3982. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Gardner W, Klima J, Chisolm D et al. Screening, triage, and referral of patients who report suicidal thought during a primary care visit. Pediatrics. 2010;125(05):945–952. doi: 10.1542/peds.2009-1964. [DOI] [PubMed] [Google Scholar]
- 15.Etter D J, McCord A, Ouyang F et al. Suicide screening in primary care: use of an electronic screener to assess suicidality and improve provider follow-up for adolescents. J Adolesc Health. 2018;62(02):191–197. doi: 10.1016/j.jadohealth.2017.08.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Davis M, Rio V, Farley A M, Bush M L, Beidas R S, Young J F. Identifying adolescent suicide risk via depression screening in pediatric primary care: an electronic health record review. Psychiatr Serv. 2021;72(02):163–168. doi: 10.1176/appi.ps.202000207. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Fein J A, Pailler M E, Barg F K et al. Feasibility and effects of a web-based adolescent psychiatric assessment administered by clinical staff in the pediatric emergency department. Arch Pediatr Adolesc Med. 2010;164(12):1112–1117. doi: 10.1001/archpediatrics.2010.213. [DOI] [PubMed] [Google Scholar]
- 18.Wintersteen M B. Standardized screening for suicidal adolescents in primary care. Pediatrics. 2010;125(05):938–944. doi: 10.1542/peds.2009-2458. [DOI] [PubMed] [Google Scholar]
- 19.Gascon B, Leung Y, Espin-Garcia O, Rodin G, Chu D, Li M. Suicide risk screening and suicide prevention in patients with cancer. JNCI Cancer Spectr. 2021;5(04):pkab057. doi: 10.1093/jncics/pkab057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Stover A M, Kurtman R, Walker Bissram J et al. Stakeholder perceptions of key aspects of high quality cancer care to assess patient-reported outcome measures: a systematic review. Cancers. 2021;13(14):3628. doi: 10.3390/cancers13143628. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Stover A M, Tompkins Stricker C, Hammelef Ket al. Using stakeholder engagement to overcome barriers to implementing patient-reported outcomes (PROs) in cancer care delivery: approaches from 3 prospective studies Med Care 201957(1, Suppl 5 Suppl 1):S92–S99. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Basch E, Deal A M, Dueck A C et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA. 2017;318(02):197–198. doi: 10.1001/jama.2017.7156. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Basch E, Deal A M, Kris M G et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol. 2016;34(06):557–565. doi: 10.1200/JCO.2015.63.0830. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Denis F, Basch E, Septans A L et al. Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer. JAMA. 2019;321(03):306–307. doi: 10.1001/jama.2018.18085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Donovan K A, Jacobsen P B. Progress in the implementation of NCCN guidelines for distress management by member institutions. J Natl Compr Canc Netw. 2013;11(02):223–226. doi: 10.6004/jnccn.2013.0029. [DOI] [PubMed] [Google Scholar]
- 26.Donovan K A, Deshields T L, Corbett C, Riba M B. Update on the implementation of NCCN guidelines for distress management by NCCN member institutions. J Natl Compr Canc Netw. 2019;17(10):1251–1256. doi: 10.6004/jnccn.2019.7358. [DOI] [PubMed] [Google Scholar]
- 27.Jacobsen P B, Ransom S. Implementation of NCCN distress management guidelines by member institutions. J Natl Compr Canc Netw. 2007;5(01):99–103. doi: 10.6004/jnccn.2007.0010. [DOI] [PubMed] [Google Scholar]
- 28.Jacobsen P B, Norton W E. The role of implementation science in improving distress assessment and management in oncology: a commentary on “Screening for psychosocial distress among patients with cancer: implications for clinical practice, healthcare policy, and dissemination to enhance cancer survivorship”. Transl Behav Med. 2019;9(02):292–295. doi: 10.1093/tbm/ibz022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Palinkas L A, Aarons G A, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38(01):44–53. doi: 10.1007/s10488-010-0314-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Kim H, Sefcik J S, Bradway C. Characteristics of qualitative descriptive studies: a systematic review. Res Nurs Health. 2017;40(01):23–42. doi: 10.1002/nur.21768. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Johnstone P AS, Lee J, Zhou J M et al. A modified Edmonton Symptom Assessment Scale for symptom clusters in radiation oncology patients. Cancer Med. 2017;6(09):2034–2041. doi: 10.1002/cam4.1125. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Johnstone P AS, Bulls H W, Zhou J M et al. Congruence of multiple patient-related outcomes within a single day. Support Care Cancer. 2019;27(03):867–872. doi: 10.1007/s00520-018-4372-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Bulls H W, Chang P H, Brownstein N C et al. Patient-reported symptom burden in routine oncology care: examining racial and ethnic disparities. Cancer Rep (Hoboken) 2022;5(03):e1478. doi: 10.1002/cnr2.1478. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Turner K, Brownstein N C, Thompson Z et al. Longitudinal patient-reported outcomes and survival among early-stage non-small cell lung cancer patients receiving stereotactic body radiotherapy. Radiother Oncol. 2022;167:116–121. doi: 10.1016/j.radonc.2021.12.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Noel C W, Forner D, Chepeha D B et al. The Edmonton Symptom Assessment system: a narrative review of a standardized symptom assessment tool in head and neck oncology. Oral Oncol. 2021;123:105595. doi: 10.1016/j.oraloncology.2021.105595. [DOI] [PubMed] [Google Scholar]
- 36.Nekolaichuk C, Watanabe S, Beaumont C. The Edmonton Symptom Assessment system: a 15-year retrospective review of validation studies (1991–2006) Palliat Med. 2008;22(02):111–122. doi: 10.1177/0269216307087659. [DOI] [PubMed] [Google Scholar]
- 37.Hoffman B M, Zevon M A, D'Arrigo M C, Cecchini T B. Screening for distress in cancer patients: the NCCN rapid-screening measure. Psychooncology. 2004;13(11):792–799. doi: 10.1002/pon.796. [DOI] [PubMed] [Google Scholar]
- 38.Weiner B J, Lewis C C, Stanick C et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(01):108. doi: 10.1186/s13012-017-0635-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Proctor E, Silmere H, Raghavan R et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(02):65–76. doi: 10.1007/s10488-010-0319-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Wood K C, Pergolotti M, Marshall T et al. Usability, acceptability, and implementation strategies for the Exercise in Cancer Evaluation and Decision Support (EXCEEDS) algorithm: a Delphi study. Support Care Cancer. 2022;30(09):7407–7418. doi: 10.1007/s00520-022-07164-6. [DOI] [PubMed] [Google Scholar]
- 41.Damschroder L J, Aron D C, Keith R E, Kirsh S R, Alexander J A, Lowery J C. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. Int J Qual Methods. 2006;5(01):80–92. [Google Scholar]
- 43.Hamilton A B, Finley E P. Qualitative methods in implementation research: an introduction. Psychiatry Res. 2019;280:112516. doi: 10.1016/j.psychres.2019.112516. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Guest G, Namey E, Chen M. A simple method to assess and report thematic saturation in qualitative research. PLoS One. 2020;15(05):e0232076. doi: 10.1371/journal.pone.0232076. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(06):349–357. doi: 10.1093/intqhc/mzm042. [DOI] [PubMed] [Google Scholar]
- 46.Nguyen H, Butow P, Dhillon H, Sundaresan P. A review of the barriers to using patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs) in routine cancer care. J Med Radiat Sci. 2021;68(02):186–195. doi: 10.1002/jmrs.421. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Foster A, Croot L, Brazier J, Harris J, O'Cathain A. The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient Rep Outcomes. 2018;2:46. doi: 10.1186/s41687-018-0072-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Glenwright B G, Simmich J, Cottrell M et al. Facilitators and barriers to implementing electronic patient-reported outcome and experience measures in a health care setting: a systematic review. J Patient Rep Outcomes. 2023;7(01):13. doi: 10.1186/s41687-023-00554-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.EUROIMPACT . Antunes B, Harding R, Higginson I J. Implementing patient-reported outcome measures in palliative care clinical practice: a systematic review of facilitators and barriers. Palliat Med. 2014;28(02):158–175. doi: 10.1177/0269216313491619. [DOI] [PubMed] [Google Scholar]
- 50.Santana M J, Haverman L, Absolom K et al. Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Qual Life Res. 2015;24(07):1707–1718. doi: 10.1007/s11136-014-0903-5. [DOI] [PubMed] [Google Scholar]
- 51.Skovlund P C, Ravn S, Seibaek L, Thaysen H V, Lomborg K, Nielsen B K. The development of PROmunication: a training-tool for clinicians using patient-reported outcomes to promote patient-centred communication in clinical cancer settings. J Patient Rep Outcomes. 2020;4(01):10. doi: 10.1186/s41687-020-0174-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Ballengee L A, Rushton S, Lewinski A A et al. Effectiveness of quality improvement coaching on process outcomes in health care settings: a systematic review. J Gen Intern Med. 2022;37(04):885–899. doi: 10.1007/s11606-021-07217-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Gilkey M B, Heisler-MacKinnon J, Boynton M H, Calo W A, Moss J L, Brewer N T. Impact of brief quality improvement coaching on adolescent HPV vaccination coverage: a pragmatic cluster randomized trial. Cancer Epidemiol Biomarkers Prev. 2023;32(07):957–962. doi: 10.1158/1055-9965.EPI-22-0866. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.National Cancer Institute PRO-CTCAE Study Group . Bennett A V, Dueck A C, Mitchell S A et al. Mode equivalence and acceptability of tablet computer-, interactive voice response system-, and paper-based administration of the U.S. National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) Health Qual Life Outcomes. 2016;14:24. doi: 10.1186/s12955-016-0426-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Owen-Smith A, Mayhew M, Leo M C et al. Automating collection of pain-related patient-reported outcomes to enhance clinical care and research. J Gen Intern Med. 2018;33 01:31–37. doi: 10.1007/s11606-018-4326-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Barak-Corren Y, Castro V M, Nock M K et al. Validation of an electronic health record-based suicide risk prediction modeling approach across multiple health care systems. JAMA Netw Open. 2020;3(03):e201262. doi: 10.1001/jamanetworkopen.2020.1262. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Walsh C G, Johnson K B, Ripperger M et al. Prospective validation of an electronic health record-based, real-time suicide risk model. JAMA Netw Open. 2021;4(03):e211428. doi: 10.1001/jamanetworkopen.2021.1428. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Nock M K, Millner A J, Ross E L et al. Prediction of suicide attempts using clinician assessment, patient self-report, and electronic health records. JAMA Netw Open. 2022;5(01):e2144373. doi: 10.1001/jamanetworkopen.2021.44373. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Haroz E E, Kitchen C, Nestadt P S, Wilcox H C, DeVylder J E, Kharrazi H. Comparing the predictive value of screening to the use of electronic health record data for detecting future suicidal thoughts and behavior in an urban pediatric emergency department: a preliminary analysis. Suicide Life Threat Behav. 2021;51(06):1189–1202. doi: 10.1111/sltb.12800. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Walker R L, Shortreed S M, Ziebell R A et al. Evaluation of electronic health record-based suicide risk prediction models on contemporary data. Appl Clin Inform. 2021;12(04):778–787. doi: 10.1055/s-0041-1733908. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Henry M, Rosberger Z, Bertrand L et al. Prevalence and risk factors of suicidal ideation among patients with head and neck cancer: longitudinal study. Otolaryngol Head Neck Surg. 2018;159(05):843–852. doi: 10.1177/0194599818776873. [DOI] [PubMed] [Google Scholar]
- 62.Bryan C J, Allen M H, Thomsen C J et al. Improving suicide risk screening to identify the highest risk patients: results from the PRImary Care Screening Methods (PRISM) study. Ann Fam Med. 2021;19(06):492–498. doi: 10.1370/afm.2729. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplementary Material
Supplementary Material