Abstract
Research-based queries about patients’ experiences often uncover suicidal thoughts. Human subjects review requires suicide risk management (SRM) protocols to protect patients, yet minimal information exists to guide researchers’ protocol development and implementation efforts. The purpose of this study was to examine the development and implementation of an SRM protocol employed during telephone-based screening and data collection interviews of depressed primary care patients. We describe an SRM protocol development process and employ qualitative analysis of de-identified documentation to characterize protocol-driven interactions between research clinicians and patients. Protocol development required advance planning, training, and team building. Three percent of screened patients evidenced suicidal ideation; 12% of these met protocol standards for study clinician assessment/intervention. Risk reduction activities required teamwork and extensive collaboration. Research-based SRM protocols can facilitate patient safety by (1) identifying and verifying local clinical site approaches and resources and (2) integrating these features into prevention protocols and training for research teams.
Keywords: Depression, Patient safety, Suicide, Primary care, Research participants, Collaborative care
Suicidal ideation (SI) and behavior are significant for healthcare management across clinical and research settings. Within research studies, ethical practice mandates suicide risk management (SRM) protocols to guide researchers’ efforts to maximize patient safety [1, 2]. Key stakeholders in the adequacy of these protocols include patients who participate in research, the researchers who study patients at risk for suicide, and the Institutional Review Boards (IRB) that oversee research activities. Several important management tasks are relevant for research-based SRM. Detection tasks include identifying those research participants who evidence high risk and require SRM protocol entry [2]. Essential management tasks include the development and implementation of specific plans and identification and training of specific personnel to perform risk management activities [1, 2].
Guidance regarding the detection of suicidal threat can be found in the suicide literature, which characterizes risk across numerous domains [3]. Socio-demographic risks for suicide include age, gender, and ethnicity, with white men and older adults accounting for disproportionately high rates of completed suicides [4]. Distal suicide risk factors such as mental and medical illnesses represent relatively stable backdrops against which SI or suicidal behavior occurs [5]. For example, existing research indicates that suicide risk is elevated among persons with mental illness generally [6] and particularly among those with mood disorders [7], substance abuse/dependence problems [8], anxiety and posttraumatic stress disorder [9–11], self-reported physical illness [7, 12], and chronic pain [13]. Psychosocial issues can be conceptualized as more proximal suicide risk factors, which exist on their own or in concert with distal risks. More proximal or transient factors in this domain include social isolation [14, 15], interpersonal relationship disruption [7], acute and severe life stressors [16], and unemployment [17]. Finally, in particularly high-risk situations, proximal and acute warning signs suggest the possibility of imminent suicidal behavior [5]. These warning signs include a patient’s expressed intent to die, preparatory behavior and planning such as gaining access to lethal means, and implementation of self-harm action plans [5].
Clinical practice guidelines for suicide and representative assessment/prevention recommendations [18, 19] comprise findings from the suicide risk literature. This work, which is generally designed for clinical use in traditional face-to-face treatment settings, recommends thorough psychiatric and diagnostic evaluation of patients, with specific focus on imminent suicidal risk (e.g., presence and lethality of plans, means accessibility, etc.). Despite the usefulness of these approaches, they do not translate seamlessly for use in research settings, which often incur goals and circumstances that differ from what occurs in clinical settings.
Researchers who attempt to translate clinical suicide risk research and existing management recommendations into research settings will face several potential challenges. First, researchers will likely uncover SI among patients who are not actively seeking treatment. In addition, non-clinician data collection personnel often conduct research interviews (e.g., [20–22]), and these persons might have minimal experience assessing risk. This limited experience might couple with time restrictions of research encounters to preclude full diagnostic determination or comprehensive psychosocial evaluation. Additional challenges arise in telephonic research encounters where distance separates researchers and patients. For example, distant researchers could lack the local clinical partnerships for care collaboration that might exist naturally in situations where patients and research clinicians are co-located. Telephonic encounters also restrict the available data upon which clinicians base their assessments of patient functioning. Because telephonic communication is restricted to auditory data, for example, visual observation of patients’ affect is not possible. Finally, consistent with recent standards for crisis-oriented telephonic interactions [23], telephonic research-based SRM requires rapid establishment of rapport in situations where the assessor has minimal control over patients’ behavior, and research personnel must often contend with minimal information regarding the patient’s history.
In summary, SRM protocols are necessary to facilitate the safety of suicidal research participants. Whereas existing literature provides a starting place for SRM protocol development, it fails to suggest parsimonious, algorithmic approaches for practical application to research interviews. Few studies have examined SRM protocols specifically (see [2]), and none appears to have documented the application of protocols in practice. This situation leaves a gap in the literature which would inform researchers’ efforts toward SRM protocol development and implementation. To make matters worse, additional challenges exist for distant researchers who initiate contact over the telephone. The present paper addresses the gap in the literature by characterizing the development, methods, and application of an SRM protocol designed for use over the telephone with depressed primary care patients, a frequently studied [24–27] group at risk for SI [28].
For our first aim, we describe the development and implementation of a comprehensive SRM protocol from a multisite depression study that recruited participants and collected data over the telephone. Our second aim uses directed content qualitative analysis to describe the suicide assessment/prevention activities that occurred during SRM protocol implementation. Data for our second aim were obtained from de-identified documentation of SRM-based patient–clinician encounters. Our results provide a framework for research-based management of primary care patients with SI and add to the literature by reporting on actual functioning and outcomes of telephone-based suicide assessment and prevention.
AIM 1: TO DESCRIBE SRM PROTOCOL DEVELOPMENT AND IMPLEMENTATION
Developing basic suicide prevention algorithms
To facilitate our first aim, we reviewed historical research study documents detailing development and implementation of the SRM protocol. Our final SRM protocol required the creation of algorithms that translated research on suicide prevention into a stepped care approach. The protocol needed to guide frontline non-clinician interviewers on when to contact supervisors and to guide supervisors on when to contact on-call study clinicians. It also needed to integrate local resources and safety management procedures for study clinicians to invoke when needed. We sought to avoid overuse of local research site clinical resources as multiple requests for assistance with low-risk patients would overburden sites and potentially reduce site responsiveness to true emergencies. Given this and the fact that relatively small proportions of primary care patients with SI have active plans for harm [29], our protocol identified successive levels of risk, culminating in initiation of full threatened suicide protections. In developing our algorithms for intervention, we considered a range of risk, contextual and buffering factors such as those found in the suicide risk and warning sign literature discussed above.
SRM protocol development was guided by a risk management process [30] already in place for the clinical intervention arm of the parent study. Briefly, it involved obtaining documentation regarding local risk management policies, testing local site procedures to ensure operability in an emergent situation, and defining a chain of responsibility, with detailed procedures for documentation and communication. Research staff drew on the knowledge gained and relationships formed with local personnel during the original SRM development process to guide SRM protocol development for the parent study. Study investigators’ developmental efforts resulted in our multistage SRM protocol, which is presented in Fig. 1. Brief descriptions of the protocol’s three phases follow (additional information available on request).
Fig 1.
The three-phase suicide risk management protocol
Phase 1: planning and development
The first step in the development process assigned responsibilities for all suicide assessment and intervention activities. Study investigators developed a brief structured risk assessment interview (see phase 2 below) for use with suicidal participants and trained layperson data collection interviewers in its use. These interviewers were assigned responsibility for detection and initial assessment of suicidal acuity based on study questionnaire responses and responses to the brief structured interview. Finally, in phase 3, on-call clinicians were tasked with immediate assessment and management of participants who were deemed at high risk by the data collection interviewers.
Procedural clarification and team building represented essential early SRM protocol developmental steps. These entailed clarification of site-level procedures and identification of personnel for local care coordination and/or urgent intervention. Procedural clarification was especially complex because of site-level variability in the degree to which specific and formal risk management procedures already existed. Additional complexity stemmed from the geographic distances that separated the research team, the research sites, and the data collection personnel. For example, because research personnel initially lacked first-hand experience with each site’s resources, these personnel were required to ascertain local suicide hotline phone numbers and local contact procedures for emergency services.
Team building resulted in a list of contact information for clinical personnel affiliated with the study at each implementation site. These local personnel were identified to ensure continuity of care for participants who required ongoing follow-up and aftercare for those requiring emergency intervention. Site-level investigators were informed that on-call clinicians would contact them regarding at-risk study participants. The researchers integrated site-specific contact and procedural information into the SRM protocol; this information was then tested through case simulation and emergency contact access [30]. Research staff compiled an On-Call Clinician Resource Guide containing information about resources and contact information at each site. This guide was disseminated to all on-call personnel in the form of wallet cards.
A contracted research firm conducted all data collection activities. The data collection team included two levels of personnel: frontline interviewers, and more senior and experienced data collection supervisors. Training the firm’s layperson interviewers and supervisors in suicide risk assessment and study-specific response procedures preceded field entry. All personnel met minimal qualifications that required formal education beyond high school and at least 1 year of data collection interviewing experience or 6 months of equivalent training. In addition, prior to performing any interviews, all data collection personnel completed an extensive 3-day workshop, during which all aspects of the parent study’s data collection were addressed. The workshop was conducted by the study’s principal investigators and two other senior project staff; three of the workshop presenters were members of the study’s on-call clinician team.
A significant portion of the training workshop was dedicated to suicide risk assessment and the study’s safety management plan. The suicide-specific component included basic information about suicide, detailed instructions about how and when to invoke suicide risk assessment, and role playing activities where the trainees simulated interviews with suicidal research respondents. These role-playing activities were observed directly by a member of the study team, and these observers provided direct feedback about the interviewers’ performances and adherence to the safety management plan. After the workshop, interviewers spent an additional week of training in which they conducted simulated interviews with mock respondents under direct supervision. These mock interviews included respondents who simulated suicidal ideation and, thus, invoked the threatened suicide procedures. Although we employed extensive training of the data collection interviewers and provided direct feedback on their training performances, we did not employ formal or quantitative assessment of their competence. Finally, prior to implementation, the final SRM protocol was reviewed and approved by IRBs at each participating site.
Phase 2: structured risk assessment
The second implementation phase was embedded in the parent study’s data collection survey and focused on structured suicide risk assessment (see Fig. 1). The SRM protocol required targeted suicide risk assessment whenever survey respondents expressed SI spontaneously or endorsed SI on the Patient Health Questionnaire—9 (PHQ-9) [31], a self-report depression measure. The interviewing software automatically initiated the structured suicide risk assessment following PHQ-9 endorsement of SI; interviewers invoked the assessment manually following spontaneous disclosure. A data collection firm supervisor joined and monitored the data collection conversation whenever structured risk assessment was invoked.
Following recommendations regarding acute warning signs and lethality [5, 23], the structured risk assessment included a question about serious consideration of suicide and suicide planning. Additional questions assessed whether SI had been shared with a local care provider and whether the respondent had disclosed any information to the interviewer that was unknown to his/her local providers. Whereas respondents’ answers to these questions facilitated determination of the necessity of immediate communication with an on-call clinician, the complexity of risk determination and our overriding concern for participant safety precluded use of a standard formula. If participants’ responses to all questions were suggestive of no or very low risk, interviewers and supervisors did not invoke the next step of assessment. Answers suggesting serious consideration of suicide, those provided by particularly upset or agitated participants, or any lingering concerns about participant safety on behalf of lay interviewers did invoke on-call clinician assessment. Phase 2 assessments were excluded from the standard parent study database to ensure confidentiality. Interviewers recorded respondents’ phase 2 responses off-line on hard copy standardized forms, and a single variable in the parent study database indicated whether phase 2 assessment was triggered.
Phase 3: on-call clinician contact
The monitoring supervisor invoked the SRM protocol’s phase 3 when the suicide assessment indicated serious SI, plans/intent for harm, or whenever the interviewer experienced concern for the respondents’ well-being. Phase 3 commenced when the data collection firm supervisor paged or telephoned the on-call coverage team. Following a brief consultation to share situation details, the on-call clinician telephoned the respondent directly with the aim of identifying and implementing actions necessary to maintain safety. The on-call coverage team included six doctoral-level clinicians with previous experience working with depressed Veterans Affairs (VA) primary care patients. Three physician clinicians were board-certified in internal medicine and all had over a decade of clinical experience. One physician held additional certification in psychiatry and the other in gerontology. Three clinicians were trained in clinical psychology; the senior psychologist had 25 years of clinical experience and provided direct supervision to the two early career clinicians.
To ensure clinician availability, two team members (one primary and one backup) were on-call during all data collection activities. On-call assignments rotated among the clinical team to spread responsibility. Phase 3 assessment and intervention activities were unscripted as clinicians followed clinical standards of care and drew procedural guidance from the On Call Clinician Resource Guide. As noted above, the guide explained how to invoke local sites’ suicide protocols, reach local healthcare personnel, and activate local emergency services. Over the entire course of the research project, study administrative staff, data collection supervisors, and members of the on-call coverage team held weekly telephone conferences. These conferences included staffing of general issues related to data collection, consultation about crisis encounters, and confirmation of ongoing on-call issues and staffing. These telephonic conferences facilitated continuity and deepened the ongoing relationships between the study team and the data collection personnel.
AIM 2: TO DESCRIBE SRM ASSESSMENT AND INTERVENTION ACTIVITIES
Method
The parent study
The present project’s parent study—The Well-being Among Veterans Enhancement Study WAVES—studied the effectiveness of a quality improvement program for collaborative care. WAVES participants included depressed patients with scheduled visits to one of ten geographically dispersed VA primary care clinics. All study protocols were approved by IRBs overseeing the participating clinics and those housing the study investigators.
A contracted telephone survey research firm conducted WAVES surveys. Prior to interviewer contact, patients received mailed study information, which included “opt out” instructions and descriptions of the limits of confidentiality. To generate the parent study sample, interviewers contacted patients and secured oral informed consent before screening for depression with the PHQ-9. Interviewers’ consent scripts explained that disclosure of information regarding harm would invoke further assessment and possible breach of confidentiality. Interviewers proceeded with a full baseline interview of consenting patients who evidenced apparent major depressive symptomatology. Patients were resurveyed via telephone at 7 months.
Measures and procedures
Measures
The present analyses employed brief measures of symptomatology and health status, which were a subset of the data collected for the parent study. The PHQ-9 [31] assessed depressive symptom severity via indication of the 2-week frequency (0 = “not at all”–3 = “every day/nearly every day”) of nine depression symptoms. The final PHQ-9 item assessed the presence of SI (“Thoughts that you would be better off dead or of hurting yourself in some way”); any response above “0” reflected SI. The Primary Care PTSD Screen [32] assessed the presence/absence of four cardinal PTSD symptoms and the presence of probable PTSD. The Alcohol Use Disorders Identification Test Consumption (AUDIT-C) questions assessed alcohol use and misuse [33]. Finally, participants described their general health with one of five response options ranging from “excellent” to “poor” [34].
Procedures
To examine our second aim, we analyzed de-identified qualitative data provided by the six doctoral-level on-call coverage clinicians who documented suicide assessment and prevention activities with high-risk patients. The SRM protocol required written documentation of clinician assessment and intervention activities in the form of “on-call notes.” No information regarding the on-call encounters was included in the WAVES database, and no demographic or other study data were linked to on-call documentation to maximize participant privacy. Instead, the database included a single variable indicating whether or not clinician contact occurred. Clinicians documented the encounters “off-line” on hard copy standardized forms. On-call notes were de-identified by an on-call coverage team member prior to qualitative analysis. De-identification followed HIPAA guidelines and included the deletion of all information regarding participant identity, collateral identity, clinical care providers, location, treatment dates, etc.
Qualitative analysis strategy
The fully de-identified on-call note information was entered into ATLAS.ti (v. 5.6.3) qualitative analysis software and analyzed by two coders (DGC and LMB) using a directed content approach [35]. Unlike conventional content analysis, in which codes are derived from the data, previous research findings guide the development of initial codes in directed content analysis. Using this method, we identified an initial set of codes early in the analysis process; these codes guided subsequent coding and code derivation. Codes reflecting salient aspects of the assessment and prevention process were attached to blocks of text. Initial coding revealed that some codes were attached to multiple text passages across notes, whereas others were attached to very few. Codes attached to multiple text passages reflected particularly important activities and were analyzed further via subcoding. Organizing meta-themes were derived from top-level and subcoding analytical procedures. Coders compared initial independent results and resolved discrepancies by consensus. Subcoding was performed independently.
RESULTS
Suicidal threat frequency and demographics
Data collection personnel initiated depression and study inclusion screening of 10,929 participants using the PHQ-9. Of the 10,929 screened baseline participants, 357 (3.3%) had suicidal ideation that triggered phase 2 suicide assessment and supervisor call monitoring. Forty-three (12.0%) of the 357 individuals who triggered a phase 2 suicide assessment had serious SI, suicidal plans, or other SRM protocol indications that required phase 3 intervention; all 43 completed the required clinician contact. At 7-month follow-up, 586 participants provided study data; 135 (23.0%) of these participant contacts resulted in phase 2 structured suicide assessment, and 9 of the 135 (6.7%) required phase 3 on-call clinician contact. In all, data collection at baseline and 7-month time periods yielded a total of 52 phase 3 on-call clinician contacts. Clinicians assessed 50 unique patients, with two participants requiring contact at baseline and follow-up. No known suicides occurred during the study window, and to our knowledge, only one participant initiated a suicidal gesture, which was self-aborted.
The dichotomous variable in the study database indicating whether or not on-call clinician contact occurred allowed derivation of aggregate data regarding illness and demographic characteristics. Table 1 presents these data for 42 of the phase 3 respondents (data were missing for eight participants). Over 90% (n = 38) of the respondents assessed by on-call clinicians’ were men and 83% (n = 35) were Caucasian. Average age was 54 (SD = 10) years. Respondents reported moderate to severe depressive symptomatology (PHQ-9 mean = 20, SD = 4.5), and all participants requiring on-call clinician assessment had PHQ-9 scores indicative of major depression [31]. Fewer than 10% of respondents self-rated their physical health as “good” or better. Nearly three fourths of respondents screened positive for probable PTSD. AUDIT-C scores indicated that 24.4% evidenced significant alcohol misuse [36]. Phase 3 assessment/intervention contacts with research participants averaged 20 min.
Table 1.
Characteristics of respondents who required phase 3 assessment (n = 42)
Characteristics | |
---|---|
Mean age (SD) | 54.2 (10.4) |
Gender (%) | |
Men | 90.5 |
Ethnicity (%) | |
White | 83.3 |
Non-white | 16.7 |
Education (%) | |
High school or less | 47.6 |
Some college/college and above | 50.0 |
Relationship status (%) | |
Married/living as married | 47.6 |
Single | 52.4 |
Employment (%) | |
Full-time or part-time | 7.1 |
Unemployed | 16.7 |
Disabled | 59.5 |
Retired | 14.3 |
Mean depressive severity: PHQ-9 (SD)a | 20.1 (4.5) |
Self-reported general health good or better (%) | <10.0 |
PTSD screening score exceeds cutoff (%) | 73.0 |
AUDIT-C | |
Abstinent from alcohol consumption (%) | 46.3 |
Exceeds cut score for alcohol misuse (%) | 24.4 |
Percentages may not sum to 100% due to missing data
Demographic data were missing for eight participants (six were excluded due to acute suicidality; two terminated interview prior to demographic data collection)
an = 50
Qualitative analysis of phase 3 clinician notes
Qualitative analysis of de-identified on-call notes identified two meta-themes. The first reflected assessment of empirical correlates of suicide risk and lethality. The second meta-theme represented on-call clinicians’ risk mitigation activities.
Meta-theme 1: in-depth assessment activities
Research on-call clinicians performed risk assessment consistent with standard practice and empirical evidence on suicide risk. The two major themes within meta-theme 1 were assessment of distal epidemiological risk factors and suicide warning signs/lethality assessment.
Epidemiological risk factors theme
Subthemes focused on risk based on mental health issues (including substance abuse), social support, or stressors.
Regarding mental health issues and psychiatric symptomatology, clinicians documented discussion of depression history, current depressive symptomatology and current/historical use and abuse of alcohol and drugs in nearly all notes. They also frequently assessed presence of other common psychiatric conditions, such as PTSD.
Clinicians documented discussion of social isolation, conflicted interpersonal relationships, and, on occasion, conflicted interactions with the healthcare system. In addition, many on-call clinicians documented the protective role of social support from family, friends, and healthcare providers. This included active support as well as respondents’ recognition that suicide would hurt loved ones. One key assessment target was the nature of the respondents’ relationships with local primary healthcare providers and/or mental health specialists. On-call clinicians assessed whether respondents were actively engaged in local care and the presence or absence of upcoming scheduled appointments. These discussions included assessment of preexisting plans to ensure safety and assessment of whether respondents would enact these plans if necessary.
Finally, on-call clinicians documented discussion of stressors and their management during the majority of encounters. Reported stressors included medical problems and complex combinations of stressors across multiple life domains.
Risks of harm and lethality theme
This theme focused on risks and warning signs related to suicidal means and/or intent. All on-call clinicians documented the presence and degree of current risk based on suicidal intent, or the respondent’s current position along a continuum from ideation to action. They also documented protective factors, including religious or moral proscriptions and the effects of suicide on surviving friends and family members. Finally, they documented whether respondents had selected readily available means of completing suicide.
Meta-theme 2: risk reduction activities
When necessary, on-call clinicians acted on the imperative to protect at-risk respondents. This role at times necessitated coordination and collaboration among different stakeholders, including other research clinicians, respondents themselves, respondents’ family members and other immediately available individuals, staff members and clinicians at local sites, and local emergency personnel. The risk reduction activities meta-theme included collaborative activities that clustered in the following four themes: (a) household member contact; (b) safety planning; (c) research on-call clinician as care extender; and (d) communication, continuity, and follow-up.
Household member contact theme
This theme focused on collaboration with immediately available individuals in the respondent’s environment. With respondents’ verbal consent and in situations when elevated or imminent risk warranted it, on-call clinicians initiated immediate contact with family members or other members of respondents’ households. These contacts occurred in roughly a quarter of all on-call interactions and facilitated assessment of whether additional crisis management activities or interventions were required. As illustrated in the following example, these persons provided the on-call clinician with potential collaborators to assist in assessment, problem solving, and determining next steps with regard to additional emergency intervention:
I called the [respondent] back and spoke to [collateral] (with his permission). [Collateral] stated that she felt safe and that if she became concerned about her safety or the [respondent’s], she would call 911. She mentioned that [another support person] was also home and that they could manage the [respondent’s] behavior.
Safety planning theme
This theme focused on engaging respondents in collaboration to maintain safety. Following suicide risk assessment, it occasionally became necessary for clinicians and respondents to develop concrete plans to maintain safety. These plans frequently culminated in the discussion of respondents’ agreement to engage in a variety of activities to ensure short-term safety and active problem solving regarding these activities:
[The respondent] agreed to mention his on-going depression to the doctor at [his upcoming appointment]. If he begins to feel worse before that appointment, he agreed to call the clinic or the emergency number provided by the study interviewers.
Some respondents were less confident in their abilities to refrain from harm and evidenced ambivalence or doubt about safety planning. As documented in one on-call note:
the [respondent] at first said that he could promise not to hurt himself before his meeting with a [local clinician] on [date deleted]; he then admitted that he could not reliably make that promise. He did say that he could make a promise not to hurt himself tonight.
On the few occasions when respondents did not agree to engage efforts for safety maintenance, additional intervention activities occurred. Thus, these activities represented a stepped approach in which different levels of care were applied based upon different levels of need.
Research on-call clinician as care extender theme
When clinicians assessed present risk as minimal, on-call clinicians adopted therapeutic and care extender roles with the goal of facilitating local care. The care extender role included active problem solving to support positive actions the respondent might take to maintain safety until a local long-term plan could be developed and implemented. Common elements of these discussions included encouragement to adhere to current treatment plans. For example, one on-call clinician documented discussion of “…how SSRIs can take as long as 6–8 weeks to be effective and that he should continue to take [his medication] as prescribed.” On-call clinicians also encouraged respondents to keep follow-up appointments with local care providers and to initiate local discussion of suicide. For example, one on-call clinician noted, “I encouraged [the respondent] to discuss his feelings of depression and thoughts of suicide with his doctor at his next appointment.” During some calls, the on-call clinician collaborated with the respondent to develop an explicit safety plan: “[the respondent] keeps [weapons] all locked up but keeps 2 of them loaded (and locked up). I suggested that he consider keeping them all unloaded.”
Communication, continuity, and follow-up theme
This theme focused on engaging collaboration from local care providers and emergency personnel. Acute suicide risk warranted direct communication between on-call clinicians and local care providers and/or emergency care personnel. Collaborative communication with local care providers occurred in about half of all phase 3 encounters and highlighted the complexity of on-call clinicians’ roles when respondents’ behavior raised acute concern regarding safety. As illustrated in the following quote, on-call clinicians communicated with local clinical personnel, such as depression care managers (DCMs), who then facilitated ongoing care efforts at the local sites:
Spoke with [DCM] about the [respondent]. She provided me with contact information for the [respondent’s] mental health providers. I first called the Outpatient clinic and requested to speak with [the respondent’s therapist].…I later called [the respondent’s psychiatrist’s] extension and left a message briefly describing my contact with the patient. [The respondent’s psychiatrist] returned my call. I explained the situation, and he noted that his clinic would follow the patient “more closely.” Paged [site principal investigator]: Spoke with him briefly and gave the [respondent’s] information.…[The site principal investigator] noted that he would arrange contact of the [respondent] as appropriate.
In situations representing the highest risk, on-call clinicians communicated with local triage providers identified in the On-Call Clinician Resource Guide (derived in phase 1). Rare occasions of acute and imminent suicide risk warranted immediate intervention via Emergency Medical Services. We breached confidentiality twice in order to communicate with emergency providers to maintain safety. Faced with this situation, one on-call clinician documented the following:
After getting off the phone with [the respondent], I called the Emergency Number…[and] relayed the gist of my conversation with [the respondent] and my concern. [The emergency dispatcher] said that they would be sending a police car…to evaluate [the respondent].
These rare emergency-based interactions required close consideration of when to involve other stakeholders as well as consultation among research on-call clinical staff. Evidence for consultation among on-call personnel was documented in the following:
I consulted with [the]…project coordinator and we agreed that a return phone call to [the respondent] was indicated and that I should explain my position and alert her to the possibility of my contacting emergency services.
Finally, to ensure continuity of care, on-call clinicians occasionally interacted with local care providers at multiple points in time beyond the initial encounter:
I will call [Depression Care Manager] in the morning on [date deleted] to f/u on the message I left this evening. If I am unable to communicate with [Depression Care Manager] or if we deem it appropriate, I will attempt to contact one or more of [the respondent’s] outpatient [Mental Health] providers.
Finally, during one on-call encounter, a research participant communicated concerning thoughts about self-harm. Although the participant denied imminent intent of harm, the on-call clinician initiated SRM protocol consistent communication with the research study’s site-level principal investigator. Later, upon review of the patient’s electronic medical record, the site principal investigator learned that the patient had made a self-aborted suicidal gesture prior to presenting for local treatment and entering the hospital voluntarily.
DISCUSSION
Existing suicide risk assessment and prevention recommendations have relevance for patients seeking care in person (e.g., [18, 19]), but guidance is needed on how to structure, staff, and implement SRM in research contexts and over the phone. Our telephone-based SRM protocol (see Fig. 1) successfully managed suicide risk and serves as a potential reference for others who are developing their own risk management approaches. We maintain that our data provide a starting point for researchers by highlighting the levels of personnel that are required at particular points in a stepped process leading patients from non-clinician interviewers to clinicians.
Our results confirm prior findings [37] suggesting that SI is fairly common among primary care patients and that high suicide risk is less so [29]. We screened nearly 11,000 patients, and 357 (3%) triggered our structured SI assessment. Only 43 patients required real-time on-call clinician assessment/intervention at baseline. Our follow-up sample included depressed patients only. Because depression confers risk of SI, the fact that nearly one in four follow-up patients required phase 2 assessment provides some evidence that our protocol functioned effectively. Collectively, our results underscore the necessity of SRM protocols [1] and highlight the importance of adequate SRM staffing by clinical research personnel.
Development of our SRM protocol required extensive advance planning, teamwork, and collaboration among geographically dispersed study personnel and local participating research sites. Mirroring the key features of SRM protocols identified in the literature [2], phase 1 included relationship building with participating sites, establishment of communication avenues, designing structured layperson interview protocols, and designing and delivering training activities for data collection personnel. Our training activities were extensive and involved a variety of activities, ranging from presentation of basic suicide information to simulated interviews with suicidal research respondents. We recommend strongly that future investigators allocate sufficient resources for developmental SRM activities and begin these collaborative tasks very early in the research design process. Moreover, in addition to providing extensive advance training for data collection personnel, future investigators might consider implementing formal means of gauging interviewer competence regarding risk assessment.
Phase 2 focused on the actual conduct of data collection interviewers. This included development and application of a brief protocol for targeted assessment of suicide risk, which was triggered by overt comments regarding harm or any non-zero response to the PHQ-9 SI question. Phase 2’s structured questions addressed acute warning signs in the suicide risk assessment literature such as suicidal planning and stated intent to engage in self-harm [5]. We recommend that future researchers consider the use of a few structured, simple to administer questions such as these to assess safety while minimizing research intrusions into participants’ experiences.
Finally, phase 3 required assessment and intervention whenever participants shared information at phase 2 that (1) suggested suicide planning or serious consideration, (2) suggested that suicide risk was not already being managed by local care providers, or (3) left phase 2 interviewers with any degree of doubt regarding participant safety. Phase 3 employed intensive assessment and intervention by doctoral-level study clinicians whose experience working with depressed patients in primary care conferred previous experience working with suicide. Thus, the phase 3 on-call clinicians were identified as the on-call clinical team on the basis of experience, not because they necessarily embodied a “gold standard” for harm assessment and intervention. Our qualitative analyses of de-identified on-call documentation suggested that phase 3 activities were often complex. Clinicians drew upon clinical skill and experience and integrated available resources instead of using a scripted intervention. This flexibility enabled clinicians to conduct in-depth suicide risk assessment (meta-theme 1) and complete collaborative risk management activities (meta-theme 2).
Findings regarding meta-theme 1 suggested that phase 3 assessment effectively translated findings from the suicide risk literature into practice. Presence and severity of mental illnesses like depression and PTSD were assessed routinely, a finding consistent with the literature regarding distal risk factors [6, 7, 9–11]. Discussion of acute stressors, social support, and present use of alcohol and drugs also suggested that phase 3 encounters examined more proximal suicide risk factors [7, 8, 14–16]. Finally, consistent with calls in the literature for targeted assessment regarding suicidal intent and imminent warning signs (e.g., [23]), phase 3 encounters included routine discussion of lethality and suicide planning.
The unscripted phase 3 approach facilitated on-call clinicians’ flexible and therapeutic stance, which was reflected in meta-theme 2’s collaborative risk management activities. Clinicians enlisted assistance from patients’ existing supports, other research clinicians, and local mental health and primary care personnel. On-call clinicians also provided considerable general support and encouragement, and we conclude that the expert research team clinicians’ support of patients contributed substantially to the success of the encounters. Generally speaking, these efforts were consistent with recommendations regarding inclusion of family members [1] and establishment of working alliances with at-risk patients [23].
The most complex phase 3 interactions involved risk management communication initiated by the on-call provider with local care personnel and local members of the research project’s clinical teams. Although no suicides occurred during the study window, some patients were hospitalized and there was at least one suicidal gesture and self-aborted attempt. These experiences underscore the reality of suicide risk in chronic illness research and research in primary care. In cases such as these when risk was high, our on-call clinicians consulted the SRM protocol and collaborated with pre-identified local triage providers or, on rare occasions, crisis personnel. Study clinicians also engaged local depression care managers and site-based principal investigators who facilitated patients’ transitions to immediate and/or ongoing local care. In rare cases involving the highest levels of risk, on-call clinicians collaborated with multiple individuals on multiple occasions to ensure safety and care continuity. While some complexity in negotiating care across geographically dispersed sites is inevitable, we credit the substantial phase 1 advance work and the resulting SRM protocol with simplifying the strategies as much as possible and facilitating their success. Finally, whereas our protocol did not require the on-call clinicians to secure additional suicide-specific training, researchers planning SRM protocols might consider providing on-call personnel with suicide-specific skills reinforcement.
Our SRM strategy can inform researchers and IRBs as they consider the development and evaluation of their own plans for safety management. Unlike the situation in which a suicidal patient initiates contact with a healthcare facility, researcher-initiated contacts often come outside of a help-seeking context. Balancing research participant autonomy, confidentiality, and safety is necessary for the successful conduct of the research upon which healthcare improvements rest. Our data collection interviewers informed patients about the limits of confidentiality and our stepped protocol minimized (though it did not eliminate) research team intrusions into patients’ decision-making autonomy. On-call clinicians became involved only when necessary to protect the patient, and confidentiality was compromised only twice. Each phase of our strategy contributed to participant safety without unduly burdening research participants or staff.
The implementation of our protocol from a distance over the telephone raised some significant challenges. First, the distance involved between the on-call clinical team, the data collection personnel, and the patients heightened the importance of advance planning and testing of communication avenues. Prior to phase 1’s extensive advance planning, for example, the research team did not have the collaborative relationships with local clinical personnel, which would have existed if the research team and patients were co-located. Second, threat assessment done over the telephone restricts data about clinical status to the patient’s verbal self-report. This required clinicians to make complex care decisions without routine availability of information from visual observation and/or review of the patients’ medical chart. The collaborative activities and contacts observed in our clinician documentation helped facilitate assessment and ensure safety and also served to mitigate some of the difficulties associated with telephonic assessment. For example, communication between the on-call clinicians and collateral informants in the patients’ homes was very helpful in increasing the available data regarding threat. Other collaborative activities such as communication with other research team members and local care personnel provided additional information that enhanced patient safety.
Although derived from a research study, our data have potential relevance for clinicians who employ telephone-based interventions or work with health problems that confer suicide risk [38, 39]. For example, prepared personnel (e.g., appointment clerks or nursing staff) could readily initiate suicide assessment over the telephone or in person via standardized measures of depression (e.g., PHQ-9). Pre-identified clinical personnel could engage in more thorough assessment of harm potential whenever warranted by questionnaire responses. In order to guide their interactions with at-risk patients, clinicians might benefit from advance preparation and availability of a brief checklist including key assessment targets identified here and abstracted from existing suicide assessment guidelines and protocols [18, 19] (e.g., acute stressors, mental health symptomatology, alcohol/substance abuse, availability of lethal means, presence/absence of immediate social support or household member, etc.). In light of findings suggesting that formal suicide assessment tools fail to assess suicidal probability with certainty [40], clinicians should use structured information to supplement their clinical judgment and not as a rote checklist [1]. Ultimately, personnel ranging from trained frontline staff to doctoral-level clinicians can be involved to detect and respond to high-risk situations. Our protocol emerged from the collaborative efforts of primary care and specialty mental healthcare providers. Just as collaborative care evidences effectiveness in depression treatment [41], we maintain that collaborative efforts among primary care and mental health providers will maximize efforts to manage suicide and suicide potential.
Limitations
The present data were collected during a VA depression care implementation study. Our sample included mostly older Caucasian men with multiple comorbid illnesses and a high PTSD and problematic alcohol use prevalence. In addition, we employed a consecutive patient sampling design among VA medical clinic attendees, which may have resulted in a sample with relatively poor health [42] and significant mental health concerns. Thus, relative to other patient populations, our analyses may overestimate suicide risk and/or the complexity of urgent assessment and intervention. Generalization of results should be done cautiously.
A second limitation relates to limited data availability and maturation. Demographic data were missing for some participants, and documentation from two contacts was not analyzed. Moreover, because our SRM protocol did not specify what was documentable, some activities might have been underrepresented. This negates neither the description of our multi-step process nor the major activities included in phase 3 on-call interactions. In an attempt to further expand the evidence base regarding suicide management and prevention strategies, researchers considering similar future studies might plan specifically to examine assessment and intervention activities and implement data tracking systems that would facilitate this from the start.
Our data collection occurred during the years 2003–2006. Despite the passage of time, our analysis still ranks among the first to systematically characterize and evaluate SRM protocol development and implementation. We maintain, therefore, that our analyses are timely and responsive to a need voiced in the literature [2]. Finally, clinical implications of the present work might apply most readily to care provided in large healthcare organizations (i.e., VA or HMOs) which enjoy considerable personnel resources. Our stepped care protocol and collaborative approach might be more difficult to implement in small private practice settings.
In conclusion, our findings offer planning guidance for clinical and implementation science researchers regarding the maintenance of research participant safety, an understudied and exceptionally important issue. Effective protection against suicidal behavior for at-risk research participants can result from team-based advance planning, primary care and mental health specialty collaboration, and initial structured assessment by trained non-professionals followed by targeted assessment and intervention by trained clinicians as necessary. The present results highlight the feasibility of telephonic suicide assessment and management protocols in which the assessment and protection of patients stem from collaboration among multiple providers and stakeholders across sites and organizational levels.
Acknowledgments
This project was funded by the Department of Veterans Affairs (VA) Health Services Research and Development (HSR&D) Service and the Quality Enhancement Research Initiative (QUERI) (Project nos. MHI 99-375, MNT 01-027, MHQ 10-06). Drs. Campbell and Bolkan received funding from the VA Office of Academic Affiliations Associated Health Postdoctoral Fellowship Programs at the Northwest Center for Outcomes Research. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. The authors wish to acknowledge the contributions of Carol Simons, Barbara Simon, Andy Lanto, Jane Uman, and Carole Oken. Finally, the authors wish to thank Barbara Simon, Carol Oken, and JoAnn Kirchner for helpful comments on an earlier draft of this manuscript.
Footnotes
Implications
Practice: Research-based suicide risk management strategies that result from extensive advance planning and include intensive collaboration contribute to research participant safety and can function effectively via telephone.
Policy: Policy makers who oversee clinical activities might consider implementation of stepped care approaches to suicide risk management that employ non-clinical and clinical personnel in support of suicide assessment and/or intervention tasks.
Research: Judgments of the Institutional Review Boards that evaluate research-based safety management strategies will benefit from our detailed analysis of experiences that occurred during research-based suicide risk management encounters.
References
- 1.Pearson JL, Stanley B, King CA, Fisher CB. Intervention research with persons at high risk for suicidality: Safety and ethical considerations. The Journal of Clinical Psychiatry. 2001;62(sup 25):17–26. [PubMed] [Google Scholar]
- 2.Vannoy S, Whiteside U, Unützer J. Current practices of suicide risk management protocols in research. Crisis. 2010;31:7–11. doi: 10.1027/0227-5910/a000004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Jacobs DG, Brewer M, Klein-Benheim M. Suicide assessment: An overview and recommended protocol. In: Jacobs DG, editor. The Harvard Medical School Guide to Suicide Assessment and Intervention. San Francisco: Josey-Bass; 1999. pp. 3–39. [Google Scholar]
- 4.McIntosh, J. L. (for the American Association of Suicidology). U.S.A. suicide 2007: Official final data. Washington, DC: American Association of Suicidology, dated May 23, 2010. Retrieved 02 February 2011 from http://www.suicidology.org.
- 5.Rudd MD, Berman AL, Joiner TE, et al. Warning signs for suicide: Theory, research, and clinical applications. Suicide & Life-Threatening Behavior. 2006;36:255–262. doi: 10.1521/suli.2006.36.3.255. [DOI] [PubMed] [Google Scholar]
- 6.Mann JJ. A current perspective of suicide and attempted suicide. Annals of Internal Medicine. 2002;136:302–311. doi: 10.7326/0003-4819-136-4-200202190-00010. [DOI] [PubMed] [Google Scholar]
- 7.Hall RCW, Platt DE, Hall RCW. Suicide risk assessment: A review of risk factors for suicide in 100 patients who made severe attempts. Psychosomatics. 1999;40:18–27. doi: 10.1016/S0033-3182(99)71267-3. [DOI] [PubMed] [Google Scholar]
- 8.Goldberg JF, Singer TM, Garno JL. Suicidality and substance abuse in affective disorders. The Journal of Clinical Psychiatry. 2001;62(suppl 25):35–43. [PubMed] [Google Scholar]
- 9.Sareen J, Cox BJ, Afifi TO, et al. Anxiety disorders and risk for suicidal ideation and suicide attempts: A population-based longitudinal study of adults. Archives of General Psychiatry. 2005;62:1249–1257. doi: 10.1001/archpsyc.62.11.1249. [DOI] [PubMed] [Google Scholar]
- 10.Oquendo MA, Friend JM, Halberstam B, et al. Association of comorbid posttraumatic stress disorder and major depression with greater risk for suicidal behavior. The American Journal of Psychiatry. 2003;160:580–582. doi: 10.1176/appi.ajp.160.3.580. [DOI] [PubMed] [Google Scholar]
- 11.Seedat S, Stein MB, Forde DR. Association between physical partner violence, posttraumatic stress, childhood trauma, and suicide attempts in a community sample of women. Violence and Victims. 2005;20:87–98. doi: 10.1891/vivi.2005.20.1.87. [DOI] [PubMed] [Google Scholar]
- 12.Goodwin RD, Kroenke K, Hoven CW, et al. Major depression, physical illness, and suicidal ideation in primary care. Psychosomatic Medicine. 2003;65:501–505. doi: 10.1097/01.PSY.0000041544.14277.EC. [DOI] [PubMed] [Google Scholar]
- 13.Tang NK, Crane C. Suicidality in chronic pain: A review of the prevalence, risk factors and psychological links. Psychological Medicine. 2006;36:575–586. doi: 10.1017/S0033291705006859. [DOI] [PubMed] [Google Scholar]
- 14.Joiner T. Why people die by suicide. Cambridge: Harvard University Press; 2005. [Google Scholar]
- 15.Pope KS, Vasquez MJT. Responding to suicidal risk. In: Pope KS, Vasquez MJT, editors. Ethics and psychotherapy and counseling, 3rd ed. San Francisco: Josey-Bass; 2007. pp. 258–279. [Google Scholar]
- 16.Mościcki EK. Epidemiology of suicide. In: Jacobs DG, editor. The Harvard Medical School Guide to Suicide Assessment and Intervention. San Francisco: Josey-Bass; 1999. pp. 40–51. [Google Scholar]
- 17.Brown GK, Beck AT, Steer RA, Grisham JR. Risk factors for suicide in psychiatric outpatients: A 20-year prospective study. Journal of Consulting and Clinical Psychology. 2000;68:371–377. doi: 10.1037/0022-006X.68.3.371. [DOI] [PubMed] [Google Scholar]
- 18.American Psychiatric Association (2003). Practice guideline for the assessment and treatment of patients with suicidal behaviors. Retrieved 3 February 2011 from http://www.psychiatryonline.com/pracGuide/pracguideChapToc_14.aspx. [PubMed]
- 19.Sánchez HG. Risk factor model for suicide assessment and intervention. Professional Psychology, Research and Practice. 2001;32:351–358. doi: 10.1037/0735-7028.32.4.351. [DOI] [Google Scholar]
- 20.Esposito E, Wang JL, Adair CE, et al. Frequency and adequacy of depression treatment in a Canadian population sample. Canadian Journal of Psychiatry. 2007;52:780–789. doi: 10.1177/070674370705201205. [DOI] [PubMed] [Google Scholar]
- 21.Fortney JC, Pyne JM, Edlund MJ, et al. A randomized trial of telemedicine-based collaborative care for depression. Journal of General Internal Medicine. 2007;22:1089–1093. doi: 10.1007/s11606-007-0201-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Unützer J, Katon W, Callahan CM, et al. Collaborative care management of late-life depression in the primary care setting: A randomized controlled trial. JAMA. 2002;288:2836–2845. doi: 10.1001/jama.288.22.2836. [DOI] [PubMed] [Google Scholar]
- 23.Joiner T, Kalafat J, Draper J, et al. Establishing standards for the assessment of suicide risk among callers to the National Suicide Prevention Lifeline. Suicide and Life-Threatening Behavior. 2007;37:353–365. doi: 10.1521/suli.2007.37.3.353. [DOI] [PubMed] [Google Scholar]
- 24.Fortney JC, Pyne JM, Edlund MJ, Robinson DE, Mittal D, Henderson KL. Design and implementation of the Telemedicine-Enhanced Antidepressant Management Study. General Hospital Psychiatry. 2006;28:18–26. doi: 10.1016/j.genhosppsych.2005.07.001. [DOI] [PubMed] [Google Scholar]
- 25.Nutting PA, Gallagher K, Riley K, White S, Dickinson WP, Korsen N, Dietrich A. Care management for depression in primary care practice: Findings from the RESPECT-Depression Trial. Annals of Family Medicine. 2008;6:30–37. doi: 10.1370/afm.742. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Solberg LI, Glasgow RE, Unützer J, et al. Partnership research: A practical trial design for evaluation of a natural experiment to improve depression care. Medical Care. 2010;48:576–582. doi: 10.1097/MLR.0b013e3181dbea62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Unützer J, Katon W, Williams JW, Jr, et al. Improving primary care for depression in late life: The design of a multicenter randomized trial. Medical Care. 2001;39:785–799. doi: 10.1097/00005650-200108000-00005. [DOI] [PubMed] [Google Scholar]
- 28.Schulberg HC, Bruce ML, Lee PW, Williams JW, Jr, Dietrich AJ. Preventing suicide in primary care patients: The primary care physician's role. General Hospital Psychiatry. 2004;26:337–345. doi: 10.1016/j.genhosppsych.2004.06.007. [DOI] [PubMed] [Google Scholar]
- 29.Schulberg HC, Lee PW, Bruce ML, et al. Suicidal ideation and risk levels among primary care patients with uncomplicated depression. Annals of Family Medicine. 2005;3:523–528. doi: 10.1370/afm.377. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Bonner L, Felker B, Chaney E, et al. Suicide risk response: Enhancing patient safety through development of effective institutional policies. In: Henriksen K, Battles JB, Marks E, Lewin DI, et al., editors. Advances in patient safety: From research to implementation, 3. Rockville: Agency for Healthcare Research and Quality; 2005. pp. 507–519. [PubMed] [Google Scholar]
- 31.Kroenke K, Spitzer RL, Williams JBW. The PHQ-9. Journal of General Internal Medicine. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Prins A, Ouimette P, Kimmerling R, et al. The primary care PTSD screen (PC-PTSD): Development and operating characteristics. Prim Care Psychiatry. 2003;9:9–14. doi: 10.1185/135525703125002360. [DOI] [Google Scholar]
- 33.Bush KR, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem drinking. Archives of Internal Medicine. 1998;158:1789–1795. doi: 10.1001/archinte.158.16.1789. [DOI] [PubMed] [Google Scholar]
- 34.Kazis LE, Ren XS, Lee A, et al. Health status in VA patients: Results from the Veterans Health Study. American Journal of Medical Quality. 1999;14:28–38. doi: 10.1177/106286069901400105. [DOI] [PubMed] [Google Scholar]
- 35.Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qualitative Health Research. 2005;15:1277–1288. doi: 10.1177/1049732305276687. [DOI] [PubMed] [Google Scholar]
- 36.Bradley KA, DeBenedetti AF, Volk RJ, Williams EC, Frank D, Kivlahan DR. AUDIT-C as a brief screen for alcohol misuse in primary care. Alcoholism, Clinical and Experimental Research. 2007;31:1208–1217. doi: 10.1111/j.1530-0277.2007.00403.x. [DOI] [PubMed] [Google Scholar]
- 37.Corson K, Gerrity MS, Dobscha SK. Screening for depression and suicidality in a VA primary care setting: 2 items are better than 1 item. Am J Manag Care. 2004;10:839–45. [PubMed] [Google Scholar]
- 38.Hailey D, Roine R, Ohinmaa A. The effectiveness of telemental health applications: A review. Canadian Journal of Psychiatry. 2008;58:767–778. doi: 10.1177/070674370805301109. [DOI] [PubMed] [Google Scholar]
- 39.Kennedy GJ. Targets for telephone-based behavioral health interventions. Primary Psychiatry. 2008;15:35–39. [Google Scholar]
- 40.Bisconer SW, Gross DM. Assessment of suicide risk in psychiatric hospital. Professional Psychology, Research and Practice. 2007;38:143–149. doi: 10.1037/0735-7028.38.2.143. [DOI] [Google Scholar]
- 41.Gilbody S, Bower P, Fletcher J, et al. Collaborative care for depression: A cumulative meta-analysis and review of longer-term outcomes. Archives of Internal Medicine. 2006;166:2314–2321. doi: 10.1001/archinte.166.21.2314. [DOI] [PubMed] [Google Scholar]
- 42.Lee ML, Yano EM, Wang M, et al. What patient population does visit-based sampling in primary care settings represent? Medical Care. 2002;40:761–770. doi: 10.1097/00005650-200209000-00006. [DOI] [PubMed] [Google Scholar]