Skip to main content
PMC home page
Internet Interventions logoLink to Internet Interventions
. 2021 Jan 18;23:100366. doi: 10.1016/j.invent.2021.100366

Psychiatric rehabilitation through teaching smartphone skills to improve functional outcomes in serious mental illness

Elena Rodriguez-Villa 1,⁎, Erica Camacho 1, John Torous 1
PMCID: PMC7822966  PMID: 33532245

Abstract

This study measured the impact of a digital competencies and skills course on participants with serious mental illness. Close to 75% of participants reported an improvement in a smartphone related skill, and the majority of participants that reported improvement in one skill reported improvement in at least one other. Qualitative feedback from participants suggests how digital competencies acquired were used to immediately support functional outcomes.

Objective

To improve functional outcomes in patients with serious mental illness through a multi-session curriculum designed to improve smartphone skills and engage participants in group learning and problem solving, targeting negative and cognitive symptoms of illness.

Methods

An eight-week smartphone digital competencies and skills course was offered to two distinct groups of youth with serious mental illness. Pre and post self-report measurements were captured for each participant for each session.

Results

Group participation varied by session, but overall 28 unique patients attended. From survey results, 75% reported improvement in smartphone related skills because of the groups. Qualitative feedback suggests how skills acquired by patients were immediately utilized to gain insight into health and support functional outcomes.

Conclusions and implications for practice

Smartphone skills groups are a means to provide practical psychiatric rehabilitation that may enable some patients to compensate for cognitive and social deficits due to illness. While ensuring groups are responsive to patients with varying degrees of skills remains a challenge, adapting lesson structures and mediums, as well as creating new measurement tools, offers a means to modify the course with the clinical need.

Keywords: Smartphones, Schizophrenia, Serious mental illness, Functional outcomes, Psychiatric rehabilitation

Highlights

  • •

    This study measured the impact of a digital competencies and skills course on participants with first episode psychosis.

  • •

    Close to 75% of participants reported an improvement in at least 1 smartphone related skill.

  • •

    Qualitative feedback suggests that digital skills acquired in groups were utilized to support functional outcomes.

  • •

    Smartphone skills groups may provide psychiatric rehabilitation to help patients compensate for cognitive or social deficits.

1. Introduction

While medication and therapy treatments offer effective interventions for positive symptoms associated with serious mental illness (SMI), there are considerably fewer treatment options for negative symptoms and cognitive deficiencies related to the illness. Although each patient's disability is unique, the impact on real world-functioning can be profound (Świtaj et al., 2012). The majority of SMI patients face career and housing challenges, often rooted in deficits that make maintaining commitments difficult (Harvey et al., 2019). While not a panacea, interventions that utilize digital technologies such as mobile apps offer a novel means to improve functional outcomes through targeting both negative symptoms and cognitive deficiencies.

As smartphones and mobile apps become more ubiquitous and entrenched in societies everywhere, patients with SMI need certain digital literacy skills to stay connected with the world and thrive. On a daily basis, negative symptoms and cognitive deficiencies related to SMI impair quality of life, sociability and productivity. Activities including but not limited to handling medications, preparing food, shopping, managing finances, maintaining hygiene, navigating public transport, and using technology are challenging (Samuel et al., 2018). Smartphones can support patients in completing tasks and maintaining regular commitments. Increasingly, comfort with mobile technology is less of an asset than it is a requirement. Many medications are now refilled via internet, groceries are ordered through apps, shopping is done online, banking and check deposits are digital, and every major metro system offers schedules designed for mobile devices. During the 2020 COVID-19 pandemic, telehealth became the only means for many patients to access mental health care (Kim et al., 2020). The widespread disruption to treatment and social isolation highlighted the need for digital literacy not only among patients with SMI, but also anyone seeking healthcare services (Triana et al., 2020). Digital skills, and confidence practicing them, are critical in utilizing technology towards health and daily functioning (Connolly et al., 2020; Sarkar et al., 2016). While not all patients require digital literacy training, there is a broad, unmet, and growing need for building technological skillsets in order to access care and basic resources.

Digital Opportunities for Outcomes in Recovery Services (DOORS) is a digital literacy intervention designed to improve functioning among patients with SMI through introducing participants to smartphone skills in a group setting (Hoffman et al., 2020). Social learning targets negative symptoms related to social drive and interest, and practicing using smartphones to make phone calls and send text messages, store contact information, write emails, and access online peer support makes building and maintaining relationships easier. While there is concern that smartphones may hinder cognition for the general population (Thornton et al., 2014), the learning and use of digital tools by those with SMI likely engages problem solving and increases attention and concentration in a manner that promotes cognitive functioning. Simply put, a smartphone can be a powerful social and cognitive aid for patients with impairments caused by SMI (Torous and Keshavan, 2016). Combined with the growing need for digital literacy outlined above – psychosocial rehabilitation centered around building smartphone skills offers a novel opportunity to improve functional outcomes in patients with SMI.

DOORS addresses negative symptoms and cognitive deficiencies in a multi-session curriculum that features topical lessons. Each session engages group learning and problem solving to teach a new smartphone skill immediately useful and applicable to daily life. DOORS began as a program to teach patients with serious mental illness the skills and competencies necessary to use a smartphone towards wellness. In this paper we outline how a new curriculum with adaptive lessons and targeted, practical skills has enabled the program to evolve towards a psychosocial rehabilitation intervention for patients with serious mental illness.

2. Towards a curriculum that reflects skills-based learning

Past iterations of DOORS have focused on wellness goals. Grounded in SAMSHA's Eight Dimensions of Wellness Model, the lessons taught participants how to use their smartphone towards these interconnected yet broad aims (Hoffman et al., 2020). DOORS was run with a total of 8 groups in diverse settings from first episode psychosis programs to clubhouses offering care to patients with a range of psychiatric conditions. The wellness focus helped participants reflect on what motivates them and how they can use their smartphones to build healthy habits. However, many participants asked for further assistance with basic smartphone skills and found the most value not in any one wellness goal but rather the skill acquitted to reach that goal. One participant emphasized in a qualitative survey: “I need to learn how to use my phone; how to navigate it.” Across groups, we observed a need, and opportunity, to introduce and practice simple smartphone skills that may yield high utility.

Added value to participants hinges on whether they acquire skills or knowledge through DOORS that they are able to recall on their own. Participants seek to draw on or utilize specific learned digital tools and techniques independently, at times at which they are personally relevant or most helpful. This may be towards wellness but also mental health, daily activities, education, and employment. Towards improving functional outcomes, the revised DOORS curriculum serves to equip participants with skills and resources that are easy to access, appealing to use, and relevant to daily life (Liberman et al., 1998).

To facilitate cumulative learning, DOORS has been adapted since its inception, detailed in Hoffman et al. (2020), to include eight distinct sessions. Each session centers around a theme or category related to phone use. Session topics were chosen based on prior use of smartphone technology in clinical settings such as at the Digital Clinic at the Beth Israel Deaconess Medical Center (Rodriguez-Villa et al., 2020) and the Massachusetts Psychosis Network for Early Treatment (MAPNET), as well as from findings from the Technology Evaluation and Assessment Criteria for Health apps (TEACH-apps) pilot also at the Beth Israel Deaconess Medical Center (Camacho et al., 2020). Within each session, 2–3 specific smartphone skills or features are introduced and practiced throughout the lesson. Most activities encourage participants to use their own phone and enact learned techniques, challenging them to sustain attention and work through frustration with the support of peers and group leaders (Fig. 1).

Fig. 1.

Fig. 1

Open in a new tab

Session themes cover eight topics related to smartphone use. Each session emphasizes key skills and facilitates discussion around how these new skills are applicable to daily life.

The curriculum and lesson plans serve as a guiding framework for sessions. Prior groups demonstrated that participants' interests and comfort levels with technology should significantly influence the content and presentation of lessons. Group engagement hinges on topics and use cases for digital skills that are relevant to participants. As such, the lessons outlined above were built to cater to a younger population of first episode psychosis patients. We anticipated participants in this group would show a higher dexterity navigating their smartphones and have prior exposure to social media and a range of smartphone apps. This curriculum still reviews foundational skills such as connecting to Wi-Fi but also introduces more involved practices that build on existing skills and are widely applicable, such as utilizing GoogleMaps to find businesses, determine their hours, and travel there in the shortest time.

While participants are learning in sessions how and why these digital resources can be useful, they are also engaging with one another and focusing attention to perform tasks – practicing skills that improve negative symptoms and, in turn, may also reduce psychiatric stigma directed towards them in public settings (Penn et al., 2000). Discussions facilitate a cooperative learning environment in which participants are encouraged to listen to one another, regulate their reactions, and offer relevant responses and insights. Although participants are not given formal social skills training, group discussion and activities are an opportunity to rehearse social skills. For example, participants practice Interactional Skills – “starting, maintaining and terminating a conversation” – through sharing which smartphones apps they visit for news updates and asking peers follow-up questions around why or how they use an app (Kopelowicz et al., 2006). Group learning encourages prosocial behavior from participants.

3. Assessment and outcomes

3.1. Participants

We piloted the new digital skills curriculum with two separate groups of young adults ranging in ages 17 through 26. Group 1 consisted of young adults with early-stage psychosis and Group 2 consisted of young adults in therapy for a broader range of mental illnesses, including psychosis and depression. Demographics such as gender and level of education, as well as clinical characteristics including pathology and severity of illness, were not collected in line with guidance from the Institutional Review Board. Both groups met at the same multi-service center for young adults with mental illness in Boston, MA between the months of January and March of 2020. Group 1 met each week for eight weeks and Group 2 met every other week over the course of sixteen weeks. Sessions lasted 45 min and were led each week by the same two research staff at Division of Digital Psychiatry to build trust and rapport over time. Attending all 8 sessions was encouraged but not mandatory. External factors – weather, competing activities at the community center, and appointments – impacted attendance week to week. As a result, session size in both groups ranged from 3 to 14 participants.

3.2. Measurement tools

Functional outcomes, characterized by capacity to perform a task versus ability to perform it, are often used to measure the efficacy of treatments for schizophrenia (Bromley and Brekke, 2010). Functional outcome scales offer a holistic assessment of areas of life impacted by SMI and, if used over time and in conjunction with interventions, can indicate cognitive changes. They make it possible to predict success or achievement in areas of daily life. Traditional functional outcome scales may take 10–15 min to complete and can help anticipate whether a patient is capable of living alone (Mausbach et al., 2007). The University of California, San Diego, Performance-Based Skills Assessment – Brief (UPSA-Brief) accounts for neurological and environmental factors and illustrates that measurable skills relate to functioning (Mausbach et al., 2007). Needing a briefer instrument for DOORS, in which sessions are only 1 h each, the surveys built for the revised curriculum aim to measure learned skills in less than 1 min. Survey content for each session is determined by the targeted skills a lesson is designed to teach. Surveys include 5–7 statements about a participant's ability to perform a smartphone task and the kind of assistance, if any, they need to complete it. Participants completed a session survey at the start and end of each session. This allowed for skill building and confidence building that took place over the course of a single lesson to be captured (Fig. 2).

Fig. 2.

Fig. 2

Open in a new tab

Surveys, unique to each session, are completed at the start and end of a session. Participants report their ability to perform a series of smartphone skills or activities introduced or reviewed during the session. The purpose of these surveys is to capture learning over the course of a single session.

3.3. Results

45 out of a total of 104 individual surveys amassed from 8 sessions of Group 1 and 5 sessions of Group 2 showed improvement in a participant's ability to complete at least one smartphone related task. Group 2 ended early because of the spread of COVID-19. Nearly 75% of participants who showed improvement in one smartphone related task (33/45) showed improvement in at least one other. Overall, 102 improvements were reported across all groups and sessions compared to only 10 instances of decreases in perceived aptitude. Nearly all of these improvements (97/102) were reported as achieving high proficiency – by a 4 or 5 ranking – in completing a smartphone task. “Checking the number of steps I've taken today” yielded the largest number of improvements in a single Group session; 8 out of 14 participants in attendance at the same group reported improvement. Skills that participants in both groups combined reported improvements in were “Checking how many hours you slept last night” (6/17), “Monitoring my screen time” (6/17), and “Setting a voicemail greeting” (8/21).

Surveys do not capture cognitive endpoints of measurements of negative symptoms. However, in each iteration of DOORS, we regularly observed personal growth in participants. Many participants who were reluctant to participate in activities or reticent about sharing personal experiences in the first session were volunteering answers to group exercises by the last. In an early lesson, one participant shared that they were intimidated by the word “smartphone.” Another participant echoed their sentiments and responded directly to them. “Thank you for saying that,” she said. We noticed that after this interaction, the second participant began asking for clarification when she didn't understand a new concept. She was more assertive asking for help in finding last week's step count on her phone and she was excited to show the group how her stepcount increased just by walking around the room. “I look forward to tracking my steps each day,” she stated.

Learning is also evident when participants share how they have applied a smartphone skill outside of DOORS. Following a session in which much of the lesson focused on utilizing map apps to determine method of travel, a participant explained how he had used Google Maps to arrive at his intermural basketball game on time. He had considered taking public transportation but chose instead to use Uber after calculating and comparing the time and cost of both routes through Google Maps. His anecdote revealed that he had mastered a new smartphone skill to the extent he was able to utilize it independently and that he had leveraged it, effectively, to solve a problem and help him make a decision. While DOORS does not measure its impact on the alleviation of negative or cognitive symptoms, we can infer success from narratives such as these. Participants are making use of digital resources and learned techniques to access information, solve problems, and engage in activities.

3.4. Discussion

Though limited, survey results and descriptive statistics illustrate that through group learning and activities that rehearse techniques, participants can improve upon smartphone skills to an extent that they are confident in utilizing learned features on their own. The high rate of participants that improved in more than one category (33/45) suggests that DOORS yields learning that is more significant for a subset of the population. In other words, not all participants benefited from DOORS but those that did reported improvement in multiple areas. Lesson content and delivery, as well as group size, may facilitate learning and build confidence for participants who already possess some experience with technology and are comfortable engaging with peers. To expand the reach of DOORS and broaden its rate of success, future iterations of DOORS need to account for and address variance in digital dexterity, cognition, and social skills.

Surveys used for Group 1 and Group 2 did not assess changes in negative or cognitive symptoms, but by asking participants to evaluate whether they were able to reproduce a certain digital technique or access a particular resource on their own, they captured skill building. There were 10 instances in which participants reported a decrease in their ability to perform a skill. This decline is most likely a result of misjudging competency in a skill before it was introduced and highlights a limitation of self-report surveys. Self-report surveys are based on self-perception. Improvements captured may illustrate learning or they may illustrate an increase in self-esteem or positive thinking. We cannot determine from the self-report surveys alone what factors caused participants' perception of their skillset to change. However, both results of increased ability or greater confidence using smartphone skills are indicative of functional outcomes. The former predicts advances in daily living and the latter denotes an alleviation of negative symptoms.

4. Opportunities for tailored learning and assessment

A challenge in running DOORS, highlighted by results discussed earlier, is that diversity of smartphone literacy and functional goals – combined with heterogeneity in cognitive deficits and negative symptoms – requires varying degrees of instruction and additional support. To ensure the groups remain productive but also responsive to induvial needs, we are modifying our curriculum further to incorporate the changes below:

4.1. Office hours

Our revised lesson plan is built to inform patients about capabilities or skills we believe are most relevant to the largest number of people and will have the greatest impact. The skills we chose to emphasize – such as adding an event to iCal or checking daily step count – are largely foundational but critical for psychiatric rehabilitation. If patients can replicate these skills independently, they can build on them to utilize personally relevant smartphone features they find on their own. Some participants may benefit more from a given lesson than others due to their unique clinical needs. Moreover, some participants have already mastered many of the skills outlined in the curriculum. Distinct diagnoses and personal treatment goals, paired with different levels of comfort with technology, complicate building a lesson plan that is relevant to every member of the group.

To offer participants learning tailored to their clinical needs and comfort with technology, future DOORS curriculums will include a combination of classroom style lessons and one-on-one office hours. Individual or smaller group meetings will be offered for participants with negative symptoms so severe they are unable to learn in a large group setting. For participants with impaired cognition such that it slows their hand-eye coordination or ability focus on completing a task, office hours will simplify lessons to consist of basic phone functionalities and provide hands-on support. High-functioning participants will use one-on-one attention in office hours to review lessons they may have missed and ask questions about smartphone features the curriculum may not cover. In all cases, office hours also offers a new opportunity for DOORS leaders to more closely assess a participant's individual progress. Through looking at phone usage patterns – such as average minutes per day spent using a given app – leaders and participants can discuss how activity has or hasn't changed and set goals together to practice new skills independently. One-on-one meetings will allow leaders to better evaluate whether the skills rehearsed in lessons are leading to functional outcomes outside the program.

4.2. Observational surveys and assessments

By increasing the number of surveys participants are asked to complete, we are increasing opportunities to capture change in skill development. Administering surveys at the start and end of each lesson allowed us to identify topics or skills participants reported themselves as more capable of recalling on their own. Going forward, we can supplement surveys with objective measures by asking participants to report passive data measures each week that are available on their smartphone such as screen time and step count. The ten cases in which participants recorded decreased perception of smartphone skills after a lesson may have in part been due to over ranking sills before the lesson and then realizing this overvaluation with a resulting lower score after. A skills assessment will thus help capture actual gains. Information from skills tests and real-world phone usage data over a week between sessions may also support participant narratives describing how they have utilized a function or feature and are activating it on their own time, as well as whether their sociability has changed (Wisniewski et al., 2019). Thus, passive data may help us capture functional outcomes that result from skills learned in lessons and through group discussions.

Additionally, group leaders can complete a weekly observational survey that pertains to each participant. We often notice that participants complete activities with more ease after a class activity. An observational survey will help leaders keep track of the progress they are noticing in participants. Combined, office hours, insights from passive smartphone data, and observational surveys from leaders will create a more comprehensive and compelling account of learning and symptom alleviation, and ultimately better predict functional outcomes.

4.3. Online curriculum

In recent months, we have built an online learning platform to reach more patients and make digital skills training more accessible to them. The online platform offers the same 8-session curriculum delivered to Group 1 and Group 2 in-person but also includes short videos that teach and demonstrate smartphone skills. Sessions, and the videos and digital surveys included in each one, can be started and stopped at any time, allowing patients to engage with the platform on their own schedule and at their own speed. We believe that making DOORS available online will increase our sample size, increase survey accuracy by allowing participants to review their saved pre-survey while completing their post survey, and that it will demonstrate the program's feasibility and replicability. Interested readers can visit the DOORS website and request access to the Learning Platform here: https://skills.digitalpsych.org/.

5. Conclusion

DOORS is a low-cost program with high-impact potential that is designed to be replicated and revised so that it can reach patients with a range of clinical needs. It offers participants a community in which they can learn and practice digital skills towards improving functional outcomes. Through collaborative, interactive learning that facilitates self-reflection and responsiveness to others, DOORS addresses negative symptoms and cognitive deficiencies that can hinder recovery. DOORS equips participants with digital learning, self-confidence, and prosocial practice that combined, may lead to significant improvements in daily living.

Funding

The educational objectives of this work are generously supported by the Sidney R. Baer, Jr. Foundation.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

  1. Bromley Elizabeth, Brekke John S. Behavioral Neurobiology of Schizophrenia and Its Treatment. Springer; Berlin, Heidelberg: 2010. Assessing function and functional outcome in schizophrenia; pp. 3–21. [DOI] [PubMed] [Google Scholar]
  2. Camacho E., Hoffman L., Lagan S., Rodriguez-Villa E., Rauseo-Ricupero N., Wisniewski H.…Torous J. Technology Evaluation and Assessment Criteria for Health Apps (TEACH-Apps): Pilot Study. Journal of Medical Internet Research. 2020;22(8) doi: 10.2196/18346. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Connolly S.L., Hogan T.P., Shimada S.L., Miller C.J. Leveraging implementation science to understand factors influencing sustained use of mental health apps: a narrative review. Journal of Technology in Behavioral Science. 2020:1–13. doi: 10.1007/s41347-020-00165-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Harvey P.D., Strassnig M.T., Silberstein J. Prediction of disability in schizophrenia: symptoms, cognition, and self-assessment. Journal of Experimental Psychopathology. 2019;10(3) [Google Scholar]
  5. Hoffman L., Wisniewski H., Hays R., Henson P, Vaidyam A, Hendel V.…Torous J. Digital Opportunities for Outcomes in Recovery Services (DOORS): A Pragmatic Hands-On Group Approach Toward Increasing Digital Health and Smartphone Competencies, Autonomy, Relatedness, and alliance for Those With Serious Mental Illness. Journal of psychiatric practice. 2020;26(2):80. doi: 10.1097/PRA.0000000000000450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Kim J., Desai E., Cole M.B. Health Affairs Blog, June, 2. 2020. How the rapid shift to telehealth leaves many community health centers behind during the COVID-19 pandemic. [Google Scholar]
  7. Kopelowicz A., Liberman R.P., Zarate R. Recent advances in social skills training for schizophrenia. Schizophrenia bulletin. 2006;32(suppl_1):S12–S23. doi: 10.1093/schbul/sbl023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Liberman R.P., Wallace C.J., Blackwell G., Kopelowicz A., Vaccaro J.V., Mintz J. Skills training versus psychosocial occupational therapy for persons with persistent schizophrenia. Am. J. Psychiatr. 1998;155(8):1087–1091. doi: 10.1176/ajp.155.8.1087. [DOI] [PubMed] [Google Scholar]
  9. Mausbach B.T., Harvey P.D., Goldman S.R., Jeste D.V., Patterson T.L. Development of a brief scale of everyday functioning in persons with serious mental illness. Schizophr. Bull. 2007;33(6):1364–1372. doi: 10.1093/schbul/sbm014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Penn D.L., Kohlmaier J.R., Corrigan P.W. Interpersonal factors contributing to the stigma of schizophrenia: social skills, perceived attractiveness, and symptoms. Schizophr. Res. 2000;45(1–2):37–45. doi: 10.1016/s0920-9964(99)00213-3. [DOI] [PubMed] [Google Scholar]
  11. Rodriguez-Villa E., Rauseo-Ricupero N., Camacho E., Wisniewski H., Keshavan M., Torous J. 2020. The Digital Clinic: Implementing Technology and Augmenting Care for Mental Health. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Samuel R., Thomas E., Jacob K.S. Instrumental activities of daily living dysfunction among people with schizophrenia. Indian J. Psychol. Med. 2018;40(2):134. doi: 10.4103/IJPSYM.IJPSYM_308_17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Sarkar U., Gourley G.I., Lyles C.R., Tieu L., Clarity C., Newmark L.…Bates D.W. Usability of commercially available mobile applications for diverse patients. Journal of general internal medicine. 2016;31(12):1417–1426. doi: 10.1007/s11606-016-3771-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Świtaj P., Anczewska M., Chrostek A., Sabariego C., Cieza A., Bickenbach J., Chatterji S. Disability and schizophrenia: a systematic review of experienced psychosocial difficulties. BMC psychiatry. 2012;12(1):193. doi: 10.1186/1471-244X-12-193. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Thornton B., Faires A., Robbins M., Rollins E. The mere presence of a cell phone may be distracting: Implications for attention and task performance. Soc. Psychol. 2014;45(6):479–488. [Google Scholar]
  16. Torous J., Keshavan M. The role of social media in schizophrenia: evaluating risks, benefits, and potential. Current opinion in psychiatry. 2016;29(3):190–195. doi: 10.1097/YCO.0000000000000246. [DOI] [PubMed] [Google Scholar]
  17. Triana A.J., Gusdorf R.E., Shah K.P., Horst S.N. Telemedicine and e-Health. 2020. Technology literacy as a barrier to telehealth during COVID-19. [DOI] [PubMed] [Google Scholar]
  18. Wisniewski H., Henson P., Torous J. Using a smartphone app to identify clinically relevant behavior trends via symptom report, cognition scores, and exercise levels: a case series. Frontiers in psychiatry. 2019;10:652. doi: 10.3389/fpsyt.2019.00652. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES