Skip to main content
Psychopharmacology Bulletin logoLink to Psychopharmacology Bulletin
. 2021 Mar 16;51(2):8–19.

Development of a Patient-Centered Software System to Facilitate Effective Management of Bipolar Disorder

Charles L Bowden 1, Richard Priesmeyer 1, Mauricio Tohen 1, Vivek Singh 1, Joseph R Calabrese 1, Terry Ketter 1, Andrew Nierenberg 1, Michael E Thase 1, Gregg Siegel 1, Leslie H Siegel 1, Jim Mintz 1, Rif S El-Mallakh 1, Susan L McElroy 1, Melissa Martinez 1
PMCID: PMC8146566  PMID: 34092819

Abstract

Objective:

Self-management of bipolar disorder (BD) is an important component of treatment.

Methods:

We developed a patient-centered computational software system based on concepts from nonlinear systems (chaos) theory with mobile access to assist in managing BD known as KIOS. KIOS tracks interacting symptoms to determine theprecise state of a BD patient. Once the patient’s state is identified and the trajectory of the patient established, specific advice is generated to help manage the course of the disease. KIOS also provides analytics that can be used by clinicians and researchers to track outcomes and the course of illness. A 12-week field test was completed.

Results:

In 20 BD subjects, use of KIOS was associated with improvements in primary symptom categories of BD. Usability and generated advice were rated as a median of 6 out of a maximum of 7.

Conclusions:

The KIOS focus on change illuminates problems in the same way that humans experience them, implying that the future state will be consequent to changes made to impact the current state. Randomized clinical trial is indicated.

Keywords: bipolar, treatment, self-assessment, patient guidance, chronic disease, smart phone

Introduction

The functional impairment and associated cost of lost productivity from bipolar disorder (BD),1,2 coupled with the insufficient number of health professionals in the field and the requirements of the Affordable Care Act call for implementation of effective self-management programs.35 Although psychiatrists routinely educate patients about clinical states and treatment options, self-management can be time-consuming, complex, and overwhelming for patients.4 Recognizing changes in symptoms and subsequently taking the appropriate action is challenging for even the most conscientious and knowledgeable patients.6 The complexity inherent in BD also creates problems in the implementation of personalized self-management programs. A nine-week online BD self-management education program experienced high attrition, in part because the information provided was too general and not adapted to the specific needs of participants.7,8 We have embarked on a project to develop patient-centered software for use in a smart phone or computer to assist in effective self-management of BD. We report herein the rationale, methodology, and results of first stage and prototype developments.

BD has been studied extensively using methodologies that aggregate data and apply traditional statistical methods to determine the efficacy of treatment interventions. These static endpoints do not account for the dynamics of the symptomatology that individual patients experience. We have developed software that employs novel nonlinear methods to track changes in multiple clinical measures in BD as they evolve over time. This focus on change illuminates problems in the same way that humans experience them, implying that the future state of the patient will be consequent to changes made to impact the current state. This removes a major contributor to a patient’s expectation of persistent function-impairing illness: the sense that the disease is out of his/her control. The software is referred to as KIOS®.

Methods

Expert Panel. We selected an Expert Panel of eight psychiatrists to complete the initial aim of the project. All had participated in large, high impact longitudinal clinical studies involving BD over the previous 15 years.911 Each panelist initially worked independently to respond to identical instructions including the following: “We are attempting to identify a limited list of variables that, assessed dynamically over time, will best reflect the current clinical state of the patient’s bipolar disorder. Please select variables with regard to the extent that a change in the variable is important in determining the patient’s current clinical state.” Each panel member selected 10 variables from an initial set of over 50 items appearing in the Montgomery-Åsberg Rating Scale (MADRS), Young Mania Rating Scale (YMRS), or the more comprehensive Bipolar Inventory of Symptoms Schedule (BISS).1214 The initial list of variables also included aggregate scale scores and some lab indices (e.g. Thyroid Stimulating Hormone). Each expert was encouraged to recommend any other variables that should be included in the study.

Initial Selections. Panelists favored variables inherently experienced by patients vs. those not (e.g., reported sadness over observed sadness), variables with higher mean severity scores (energetic over hyperactive) in independent samples of bipolar patients studied in a randomized, blinded 8 month treatment study8 and studies conducted in psychometric characterization of the BISS.1416 Following selection of individual variables, panelists were asked to relate variables to at least one of the primary clinical states of BD: depression, mania/hypomania or mixed state. This step allowed us to apply quantitative data to the final sets of variables and variable pairs. We required that each variable have exploratory factor analysis loading ⩾ 0.40 on 1 of the 5 symptom domains of bipolar disorder (depression, irritability, mania, anxiety, psychosis).16 We also required that the mean severity of each variable be ⩾ the mean severity (i.e., ⩾ 1.8) of the variable in patients who were syndromally manic/hypomanic, mixed state or depressed.

Eight symptoms were selected as essential to characterizing the current state of a bipolar patient (Table 1). The symptoms, though selected independently by each expert, were highly overlapping. Two symptoms, psychomotor slowing and affective lability, are absent in the MADRS and YMRS, but present in the BISS, which we developed over the past decade as part of an NIMH Center grant.10 The BISS offers a clinician as well as a self-assessment version which was readily adaptable to KIOS.

The expert team identified the key symptoms that, collectively, define the following clinical states of a BD patient: remitted; i.e., euthymia, manic, mixed, depressed, subsyndromal manic/mixed and subsyndromal depressed. For variables that seemed too abstruse for a broad constituency we substituted less technical terms. For example, psychomotor slowing became slowed down, and affective lability became shifting emotions. Based on changes in these variables, the expert team was able to provide specific information about the state of the patient and embed KIOS with both non-pharmacological and pharmacological intervention advice for patients and practitioners. KIOS provides a brief narrative interpretation of clinical changes in illness trajectory. Because KIOS focuses on dynamic, not static information, it comports with learning processes by which individuals understand complex but common every day matters in other realms of life.

KIOS is based on a novel approach to nonlinear systems theory. The software evaluates bipolar symptoms dynamically over time to establish a behavioral trajectory, enabling individualized treatment recommendations and analyses that can yield new outcome measures. KIOS was used to assess variable characteristics in a de-identified data set from an NIMH-funded 8 month randomized study of two treatments for bipolar patients in depressive episodes.9 Rather than apply traditional statistics that aggregate the sample data, the KIOS software performed analyses on the unique trajectories of each subject. The resultant data set contained all individual variable changes for 86 subjects, consisting of 628 assessments. This analysis provided empirical evidence of each variable’s dynamic characteristics to the Expert Panel. The analyses constitute an evidenced-based screening to ensure that the variables selected by the team of experts would be adequately responsive to changes in illness state.

Subsequent to identifying and selecting the variables, the experts were able to relate variable changes to bipolar clinical states, and couple them with suggestions for managing the patients’ evolving condition. The ability to associate clinical states and intervention strategies to the dynamic states defined by the variables is paramount to the application of this technology. The process for developing first step actions for patients was straight-forward, and there was general concurrence among the experts regarding the advice to be offered.

The suggestions to the patient are generally in the 10–40 word range. The recommendations are intended to encourage a mind set and actions that are involved in mindfulness-based treatment.17 Such approaches encourage patients to remain in contact with, and cope effectively with, emotions, urgent but harmful desires (e.g., overeating), and environmental cues that provoke undesirable emotions.18 These strategies contrast with relapse prevention therapy, which focuses on avoidance based goals and actions.

Building the User Interface. The KIOS user interface is optimized for compatibility with smart phones and is also accessible via tablets and personal computers. The user-friendly navigation system provides a sliding cursor to allow the patient to input their data by tapping on a visual analog scale rather than using a numerical scale. Each variable on the scale is anchored at the two ends by “Not at all” and “Severely.” This provides the severity measure of each symptom variable and facilitates KIOS analytics dealing with ties or unclear trajectory dominance, particularly when symptom pair changes are ambiguous at face value. The color intensity of the visual analog scale rises as the severity increases. The current version of KIOS is HIPAA compliant as protections were built into the prototype as it was developed, an important distinction from the majority of patient-centered apps.19

Table 2 shows two examples of changes in Sadness and Pessimism in which both variables have changed from a previous assessment. The automatic employment of a non-parametric severity of a symptom variable, made possible by the patient’s use of a sliding cursor for each variable on a line anchored at the two ends by “None” and “Severely,” facilitates the KIOS analytics dealing with ties, or unclear trajectory dominance, particularly when symptom pair changes are ambiguous at face value. The medication and non-medication intervention examples displayed in Table 2 are representative of a broad range of the full spectrum of recommendations in the actual KIOS tool. All have been screened to be clearly understood by a broad spectrum of patients; medication advice was excluded from the Field Trial and therefore not accessible to the patients in this initial study. As previously described, the patient-friendly recommendations are based on a mindfulness approach.17,18,20 KIOS technology is capable of producing more than 43,000,000 unique patient reports, indicative of the complexity of bipolar disorder.

KIOS processes the assessment data and provides the advice from experts in report form along with graphics showing trajectories of bipolar symptoms over time. An example of symptom trajectories for ‘shifting emotions’ and ‘energetic’ is displayed in Figure 1.

Software Development. The software is accessible and compatible with smart phones, (both IOS and Android systems) and is also operational on computers and tablets if patients prefer larger screens. The ongoing work to refine the patient and clinician software can be divided into two components: (1) internal services such as data storage and network interfaces and (2) user interfaces. The basic functionality of KIOS includes an ability to accept patient data, provide patient-specific non-medication suggestions and graphic depictions of current severity and history of symptoms.

Usability. KIOS was evaluated in a 12-week, IRB approved trial of 20 BD patients to assess usability of the software, compliance and patient acceptance. Subjects were individuals diagnosed By SCID criteria with bipolar I or II disorders, all recruited from a mood and anxiety disorder clinic professionally staffed by Drs. Singh, Tohen, and Bowden. Post study, participants were asked to complete a modified IBM CSUQ (Computer System Usability Questionnaire)21 to rate usability and utility. Study participants were also asked to evaluate the graphs and advice that KIOS provides. Answers were provided on a 1 to 7 Likert-type scale. Open-ended questions were included to solicit suggestions, criticisms and comments.

Results

Seventeen of the twenty (85%) subjects completed the trial, performing at least one assessment weekly. As shown in Table 3, seven of the eight items were scored to yield more symptom decreases than increases. Only “slowed down” increased more than decreased. Variables showing the greatest improvement were, in order, shifting emotions, sadness and irritability. For shifting emotions, 37.4% of time intervals were decreases, 32.0% were increases, and 30.6% were unchanged.

Most patients rated each of the usability parameters as 6 or 7 on the 1–7 (7=best) scale (Table 4). Likewise, the benefits of KIOS and the advice it provides were received favorably. The survey results identified the graphs as an area that needs improvement, particularly with regard to making the graphs easy to understand. Most participants preferred having a weekly assessment vs. more frequent or daily interactions. Email reminders or push notifications to take the assessment were also requested by some participants.

In a short query of opinions of the subjects who tried KIOS for 12 weeks, the following are the most common free entries of the participants:

“I would like to use KIOS every time a bipolar incident occurs.”

“It was easy and simple and I liked tracking my progress.”

“I really used the advice. So much of it was spot on.”

“It told me what my results mean and what to tell my doctor.”

“It helps me look at my feelings and behavior for the week objectively, which is helpful.”

“It showed me I knew what I was doing.”

Discussion

As with any repetitive daily or weekly activity undertaken for one’s physical or emotional well-being, e.g., exercise, hair and nail care; individuals vary in the degree that they maintain participatory interest. The two strongest motivations expected to impact KIOS use are the short term benefit it can potentially contribute to one’s sense of well-being, and in the experience of being free of disincentives (e.g., a tedious interface, ambiguous recommendations, or excessive time commitments). The current phase of KIOS development includes such details in software as well as input/output procedures. Although not formally studied to this point, two linked benefits are noteworthy. First, KIOS provides a self-assessment that could be available to psychiatrists and other members of a patient’s health care team. This could strengthen meaningful long-term health care provision by providing reassurances of sustained benefit, or of early recognition of new problems. Moreover, the information is principally of the trajectory, not just the less useful cross-sectional score. Second, a major impediment to the study of chronic, waxing and waning disorders is that cost and adequate methodology are both barriers. Appropriately tested for such, KIOS could be a component of long-term self-assessment of illness/health status. This could aid outcomes, contribute to intervention studies, and yield information important for public health data.2225

The concepts and methods summarized here could be of similar utility in other diseases, both psychiatric and non-psychiatric. However, systematic attention to the unique features of the disorder and vetting of the app would require the same efforts summarized herein.

Full development and examination of the KIOS app would ultimately require a 12-month, randomized, open comparison study with a currently available web-based mood chart, such as eMoods. The study would include the aim of testing for safety, ease of use by persons who are in treatment for BD, and comparative indices of effectiveness of the two patient accessed tools. Such a study has been performed and the data are being analyzed.

Limitations

The concepts of nonlinear dynamics are not routinely utilized by clinicians, or for that matter, investigators working with BD. Other constraints stem from the difficulty of testing hypothesis when familiar concepts of linear causality and statistical correlation no longer apply. Although mathematical complexity could deter clinician and researcher interest, the clinician already faces challenging complexity in deciphering how best to treat a bipolar patient over time. Furthermore, neither the clinician, nor the patient needs to utilize mathematical knowledge for using the interface. The data set which we will accumulate will also be addressable through latent class analysis procedures. The net results can yield practical guidance through integrating several variables into a single decision tree, e.g., what treatment(s) for a bipolar mixed state are effective, well tolerated .and persisting in their benefits? The program of the app is flexible and in a state of continuous improvement.

Conclusions

Self-management of symptoms in BD is a growing desire among patients afflicted with this condition. Currently available apps leave much to be desired. The KIOS app will be able to both assess symptom change trajectory of patients, and provide practical, personalized feedback that can alter disease progression. The KIOS app is expected to be the first self-management tool to be studied in a randomized clinical trial.

Table 1. Symptoms Identified by the Expert Panel.

Sadness Less Need for Sleep
Pessimism Energetic
Slowed Down Shifting Emotions
Irritability Anxiety

Table 2. Guidance Regarding Actions Consequent to Symptom Change.

Symptom Changes Clinical State/Diagnosis What you can do/consider now Things you or a family member can do
Decrease in both Sadness and Pessimism Improvement in depression;
Subsyndromal depression
Continue with current interventions. Discuss your long term treatment strategies with your doctor, counselor, loved ones With sadness and pessimism decreasing, you are moving away from depression. Be sure to maintain a good sleep cycle to sustain your progress. This is also a good time to consider work, hobby, social, recreational, and exercise pursuits
Increase in Sadness and decrease in Pessimism Inconsistent change in level of depression;
Subsyndromal depression
Continue your mood stabilizer(s)
Continue, or start a drug for depression, particularly with energy benefits.
Consider combining mood stabilizers with each other to reduce side effects
Review your social, vocational, recreational/exercise pursuits. Are they optimal?

Table 3. KIOS Bipolar Field Trial Results.

Symptom Changes Sadness Pessimism Slowed Down Irritability Less Need For Sleep Energetic Shifting Emotions Anxiety TOTALS
Symptom Decreases 78 71 60 74 56 51 83 68 541
Symptom Increases 69 65 68 66 47 45 71 58 489
Net Improvement 9 6 −8 8 9 6 12 10 52
No Change 75 86 94 82 119 126 68 96 746
Percent Decrease 35.1% 32.0% 27.0% 33.3% 25.2% 23.0% 37.4% 30.6% 30.5%
Percent Increase 31.1% 29.3% 30.6% 29.7% 21.2% 20.3% 32.0% 26.1% 27.5%
Percent No Change 33.8% 38.7% 42.3% 36.9% 53.6% 56.8% 30.6% 43.2% 42.0%

Table 4. Usability Results and Evaluation of Features.

Question Group Median Response
User Experience 6.0
Benefits 6.0
Graphs 4.0
Advice 6.0

Figure 1.

Figure 1

An Example of Symptom Trajectories for ‘Shifting Emotions’ and ‘Energetic’ as Viewed by Users or the App

Acknowledgments

Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number R42-MH091997 to Biomedical Development Corporation. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This technology was also supported in part by an award from the Kentucky Cabinet of Economic Development, Office of Entrepreneurship, under the Grant Agreement KSTC-184-512-15-223 with the Kentucky Science and Technology Corporation.

Footnotes

Conflicts of Interest

Gregg Siegel and Leslie Siegel are shareholders of Biomedical Development Corporation. None of the other coauthors have relevant conflicts of interest to report.

References

  • 1.Ferrari AJ, Stockings E, Khoo JP, Erskine HE, Degenhardt L, Vos T, Whiteford HA. The prevalence and burden of bipolar disorder: findings from the Global Burden of Disease Study 2013. Bipolar Disord. 2016;18(5):440–450. doi: 10.1111/bdi.12423. doi: [DOI] [PubMed] [Google Scholar]
  • 2.Cloutier M, Greene M, Guerin A, Touya M, Wu E. The economic burden of bipolar I disorder in the United States in 2015. J Affect Disord. 2018;226:45–51. doi: 10.1016/j.jad.2017.09.011. doi: [DOI] [PubMed] [Google Scholar]
  • 3.Jones S, Deville M, Mayes D, Lobban F. Self-management in bipolar disorder: The story so far. J Ment Health. 2011;20(6):583–592. doi: 10.3109/09638237.2011.600786. [DOI] [PubMed] [Google Scholar]
  • 4.Nicholas J, Boydell K, Christensen H. Self-management in young adults with bipolar disorder: Strategies and challenges. J Affect Disord. 2017;209:201–208. doi: 10.1016/j.jad.2016.11.040. doi: [DOI] [PubMed] [Google Scholar]
  • 5. [16 September 2020];2019 Sep 27; https://www.fda.gov/media/80958/download U.S. Department of Health and Human Services. Policy for Device Software Functions and Mobile Medical Applications—Guidance for Industry and Food and Administration Staff. Food and Drug Administration, Center for Devices and Radiological Health. Rockville, MD. Accessed. [Google Scholar]
  • 6.Blixen C, Perzynski AT, Bukach A, Howland M, Sajatovic M. Patients’ perceptions of barriers to self-managing bipolar disorder: A qualitative study. Int J Soc Psychiatry. 2016;62(7):635–644. doi: 10.1177/0020764016666572. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Proudfoot J, Parker G, Hyett M, Manicavasagar V, Smith M, Grdovic S, Greenfield L. Next generation of self-management education: Web-based bipolar disorder program. Aust N Z J Psychiatry. 2007;41(11):903–909. doi: 10.1080/00048670701634911. [DOI] [PubMed] [Google Scholar]
  • 8.Nicholas J, Proudfoot J, Parker G, Gillis I, Burckhardt R, Manicavasagar V, Smith M. The ins and outs of an online bipolar education program: a study of program attrition. J Med Internet Res. 2010;12(5):e57. doi: 10.2196/jmir.1450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Sachs GS, Thase ME, Otto MW, Bauer M, Miklowitz D, Wisniewski SR, Lavori P, Lebowitz B, Rudorfer M, Frank E, Nierenberg AA, Fava M, Bowden C, Ketter T, Kupfer D, Rosenbaum JF, Marangell L, Calabrese J. Rationale, design, and methods of the systematic treatment enhancement program for bipolar disorder (STEP-BD) Biol Psychiatry. 2003;53(11):1028–1042. doi: 10.1016/s0006-3223(03)00165-3. [DOI] [PubMed] [Google Scholar]
  • 10.Nierenberg AA, Sylvia LG, Leon AC, Reilly-Harrington NA, Ketter TA, Calabrese JR, Thase ME, Bowden CL, Friedman ES, Ostacher MJ, Novak L, Iosefescu DV. LiTMUS Study Group. Lithium treatment—moderate dose use study (LiTMUS) for bipolar disorder: rationale and design. Clin Trials. 2009;6(6):637–648. doi: 10.1177/1740774509347399. [DOI] [PubMed] [Google Scholar]
  • 11.Nierenberg AA, Sylvia LG, Leon AC, Reilly-Harrington NA, Shesler LW, McElroy SL, Friedman ES, Thase ME, Shelton RC, Bowden CL, Tohen M, Singh V, Deckersbach T, Ketter TA, Kocsis JH, McInnis MG, Schoenfeld D, Bobo WV, Calabrese JR. Bipolar CHOICE Study Group Clinical and Health Outcomes Initiative in Comparative Effectiveness for Bipolar Disorder (Bipolar CHOICE): a pragmatic trial of complex treatment for a complex disorder. Clin Trials. 2014;11(1):114–127. doi: 10.1177/1740774513512184. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Young RC, Biggs JT, Ziegler VE, Meyer DA. A rating scale for mania: Reliability, validity and sensitivity. Br J Psychiatry. 1978;133:429–435. doi: 10.1192/bjp.133.5.429. [DOI] [PubMed] [Google Scholar]
  • 13.Montgomery SA, Åsberg M. A new depression scale designed to be sensitive to change. Br J Psychiatry. 1979;134:382–389. doi: 10.1192/bjp.134.4.382. [DOI] [PubMed] [Google Scholar]
  • 14.Bowden CL, Singh V, Thompson P, Gonzalez JM, Katz MM, Dahl M, Prihoda TJ, Chang X. Development of the bipolar inventory of symptoms scale. Acta Psychiatr Scand. 2007;116(3):189–194. doi: 10.1111/j.1600-0447.2006.00955.x. [DOI] [PubMed] [Google Scholar]
  • 15.Gonzalez JM, Bowden CL, Katz MM, Thompson P, Singh V, Prihoda TJ, Dahl M. Development of the Bipolar Inventory of Symptoms Scale: Concurrent validity, discriminant validity and retest reliability. Int J Method Psychiatr Res. 2008;17(4):198–209. doi: 10.1002/mpr.262. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Thompson PM, Gonzalez JM, Singh V, Schoolfield JD, Katz MM, Bowden CL. Principal domains of behavioral psychopathology identified by the Bipolar Inventory of Signs and Symptoms Scale (BISS) Psychiatry Res. 2010;175(3):221–226. doi: 10.1016/j.psychres.2009.04.017. [DOI] [PubMed] [Google Scholar]
  • 17.Ives-Deliperi VL, Howells F, Stein DJ, Meintjes EM, Horn N. The effects of mindfulness-based cognitive therapy in patients with bipolar disorder: a controlled functional MRI investigation. J Affect Disord. 2013;150(3):1152–1157. doi: 10.1016/j.jad.2013.05.074. [DOI] [PubMed] [Google Scholar]
  • 18.Howells FM, Rauch HGL, Ives-Deliperi VL, Horn NR, Stein DJ. Mindfulness based cognitive therapy may improve emotional processing in bipolar disorder: Pilot ERP and HRV study. Metab Brain Dis. 2014;29(2):367–375. doi: 10.1007/s11011-013-9462-7. [DOI] [PubMed] [Google Scholar]
  • 19.Nicholas J, Larsen ME, Proudfoot J, Christensen H. Mobile apps for bipolar disorder: A systematic review of features and content quality. J Med Internet Res. 2015;17(8):e198. doi: 10.2196/jmir.4581. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Bowen S, Witkiewitz K, Clifasefi SL, Grow J, Chawla N, Hsu SH, Carroll HA, Harrop E, Collins SE, Lustyk MK, Larimer ME. Relative efficacy of mindfulness-based relapse prevention, standard relapse prevention, and treatment as usual for substance use disorders: a randomized clinical trial. JAMA Psychiatry. 2014;71(5):547–556. doi: 10.1001/jamapsychiatry.2013.4546. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Lewis JR. IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. [16 September 2020];Int J Hum-Computer Interaction. 1995 7(1):57–78. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.584.6610&rep=rep1&type=pdf Accessed . [Google Scholar]
  • 22.Nielsen J. Usability Engineering. Kaufmann Morgan., editor. 1993:309. San Francisco. pp. [Google Scholar]
  • 23.Holtzblatt K, Beyer HR. Contextual Design (Chapter 8). In: The Encyclopedia of Human-Computer Interaction. [16 September 2020];2013 2nd Edition. (Edited by: Interaction Design Foundation). (Published Online First). https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/contextual-design. Accessed . [Google Scholar]
  • 24. [16 September 2020];2002 Jan 11; https://www.fda.gov/media/73141/download U.S. Department of Health and Human Services. General Principles of Software Validation—Final Guidance for Industry and FDA Staff. Food and Drug Administration, Center for Devices and Radiological Health, Rockville, MD. Accessed. [Google Scholar]
  • 25. [16 September 2020];2018 Oct 18; U.S. Depatment of Health and Human Services. Content of Premarket Submissions for Management of Cybersecurity in Medical Devices—Draft Guidance for Industry and Food and Drug Administration Staff. Food and Drug Administration, Center for Devices and Radiological Health, Rockville, MD. https://www.fda.gov/media/119933/download. Accessed . [Google Scholar]

Articles from Psychopharmacology Bulletin are provided here courtesy of MedWorks Media Inc.

RESOURCES