Sensor, Wearable, and Remote Patient Monitoring Competencies for Clinical Care and Training: Scoping Review

Donald M Hilty; Christina M Armstrong; Amanda Edwards-Stewart; Melanie T Gentry; David D Luxton; Elizabeth A Krupinski

doi:10.1007/s41347-020-00190-3

. 2021 Jan 22;6(2):252–277. doi: 10.1007/s41347-020-00190-3

Sensor, Wearable, and Remote Patient Monitoring Competencies for Clinical Care and Training: Scoping Review

Donald M Hilty ^1,^✉, Christina M Armstrong ², Amanda Edwards-Stewart ³, Melanie T Gentry ⁴, David D Luxton ⁵, Elizabeth A Krupinski ⁶

PMCID: PMC7819828 PMID: 33501372

Abstract

Sensor, wearable, and remote patient monitoring technologies are typically used in conjunction with video and/or in-person care for a variety of interventions and care outcomes. This scoping review identifies clinical skills (i.e., competencies) needed to ensure quality care and approaches for organizations to implement and evaluate these technologies. The literature search focused on four concept areas: (1) competencies; (2) sensors, wearables, and remote patient monitoring; (3) mobile, asynchronous, and synchronous technologies; and (4) behavioral health. From 2846 potential references, two authors assessed abstracts for 2828 and, full text for 521, with 111 papers directly relevant to the concept areas. These new technologies integrate health, lifestyle, and clinical care, and they contextually change the culture of care and training—with more time for engagement, continuity of experience, and dynamic data for decision-making for both patients and clinicians. This poses challenges for users (e.g., keeping up, education/training, skills) and healthcare organizations. Based on the clinical studies and informed by clinical informatics, video, social media, and mobile health, a framework of competencies is proposed with three learner levels (novice/advanced beginner, competent/proficient, advanced/expert). Examples are provided to apply the competencies to care, and suggestions are offered on curricular methodologies, faculty development, and institutional practices (e-culture, professionalism, change). Some academic health centers and health systems may naturally assume that clinicians and systems are adapting, but clinical, technological, and administrative workflow—much less skill development—lags. Competencies need to be discrete, measurable, implemented, and evaluated to ensure the quality of care and integrate missions.

Keywords: Competencies, Monitoring, Training, Sensor, Wearable, Mobile health, Education, Implementation

Introduction

The number of connected wearable devices worldwide is expected to increase from 325 million in 2016 to 929 million by 2021, with 1 in 6 Americans using a wearable by 2021 (Piwek et al. 2016; Statistica 2018). Sensor, wearable, and remote patient monitoring (RPM) technologies are not only being integrated into care but are also increasingly used as the primary assessment and/or treatment modality (Hermens and Vollenbroek-Hutton 2008; Kvedar et al. 2014). Examples of their use include the collection of behavioral data, monitoring symptoms and adherence, and providing preventive and therapeutic interventions (Hilty et al. 2020; Luxton 2016; Silva et al. 2015; Steinhubl et al. 2013), making care more person- and patient-centered.

Sensor, wearable, and RPM technologies can transform care by moving from cross-sectional, manual transfer of data at a healthcare appointment to a 24 × 7, longitudinal framework (Areàn et al. 2016; Luxton 2016; Torous and Roberts 2017; Ariga et al. 2019). Enabling technologies, including cloud computing, ambient sensors, and artificial intelligence (AI) (e.g., machine learning (ML)), make the collection, processing, and sharing of data far more integrated and provides the opportunity for real-time feedback based on the ecology (home, health, lifestyle, social) of patients in natural settings (Seko et al. 2014). These data support “in-time” clinical decision support (Rohani et al. 2018; CDS) (Greenes et al. 2018) and automatic monitoring systems (Garcia-Ceja et al. 2018; National Academy of Sciences, Engineering and Medicine 2020). For example, digital phenotyping or behavioral markers can correlate multimodal sensor data, cognitions and depressive mood symptoms (Mohr et al. 2017) to detect illness and/or predict relapses.

The COVID-19 pandemic forced healthcare providers and systems to rapidly adapt their way of providing services. This also necessitated the need for rapidly attainment of technological capabilities and practical skills to assure competent, legal, and ethical practice. Fortunately, competencies are identified and training and knowledge resources for video-based telehealth services were well established and are generally accessible. However, training and competency resources for wearables, sensors, and remote monitoring technologies are presently limited, and the existing literature mostly describe methods, interventions, technologies, and care outcomes, rather than required clinical skills to use technologies (Hilty et al. 2020). Even clinicians familiar with telehealth may not see that these technologies are part of care, struggle to use them, or fail to realize their limitations (Hilty et al. 2018; Hilty et al. 2019a, b; Hilty et al. 2020). Graduate education, graduate education councils, and professional organizations/boards are providing guidance on use of technology but have not yet put forward competencies to ensure quality care for sensor, wearable, and RPM (Hilty et al. 2017; Maheu et al. 2019).

Given the increasing capabilities and benefits of sensors, wearables, and RPM and the current lack of identified competencies regarding use in clinical care, we aim to identify essential clinical skills and organize them into competencies within an educational theory, implementation science, and evaluation practices framework. We also consider that, like clinical interventions, competency implementation and effectiveness depend on assessment of acceptability, adoption, feasibility, costs, and sustainability (Proctor et al. 2010; Curran et al. 2012; Gargon et al. 2019; Marcolino et al. 2019). Identified and clarified competencies will help healthcare providers and systems adapt to and integrate these technologies into workflow and advance person—and patient-centered care.

This scoping review does not specifically address accreditation, licensing, and other regulatory boards/agencies but briefly summarizes position statements and guidelines on the integration of technology in clinical settings.

Methods

Approach

The original six-stage process (Arksey and O’Malley 2005) and updated modifications (Levac et al. 2010) for scoping reviews were used.

Research Question

This review addresses the overarching question: “What are the skills, attitudes, and knowledge that trainees and clinicians need to incorporate sensors, wearables and RPM into clinical practice?” It synthesizes clinical data based on research findings and considers current and target (i.e., future/ideal) states for those using sensors and wearable technologies.

Identifying Relevant Studies

The literature key word search was conducted for articles published through May 2020 using PubMed/Medline, American Psychological Association (APA) PsycNET, Cochrane, Embase, PsycINFO, Web of Science and Scopus, Science Citation Index, Social Sciences Citation Index, Telemedicine Information Exchange database, Centre for Reviews and Dissemination, and The Cochrane Library Controlled Trial Registry database. The initial search targeted four concept areas and associated search terms:

Competencies (adult, assessment, attitude, behavior, cognition, cognitive, curriculum(a), education(al), evaluation, framework, goals, graduate, improvement, knowledge, learner, learning, literacy, objectives, pedagogy, postgraduate, process, quality, resident, skill, student, supervision, teaching, trainee, training, milestones)
Sensor, wearable, and remote monitoring (actigraphy, activity, assessment, data, diagnosis, ecological, geographical, global, GPS, momentary, monitoring, passive, remote, storage, telemonitoring)
Mobile (android, app, asynchronous, cellular, device, e, ehealth, ebehavioral, e-behavioral, e-mental, health, mhealth, m-health, mobile, phone, smartphone, tablet, text), asynchronous (app, device, eConsult, e-consult, e-mail, mobile, monitoring, sensor, social media, store-and-forward, technology, text, wearable) and synchronous telepsychiatry, telebehavioral or telemental health (video, web-based, Internet)
Behavioral health (BH), psychiatry, and psychology (behavioral, client, clinician, care, diagnosis, emotional, health, intervention, medicine, mental, patient, services, psychiatry(ic), psychology(ical), therapist, treatment)

Study Selection

Two authors (DH, CA) searched then screened titles and abstracts for relevant studies. A third reviewer (EK) determined outcomes if disagreements occurred. Full-text articles were reviewed for final inclusion based on the key word search. Reviewers met at the beginning, midpoint, and final stages of review to discuss challenges and uncertainties and refine the search strategy. The flow diagram is in Fig. 1. Inclusion criteria were the terms in combination from the four concept areas: 1 and 2; 1, 2, and 4; or 1, 3, and 4. Exclusion criteria included the following: terms out of context (e.g., abstract use of competency, not mentioning a single skill, non-measurable behavior); data unrelated to BH; not applicable to clinical care; wrong report type (e.g., abstract, editorial, column, literature review); models, principles, and concepts (e.g., cognitive, milestones, patient) with no educational outcomes; non-English language; and other.

Fig. 1 — Search flowchart: diagram of papers reviewed, excluded, and selected

Data Charting

A data charting form was used to extract data, and notes were organized consistent with a narrative review or descriptive analytical method by each reviewer. The reviewers compared and consolidated information using a qualitative content analysis approach. The information was shared with selected experts (see below), their input summarized, and themes extracted.

Analysis, Reporting, and the Meaning of Findings

This phase organized results in a table, with skills outlined and parsed. There were so few papers that findings were reported individually since the depth of existing research was less than expected. Therefore, the reporting used excerpts from published framework tables. The Accreditation Council of Graduate Medical Education (ACGME) Milestone framework was selected as it is well-known across medicine (ACGME 2013). Its six domains are patient care, medical knowledge, system-based practice, professionalism, practice-based learning and improvement, and interpersonal and communication skills. Two authors (DH, CA) examined the ACGME milestones from addiction medicine, clinical informatics, family medicine, internal medicine, psychiatry, and surgery, the CanMEDs roles for physicians (Royal College of Physicians and Surgeons Canada 2015), and the Coalition of Technology in Behavioral Science (CTiBS) telebehavioral health competency framework (Maheu et al. 2018) to build the current competency framework (Hilty et al. 2018). The content was organized into a 3-level learner system (i.e., novice/advanced beginner, competent/proficient, and advanced/expert) (Dreyfus and Dreyfus 1980; ACGME 2013; Hilty et al. 2015).

Expert Opinions

Expert opinions were solicited to review preliminary findings and suggest additional steps for improvement. A list of relevant experts was compiled from (1) recent regional and national presentations (e.g., American Telemedicine Association (ATA), American Psychiatric Association, American Association of Directors of Psychiatry Residency Training), (2) technology-related special interest groups of organizations, (3) national behavioral health organizations (e.g., CTiBS), and (4) authors of recent articles. The goal was to have representatives with mobile health and informatics expertise in behavioral analysis, counseling, couples/marriage/family, psychiatry/medicine, psychology, social work, and substance use professions.

Results

Expert Feedback

Fifty-two experts were invited from the Department of Veterans Affairs, Department of Defense, and academic institutions across four countries (Canada, India, Italy, USA). Disciplines included 29 (55.8%) psychiatrists, 16 (30.8%) psychologists, 3 (5.8%) engineers, 1 (1.9%) social worker, 1 (1.9%) occupational therapist, 1 (1.9%) dietitian, and 1 (1.9%) primary care physician.

Of the 52 experts invited, 26 responded and 24 (46.1%) participated, with 11 (45.8%) via email, 9 (37.5%) via 1 of 4 videoconference sessions, and 4 (16.7%) used both options. Feedback was collated based on previous work using consensus and modified Delphi processes (Hilty et al. 2018, Maheu et al. 2018). The 24 experts were asked to complete a brief 6-item Likert-scale survey and 15 (62.5%) responded. Results showed that the majority agreed or strongly agreed that:

“Table 1 was helpful in understanding the scope of what is currently being used” (100%).
“The competencies provided in Table 3 are in the ballpark and relatively ‘complete’” (85.7%).
3 “The length of Table 3 is good (not too long, not too short)” (78.6%).
“The subdomains in Table 3 are helpful” (92.9%).
“The behaviors/skills in Table 3 are learnable, teachable and measurable” (100%).
“The level of clarity and detail in Table 3 is adequate (not too much, not too little)” (87.6%).

Table 1.

Sensor, wearable, and remote patient monitoring technologies used for mobile health interventions

Hardware (devices)	Device analytics
Blood pressure cuff	Usage, duration, frequency, quality of data transfer
Flexible sensors (i.e., patch, printed technology)	Usage, duration, frequency, quality of data transfer
Headband/headset	Usage, duration, frequency, quality of data transfer
Holter monitor	Usage, duration, frequency, quality of data transfer
Pulse oximeter	Usage, duration, frequency, quality of data transfer
Smartphone	Usage, call logs: no., duration, frequency, missed SMS text logs: no. of frequency, characters
Textiles with imbedded sensors	Usage, duration, frequency, quality of data transfer
Wristband (i.e., smartwatch, band)	Usage, duration, frequency, quality of data transfer
Software (apps)	App analytics
Mobile app	Downloads, active users, sessions, average visit time, average screen views per visit, app retention (# return users), app churn (%age of users that stop using), app event tracking (actions users take in app (i.e., clicks, purchase), social media tracking
Sensors	Sensor analytics
Accelerometer	Autocorrelation, distance, speed, stillness/inactivity, time periods, vigorous activity, sleep sensor analytics
Actigraph	Physical movement, rest-activity cycles, circadian-rhythm activity cycles, steps
Air resistance sensor	Movement in wind field
Barometer	Air pressure
Barometric altimeter	Air pressure and altitude
Bluetooth	Nearby Bluetooth capable devices
Blood pressure monitor	Systolic and diastolic blood pressure
Camera	Eye gaze, light
Electrocardiogram (ECG)	Heart rate activity, heart rate variability
Electrodermal sensor	Skin conductance, galvanic skin response, psychogalvanic reflex, sympathetic skin response
Electroencephalogram (EEG)	Brain activity
Electroglottogram (EGG)	Vibration of vocal folds during voice production
Electromyogram (EMG)	Response of muscle against nerve stimulation
Electrooculography (EOG)	Drifting eye movements
Esophageal pH monitor	Frequency of stomach acid in gastrointestinal tract
Fingerprint	Distances and pattern between ridges on finger surface for biometric user verification
Global Positioning System (GPS)	Location, circadian rhythm, entropy, location clusters, movement duration, speed, proximity
Glucometer	Blood glucose
Gyroscope	Device’s rate of rotation
Heart rate monitor	Heart rate based on the difference between light intensity between blood vessel pulsations using LED light
Incident detection	Sudden deceleration
Infrared light	Distance/depth of objects in a spatial area
Light Detection and Ranging Sensor (LiDAR)	Distance/depth of objects in a spatial area
Light sensor	Ambient and direct light
Magnetometer	Geomagnetic field strength
Microphone	Loudness of sound, speech recognition, natural language processing, ambient sounds
Pedometer	Steps, sedentary activity
Photoplethysmogram (PPG)	Perfusion of blood to dermis and subcutaneous tissues of skin, often measured with pulse oximeter
Proximity sensor	Detection of any close object using infrared LED and light detector
Spirometer	Lung ventilation
Tactile sensor	Touch pressure and mapping
Thermometer	Skin temperature, ambient temperature
Time-of-flight (ToF) camera	Distance/depth of objects in a spatial area
Wi-Fi	Location based on service-set identifier (SSID) and signal strength of Wi-Fi networks

Open in a new tab

Table 3.

Sensor, wearable, and remote monitoring competencies for clinicians and trainees organized by accreditation council of graduate medical education domains

Area/topic

Novice/advanced beginner (ACGME milestone levels 1–2)

Competent/proficient
(ACGME milestone levels 3–4)

Advanced/expert (ACGME milestone level 5)

Patient care

Patient assessment/evaluation

Change patient care approach to gather information, develop differential diagnosis and workup common conditions across populations, with supervision

Educate patients on technology and inquire about preferences for use

Screen use of mobile phone, apps, and for what (e.g., exercise, music, health care)

Screen for ability to text, install/ update app and use software

Ask for advice/consultation if concerns arise (e.g., data security) and document

Change patient care approach to develop the diagnostic assessment, workup, and treatment plan independently, including secondary data, for common and complex conditions

Evaluate digital literacy and interest in app, text (i.e., generic, motivational or tailored) or wearable

Screen use of apps, Bluetooth, wearable (e.g., activity tracker, Holter monitor sensor-imbedded textile) and for what (e.g., glucometer for diabetes)

Assess quality of current data via self-report, manual or other process on symptom (i.e., mood, activity, weight)

Develop/adapt clinical pathways, guidelines and procedures for management of complex conditions for all points-of-service

Use/teach/develop evidence-based strategies for clinical, technical, and administrative tasks

• Consent, policy, and procedures

• Populations and settings

• Medico-legal considerations

Train/supervise/consult to optimize assessment and flexibility (e.g., activity option to detect depression and/or mania; light, sound, proximity or physiological options)

Engagement and interpersonal skills

Inquire about experience, comfort, and trust with technology

Discuss impact on relationships with others, professional life and health care

Field questions and requests for use and respond in time (e.g., video, in-person) or asynchronously, with supervision if needed

Learn about informational, educational, motivational, tailored and other feeds to patients—and directionality—by discussing with patients, supervision, and if possible, by receiving examples as if a patient

Encourage reflection on pros/cons (i.e., privacy) and purchases (e.g., compatibility)

Gauge interest and capacity for learning to incorporate data/feedback, making a behavior change, responding to clinician/team, sharing new data and responding to clinical recommendations made based on that and other in time data

Assess/weigh benefits (e.g., interaction, engagement, change) with concerns (e.g., privacy, being monitored)

Assess impact on therapeutic relationship (e.g., communication, intimacy, boundaries

Adjust approach to express empathy) and compare to other technologies and in-person care

Provide guidance to patient, family, clinician and teams on effective communication

Instruct on best ways to use technology

• Simplicity with purpose

• Match use with goal/intention and users’ capacity

• Costs for effective use: time, technology, learning, teaching

• Compare technologies’ impact on relationships

Discuss/clarify expectations of participants

Determine/teach alliance-building practices, prevention of problems and responses to barriers

Management and treatment planning

Monitor/adjust ongoing use for issues (e.g., consider speech processing if physical problem)

Make clear communications (e.g., text, notifications, symbols)

Pilot/monitor/evaluate a technology for a specific purpose

• Use supervision to adhere to health system policies and inquire about data feed to EHR

• Clarify when data is reviewed “outside” regular appointments to set appropriate expectations

Assess comfort with wearable in daily life (e.g., ease, fit with contact/water sports)

Select option based on patient preference, feasibility and purpose

• Sensor to monitor mood, or

• Sensor, app and text reminders to self-report data in time (i.e., EMA) by structured (e.g., 9 am), event (e.g., pain) or signal (i.e., random) prompts

Survey benefits and satisfaction of use and modify approach if necessary to match treatment objectives with patient preference and capacity

Tailor recommendations to resources, culture and patient preference

Research/disseminate procedures to prevent problems and manage clinical and administrative issues

Offer practical, proven options to add

Liaison between clinicians, informatics and system administrators

• Improved workflow

• Trouble-shoot and plan rollouts

Teach best practices (e.g., evidence-based sensor or wearable within an evidence-based approach)

Administration and medico-legal issues: privacy, confidentiality, safety, data protection/integrity, and security

Adhere to clinic, health system and professional requirements for in-person care

Identify and adhere to relevant laws and regulations in the practice jurisdiction(s) of patient

Clarify public or private access related to communication (e.g., email within EHR not non-HIPAA compliant systems)

Seek supervision/advice, if needed, to learn the system’s path for use of a technology

Synthesize applicable ethical guidelines

Identify and articulate security and privacy concerns and solutions related to using technology in clinical practice

Use standard language for these technologies for consent form, treatment plan and progress notes

Identify, and support others to navigate, options for seeking technical help (e.g., mobile phone company, IT system)

Verify identity and monitor security, privacy and confidentiality

Apply in-person relevant laws and regulations in any/all jurisdiction(s) to these technologies and adjust clinical care

Promote trust and responsiveness

Teach/consult on in-person laws and regulations applied to technologies

Develop legal, billing and regulatory strategies (e.g., emergencies)

Update and consult with regulatory boards and health authorities

Instruct on technology related to documentation, privacy, billing and reimbursement

Develop standard language for these technologies for consent form, treatment plan and progress notes, particularly for non-routine telepractice; seek consultation

Procedures

Encourage teaching demonstrations

Verify technology type and fit with purpose used

Ask for advice/consultation on current and potential options

Check settings of software

Identify how/if the integration of health technology may modify workflow and customize approach to maximize efficiency

Enhance continuum of automated data

Prevent, identify and manage obstacles

Help patient and others obtain assistance

Implement and validate options

Customize technology options into workflow (e.g., EMA)

Research/teach/consult on approaches for clinical quality (e.g., hard/software; accessories)

Serve as primary assessor of options

Decision-making, synthesis, and information systems (IS)

Effectively locate regular and wearable data in EHR

Consider CDS tools of CDSS (e.g., for disease management)

Identify basic principles of effective DS and decision science

Demonstrate familiarity with how informatics tools coordinate, integrate and document care

Prioritize EHR–compatible options to facilitate integration of information

Adjust parameter(s) for decision-making and documentation using wearable data

Help patients, learners and staff use decision support tools and customize when possible (e.g., mood and activity trigger for mania)

Weigh accelerometer, actigraphy, GPS and sensor subtype (e.g., light, sound) options

Teach how to use pre- and intra-platform data feeds into EHR

Implement/evaluate patient (e.g., apps, messaging, symptom monitoring) and clinician (e.g., tools, algorithms) tools with clinical informatics

Offer regular (e.g., generic/all patients) and customized (e.g., specific to stage, demographic, treatment plan) tools

Interpersonal and communication skills

Therapeutic relationship with patients and families

Create non-judgmental, safe space

Listen, ask questions, and organize information for patients

Ask advice/consultation to improve care and resolve problems

Build rapport that fosters trust, respect and understanding, including discussing problems and managing conflict

Identify physical, cultural, psychological and social barriers to communication

Role model effective, continuous, personal and professional relationships

Coach flexibility/coordinate input to identify and resolve conflict

Educate/consult on system workflow

Patient-centered care

Ask about goals and preferences

Identify positive and negative trends in patient populations using technology (e.g., generation Y)

Develop skill with text and e-mail and avoid technical jargon

Be of service to clarify technology questions

Engage patient perspective and clarify goals, values, preferences and expectations to foster shared decision-making

Ensure timely documentation, keeping of databases (e.g., problem list, registries)

Seek family, cultural, language and technology consultants for advice

Improve systems and communication with all care participants

Develop policies/adapts “best practices” based on aggregate data

Interprofessional education (IPE) and team-based care (TBC) (e.g., e-consultation)

Demonstrate the importance of the team for care and learning

Define TBC (i.e., coordination, collaboration, teamwork) and share information

Ask advice/consultation on conflicts and communication problems

Build, re-design and lead team in to share data, share decisions and provide care (i.e., practice/lead TBC via wearables)

Present, document and communicate in a clear, concise and organized manner

Evaluate if/how communication and workflow are affected and adjust plans

Teach/consult to IPE TBC via technology in terms of options, roles, practices, and communication

Help others manage uncertainty/conflict on virtual care and communication

Keep focus on patient during team, technology and system dilemmas

Cultural, diversity, and social determinants of health; attend to language issues

Identify technology utilization patterns across demographic variables (age, ethnicity, race, national origin, literacy level, disability and identify inaccurate biases that are not supported by data

Promote reflection and awareness how cultural, diversity issues, and social determinants of health might impact readiness to adopt, availability of, and access to health technology

Identify potential biases regarding use and how these could impact delivery of care

Utilize a cultural competency framework to understand how experiences with technology may impact integration into care

Utilize strategies to increase technological cultural competency and decrease biases and/or stereotypes

Determine patient regarding readiness of use, and if/how culture impacts use

Adjust to preferences based on patient need/interest

Observe, adjust and manage language and communication issues (e.g., emoticon use)

Instruct/use components of a cultural formulation interview, if applicable

Instruct on utilization of cultural competency frameworks to shift from ethnocentric to ethno-relative perceptive for use of health technology Instruct on how to adapt assessment and management approaches according to differences

Medical knowledge

Clinical care as it relates to the use of technology

Learn

• Benefits and barriers of the use of a given health technology

• Diseases treated with wearables (e.g., depression, diabetes)

Ask for advice/consultation for resources to learn

Assess what works and does not, based on patient, technical and other factors

Anticipate clinical requests/dilemmas

• If data feed to patient, consider the type, frequency, formatting and “quantity”

• Develop plan to review data on/off hours, make decision and communicate

Research/teach new ways to facilitate/improve care (e.g., AI, machine learning, algorithms)

Teach/consult on patients/diagnoses that require special preparation

Evaluate need for 24-h, 7-day data review and decision-making

Fundamentals of technology and IS

Understand basics of components and IS (e.g., network, Internet, hardware, software, CPOE) workflow

Understand complex components and healthcare IS (e.g., EHR, algorithms, databases)

Reads on wearable, IS and human factors to answer questions and teach others

Research/teach/consult enterprise-wide architecture, system integration

Teach on input, process, output, feedback and control workflows

Definitions of technology

Define SP/device and apps and list some pros/cons

Understand “native app” on a SP/device versus others

Recognize wearable components

Define components (e.g., wearable, IS) and educate others on information and data flow from patient to clinician

Teach/consult on multiple SP/devices platforms, app varieties, and selection

Evidence-base (e.g., app)

Identify the level of evidence-base for the general technology category in improving health outcomes (i.e., Holter monitor in chronic heart disease)

Describe the level of available evidence for the specific sensor/wearable technology to colleagues and patients

• Content within the technology (interventions, algorithms, etc.)

• Results of objective evaluations (i.e., published studies, quality of data)

• Evaluate and employ the evidence base available in evidence-based treatment

Instruct colleagues on best practices (e.g., guidelines) and approaches in the evaluation of the evidence-based for sensor/wearable technology

Adapt/translate best practices and approaches from other health and behavioral health professions

Problem-solving and prevention

Recognize and report problems

Ask for advice/consultation

Assess user requirements for fit, diagnose problems and provide technical assistance

Disseminate steps, approaches and/or techniques identify/solve problems

Decision support (DS)

Understands how to help patient learn and for decision-making

Pilot tools with supervision

Use and evaluate ability of mH, SP/device and/or apps to help decision-making

Compare manual and technology options

Teach steps (i.e., gather information, weigh data and alternatives)

Use principles and provide resources

Systems-based practice

Patient safety and quality improvement (QI)

Learn safety/risk issues (e.g., physical, privacy)

Use reporting systems for problems

Discuss possible quality gaps for care delivery with supervisors

Participate in chart review, case conference and other activities

Outline factors/causal chains contributing to quality gaps, with management

Identify problems/unintended consequences

Recognize system error and activates others

Participate in conferences on errors and uses structured tools (e.g., checklists, hand-off procedures) to prevent adverse events

Match clinical workflow with IS and technology processes

Analyze/teach aggregate data, root cause analysis, and change options

Teach/consults on how to analyze, select and evaluate QI options (e.g., quality measures, error prevention)

Resource utilization

Recognize need for efficient and equitable use of resources

Inquire about providing options to use short- and/or long-term

Recognize disparities and know the relative cost (e.g., technology, procedures) versus level of care/benefit to individuals, system and society

Teach cost-effective, high-value care, using tools and IT for decision making

Weigh current and future state quality of care (and life), as well as costs

Role in system navigation, workflow, and data

Communicate data and workflow, via EHR to others for assistance

Identify key elements for effective transitions across settings

Participate in a workflow analysis

Coordinate/lead complex transitions

Provide patient and system consultations

Use sensors/wearables to enhance continuity

Interpret analyses/devise solutions

Role model, advocate and improve transitions across systems

Coach others to anticipate future needs

Practice-based learning

Evidence-base of a technology

Access literature and learn/apply evidence-base specific to a technology intervention

Demonstrate an understanding of the benefits and barriers of health technology to support health outcomes

Evaluate and analyze the evidence-base and employ best current state practices

Utilize best practices in interpretation of specific data (e.g., screen activity duration and physical activity, which predict depression severity better than distance travelled or number of SMS text messages) to support treatment goals

Interpret/teach on disparate data, evidence and healthcare settings

Evaluate options for multiple purposes and teach on specificity of data

Reflection for practice improvement (PI)

Evaluate regular/ manual versus technology-based care; improve performance with help from others and technology

Ensure quality care by adjusting options

Reflect consistently and integrate performance feedback (e.g., audit), peer input and practice/technology data

Teach/consult on practice improvement standards and evaluation methods

Consult/mentor others gaps between performance and expected practice

Learning practices

Add technology-based learning and use wearable to empathize

Seek out technology-specific education

Develop/promote/role model attitudes

Determine best contexts/venues to emphasize technology resources

Professionalism

Accountability, ethics, and responsibility

Quantify the impact of technology-based fatigue/stress

Adhere to core professional and governmental guidelines for conduct (e.g., boundaries, being sensitive to having data around the clock across life settings/activities)

Perform timely patient care (i.e., response to text within business hours, planning for off hours; reviewing data; documentation)

Ask for advice/consultation to clarify and manage problems

Reflect on well-being of self/others and role model healthy habits; inquire about technology fatigue

Reflect on and follow institutional policies for conduct (e.g., texting or other asynchronous communication, use of data and its transfer)

Perform timely patient care (e.g., analyzing in time data regularly rather than waiting for the next scheduled appointment; being responsive to emergent symptoms; administrative tasks and responsibilities

Practice within scope and educate others

Promote well-being among trainees, clinicians and IPE team members (e.g., offer sensors to monitor employee fitness, wellness and job performance)

Role model/teach/consult ethical issues to maintain professional identity

Monitor/report/provide feedback on technology errors/lapses

Assess/advise if clinical responsibilities shift as care model changes, to ensure practice within license and regulations (e.g., state, federal)

Reflection, feedback, and growth

Demonstrate openness to feedback, both regular and unsolicited

Join professional community for peer support and input to embrace technology

Learn from/participate in global evaluations to increase perspective

Participate/lead activities in professional societies, patient advocacy groups and community service organizations

Educate/evaluate how technology affects communication and care quality

Seek 360-degree evaluation, performance audit and other methodologies

Compare/contrast information across professions for growth opportunities

Provide leadership to colleagues on organizational policy or curricula

Promote technology as an opportunity for academic missions (e.g., informatics, competencies)

Scope of practice and therapeutic objective(s)

Attend to in-person scope and learn with supervision how a technology may change it and responsibilities

Demonstrate flexibility and openness to learning, but keep focus on shared primary objective of care

Apply in-person and video principles of scope to other technologies to anticipate and prevent problems

Educate patient on reasonable/acceptable options and the direct/indirect potential risks of its use

Teach/consult on in-person and technology adaptations

Evaluate high-risk technologies and use in complex populations, particularly with regard to legal, regulatory and fiscal issues

Open in a new tab

app application, AI artificial intelligence, CDS clinical decision support, CDSS clinical decision support system, CPOE computerized provider order entry, EMA ecological momentary assessment, EHR electronic health record, ED emergency department, IS information system, IT information technology, mH mobile health, PCP primary care provider, SP smartphone

The abstract, tables, and manuscript were edited based on the feedback.

Literature

Figure 1 shows that from a total of 2846 potential references, two authors (DH, CA) found 2828 eligible for title/abstract review, and 521 for full text review. The majority of studies were excluded due to terms used out of review context and reviewing references revealed more papers. The final number of papers included in the review was 111 (highlighted in grey in References).

Sensor, Wearable, and Remote Patient Monitoring Technologies Evidence-Based

Technologies

A variety of technologies were used for mood and anxiety disorders (Hilty et al. 2020), with smartphones (66.3%), wristbands or smartwatches (22.8%), and holters (6.5%) the most common (Table 1). The most common sensors (e.g., smartwatches, heart rate monitors, smart glasses; L’Hommedieu et al. 2019) were accelerometers (50.0%), phones (39.1%), GPS (35.9%), microphones (30.4%), actigraphs (25.0%), and electrocardiograms (ECG) (25.0%), and implantable medical devices are used for other conditions (Ferguson and Redish 2011). The sensors convert human biological data into digital signals for a microprocessor to interpret, and then, the flow of information continues (Heikenfeld Jajack et al. 2018).

Most studies used instruments relevant to their application domain, including questionnaires, ecologic momentary assessment (EMA), and clinician assessments. These are objective, accurate (although not always validated as such), and can validate self-report data (e.g., Fitbit). There was significant variation in the timing of validation measures, from one-time measures to many per day or pre-post assessments. For data processing, software applications that communicated with a remote server were used in most studies to save data for processing and, at times, provide within-study feedback to participants.

Remote monitoring and remotely supervised treatment or training (RMT) uses complex distributed information and communication technology (ICT) systems to measure, transport, and store information (ATA 2014). Feedback helps clinicians with clinical decision support and to increase patient awareness of their health status, gives them a feeling of safety and motivate, and enables them to change/improve their health status (Maddison Cartledge et al. 2019).

Evidence-Based

One major subset of studies focused on feedback in the form of applications that display graphs representing mental health status (Abdullah et al. 2016; Faurholt-Jepsen et al. 2016). Many studies found that motivational messages (e.g., specific suggestion for action), tailored messages (e.g., specific to disease, stage of progress, demographic information, treatment plan), and health professional contact-enhanced therapeutic engagement are generally more effective than receiving (typically numeric) health measure feedback (Ma et al. 2014). These individualized (or similar user) models appear more effective (Wahle et al. 2016) than generalized models (e.g., generic message for a depression cohort). Individualized design of training and exercise programs may be helpful to maximize the improvements in performance and avoid risks (Cardinale and Varley 2017).

Another area of research is related to validating diagnosis and/or determining predictors of outcomes. Within the depression studies, cell tower identification (i.e., geographic movement), screen active duration, and activity measurement have the strongest linkage with depression severity compared with distance travelled and SMS text message received, whereas manic patients have longer calls, more messages sent, and more travel (Hilty et al. 2020). For bipolar disorder, studies generally support the interpretation of sensed data by employing traditional measures or assessments for depression (e.g., PHQ-8 or PHQ-9 self-report instruments) and clinician assessment/batteries (Faurholt-Jepsen et al. 2016). Mood and anxiety studies use algorithms to interpret or predict participant’s status (e.g., support vector machine, naïve Bayes classifiers, decision trees, random forests, linear regression, Bayesian networks, logistic regression, and other machine learning methods) (Elgendi and Menon 2019). The College Sleep Project used multi-modal sensors (physiological, behavioral, environmental, social) from smartphones to identify associations among and develop predictors for sleep behaviors, stress, mental health, and academic performance (Sano et al. 2015).

Behavioral Health Profession and International Organization Position Statements and Guidelines on Telehealth and E-health

Our current review builds on past comparisons (Hilty et al. 2017; Hilty et al. 2018; Hilty et al. 2020) of health and behavioral health professional organizations’ documents that pertain to technologies by focusing on sensors and wearables (Table 2). US and Canadian guidelines were reviewed (American Psychiatric Association 2014, 2017, 2018; American Psychiatric Association-ATA 2018; ATA 2009, 2013, 2017; Royal College of Physicians and Surgeons Canada 2015). Only the ATA child and adolescent guideline mentions monitoring and wearables (ATA 2017). The only competencies have come from individual research groups for video (Hilty et al. 2015, 2018), social media (Hilty et al. 2018; Zalpuri et al. 2018), mobile health (Hilty et al. 2019a, b; Hilty et al. 2020), and asynchronous (Hilty et al. 2019a, b; Hilty et al. 2020) technologies.

Table 2.

A comparison of telebehavioral guidelines and position statements across professions related to asynchronous technologies

Domain/competency	Psychology (Australian, British, Canadian and U.S.)	ACA	Social work	C/MFT	Psychiatry (individual researchers > U.S. or Canadian)	ATA adult	ATA child and adolescent	CTIBS
Synchronous video
All AGCME domains in summary	✓ - ✓ ✓	✓ - ✓ ✓	✓	✓	✓ ✓ ✓ - ✓ ✓ ✓ ✓	✓ ✓	✓ ✓	✓ ✓ ✓ - ✓ ✓ ✓ ✓
Asynchronous technologies
Apps	✓	✓			✓ ✓ ✓ ✓		✓	✓ ✓ ✓
Asynchronous	✓		✓		✓ ✓	✓	✓	✓ ✓
Device (mobile/smart)	✓^a	✓	✓ ✓		✓ ✓	✓	✓	✓ ✓
E-consultation	✓^a				✓	✓	✓
E-mail	✓	✓	✓ ✓		✓ ✓		✓	✓
Mobile/portable	✓				✓ ✓ ✓ ✓		✓	✓ ✓ ✓
Monitoring (remote)
Sensor	✓^b				✓		✓	✓ ✓
Social media	✓ ✓	✓ ✓	✓ ✓	✓	✓ ✓ ✓ ✓		✓	✓ ✓ ✓
Store-and-forward					✓ ✓		✓
Text	✓^a	✓	✓	✓	✓ ✓ ✓ ✓		✓	✓ ✓ ✓
Text	✓ ✓^b
Wearable	✓^b				✓		✓
Web-based			✓

Open in a new tab

✓, important, mentioned and terminology defined; ✓✓, important, discussed in-depth; ✓✓✓, limited scope of measureable behaviors/competencies based on evidence and/or consensus; ✓✓✓✓, broad scope and depth of measureable behaviors/competencies based on evidence and/or consensus

ACA American Counseling Association, C/MFT Couples/Marriage Family Therapists, CTiBS Coalition for Technology in Behavioral Science

^aAustralian Psychological Society only

^bCanadian Psychological Association only

Contributions from behavioral health come from American, Australian, British, and Canadian psychology associations (APA 2013, 2019; Association of State and Provincial Psychology Boards 2014; Australian Psychological Society 2011; British Association for Counselling and Psychotherapy 2019; Canadian Psychological Association 2006; Goss et al. 2001) and U.S. social work (National Association of Social Workers 2017), counseling (American Counseling Association 2014; Canadian Counselling and Psychotherapy Association 2019), and couples/marriage family therapy (American Association for Marriage and Family Therapy 2015; American Association for Marriage and Family Therapy Regulatory Board 2016). The most commonly discussed technologies were e-mail and social media, followed by texting, with note of sensors and wearables by the Canadians (sensor, text, wearable) and Australians (device, e-consultation, mobile, text). Only the CTiBS competencies for telebehavioral health (Maheu et al. 2018) mentioned briefly sensors and wearables as part of a mobile health domain.

Institutional and professional organizations are overseen by accreditation, licensing and boards/agencies to provide guidance to ensure professionals provide care proficiently, legally and ethically. They proactively inform and collaborate with regulatory boards/agencies to address broad, overarching issues related to quality of care, standards for practice and maintenance of certification. They also employ competency benchmarks (APA 2012), skills-based exams, and decision-making exams (Hilty et al. 2020). They tend to make changes incrementally and adjust to sentinel events one-by-one.

Based on the current review, the landscape related to sensors and wearables reveals several themes regarding mobile health technologies, some of which are concerning with respect to competencies (Seppälä et al. 2019; Hilty et al. 2020). First, there is incremental adaptation of technology added to care systems with the assumption that it is just part of routine delivery rather than a new form of care (Hilty et al. 2013). Second, innovation is siloed within research programs and/or pilot projects. Third, most innovation is related to video, e-mail, and electronic health record (EHR) portals, not mobile health, sensors, and wearables. Fourth, few if any academic health centers have yet to create or adopt technology competencies or install mobile health (apps, text mobile phones) and wearable sensors into clinical decision support systems (CDSS), so impact is limited.

Shifting Perspectives for Patients, Clinicians, Teams, and Systems

Overview

The future state is an “open” mobile health architecture that offers universal data standards and a global interconnected network, with patient and clinician facing dashboards, rather than apps and other technologies limited by siloed or un-integrated proprietary data, management, and analytic tools (Ramkumar et al. 2019). Clinicians may have to reconsider patient’s skills, needs, and preferences to determine suitability for any specific technology as part of the consent process, and again later, when needs/preferences among available choices are understood or care plans change (Hilty et al. 2015; Maheu et al. 2018).

The impact of new technology has increased exponentially (Mermelstein et al. 2017; Sturmberg et al. 2014), with users currently compared as digital natives versus immigrants (Wang et al. 2013). A broader perspective from education, business, health care, and other disciplines suggests that technology plays different roles along an augmentation-exploration-innovation spectrum (Hilty et al. 2019). It may affirm, complement, accentuate, and augment what people do. Further, it may help people explore, engage, and experiment with ideas and partners in new ways, “changing” dynamics or facilitating a different culture, making new things possible.

Paradigm Shifts in Integrating Sensors and Wearables with In-person and Synchronous Care

The first paradigm shift is including technology in traditional health care or adapting that care to technology. This can be difficult for clinicians, training programs, and health systems. The first example of this shift was from in-person to synchronous and asynchronous video (Yellowlees et al. 2018; Hilty et al. 2020). Even low-cost systems can facilitate engagement and “social presence” for participants to share a virtual space, get to know one another, and discuss complex issues (Hilty, Nesbitt et al. 2002). The most important goal with technology is to keep the exchange of information, clarity, responsiveness, and comfort close to real-time experiences (Hilty et al. 2020). Empathy, positive regard, and genuineness play a critical role in therapy effectiveness (Grekin et al. 2019; Kemp et al. 2020). Clinicians must adjust techniques to inquire, engage, communicate, and listen effectively (Hilty et al. 2020).

A second shift is correspondence via technology outside the office visit creating a new sense of continuity, connection, and exchange of information. Patients, clinicians, and healthcare researchers are trying to understand how this affects the therapeutic frame, communication, boundaries, and trust (Hilty et al. 2020; Li et al. 2016). A primary example of this is social media, which became a key part of patients’ lives, and by default part of care, though many clinicians did not understand or embrace it (Hilty et al. 2018). Asynchronous communications have a sense of immediacy and interaction that build “trust” but, despite a sense of control, are not always clear and can lead to ill-intended consequences (Hilty et al. 2020). Administrative planning and training can manage risks (e.g., privacy, self-disclosure, cyberbullying) (Armontrout et al. 2016; Joshi et al. 2019).

A third shift is how to integrate mobile health, text, sensors, social media, and other technologies into clinical workflow for a virtual office. As apps became popular, clinicians adapted more quickly, and now clinicians need to assess sensors and wearables for potential applications (e.g., entertainment, health care, behavioral health) (Hilty et al. 2019b). This is a key part of the therapeutic relationship, and there is a flow to the process (Fig. 2). For example, in-time data collection can alert a patient to self-assess mood regularly, with data transmitted automatically to the clinician. In-time data collection (e.g., EMA) is more valid, reliable, and meaningful than sporadic, retrospective reflection/recall, as long as the time to input data does not adversely impact patient motivation (Russell and Gajos 2020). This process also facilitates in-time clinical decision-making by adding a cognitive scaffolding and analytic methods that enhance confidence and skills (Clark 2013; Luxton 2016) (e.g., mood and medication management) (Thompson et al. 2014). This requires flexibility, between visit review of data, and supervision for trainees.

Fig. 2 — Data flow visualization from and applicable competency domains across clinical workflow

A fourth shift relates to organizational leadership and change. Many organizations are reluctant to change regarding technology despite telehealth’s impact (Hilty et al. 2020). Progressive businesses integrate core divisions (e.g., research, operations, finance) with information technology (IT) (i.e., shared IT-business framework) to leverage knowledge and capital (Ray et al. 2007). This paradigm has been applied to telehealth to organize/integrate health care rather than adding technology to existing systems (Hilty et al. 2019), suggesting that organizing care with technology will have better outcomes than appending it to health care.

Impact on Clinical, Technological, and Administrative Workflows

For healthcare systems, these new technologies have many implications, including clinical, technical, policy, and other administrative workflows to comply with legal and ethical mandates (American Psychiatric Association-ATA 2018; ATA 2009, 2013, 2017). Technological factors relate to access (i.e., equipment, software) and ease of use, as well as participant literacy and skill. Some tools are only new versions of in-person tools (e.g., smartwatch to collect photoplethysmography (PPG) signals to capture heart rate variability, an index associated with overall emotional and physical well-being), while other technologies shift perspectives dramatically for trainees and clinicians, requiring new competencies. For example, speech recognition and natural language processing (NLP) simplify data processing with a more natural and transparent interaction experience (e.g., Siri, Watson) (Witt et al. 2019) but require new skills.

The PACMAD (People at the Centre of Mobile Application Development) usability model emphasizes seven factors: effectiveness, efficacy, satisfaction, learnability, memorability, errors, and cognitive load (Harrison et al. 2013). A structured approach is needed to fit a sensor, app, or other technology to a user and evaluate its success, impact and value (Caulfield et al. 2019). Steps could include requirement definition (e.g., unobtrusive, body location, degree of measurement accuracy), device search, device evaluation criteria (e.g., human factor measurement tested, regulatory compliance, technical specifications/capabilities), and post-implementation evaluation of fit (or alternatively, use in others who are very similar to a give patient). While most clinicians currently lack these skills, the increased need to offer RPM and telehealth options during the COVID-19 pandemic has now increased utilization, and training should be based on established competencies when possible.

Many fields use human-centered co-design studios to develop and pre-test technological options (Hetrick et al. 2018; Russell et al. 2020). Focus groups, studies, and surveys have suggested ways to improve technology in health care through involvement and supervision of caregiver(s), better visuals, more immediate chat options and feedback between patients and clinicians, gamification and incentives, reminder alarms/alerts (MacKintosh et al. 2019), and seamless integration of data from apps and mobile phones into patient portals (Bennett et al. 2012).

Remote monitoring helps clinicians and trainees see or experience how important technology is for patients (e.g., safety, empowerment, control, convenience), healthcare systems (outpatient/in home, stepped care), and payers (e.g., insurance, cost reduction) (Hermens et al. 2008; Ramkumar et al. 2019). More importantly, they see how interprofessional teams can be effective, efficient, and collaborative in sharing expertise and roles (e.g., care coordinators, nurses) (Barisone et al. 2018; Winters et al. 2018).

Ethical issues related to informed consent, privacy, and data security are important to consider. Ethical issues concern providing and obtaining adequate informed consent from participants, information quality in monitoring apps, personal privacy, and international legal and cultural differences (Luxton 2016; Torous and Roberts 2017). For example, it is possible that with a combination of a great deal of physiological data, a person could theoretically be identifiable. Mobile health may also create privacy issues associated with third parties (e.g., images and sounds from nonparticipants). All of these issues should be evaluated during the technology design process so appropriate encryption and data security features are built in (Luxton et al. 2012). Other technological challenges include sensor precision, power, location, analytic procedure, communication, data acquisition, and processing, as well as performance issues between devices, accuracy of date versus gold standards, error rates, and other parameters (Patel et al. 2002).

Sensor and wearable technologies have not been adopted significantly to date, partly due to security, affordability, user-friendliness, compatibility with EHR systems (Bennett et al. 2012; Dinh-Le et al.2019), and financial and policy issues that have not been resolved even though virtual care codes were approved for some services in the U.S. in 2019 (Hilty et al. 2020). Manufacturers and healthcare providers also must consider certification and approval requirements related to US, European Union, and other international regulatory standards (Loncar-Turukalo et al. 2019; Ravizza De Maria et al. 2019).

Health System and Institutional Approaches to Sensor and Wearable Competencies

Overview

Competency-based education focuses on clinical skill development in addition to knowledge acquisition (Iobst et al. 2010). The Clinical Informatics Milestones have been suggested for the EHR, data use (e.g., safety, quality, privacy), and CDS (Hersh et al. 2014). These were followed by adult telepsychiatry (Hilty et al. 2015, 2018), CTiBS’ telebehavioral health (Maheu et al. 2018), social media (Hilty et al. 2018; Zalpuri et al. 2018), mobile health (Hilty et al. 2019a; Hilty et al. 2020), and asynchronous (Hilty et al. 2020) competency sets. They include suggestions for training, faculty development, and institutional change and use the ACGME domains (ACGME 2013) with novice/advanced beginner, competent/proficient, and expert levels. Sensors and wearables have yet to be addressed.

The slow integration of telehealth into clinical care and curricula may be a revealing story for sensors and wearables (Hilty et al. 2019). Interest by interns, residents, and fellows has significantly increased, but trainees across behavioral health have attributed inadequate experience, false beliefs of teachers, and speculation about technology efficacy as key limiting factors (Glover et al. 2013; Hilty et al. 2015). Residents have had valuable tele-experiences in establishing rapport and engaging with patients, working collaboratively, identifying different approaches used, and becoming aware of how to handle complex cases (Teshima et al. 2016). Many experts have called for more work in telehealth education (Balon et al. 2015; Sunderji et al. 2015; Crawford et al. 2016; Khan and Ramtekkar 2019), and one survey revealed only 21 of 46 programs have a telehealth curriculum or informal clinical experience (Hoffman and Kane 2015). The Veterans Affairs’ national telemental health training program provides a model, with comprehensive web-based modules, national satellite broadcasts, a national teleconferenced journal club, interactive Internet applications, and live real-time simulated remote competency assessments (Gifford et al. 2012; Godleski 2012).

Mobile health, social media, and informatics practices require more substantial planning and organizational change (Hilty et al. 2018; Hilty et al. 2019a, b, 2020; Hilty et al. 2020). Building on the IT-business model, this requires a multi-dimensional shift in training and clinical culture (Fig. 3). It requires positive attitudes in clinicians, teachers, and faculty to positively support learners (Mostaghimi et al. 2017) and reflect on practice change, share information, and adjust curriculum (e.g., courses, rotations, supervisory approaches). For sensors and wearables, health systems need to integrate objective data into clinical workflow (Dogan et al. 2017). EHRs need to develop comprehensive use-case scenarios that include details regarding the type of data needed to support care processes, clinical and community-based personnel, and standardized interventions that are already in place or need to be established to best utilize the information to be collected and integrated. These use-case scenarios should also be specific for different clinical populations (Lobelo et al. 2016).

Fig. 3 — Clinical and training contexts for sensor, wearable, and remote patient monitoring competencies

Sensor and wearable curricular approaches are similar to those for mobile health and asynchronous competencies (Hilty, Chan et al. 2019a, b, 2020; Hilty et al. 2020). In addition to the sensors being the focus (i.e., content) of these competencies, sensors can shape the learning process, teaching learners, and simultaneously giving them experience (e.g., armband to measure proper hand washing via signal processing and ML) (Kutafina et al. 2015). Learning analytics can be applied to cluster, depict, and process data to support a number of objectives (González-Crespo and Burgos 2019), since teachers need to reshape the course plan for different learners (e.g., styles, motivation, performance) and integrate that information with real-time analytical information to supervise, assess, adapt, and offer feedback. Sensors and wearables can be applied to a wide range of educational contexts, including but not limited to:

learning management systems (LMS) or content management systems (CMS) that process user data collected
studying relationships between type of device from which users access LMS and how it affects performance
live or virtual reality (VR)-based sessions with teams or patients so oral and body language skills are shaped and improved
game-based learning with virtual/augmented reality to improve procedures
recommendations to users (e.g., explore impact of order given, behavioral response, impact of choices on engagement via algorithm) (Baldominos and Quintana 2019; Roque et al. 2019)

Many healthcare institutions already provide students with iPads, smartphones, and other technologies (Gaglani and Topol 2014), so adding sensors and wearables is the next logical step, allowing for enhanced guidance from preceptors on technique and interpretation of findings. Inter-institutional collaborations also improve teaching, reduce costs, and integrate a variety of patient populations (Metter et al. 2006).

Sensor, Wearable, and Remote Patient Monitoring Competencies

The proposed framework (Table 3; domains vertically, learner levels horizontally) uses the six ACGME domains (ACGME 2013). Most content came from the telepsychiatry (general themes) (Hilty et al. 2015; Hilty et al. 2018), social media (privacy, security, professionalism) (Hilty et al. 2018; Zalpuri et al. 2018), mobile health (CDS, mobile phones, apps) (Hilty et al. 2019a, b, 2020), and asynchronous care (procedures, resource utilization, role in system navigation and workflow, and accountability, ethics, and responsibility) competencies (Torous and Roberts 2017; Hilty et al. 2020). The CTiBS (Hilty et al. 2017; Maheu et al. 2018), clinical informatics (Torous et al. 2017), and the Clinical Informatics Milestones (ACGME 2013) were also helpful.

Examples are:

Within patient care, the subdomains are patient assessment/evaluation, engagement and interpersonal skills, management and treatment planning, administration and medico-legal issues, procedures, and clinical decision-making, synthesis, and information systems (IS). One challenge is optimizing the match between the sensor and a specific application context, so a structured approach for requirements and evaluation would help a novice/advanced beginner. A healthy middle-aged adult engaging in employee wellness in a large organization will have different needs than a 70-year-old at home suffering from depression (Caulfield et al. 2019). They require different devices, indifferent sensor locations (wrist versus wearable vest), steps for synchronization (from own device to data aggregator), and a software change (from “reliable” to validated measurement). This may proceed smoothly if the healthcare system prepares options, has staff to assist, and trains faculty supervisors.
Also in patient care, a novice/advanced beginner should understand what a device is used for (e.g., behavior, physiology), its capacity (i.e., connection with databases), weigh the value of data, monitor/adjust treatment with supervision, and help patients reflect on use and output. A supervisor would need to adjust teaching, supervision/availability, and workflow (e.g., between visit documentation) based on the technology, learner skill level, and patient care outcomes. The competent/proficient clinician would base the selection of a wearable on patient preference, skill, and need (i.e., purpose), and show proficiency when moving between regular to synchronous, continuous care. They need to remember that the therapeutic frame includes in-person, video, and mobile health engagement, in order to create a positive virtual environment.
Under professionalism, subdomains include attitude, accountability, integrity and ethics, reflection, feedback, and growth. A novice/advanced beginner would be open to learning and adhere to professional guidelines (i.e., timely response to text, documentation). Learning boundaries may come naturally and require small but important adjustments (e.g., learning to see if self-assessments match data reports). A competent/proficient clinician would understand boundaries/ethical dilemmas with technological options and how to engage patients (e.g., sensitively inquire how it feels to have self-assessments double-checked with in-time data or behavior). If symptoms like suicidal thoughts or worsening depression appear linked to a particular geographic setting, the clinician may ask if the patient has noticed a pattern before simply announcing a conclusion. An advanced/expert clinician would provide leadership, teaching, and consultation, including assessing/advising if clinical responsibilities shift as care models changes to ensure practice within license and regulations (e.g., how often clinicians are expected to check data inflow and make decisions).

Teaching, Evaluation and Assessment: Clinical Care and Supervision in Training

Teaching sensor and wearable competencies successfully will require a mix of methods layered and adjusted for increasing skill level over time. This requires evaluation methods that can determine whether competencies have been achieved and for supervisors to provide formative feedback to learners (Table 4) (Tekian et al. 2015). The following case examples illustrate approaches to the teaching and supervising sensor and wearable competencies.

Within the patient care, one subdomain is patient assessment/evaluation. An academic medical center may decide to offer a weight management program to support patient goals by collecting sensor data on step count via an app. Novice/advanced beginners must be able to screen for technology use among eligible patients, educate patients on the use of technology, and monitor/adjust treatment with supervision. The competent/proficient clinician would teach in the curricular for trainees, seminars for interprofessional teams, and on supervisory rounds. A case or simulation could introduce concepts to learners and be linked to clinical workflow. The advanced/expert clinician would develop the educational approach, train teachers, and evaluate process of change, aggregate sensor data, and on clinical quality and cost-effectiveness outcomes.
Within the patient care, the clinical decision-making, synthesis, and IS subdomain focus on EHR and documentation (Table 3). A novice/beginner should effectively read and write in the EHR, consider CDS tools, and use tools to coordinate and document key care events (Fig. 2). If a sensor is inputting data into the EHR between appointments, will the trainee know to review it or understand prompts? Can they adjust treatment plans “in time”, or do they need the supervisor to adjust practices (Table 4)? Typically, trainees wait and review data in a 1-h, scheduled, weekly supervision of a caseload, but “curbside consultations” are frequently needed. This could work better with two, half-hour meetings, more “walk-in” supervision time or more flexible coverage policies (see professional reflection, monitoring and hygiene related to patient care and supervision and subdomain in-time supervision in-person or at distance on critical incident). A competent/proficient clinician should select a wearable to ensure good usability, durability, interactivity (e.g., chat, feedback), and fit with the EHR for decision-making and documentation. They educate patients, trainees, and staff on tools’ pros/cons, and the meaning of the data, and like supervisors, has to consider questions like, “What is reasonable checking of data between visits? What standard of care is sought for?”
Case-based learning (Table 4) uses real life examples or vignettes in seminar, site-based case conferences and other presentations. Trainee’s experience with patients could be presented in a weekly group supervision or site-based case conference with an interprofessional team. Interactive methods like role-play using peers can highlight issues, practice communication skills, identify options for decisions, and propose solutions for patients. This provides an opportunity to build and/or solidify the trainee role as an educator (Table 4), working with an interprofessional team and adapting communication skills to multiple people.

Table 4.

Teaching, assessment and evaluation methods for sensor, wearable and remote patient monitoring clinical competencies

Open in a new tab

*Learning objective based on Bloom’s taxonomy

Institutional Competencies

The original telepsychiatric competencies suggested that institutions develop an approach for video and other technologies on the e-health (or e-behavioral health) spectrum (Hilty et al. 2015). The IT-business model (Hilty et al. 2019; Hilty et al. 2019; Ray et al. 2007) provides six steps for institutions to implement new technologies: (1) assess readiness; (2) create/hardwire the culture; (3) write policies and procedures; (4) establish the curriculum and competencies; (5) train learners and faculty; and (6) evaluate/manage change. One literature review focused on additional suggestions for synchronous and asynchronous telehealth (Hilty et al. 2020) and organized institutional competencies into 5 focus areas: patient-centered care; evaluation and outcomes; roles/needs of participants (e.g., trainees, faculty, teams, professions); teams, professions and systems within institutions; and the academic health center institutional structure, process, and administration.

Discussion

A variety of sensors and wearables are being used today and they are creating new options for patient care, clinician decision-making, and population health. These options reduce geographical, cost, and temporal barriers (Naslund et al. 2016; Tachakra et al. 2003; Cowan et al 2019) and provide an opportunity to bring patients and clinical teams together for communication, support, and intervention (Kumari et al. 2017; Torous and Roberts 2017). Successful integration of these technologies into care requires a person- and patient-centered approach, with patients moving from mere passengers to valued, responsible drivers of their care. Clinicians also must be aware of required skills, knowledge, and training to assure ethical and competent practice.

Our review of established guidelines and position statements highlights several important questions and concerns. Professional organizations have prioritized and made suggestions on how to integrate video, phones and e-mail, but mobile health (e.g., apps, texts) and wearable sensors are underrepresented or not mentioned at all. Nor are boards, agencies, and other government sectors showing significant movement on mobile health technologies that correspond with current and emerging paradigm shifts.

Research on sensors and wearables, remote monitoring, coupled with video, social media, mobile health, and asynchronous competencies, provides a foundation for the competencies we identified in this review. The domains and subdomains are a starting place for further discussion, elaboration, and improvement. The training approach with cases and curricular options we presented provides a source of options for clinicians, teachers, and program directors to help others develop knowledge, skills, and attitudes and has worked for other complex endeavors (e.g., culture and diversity, rural health) (Gifford et al. 2012; Office of Rural Health and Department of Veterans Affairs Employee Education System 2017; Hilty et al. 2018; Hilty et al. 2018; Kurtz 2018). Methodologies are needed to develop, implement, and evaluate (quantitative, qualitative) clinically and competency-based interventions (Proctor et al. 2010; Curran et al. 2012; Gargon et al. 2019; Marcolino et al. 2019; Solino-Fernandez et al. 2019). These learner-centered approaches parallel patient-centered ones and ensure quality care via technology (Hilty et al. 2020). Institutional competencies for synchronous and asynchronous technologies (Hilty et al. 2019; Hilty et al. 2020) could guide the paradigm shift for sensors and wearables for behavioral health in primary, integrated, and community care settings (Hilty et al. 2018; Hilty et al. 2019; Winters et al. 2019; Hilty et al. 2020).

Wearable, sensors, and other remote monitoring technologies are providing opportunities for additional data collection and sharing that have the potential to improve care. Moreover, traditional applications of clinical decision support technology that have existed for decades, such as electronic health records and expert systems, are being enhanced by newer technologies, such as cloud computing, artificial intelligence (AI), ML, and algorithms to provide access to new data that help clinicians constantly reevaluate, intervene, and make new observations and decisions (Luxton 2016; Patel 2002). Clinicians and those who train clinicians must embrace these technological advancements as part of healthcare reform so students’ experiences with technology enhance digital professionalism (Mostaghimi et al. 2017) and health systems can foster an e-culture with a wide range of evidence-based technology options (Hilty et al. 2019; Hilty et al. 2020). Computing tools can improve structure, process, and synergy with the way clinicians’ approach their work, particularly as research and clinical trials build upon empirical foundations and standardize methods, interventions, and evaluation (Armstrong et al. 2017; Bush et al. 2019).

Some organizations may be reluctant to adapt to change in general and specifically related to technology (Hilty et al. 2015). This includes academic health centers that have to balance missions with partner institutions, regional health campuses, and many health networks. Health systems are still working on basic clinical, technological, and administrative workflows, so most have not prioritized competencies. Their buy-in is needed but requires integration of missions and institutional change. Development and use of competencies for any clinical care change is an ongoing process (Holmboe, Sherbin, Donlin, Swing and Frank 2010), and the first steps involve discussion, needs assessment, implementation, and evaluation from faculty and program directors.

Even if competencies are prioritized, there are other competing priorities within psychiatric, medical, and interprofessional education (Hilty et al. 2015; Sockalingham et al. 2018), each requiring faculty development for teaching, supervision, and evaluation (Steinert et al. 2016). Some clinicians, training directors, and healthcare leaders familiar with telehealth are concerned about patient safety, information quality, professionalism, informed consent, privacy, and other medico-legal matters related to sensors and wearables. What they may fail to realize is that these and other risks are already in play with basic technologies (e.g., e-mail, phone call), growing proportionately with others (e.g., apps, texts, digital image transfer) and exponentially with sensors and wearables. Thus, the easier, more direct way to approach these issues may be to build an IT-business-health model, with input and collaboration by diverse partners such as clinicians, teachers, informaticians, and technology (i.e., developers, engineers).

Health care and medicine incrementally change over time based on social, political and scientific roots (Sturmberg et al. 2014). New paradigms of teaching and clinical care are needed to shape institutional response to innovative shifts technology disruptions and organizational contradictions that complicate institutional change (Bowe et al. 2003). A consolidated effort by training directors, national organizations, accreditation agencies, medical boards, and government could make change happen (Hilty et al. 2015). The result may be a change in the model of care, funding, and reimbursement beyond the in-person/video model and underserved populations, consistent with the call to make the temporary waivers during the COVID-19 pandemic permanent.

Across healthcare settings, training leaders, researchers, clinical champions, and technology specialists should collaborate to focus priorities on a general set of technology competencies that could be integrated into each profession’s existing set. Such a framework may avoid traditional pitfalls of trying to add completely new competencies, confusion about where they fit, concern about too many levels, and more generic pedagogic distinctions (e.g., competencies versus activities/tasks knowledge versus skills/behaviors) (Fairchild and Bayer 2004; Hilty et al. 2015).

Limitations of this paper include the scoping review style that was used that examines the extent, range, and nature of research in a topic area and identifies gaps in knowledge, rather than examine more specific, narrow topics as with systematic reviews (Arksey and O’Malley 2005; Levac et al. 2010). Second, each of the concept areas is unique and complex, so greater depth of analysis is needed to determine the “what, when, why and how” they work—and the human factors for patients and teams with varying skill levels and backgrounds. Third, country/province, state/federal, and healthcare institutions are unique, so one approach may not fit all systems. Fourth, core competencies must be clearly defined, implemented, and evaluated for each clinical specialty, in conjunction with professional development, change management, and building of a positive e-culture for participants to succeed.

Conclusions

Research is promising for sensors and wearables, and it is shifting toward standardized methods, interventions, and evaluation measures. As these technologies are integrated into health care, steps are needed to ensure quality care, clinicians’ skills, and outcomes that are in line with in-person and synchronous telehealth care. Clinician skillsets for sensors and wearables are similar to and overlap with those required for in-office or telehealth-based care, but there are additional components. Clinician and institutional competencies need to be implemented and evaluated to ensure quality of care, integration of missions, and institutional change. Attention to clinical, educational, administrative and technical factors is needed to build a positive e-culture of care, and collaboration is needed between clinicians, teachers, informaticians, developers, and engineers.

Acknowledgments

Accreditation Council of Graduate Medical Education, American Telemedicine Association and Telemental Health Interest Group, Northern California Veterans Administration Health Care System, UC Davis School of Medicine, Office of Connected Care, Department of Veterans Affairs, Defense Health Agency, Department of Defense.

Footnotes

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Donald M. Hilty, Email: donh032612@gmail.com

Christina M. Armstrong, Email: Christina.Armstrong@va.gov

Elizabeth A. Krupinski, Email: ekrupin@emory.edu

References

Abdullah S, Matthews M, Frank E, Doherty G, Gay G, Choudhury T. Automatic detection of social rhythms in bipolar disorder. Journal of the American Medical Informatics Association. 2016;23:538–543. doi: 10.1093/jamia/ocv200. [DOI] [PubMed] [Google Scholar]
Accreditation Council on Graduate Medical Education. (2013). Common program requirements. Retrieved from: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf.
American Association for Marriage and Family Therapy. (2015). Standard VI technology-assisted professional services. Retrieved from: https://www.aamft.org/iMIS15/AAMFT/Content/legal_ethics/code_of_ethics.aspx.
American Association for Marriage and Family Therapy Regulatory Board. (2016). Teletherapy guidelines. Retrieved from: https://amftrb.org/wp-content/uploads/2017/05/Proposed-Teletherapy-Guidelines-DRAFT-as-of-09.12.16.pdf.
American Counseling Association. (2014). Code of ethics. Retrieved from: https://www.counseling.org/resources/aca-code-of-ethics.pdf.
American Psychiatric Association and American Telemedicine Association. (2018). Best practices in videoconferencing-based telemental health. Retrieved from: https://www.psychiatry.org/psychiatrists/practice/telepsychiatry/blog/apa-and-ata-release-new-telemental-health-guide. [DOI] [PubMed]
American Psychiatric Association. (2017). Resource document on telemedicine: synchronous videoconferencing in psychiatry. Retrieved from: https://www.resource-document-telemedicine-synchronous-video-conferencing-in-psychiatry-2017.
American Psychiatric Association. (2014). Resource document on telepsychiatry and related technologies in clinical psychiatry. Retrieved from: https://www.Resource-2014-Telepsychiatry-Clinical-Psychiatry.
American Psychological Association Educational Directorate. (2012). A practical guidebook for the competency benchmarks. Retrieved from: http://www.apa.org/ed/graduate/benchmarks-guide.aspx.
American Psychological Association. (2013). Guideline for the practice of telepsychology. Retrieved from: http://www.apapracticecentral.org/ce/guidelines/telepsychology-guidelines.pdf.
American Psychological Association. (2019). Proposed guidelines for the optimal use of social media in professional psychological practice. In Press.
American Telemedicine Association ATA practice guidelines for video-based online mental health services. Telemedicine Journal and E-Health. 2013;19(9):722–330. doi: 10.1089/tmj.2013.9989. [DOI] [PubMed] [Google Scholar]
American Telemedicine Association. (2017). Practice guideline for child and adolescent telemental health. Telemedicine Journal and E-Health. Retrieved from: https://higherlogicdownload.s3.amazonaws.com/AMERICANTELEMED/618da447-dee1-4ee1-b941-c5bf3db5669a/UploadedImages/Practice%20Guideline%20Covers/NEW_ATA%20Children%20&%20Adolescents%20Guidelines.pdf.
American Telemedicine Association. (2014). Practice guidelines for live, on demand primary and urgent care. Retrieved from: http://www.medicalinfo.ch/images/articles/ATA-Practice-Guidelines-for-Live-On-Demand-Primary-and-Urgent-Care.pdf. [DOI] [PubMed]
American Telemedicine Association. (2009). Practice guidelines for videoconferencing-based telemental health. Retrieved from: http://www.americantelemed.org/docs/default-source/standards/practice-guidelines-for-videoconferencing-based-telemental-health.pdf?sfvrsn=6. [DOI] [PubMed]
Areàn PA, Ly KH, Andersson G. Mobile technology for mental health assessment. Dialogues in Clinical Neuroscience. 2016;18(2):163–169. doi: 10.31887/DCNS.2016.18.2/parean. [DOI] [PMC free article] [PubMed] [Google Scholar]
Ariga K, Makita T, Ito M, Mori T, Watanabe S, Takeya J. Review of advanced sensor devices employing nanoarchitectonics concepts. Beilstein Journal of Nanotechnology. 2019;10:2014–2030. doi: 10.3762/bjnano.10.198. [DOI] [PMC free article] [PubMed] [Google Scholar]
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005;8(1):19–32. [Google Scholar]
Armontrout J, Torous J, Fisher M, Drogin E, Gutheil T. Mobile mental health: navigating new rules and regulations for digital tools. Current Psychiatry Reports. 2016;18(10):91. doi: 10.1007/s11920-016-0726-x. [DOI] [PubMed] [Google Scholar]
Armstrong, C. M., Hoyt, T., Kinn, J. T., Ciulla, R. P., & Bush, N. E. (2017). Mobile behavioral health applications for the military community: evaluating the emerging evidence base. Best Practices in Mental Health: Mental Health Practice with Military and Veteran Connected Populations, 13(1), 106–119, Vandiver & Corcoran (Editors). ISSN 1553–555x.
Association of State and Provincial Psychology Boards. (2014). Maintenance of competence for licensure (MOCL) white paper. Retrieved from: https://cdn.ymaws.com/asppb.site-ym.com/resource/resmgr/Guidelines/Maintenance_of_Competence_fo.pdf.
Australian Psychological Society. (2011). Guidelines for providing psychological services and products using the internet and telecommunications technology. Retrieved from: https://aaswsocialmedia.wikispaces.com/file/view/EG-Internet.pdf.
Baldominos A, Quintana D. Data-driven interaction review of an ed-tech application. Sensors. 2019;19:1910. doi: 10.3390/s19081910. [DOI] [PMC free article] [PubMed] [Google Scholar]
Balon R, Beresin EV, Coverdale JH, Louie AK, Roberts LW. Strengthening telepsychiatry’s role in clinical care and education. Academic Psychiatry. 2015;39(1):6–9. doi: 10.1007/s40596-014-0265-x. [DOI] [PubMed] [Google Scholar]
Barisone M, Bagnasco A, Timmins F, Aleo G, Sasso L. Approaches to nurse education and competence development in remote telemonitoring of heart failure patients with implanted heart devices in Italy: a cause for concern. European Journal of Cardiovascular Nursing. 2018;17(5):388–389. doi: 10.1177/1474515117742132. [DOI] [PubMed] [Google Scholar]
Bennett CC, Doub TW, Selove R. EHRs connect research and practice: where predictive modeling, artificial intelligence, and clinical decision support inter- sect. Health Policy and Technology. 2012;1(2):105–114. [Google Scholar]
Bowe, C. M., Lahey, L., Kegan, R., & Armstrong, E. (2003). Questioning the “big assumptions.” Part II: recognizing organizational contradictions that impede institutional change. Medical Education, 37(8), 723–733. [DOI] [PubMed]
British Association for Counselling and Psychotherapy. (2019). Working online in the counselling professions. Retrieved from: https://www.bacp.co.uk/media/2162/bacp-working-online-supplementary-guidance-gpia047.pdf.
Bush NE, Armstrong CM, Hoyt TV. Smartphone apps for psychological health: a brief state of the science review. Psychological Services. 2019;16(2):188–195. doi: 10.1037/ser0000286. [DOI] [PubMed] [Google Scholar]
Canadian Counselling and Psychotherapy Association. (2019). Technology and innovative solutions, chapter project. Guidelines for uses of technology in counselling and psychotherapy. Retrieved from: https://www.ccpa-accp.ca/wp-content/uploads/2019/04/TISCGuidelinesMar2019EN.pdf.
Canadian Psychological Association. (2006). Providing psychological services via electronic media (draft ethical guidelines). Retrieved from: http://www.cpa.ca/aboutcpa/committees/ethics/psychserviceselectronically/.
Cardinale M, Varley MC. Wearable training-monitoring technology: applications, challenges, and opportunities. International Journal of Sports Physiology and Performance. 2017;12:S255–S262. doi: 10.1123/ijspp.2016-0423. [DOI] [PubMed] [Google Scholar]
Caulfield B, Reginatto B, Slevin P. Not all sensors are created equal: a framework for evaluating human performance measurement technologies. NPJ Digital Medicine. 2019;2:7. doi: 10.1038/s41746-019-0082-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
Clark A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences. 2013;36(3):181–204. doi: 10.1017/S0140525X12000477. [DOI] [PubMed] [Google Scholar]
Cowan K, Gentry M, McKean A, Hilty DM. Barriers to use of telepsychiatry: clinicians as gatekeepers. Mayo Clinic Proceedings. 2019;94(12):2510–2523. doi: 10.1016/j.mayocp.2019.04.018. [DOI] [PubMed] [Google Scholar]
Crawford A, Sunderji N, Lopez J, Soklaridis S. Defining competencies for the practice of telepsychiatry through an assessment of resident learning needs. BMC Medical Education. 2016;16:28. doi: 10.1186/s12909-016-0529-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care. 2012;50(3):217–226. doi: 10.1097/MLR.0b013e3182408812. [DOI] [PMC free article] [PubMed] [Google Scholar]
Dinh-Le C, Chuang R, Chokshi S, Mann D. Wearable health technology and electronic health record integration: scoping review and future directions. Journal of Medical Internet Research mhealth uhealth. 2019;7(9):e12861. doi: 10.2196/12861. [DOI] [PMC free article] [PubMed] [Google Scholar]
Dogan E, Sander C, Wagner X, Hegerl U, Kohls E. Smartphone-based monitoring of objective and subjective data in affective disorders: where are we and where are we going? Systematic review. Journal of Medical Internet Research. 2017;19(7):e262. doi: 10.2196/jmir.7006. [DOI] [PMC free article] [PubMed] [Google Scholar]
Dreyfus, S. E. & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition. Berkeley, CA: University of California, Operations Research Center. Retrieved from: https://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA084551.
Elgendi M, Menon C. Assessing anxiety disorders using wearable devices: challenges and future directions. Brain Sci. 2019;9(3):50. doi: 10.3390/brainsci9030050. [DOI] [PMC free article] [PubMed] [Google Scholar]
Fairchild AL, Bayer R. Public health: ethics and the conduct of public health surveillance. Science. 2004;303(5658):631–632. doi: 10.1126/science.1094038. [DOI] [PubMed] [Google Scholar]
Faurholt-Jepsen M, Munkholm K, Frost M, Bardram JE, Kessing LV. Electronic self-monitoring of mood using IT platforms in adult patients with bipolar disorder: a systematic review of the validity and evidence. BMC Psychiatry. 2016;16(1):7. doi: 10.1186/s12888-016-0713-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
Ferguson JE, Redish AD. Wireless communication with implanted medical devices using the conductive properties of the body. Expert Review of Medical Devices. 2011;8(4):427–433. doi: 10.1586/erd.11.16. [DOI] [PMC free article] [PubMed] [Google Scholar]
Gaglani SM, Topol EJ. iMedEd: the role of mobile health technologies in medical education. Academic Medicine. 2014;89(9):1207–1209. doi: 10.1097/ACM.0000000000000361. [DOI] [PMC free article] [PubMed] [Google Scholar]
Garcia-Ceja E, Riegler M, Nordgreen T, Jakobsen P, Oedegaard KJ, Tørresen J. Mental health monitoring with multimodal sensing and machine learning: a survey. Pervasive Mobile Computing. 2018;51:1–26. [Google Scholar]
Gargon E, Gorst SL, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLoS ONE. 2019;14(12):e0225980. doi: 10.1371/journal.pone.0225980. [DOI] [PMC free article] [PubMed] [Google Scholar]
Gifford V, Niles B, Rivkin I, Koverola C, Polaha J. Continuing education training focused on the development of behavioral telehealth competencies in behavioral healthcare providers. Rural and Remote Health. 2012;12(2108):1–15. [PubMed] [Google Scholar]
Glover JA, Williams E, Hazlett LJ, Campbell N. Connecting to the future: telepsychiatry in postgraduate medical education. Telemedicine Journal and E-Health. 2013;19(6):474–479. doi: 10.1089/tmj.2012.0182. [DOI] [PubMed] [Google Scholar]
Godleski L. A comprehensive national telemental health training program. Academic Psychiatry. 2012;36(5):408–410. doi: 10.1176/appi.ap.10090137. [DOI] [PubMed] [Google Scholar]
González-Crespo R, Burgos D. Advanced sensors technology in education. Sensors (Basel) 2019;19(19):4155. doi: 10.3390/s19194155. [DOI] [PMC free article] [PubMed] [Google Scholar]
Goss S, Anthony K, Jamieson A. Guidelines for online counselling and psychotherapy. Lutterworth, UK: British Association for Counselling and Psychotherapy; 2001. [Google Scholar]
Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. Journal of Biomedical Informatics. 2018;78:134–143. doi: 10.1016/j.jbi.2017.12.005. [DOI] [PubMed] [Google Scholar]
Grekin ER, Beatty JR, Ondersma SJ. Mobile health interventions: exploring the use of common relationship factors. Journal of Medical Internet Research mhealth uhealth. 2019;7(4):e11245. doi: 10.2196/11245. [DOI] [PMC free article] [PubMed] [Google Scholar]
Harrison R, Flood D, Duce D. Usability of mobile applications: literature review and rationale for a new usability model. Journal of Interactive Science. 2013;1(1):1–16. [Google Scholar]
Heikenfeld, J., Jajack, A., Rogers, J., Gutruf, P., Tian, L., Pan, T., ... & Kim, J. (2018). Wearable sensors: modalities, challenges, and prospects. Laboratory Chip,18, 217–248. [DOI] [PMC free article] [PubMed]
Hermens HJ, Vollenbroek-Hutten MM. Towards remote monitoring and remotely supervised training. Journal of Electromyography and Kinesiolology. 2008;18(6):908–919. doi: 10.1016/j.jelekin.2008.10.004. [DOI] [PubMed] [Google Scholar]
Hersh WR, Gorman PN, Biagioli FE, Mohan V, Gold JA, Mejicano GC. Beyond information retrieval and electronic health record use: competencies in clinical informatics for medical education. Advances in Medical Education and Practice. 2014;1(5):205–212. doi: 10.2147/AMEP.S63903. [DOI] [PMC free article] [PubMed] [Google Scholar]
Hetrick, S. E., Robinson, J., Burge, E., Blandon, R., Mobilio, B., Rice, S. M., ... & Davey, C. (2018). Youth codesign of a mobile phone app to facilitate self-monitoring and management of mood symptoms in young people with major depression, suicidal ideation, and self-Harm. Journal of Medical Internet Research Mental Health,5(1), e9. 10.2196/mental.9041. [DOI] [PMC free article] [PubMed]
Hilty, D. M., Armstrong, C. M., Luxton, D. D., Gentry, M., & Krupinski, E. (2020). A scoping review of sensors, wearables and remote monitoring for behavioral health: uses, outcomes, clinical competencies and research directions. Journal of Technology in Behavioral Science: In Press. [DOI] [PMC free article] [PubMed]
Hilty DM, Chan S, Torous J, Luo J, Boland RJ. A telehealth framework for mobile health, smartphones and apps: competencies, training and faculty development. Journal of Technology in Behavioral Science. 2019;4(2):106–123. [Google Scholar]
Hilty DM, Chan S, Torous J, Luo J, Boland RJ. A competency-based framework for psych/behavioral health apps for trainees, faculty, programs and health systems. Psychiatric Clinics in North America. 2019;42:513–534. doi: 10.1016/j.psc.2019.05.007. [DOI] [PubMed] [Google Scholar]
Hilty, D. M., Chan, S., Torous, J., Luo, J., & Boland, R. J. (2020). A framework for competencies for the use of mobile technologies in psychiatry and medicine. Journal of Medical Internet Research uhealth and mhealth, 8(2) http://mhealth.jmir.org/2020/2/e12229/. [DOI] [PMC free article] [PubMed]
Hilty DM, Chan S, Torous J, Mahautmr J, Mucic DM. New frontiers in healthcare and technology: Internet- and web-based mental options emerge to complement in-person and telepsychiatric care options. Journal of Health and Medical Informatics. 2015;6(4):1–14. [Google Scholar]
Hilty DM, Crawford A, Teshima J, Chan S, Sunderji N, Yellowlees PM. A framework for telepsychiatric training and e-health: competency-based education, evaluation and implications. International Review of Psychiatry. 2015;27(6):569–592. doi: 10.3109/09540261.2015.1091292. [DOI] [PubMed] [Google Scholar]
Hilty DM, Evangelatos G, Valasquez A, Sosa J. Telebehavioral health for rural culturally diverse populations: approaches for clinical services, competencies and administration. Journal of Technology in Behavioral Science. 2018;3:206–220. [Google Scholar]
Hilty DM, Feliberti J, Evangelatos G, Lu FG, Lim RF. Competent cultural telebehavioral healthcare to diverse populations: administration, evaluation, and financing. Journal of Technology in Behavioral Science. 2018;4(3):186–200. [Google Scholar]
Hilty DM, Ferrer D, Johnston B, Johnston B, Callahan EJ, Yellowlees PM. The effectiveness of telemental health: a 2013 review. Telemedicine Journal and E-Health. 2013;19(6):444–454. doi: 10.1089/tmj.2013.0075. [DOI] [PMC free article] [PubMed] [Google Scholar]
Hilty, D. M., Gentry, M, McKean, A. J., Cohen, K. E., Lim, R. F., Lu, F. G. (2020). Telehealth for rural diverse populations: telebehavioral and cultural competencies, clinical outcomes and administrative approaches. mHealth, 6, 20. 10.21037/mhealth.2019.10.04. [DOI] [PMC free article] [PubMed]
Hilty DM, Maheu M, Drude KP, Hertlein KM, Wall K, Long RP, Luoma TL. Telebehavioral health, telemental health, e-therapy and e-health competencies: the need for an interprofessional framework. Journal of Technology in Behavioral Science. 2017;2:171–189. doi: 10.1007/s41347-017-0036-0. [DOI] [Google Scholar]
Hilty DM, Maheu M, Drude K, Hertlein KM. The need to implement and evaluate telehealth competency frameworks to ensure quality care across behavioral health professions. Academic Psychiatry. 2018;42(6):818–824. doi: 10.1007/s40596-018-0992-5. [DOI] [PubMed] [Google Scholar]
Hilty DM, Nesbitt TS, Marks SL, Callahan EJ. How telepsychiatry affects the doctor-patient relationship: communication, satisfaction, and additional clinically relevant issues. Primary Psychiatry. 2002;9(9):29–34. [Google Scholar]
Hilty DM, Parish M, Chan S, Torous J, Xiong G, Scher L, Yellowlees PM. A comparison of in-person, synchronous and asynchronous telepsychiatry: skills/competencies, teamwork and administrative workflow. Journal of Technology in Behavioral Science. 2020 doi: 10.1007/s41347-020-00137-8. [DOI] [Google Scholar]
Hilty DM, Rabinowitz TR, McCarron RM, Katzelnick DJ, Chang T, Bauer AM, Fortney J. Telepsychiatry and e-mental health models leverage stepped, collaborative, and integrated services to primary care. Psychosomatics. 2018;59(3):227–250. doi: 10.1016/j.psym.2017.12.005. [DOI] [PubMed] [Google Scholar]
Hilty DM, Randhawa K, Maheu MM, McKean A, Pantera R, Mishkind MC, Rizzo A. A review of telepresence, virtual reality and augmented reality applied to clinical care. Journal of Technology in Behavioral Science. 2020;5:178–205. [Google Scholar]
Hilty DM, Sunderji N, Suo S, Chan S, McCarron RM. Telepsychiatry and other technologies for integrated care: evidence-base, best practice models and competencies. International Review of Psychiatry. 2019;1:1–18. doi: 10.1080/09540261.2019.1571483. [DOI] [PubMed] [Google Scholar]
Hilty DM, Torous J, Parish M, Chan S, Xiong G, Scher L, Yellowlees PM. A literature review comparing clinicians’ approaches and skills to in-person, synchronous and asynchronous care: moving toward asynchronous competencies to ensure quality care. Telemedicine Journal and E-Health. 2020 doi: 10.1089/tmj.2020.0054. [DOI] [PubMed] [Google Scholar]
Hilty DM, Uno J, Chan S, Torous J, Boland RJ. Role of technology in professional development. Psychiatric Clinics in North America. 2019;42:493–512. doi: 10.1016/j.psc.2019.05.013. [DOI] [PubMed] [Google Scholar]
Hilty DM, Unützer J, Ko DG, Luo J, Worley L, Yager J. Approaches for departments, schools, and health systems to better implement technologies used for clinical care and education. Academic Psychiatry. 2019;43(6):611–616. doi: 10.1007/s40596-019-01074-2. [DOI] [PubMed] [Google Scholar]
Hilty, D. M., Zalpuri, I., Stubbe, D., Snowdy, C. E., Shoemaker, E. Z., Myint, M. T., ... & Liu, H. Y. (2018). Social media/networking as part of e-behavioral health and psychiatric education: competencies, teaching methods, and implications. Journal of Technology in Behavioral Science,3(4), 268–293.
Hoffman P, Kane JM. Telepsychiatry education and curriculum development in residency training. Academic Psychiatry. 2015;39(1):108–109. doi: 10.1007/s40596-013-0006-6. [DOI] [PubMed] [Google Scholar]
Holmboe ES, Sherbin J, Donlin ML, Swing SR, Frank JR. The role of assessment in competency-based medical education. Medical Teacher. 2010;32:676–682. doi: 10.3109/0142159X.2010.500704. [DOI] [PubMed] [Google Scholar]
Iobst, W. F., Sherbino, J., Cate, T. O., Richardson, D. L., Dath, D., Swing, S. R., ... & Frank, J. R. (2010). Competency-based medical education in postgraduate medical education. Medical Teacher,32(8), 651–656. [DOI] [PubMed]
Joshi SV, Stubbe D, Li ST, Hilty DM. Correction to the use of technology by youth: implications for psychiatric educators. Academic Psychiatry. 2019;43(4):466. doi: 10.1007/s40596-018-1013-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
Kemp, J., Zhang, T., Inglis, F., Wiljer, D., Sockalingam, S., Crawford, A., ... & Strudwick, G. (2020). Delivery of compassionate mental health care in a digital technology–driven age: scoping review. Journal of Medical Internet Research,22(3), e16263. 10.2196/16263. [DOI] [PMC free article] [PubMed]
Khan S, Ramtekkar U. Child and adolescent telepsychiatry education and training. Psychiatric Clinics in North America. 2019;42(4):555–562. doi: 10.1016/j.psc.2019.08.010. [DOI] [PubMed] [Google Scholar]
Kumari P, Mathew L, Syal P. Increasing trend of wearables and multimodal interface for human activity monitoring: a review. Biosensors and Bioelectronics. 2017;90:298–307. doi: 10.1016/j.bios.2016.12.001. [DOI] [PubMed] [Google Scholar]
Kurtz D, Janke R, Vinek J, Wells T, Wells T, Hutchinson P, Froste A. Health sciences cultural safety education in Australia, Canada, New Zealand, and the United States: a literature review. International Journal of Medical Education. 2018;9:271–285. doi: 10.5116/ijme.5bc7.21e2. [DOI] [PMC free article] [PubMed] [Google Scholar]
Kutafina E, Laukamp D, Jonas SM. Wearable sensors in medical education: supporting hand hygiene training with a forearm EMG. Studies in Health Technology and Informatics. 2015;211:286–291. [PubMed] [Google Scholar]
Kvedar J, Coye MJ, Everett W. Connected health: a review of technologies and strategies to improve patient care with telemedicine and telehealth. Health Affairs (Millwood) 2014;33(2):194–199. doi: 10.1377/hlthaff.2013.0992. [DOI] [PubMed] [Google Scholar]
Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implemenation Science. 2010;5:69. doi: 10.1186/1748-5908-5-69. [DOI] [PMC free article] [PubMed] [Google Scholar]
L'Hommedieu M, L'Hommedieu J, Begay C, Schenone A, Dimitropoulou L, Margolin G, Narayanan S, et al. Lessons learned: recommendations for implementing a longitudinal study using wearable and environmental sensors in a health care organization. Journal of Medical Internet Research mhealth uhealth. 2019;7(12):e13305. doi: 10.2196/13305. [DOI] [PMC free article] [PubMed] [Google Scholar]
Li, H., Wu, J., Gao, Y., & Shi, Y. (2016). Examining individuals’ adoption of healthcare wearable devices: an empirical study from privacy calculus perspective. International Journal of Medical Informatics, 88, 8–17. http://www.ncbi.nlm.nih.gov/pubmed/26878757. [DOI] [PubMed]
Lobelo F, Kelli HM, Tejedor SC, Pratt M, McConnell MV, Martin SS, Welk GJ. The wild wild west: a framework to integrate mhealth software applications and wearables to support physical activity assessment, counseling and interventions for cardiovascular disease risk reduction. Progress in Cardiovascular Disease. 2016;58(6):584–594. doi: 10.1016/j.pcad.2016.02.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
Loncar-Turukalo T, Zdravevski E, Silva J, Chouvarda I, Traijkovik V. Literature on wearable technology for connected health: scoping review of research trends, advances, and barriers. Journal of Medical Internet Research. 2019;21(9):e14017. doi: 10.2196/14017. [DOI] [PMC free article] [PubMed] [Google Scholar]
Luxton DD, Kayl RA, Mishkind MC. mHealth data security: the need for HIPAA-compliant standardization. Telemedicine Journal and E-Health. 2012;18:284–288. doi: 10.1089/tmj.2011.0180. [DOI] [PubMed] [Google Scholar]
Luxton DD. Artificial intelligence in behavioral health care. Boston, MA: Elsevier; 2016. [Google Scholar]
Ma Y, Xu B, Bai Y, Sun G, Zhu R. Infer daily mood using mobile phone sensing. Ad Hoc Sensor Wireless Network. 2014;20:133–152. [Google Scholar]
Mackintosh, K. A., Chappel, S. E., Salmon, J., Timperio, A., Ball, K., Brown, H., ... & Ridgers, K. D. (2019). Parental perspectives of a wearable activity tracker for children younger than 13 years: acceptability and usability study. Journal of Medical Internet Research mhealth uhealth,7(11), e13858. 10.2196/13858. [DOI] [PMC free article] [PubMed]
Maddison, R., Cartledge, S., Rogerson, M., Goedhart, N. S., Ragbir Singh, T., Neil, C., ... & Ball, K. (2019). Usefulness of wearable cameras as a tool to enhance chronic disease self-management: scoping review. Journal of Medical Internet Research mhealth uhealth,7(1), e10371. 10.2196/10371. [DOI] [PMC free article] [PubMed]
Maheu M., Drude, K., Hertlein, K., Lipschutz, R., Wall, K., Long, R., et al., & Hilty, D. M. (2019). An interdisciplinary framework for telebehavioral health competencies. Journal of Technology in Behavioral Science, 3(2), 108–40; correction 3(2):107. 10.1007/s41347-019-00113. [DOI]
Maheu M, Drude K, Hertlein KM, Hilty DM. A framework for interprofessional telebehavioral health competencies: implementation and challenges moving forward. Academic Psychiatry. 2018;42(6):825–833. doi: 10.1007/s40596-018-0988-1. [DOI] [PubMed] [Google Scholar]
Marcolino, M. S., Alkmim, M. B., Pessoa, C. G., Maia, J. X., & Cardoso, C. S. (2019). Development and implementation of a methodology for quality assessment of asynchronous teleconsultations. Telemedicine Journal and E-Health, 26(5). 10.1089/tmj.2019.0049. [DOI] [PubMed]
Mermelstein H, Krupinski E, DeGuzman E, Rabinowitz T, Hilty DM. The application of technology to health: the evolution of telephone to telemedicine and telepsychiatry: a historical review and look at human factors. Journal of Technology in Behavioral Science. 2017;1(2):5–20. [Google Scholar]
Metter, D. F., Woolley, F. R., Bradley, Y., Morrison, J., Phillips, W. T., Salman, U., ... & Dodd, G. D, 3rd (2006). Teaching radiology resident didactics using videoconferencing development of a videoconferencing network to increase the efficiency of radiology resident training. Academic Radiology,13(10), 1276–1285. 10.1016/j.acra.2006.06.023. [DOI] [PubMed]
Mohr D, Zhang M, Schueller S. Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annual Review of Clinical Psychology. 2017;13:23–47. doi: 10.1146/annurev-clinpsy-032816-044949. [DOI] [PMC free article] [PubMed] [Google Scholar]
Mostaghimi A, Olszewski AE, Bell SK, Roberts DH, Crotty BH. Erosion of digital professionalism during medical students’ core clinical clerkships. Journal of Medical Internet Research Medical Education. 2017;3(1):e9. doi: 10.2196/mededu.6879. [DOI] [PMC free article] [PubMed] [Google Scholar]
Naslund JA, Aschbrenner KA, Bartels SJ. Wearable devices and smartphones for activity tracking among people with serious mental illness. Mental Health and Physical Activity. 2016;10:10–17. doi: 10.1016/j.mhpa.2016.02.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
National Academy of Sciences, Engineering, and Medicine (2020). Developing a patient-centered approach to optimizing veterans access to health care services. Retrieved from: https://www.nationalacademies.org/event/03-12-2020/developing-a-patient-centered-approach-to-optimizing-veterans-access-to-health-care-services-a-workshop.
National Association of Social Workers. (2017). Association of social work boards, council on social work education, and clinical social work association, standards for technology in social work practice. Retrieved from: https://www.naswpress.org/publications/standards/technology.html.
Office of Rural Health and Department of Veterans Affairs (2017). Employee education system, Veterans Health Administration Office of Care Coordination services. Cultural competence in telehealth clinics. Retrieved from: https://www.ruralhealth.va.gov/docs/cultural-competence-in-telehealth-clinics.pdf.
Patel VL, Kaufman DR, Arocha JF. Emerging paradigms of cognition in medical decision-making. Journal of Biomedical Informatics. 2002;35(1):52–75. doi: 10.1016/s1532-0464(02)00009-6. [DOI] [PubMed] [Google Scholar]
Piwek L, Ellis DA, Andrews S, Joinson A. The rise of consumer health wearables: promises and barriers. PLoS Medicine. 2016;13:e1001953. doi: 10.1371/journal.pmed.1001953. [DOI] [PMC free article] [PubMed] [Google Scholar]
Proctor, E., Silmere, H., Raghavan, R., Raghavan, R., Hovmand, P., Aarons, G., ... & Hensley, M. (2010). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administrative Policy in Mental Health,38(2), 65–76. [DOI] [PMC free article] [PubMed]
Ramkumar PN, Haeberle HS, Bloomfield MR, Schaffer JL, Kamath AT, Patterson BM, Krebs VE. Artificial intelligence and arthroplasty at a single institution: real-world applications of machine learning to big data, value-based care, mobile health, and remote patient monitoring. Journal of Arthroplasty. 2019;34(10):2204–2209. doi: 10.1016/j.arth.2019.06.018. [DOI] [PubMed] [Google Scholar]
Ravizza A, De Maria C, Di Pietro L, Sternini F, Audenino AL, Bignardi C. Comprehensive review on current and future regulatory requirements on wearable sensors in preclinical and clinical testing. Frontiers in Bioengineering and Biotechnology. 2019;7:313. doi: 10.3389/fbioe.2019.00313. [DOI] [PMC free article] [PubMed] [Google Scholar]
Ray G, Muhanna WA, Barney JB. Competing with IT: the role of shared IT-business understanding. Communications of the ACM. 2007;50(12):87–91. [Google Scholar]
Rohani DA, Faurholt-Jepsen M, Kessing LV, Bardram JE. Correlations between objective behavioral features collected from mobile and wearable devices and depressive mood symptoms in patients with affective disorders: systematic review. Journal of Medical Internet Research mhealth uhealth. 2018;6(8):e165. doi: 10.2196/mhealth.9691. [DOI] [PMC free article] [PubMed] [Google Scholar]
Roque F, Cechinel C, Weber TO, Lemos R, Villarroel R, Miranda D, Munoz R. Using depth cameras to detect patterns in oral presentations: a case study comparing two generations of computer engineering students. Sensors. 2019;19:3493. doi: 10.3390/s19163493. [DOI] [PMC free article] [PubMed] [Google Scholar]
Royal College of Physicians and Surgeons Canada. (2015). CanMEDS physician competency framework. Retrieved from: http://canmeds.royalcollege.ca/uploads/en/framework/CanMEDS%202015%20Framework_EN_Reduced.pdf
Russell MA, Gajos JM. Annual research review: ecological momentary assessment studies in child psychology and psychiatry. Journal of Child Psychology and Psychiatry. 2020;61(3):376–394. doi: 10.1111/jcpp.13204. [DOI] [PMC free article] [PubMed] [Google Scholar]
Sano A, Phillips AJ, Yu AZ, McHill AW, Taylor S, Jaques N, Czeisler CA, Klerman EB, Picard RW. Recognizing academic performance, sleep quality, stress level, and mental health using personality traits, wearable sensors and mobile phones. International Conference on Wearable and Implantable Body Sensor Networks. 2015 doi: 10.1109/BSN.2015.7299420. [DOI] [PMC free article] [PubMed] [Google Scholar]
Seko Y, Kidd S, Wiljer D, McKenzie K. Youth mental health interventions via mobile phones: a scoping review. Cyberpsycholology Behavioral and Social Networking. 2014;17(9):591–602. doi: 10.1089/cyber.2014.0078. [DOI] [PubMed] [Google Scholar]
Seppälä, J., De Vita, I., Jämsä, T., Miettunen, J., Isohanni, M., Rubinsten, K., ... & Bulgheroni, M. (2019). Mobile phone and wearable sensor-based mhealth approaches for psychiatric disorders and symptoms: systematic review. Journal of Medical Internet Research Mental Health,6(2), e9819. 10.2196/mental.9819. [DOI] [PMC free article] [PubMed]
Silva B, Rodrigues J, Diez I, Lopez-Coronado M, Saleem K. Mobile-health: a review of current state in 2015. Journal of Biomed Informatics. 2015;56:265–272. doi: 10.1016/j.jbi.2015.06.003. [DOI] [PubMed] [Google Scholar]
Sockalingam S, Arena A, Serhal E, Mohri L, Alloo J, Crawford A. Building provincial mental health capacity in primary care: an evaluation of a project ECHO mental health program. Academic Psychiatry. 2018;42(4):451–457. doi: 10.1007/s40596-017-0735-z. [DOI] [PubMed] [Google Scholar]
Solino-Fernandez D, Ding A, Bayro-Kaiser E, Ding EL. Willingness to adopt wearable devices with behavioral and economic incentives by health insurance wellness programs: results of a US cross-sectional survey with multiple consumer vignettes. BMC Public Health. 2019;19(1):1649. doi: 10.1186/s12889-019-7920-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
Statista. (2018). Connected wearable devices worldwide 2016–2021. Retrieved from: https://www.statista.com/statistics/487291/global-connected-wearable-devices/.
Steinert, Y., Mann, K., Anderson, B., Barnett, B. M., Centeno, A., Naismith, L., ... & Dolmans, D. (2016). A systematic review of faculty development initiatives designed to enhance teaching effectiveness: a 10-year update: BEME Guide No. 40. Medical Teacher, 38(8), 769–786. 10.1080/0142159X.2016.1181851. [DOI] [PubMed]
Steinhubl SR, Muse ED, Topol EJ. Can mobile health technologies transform health care? JAMA. 2013;310(22):2395–2396. doi: 10.1001/jama.2013.281078. [DOI] [PubMed] [Google Scholar]
Sturmberg JP, O'Halloran D, Colagiuri R, Fernandez A, Lukersmith S, Torkfar G, Salvador-Carulla L. Health care frames - from Virchow to Obama and beyond: the changing frames in health care and their implications for patient care. Journal of Evaluation and Clinical Practice. 2014;20(6):1036–1044. doi: 10.1111/jep.12266. [DOI] [PubMed] [Google Scholar]
Sunderji N, Crawford A, Jovanovic M. Telepsychiatry in graduate medical education: a narrative review. Academic Psychiatry. 2015;39(1):55–62. doi: 10.1007/s40596-014-0176-x. [DOI] [PubMed] [Google Scholar]
Tachakra S, Wang XH, Istepanian RS, Song YH. Mobile e-health: the unwired evolution of telemedicine. Telemedicine Journal and E-Health. 2003;9(3):247–257. doi: 10.1089/153056203322502632. [DOI] [PubMed] [Google Scholar]
Tekian A, Hodges BD, Roberts TE, Schuwirth L, Norcini J. Assessing competencies using milestones along the way. Medical Teacher. 2015;37(4):399–402. doi: 10.3109/0142159X.2014.993954. [DOI] [PubMed] [Google Scholar]
Teshima J, Hodgins M, Boydell KM, Pignatiello A. Resident evaluation of a required telepsychiatry clinical experience. Academic Psychiatry. 2016;40(2):348–352. doi: 10.1007/s40596-015-0373-2. [DOI] [PubMed] [Google Scholar]
Thompson WG, Kuhle CL, Koepp GA, McCrady-Spitzer SK, Levine JA. Go4Life exercise counseling, accelerometer feedback, and activity levels in older people. Archives of Gerontology and Geriatrics. 2014;58(3):314–319. doi: 10.1016/j.archger.2014.01.004. [DOI] [PubMed] [Google Scholar]
Torous J, Roberts LW. The ethical use of mobile health technology in clinical psychiatry. Journal of Nervous and Mental Disorders. 2017;205(1):4–8. doi: 10.1097/nmd.0000000000000596. [DOI] [PubMed] [Google Scholar]
Torous J, Chan S, Luo J, Boland R, Hilty D. Clinical informatics in psychiatric training: preparing today’s trainees for the already present future. Academic Psychiatry. 2017;42(5):694–697. doi: 10.1007/s40596-017-0811-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
Wahle F, Kowatsch T, Fleisch E, Rufer M, Weidt S. Mobile sensing and support for people with depression: a pilot trial in the wild. Journal of Medical Internet Research mhealth uhealth. 2016;4(3):e111. doi: 10.2196/mhealth.5960. [DOI] [PMC free article] [PubMed] [Google Scholar]
Wang QE, Myers MD, Sundaram D. Digital natives and digital immigrants towards a model of digital fluency. Business and Information Systems Engineering. 2013;5(6):409–419. [Google Scholar]
Winters, N., Langer, L., Nduku, P., Robson, J., O’Donovan, J., Maulik, P., ... & Nagraj, S. (2019). Using mobile technologies to support the training of community health workers in low-income and middle-income countries: mapping the evidence. BMJ Global Health,4(4), e001421. 10.1136/bmjgh-2019-001421. [DOI] [PMC free article] [PubMed]
Winters N, Langer L, Geniets A. Scoping review assessing the evidence used to support the adoption of mobile health (mHealth) technologies for the education and training of community health workers (CHWs) in low-income and middle-income countries. British Medical Journal Open. 2018;8(7):e019827. doi: 10.1136/bmjopen-2017-019827. [DOI] [PMC free article] [PubMed] [Google Scholar]
Witt DR, Kellogg RA, Snyder MP, Dunn J. Windows into human health through wearables data analytics. Current Opinion in Biomedical Engineering. 2019;9:28–46. doi: 10.1016/j.cobme.2019.01.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
Yellowlees, P., Parish, M. B., González, Á., Chan, S., Hilty, D., Iosif, A., ... & Xiong, G. (2018). Asynchronous telepsychiatry: a component of stepped integrated care. Telemedicine Journal and E-Health,24(5), 375–378. [DOI] [PMC free article] [PubMed]
Zalpuri I, Liu H, Stubbe D, Wrozek M, Hadsu J, Hilty DM. A competency-based framework for social media for trainees, faculty and others. Academic Psychiatry. 2018;42(6):808–817. doi: 10.1007/s40596-018-0983-6. [DOI] [PubMed] [Google Scholar]

[CR1] Abdullah S, Matthews M, Frank E, Doherty G, Gay G, Choudhury T. Automatic detection of social rhythms in bipolar disorder. Journal of the American Medical Informatics Association. 2016;23:538–543. doi: 10.1093/jamia/ocv200. [DOI] [PubMed] [Google Scholar]

[CR2] Accreditation Council on Graduate Medical Education. (2013). Common program requirements. Retrieved from: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf.

[CR3] American Association for Marriage and Family Therapy. (2015). Standard VI technology-assisted professional services. Retrieved from: https://www.aamft.org/iMIS15/AAMFT/Content/legal_ethics/code_of_ethics.aspx.

[CR4] American Association for Marriage and Family Therapy Regulatory Board. (2016). Teletherapy guidelines. Retrieved from: https://amftrb.org/wp-content/uploads/2017/05/Proposed-Teletherapy-Guidelines-DRAFT-as-of-09.12.16.pdf.

[CR5] American Counseling Association. (2014). Code of ethics. Retrieved from: https://www.counseling.org/resources/aca-code-of-ethics.pdf.

[CR6] American Psychiatric Association and American Telemedicine Association. (2018). Best practices in videoconferencing-based telemental health. Retrieved from: https://www.psychiatry.org/psychiatrists/practice/telepsychiatry/blog/apa-and-ata-release-new-telemental-health-guide. [DOI] [PubMed]

[CR7] American Psychiatric Association. (2017). Resource document on telemedicine: synchronous videoconferencing in psychiatry. Retrieved from: https://www.resource-document-telemedicine-synchronous-video-conferencing-in-psychiatry-2017.

[CR8] American Psychiatric Association. (2014). Resource document on telepsychiatry and related technologies in clinical psychiatry. Retrieved from: https://www.Resource-2014-Telepsychiatry-Clinical-Psychiatry.

[CR9] American Psychological Association Educational Directorate. (2012). A practical guidebook for the competency benchmarks. Retrieved from: http://www.apa.org/ed/graduate/benchmarks-guide.aspx.

[CR10] American Psychological Association. (2013). Guideline for the practice of telepsychology. Retrieved from: http://www.apapracticecentral.org/ce/guidelines/telepsychology-guidelines.pdf.

[CR11] American Psychological Association. (2019). Proposed guidelines for the optimal use of social media in professional psychological practice. In Press.

[CR12] American Telemedicine Association ATA practice guidelines for video-based online mental health services. Telemedicine Journal and E-Health. 2013;19(9):722–330. doi: 10.1089/tmj.2013.9989. [DOI] [PubMed] [Google Scholar]

[CR13] American Telemedicine Association. (2017). Practice guideline for child and adolescent telemental health. Telemedicine Journal and E-Health. Retrieved from: https://higherlogicdownload.s3.amazonaws.com/AMERICANTELEMED/618da447-dee1-4ee1-b941-c5bf3db5669a/UploadedImages/Practice%20Guideline%20Covers/NEW_ATA%20Children%20&%20Adolescents%20Guidelines.pdf.

[CR14] American Telemedicine Association. (2014). Practice guidelines for live, on demand primary and urgent care. Retrieved from: http://www.medicalinfo.ch/images/articles/ATA-Practice-Guidelines-for-Live-On-Demand-Primary-and-Urgent-Care.pdf. [DOI] [PubMed]

[CR15] American Telemedicine Association. (2009). Practice guidelines for videoconferencing-based telemental health. Retrieved from: http://www.americantelemed.org/docs/default-source/standards/practice-guidelines-for-videoconferencing-based-telemental-health.pdf?sfvrsn=6. [DOI] [PubMed]

[CR16] Areàn PA, Ly KH, Andersson G. Mobile technology for mental health assessment. Dialogues in Clinical Neuroscience. 2016;18(2):163–169. doi: 10.31887/DCNS.2016.18.2/parean. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR17] Ariga K, Makita T, Ito M, Mori T, Watanabe S, Takeya J. Review of advanced sensor devices employing nanoarchitectonics concepts. Beilstein Journal of Nanotechnology. 2019;10:2014–2030. doi: 10.3762/bjnano.10.198. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR18] Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005;8(1):19–32. [Google Scholar]

[CR19] Armontrout J, Torous J, Fisher M, Drogin E, Gutheil T. Mobile mental health: navigating new rules and regulations for digital tools. Current Psychiatry Reports. 2016;18(10):91. doi: 10.1007/s11920-016-0726-x. [DOI] [PubMed] [Google Scholar]

[CR20] Armstrong, C. M., Hoyt, T., Kinn, J. T., Ciulla, R. P., & Bush, N. E. (2017). Mobile behavioral health applications for the military community: evaluating the emerging evidence base. Best Practices in Mental Health: Mental Health Practice with Military and Veteran Connected Populations, 13(1), 106–119, Vandiver & Corcoran (Editors). ISSN 1553–555x.

[CR21] Association of State and Provincial Psychology Boards. (2014). Maintenance of competence for licensure (MOCL) white paper. Retrieved from: https://cdn.ymaws.com/asppb.site-ym.com/resource/resmgr/Guidelines/Maintenance_of_Competence_fo.pdf.

[CR22] Australian Psychological Society. (2011). Guidelines for providing psychological services and products using the internet and telecommunications technology. Retrieved from: https://aaswsocialmedia.wikispaces.com/file/view/EG-Internet.pdf.

[CR23] Baldominos A, Quintana D. Data-driven interaction review of an ed-tech application. Sensors. 2019;19:1910. doi: 10.3390/s19081910. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR24] Balon R, Beresin EV, Coverdale JH, Louie AK, Roberts LW. Strengthening telepsychiatry’s role in clinical care and education. Academic Psychiatry. 2015;39(1):6–9. doi: 10.1007/s40596-014-0265-x. [DOI] [PubMed] [Google Scholar]

[CR25] Barisone M, Bagnasco A, Timmins F, Aleo G, Sasso L. Approaches to nurse education and competence development in remote telemonitoring of heart failure patients with implanted heart devices in Italy: a cause for concern. European Journal of Cardiovascular Nursing. 2018;17(5):388–389. doi: 10.1177/1474515117742132. [DOI] [PubMed] [Google Scholar]

[CR26] Bennett CC, Doub TW, Selove R. EHRs connect research and practice: where predictive modeling, artificial intelligence, and clinical decision support inter- sect. Health Policy and Technology. 2012;1(2):105–114. [Google Scholar]

[CR27] Bowe, C. M., Lahey, L., Kegan, R., & Armstrong, E. (2003). Questioning the “big assumptions.” Part II: recognizing organizational contradictions that impede institutional change. Medical Education, 37(8), 723–733. [DOI] [PubMed]

[CR28] British Association for Counselling and Psychotherapy. (2019). Working online in the counselling professions. Retrieved from: https://www.bacp.co.uk/media/2162/bacp-working-online-supplementary-guidance-gpia047.pdf.

[CR29] Bush NE, Armstrong CM, Hoyt TV. Smartphone apps for psychological health: a brief state of the science review. Psychological Services. 2019;16(2):188–195. doi: 10.1037/ser0000286. [DOI] [PubMed] [Google Scholar]

[CR30] Canadian Counselling and Psychotherapy Association. (2019). Technology and innovative solutions, chapter project. Guidelines for uses of technology in counselling and psychotherapy. Retrieved from: https://www.ccpa-accp.ca/wp-content/uploads/2019/04/TISCGuidelinesMar2019EN.pdf.

[CR31] Canadian Psychological Association. (2006). Providing psychological services via electronic media (draft ethical guidelines). Retrieved from: http://www.cpa.ca/aboutcpa/committees/ethics/psychserviceselectronically/.

[CR32] Cardinale M, Varley MC. Wearable training-monitoring technology: applications, challenges, and opportunities. International Journal of Sports Physiology and Performance. 2017;12:S255–S262. doi: 10.1123/ijspp.2016-0423. [DOI] [PubMed] [Google Scholar]

[CR33] Caulfield B, Reginatto B, Slevin P. Not all sensors are created equal: a framework for evaluating human performance measurement technologies. NPJ Digital Medicine. 2019;2:7. doi: 10.1038/s41746-019-0082-4. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR34] Clark A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences. 2013;36(3):181–204. doi: 10.1017/S0140525X12000477. [DOI] [PubMed] [Google Scholar]

[CR35] Cowan K, Gentry M, McKean A, Hilty DM. Barriers to use of telepsychiatry: clinicians as gatekeepers. Mayo Clinic Proceedings. 2019;94(12):2510–2523. doi: 10.1016/j.mayocp.2019.04.018. [DOI] [PubMed] [Google Scholar]

[CR36] Crawford A, Sunderji N, Lopez J, Soklaridis S. Defining competencies for the practice of telepsychiatry through an assessment of resident learning needs. BMC Medical Education. 2016;16:28. doi: 10.1186/s12909-016-0529-0. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR37] Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care. 2012;50(3):217–226. doi: 10.1097/MLR.0b013e3182408812. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR38] Dinh-Le C, Chuang R, Chokshi S, Mann D. Wearable health technology and electronic health record integration: scoping review and future directions. Journal of Medical Internet Research mhealth uhealth. 2019;7(9):e12861. doi: 10.2196/12861. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR39] Dogan E, Sander C, Wagner X, Hegerl U, Kohls E. Smartphone-based monitoring of objective and subjective data in affective disorders: where are we and where are we going? Systematic review. Journal of Medical Internet Research. 2017;19(7):e262. doi: 10.2196/jmir.7006. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR40] Dreyfus, S. E. & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition. Berkeley, CA: University of California, Operations Research Center. Retrieved from: https://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA084551.

[CR41] Elgendi M, Menon C. Assessing anxiety disorders using wearable devices: challenges and future directions. Brain Sci. 2019;9(3):50. doi: 10.3390/brainsci9030050. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR42] Fairchild AL, Bayer R. Public health: ethics and the conduct of public health surveillance. Science. 2004;303(5658):631–632. doi: 10.1126/science.1094038. [DOI] [PubMed] [Google Scholar]

[CR43] Faurholt-Jepsen M, Munkholm K, Frost M, Bardram JE, Kessing LV. Electronic self-monitoring of mood using IT platforms in adult patients with bipolar disorder: a systematic review of the validity and evidence. BMC Psychiatry. 2016;16(1):7. doi: 10.1186/s12888-016-0713-0. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR44] Ferguson JE, Redish AD. Wireless communication with implanted medical devices using the conductive properties of the body. Expert Review of Medical Devices. 2011;8(4):427–433. doi: 10.1586/erd.11.16. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR45] Gaglani SM, Topol EJ. iMedEd: the role of mobile health technologies in medical education. Academic Medicine. 2014;89(9):1207–1209. doi: 10.1097/ACM.0000000000000361. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR46] Garcia-Ceja E, Riegler M, Nordgreen T, Jakobsen P, Oedegaard KJ, Tørresen J. Mental health monitoring with multimodal sensing and machine learning: a survey. Pervasive Mobile Computing. 2018;51:1–26. [Google Scholar]

[CR47] Gargon E, Gorst SL, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLoS ONE. 2019;14(12):e0225980. doi: 10.1371/journal.pone.0225980. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR48] Gifford V, Niles B, Rivkin I, Koverola C, Polaha J. Continuing education training focused on the development of behavioral telehealth competencies in behavioral healthcare providers. Rural and Remote Health. 2012;12(2108):1–15. [PubMed] [Google Scholar]

[CR49] Glover JA, Williams E, Hazlett LJ, Campbell N. Connecting to the future: telepsychiatry in postgraduate medical education. Telemedicine Journal and E-Health. 2013;19(6):474–479. doi: 10.1089/tmj.2012.0182. [DOI] [PubMed] [Google Scholar]

[CR50] Godleski L. A comprehensive national telemental health training program. Academic Psychiatry. 2012;36(5):408–410. doi: 10.1176/appi.ap.10090137. [DOI] [PubMed] [Google Scholar]

[CR51] González-Crespo R, Burgos D. Advanced sensors technology in education. Sensors (Basel) 2019;19(19):4155. doi: 10.3390/s19194155. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR52] Goss S, Anthony K, Jamieson A. Guidelines for online counselling and psychotherapy. Lutterworth, UK: British Association for Counselling and Psychotherapy; 2001. [Google Scholar]

[CR53] Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. Journal of Biomedical Informatics. 2018;78:134–143. doi: 10.1016/j.jbi.2017.12.005. [DOI] [PubMed] [Google Scholar]

[CR54] Grekin ER, Beatty JR, Ondersma SJ. Mobile health interventions: exploring the use of common relationship factors. Journal of Medical Internet Research mhealth uhealth. 2019;7(4):e11245. doi: 10.2196/11245. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR55] Harrison R, Flood D, Duce D. Usability of mobile applications: literature review and rationale for a new usability model. Journal of Interactive Science. 2013;1(1):1–16. [Google Scholar]

[CR56] Heikenfeld, J., Jajack, A., Rogers, J., Gutruf, P., Tian, L., Pan, T., ... & Kim, J. (2018). Wearable sensors: modalities, challenges, and prospects. Laboratory Chip,18, 217–248. [DOI] [PMC free article] [PubMed]

[CR57] Hermens HJ, Vollenbroek-Hutten MM. Towards remote monitoring and remotely supervised training. Journal of Electromyography and Kinesiolology. 2008;18(6):908–919. doi: 10.1016/j.jelekin.2008.10.004. [DOI] [PubMed] [Google Scholar]

[CR58] Hersh WR, Gorman PN, Biagioli FE, Mohan V, Gold JA, Mejicano GC. Beyond information retrieval and electronic health record use: competencies in clinical informatics for medical education. Advances in Medical Education and Practice. 2014;1(5):205–212. doi: 10.2147/AMEP.S63903. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR59] Hetrick, S. E., Robinson, J., Burge, E., Blandon, R., Mobilio, B., Rice, S. M., ... & Davey, C. (2018). Youth codesign of a mobile phone app to facilitate self-monitoring and management of mood symptoms in young people with major depression, suicidal ideation, and self-Harm. Journal of Medical Internet Research Mental Health,5(1), e9. 10.2196/mental.9041. [DOI] [PMC free article] [PubMed]

[CR60] Hilty, D. M., Armstrong, C. M., Luxton, D. D., Gentry, M., & Krupinski, E. (2020). A scoping review of sensors, wearables and remote monitoring for behavioral health: uses, outcomes, clinical competencies and research directions. Journal of Technology in Behavioral Science: In Press. [DOI] [PMC free article] [PubMed]

[CR61] Hilty DM, Chan S, Torous J, Luo J, Boland RJ. A telehealth framework for mobile health, smartphones and apps: competencies, training and faculty development. Journal of Technology in Behavioral Science. 2019;4(2):106–123. [Google Scholar]

[CR62] Hilty DM, Chan S, Torous J, Luo J, Boland RJ. A competency-based framework for psych/behavioral health apps for trainees, faculty, programs and health systems. Psychiatric Clinics in North America. 2019;42:513–534. doi: 10.1016/j.psc.2019.05.007. [DOI] [PubMed] [Google Scholar]

[CR63] Hilty, D. M., Chan, S., Torous, J., Luo, J., & Boland, R. J. (2020). A framework for competencies for the use of mobile technologies in psychiatry and medicine. Journal of Medical Internet Research uhealth and mhealth, 8(2) http://mhealth.jmir.org/2020/2/e12229/. [DOI] [PMC free article] [PubMed]

[CR64] Hilty DM, Chan S, Torous J, Mahautmr J, Mucic DM. New frontiers in healthcare and technology: Internet- and web-based mental options emerge to complement in-person and telepsychiatric care options. Journal of Health and Medical Informatics. 2015;6(4):1–14. [Google Scholar]

[CR65] Hilty DM, Crawford A, Teshima J, Chan S, Sunderji N, Yellowlees PM. A framework for telepsychiatric training and e-health: competency-based education, evaluation and implications. International Review of Psychiatry. 2015;27(6):569–592. doi: 10.3109/09540261.2015.1091292. [DOI] [PubMed] [Google Scholar]

[CR66] Hilty DM, Evangelatos G, Valasquez A, Sosa J. Telebehavioral health for rural culturally diverse populations: approaches for clinical services, competencies and administration. Journal of Technology in Behavioral Science. 2018;3:206–220. [Google Scholar]

[CR67] Hilty DM, Feliberti J, Evangelatos G, Lu FG, Lim RF. Competent cultural telebehavioral healthcare to diverse populations: administration, evaluation, and financing. Journal of Technology in Behavioral Science. 2018;4(3):186–200. [Google Scholar]

[CR68] Hilty DM, Ferrer D, Johnston B, Johnston B, Callahan EJ, Yellowlees PM. The effectiveness of telemental health: a 2013 review. Telemedicine Journal and E-Health. 2013;19(6):444–454. doi: 10.1089/tmj.2013.0075. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR69] Hilty, D. M., Gentry, M, McKean, A. J., Cohen, K. E., Lim, R. F., Lu, F. G. (2020). Telehealth for rural diverse populations: telebehavioral and cultural competencies, clinical outcomes and administrative approaches. mHealth, 6, 20. 10.21037/mhealth.2019.10.04. [DOI] [PMC free article] [PubMed]

[CR70] Hilty DM, Maheu M, Drude KP, Hertlein KM, Wall K, Long RP, Luoma TL. Telebehavioral health, telemental health, e-therapy and e-health competencies: the need for an interprofessional framework. Journal of Technology in Behavioral Science. 2017;2:171–189. doi: 10.1007/s41347-017-0036-0. [DOI] [Google Scholar]

[CR71] Hilty DM, Maheu M, Drude K, Hertlein KM. The need to implement and evaluate telehealth competency frameworks to ensure quality care across behavioral health professions. Academic Psychiatry. 2018;42(6):818–824. doi: 10.1007/s40596-018-0992-5. [DOI] [PubMed] [Google Scholar]

[CR72] Hilty DM, Nesbitt TS, Marks SL, Callahan EJ. How telepsychiatry affects the doctor-patient relationship: communication, satisfaction, and additional clinically relevant issues. Primary Psychiatry. 2002;9(9):29–34. [Google Scholar]

[CR73] Hilty DM, Parish M, Chan S, Torous J, Xiong G, Scher L, Yellowlees PM. A comparison of in-person, synchronous and asynchronous telepsychiatry: skills/competencies, teamwork and administrative workflow. Journal of Technology in Behavioral Science. 2020 doi: 10.1007/s41347-020-00137-8. [DOI] [Google Scholar]

[CR74] Hilty DM, Rabinowitz TR, McCarron RM, Katzelnick DJ, Chang T, Bauer AM, Fortney J. Telepsychiatry and e-mental health models leverage stepped, collaborative, and integrated services to primary care. Psychosomatics. 2018;59(3):227–250. doi: 10.1016/j.psym.2017.12.005. [DOI] [PubMed] [Google Scholar]

[CR75] Hilty DM, Randhawa K, Maheu MM, McKean A, Pantera R, Mishkind MC, Rizzo A. A review of telepresence, virtual reality and augmented reality applied to clinical care. Journal of Technology in Behavioral Science. 2020;5:178–205. [Google Scholar]

[CR76] Hilty DM, Sunderji N, Suo S, Chan S, McCarron RM. Telepsychiatry and other technologies for integrated care: evidence-base, best practice models and competencies. International Review of Psychiatry. 2019;1:1–18. doi: 10.1080/09540261.2019.1571483. [DOI] [PubMed] [Google Scholar]

[CR77] Hilty DM, Torous J, Parish M, Chan S, Xiong G, Scher L, Yellowlees PM. A literature review comparing clinicians’ approaches and skills to in-person, synchronous and asynchronous care: moving toward asynchronous competencies to ensure quality care. Telemedicine Journal and E-Health. 2020 doi: 10.1089/tmj.2020.0054. [DOI] [PubMed] [Google Scholar]

[CR78] Hilty DM, Uno J, Chan S, Torous J, Boland RJ. Role of technology in professional development. Psychiatric Clinics in North America. 2019;42:493–512. doi: 10.1016/j.psc.2019.05.013. [DOI] [PubMed] [Google Scholar]

[CR79] Hilty DM, Unützer J, Ko DG, Luo J, Worley L, Yager J. Approaches for departments, schools, and health systems to better implement technologies used for clinical care and education. Academic Psychiatry. 2019;43(6):611–616. doi: 10.1007/s40596-019-01074-2. [DOI] [PubMed] [Google Scholar]

[CR80] Hilty, D. M., Zalpuri, I., Stubbe, D., Snowdy, C. E., Shoemaker, E. Z., Myint, M. T., ... & Liu, H. Y. (2018). Social media/networking as part of e-behavioral health and psychiatric education: competencies, teaching methods, and implications. Journal of Technology in Behavioral Science,3(4), 268–293.

[CR81] Hoffman P, Kane JM. Telepsychiatry education and curriculum development in residency training. Academic Psychiatry. 2015;39(1):108–109. doi: 10.1007/s40596-013-0006-6. [DOI] [PubMed] [Google Scholar]

[CR82] Holmboe ES, Sherbin J, Donlin ML, Swing SR, Frank JR. The role of assessment in competency-based medical education. Medical Teacher. 2010;32:676–682. doi: 10.3109/0142159X.2010.500704. [DOI] [PubMed] [Google Scholar]

[CR83] Iobst, W. F., Sherbino, J., Cate, T. O., Richardson, D. L., Dath, D., Swing, S. R., ... & Frank, J. R. (2010). Competency-based medical education in postgraduate medical education. Medical Teacher,32(8), 651–656. [DOI] [PubMed]

[CR84] Joshi SV, Stubbe D, Li ST, Hilty DM. Correction to the use of technology by youth: implications for psychiatric educators. Academic Psychiatry. 2019;43(4):466. doi: 10.1007/s40596-018-1013-4. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR85] Kemp, J., Zhang, T., Inglis, F., Wiljer, D., Sockalingam, S., Crawford, A., ... & Strudwick, G. (2020). Delivery of compassionate mental health care in a digital technology–driven age: scoping review. Journal of Medical Internet Research,22(3), e16263. 10.2196/16263. [DOI] [PMC free article] [PubMed]

[CR86] Khan S, Ramtekkar U. Child and adolescent telepsychiatry education and training. Psychiatric Clinics in North America. 2019;42(4):555–562. doi: 10.1016/j.psc.2019.08.010. [DOI] [PubMed] [Google Scholar]

[CR87] Kumari P, Mathew L, Syal P. Increasing trend of wearables and multimodal interface for human activity monitoring: a review. Biosensors and Bioelectronics. 2017;90:298–307. doi: 10.1016/j.bios.2016.12.001. [DOI] [PubMed] [Google Scholar]

[CR88] Kurtz D, Janke R, Vinek J, Wells T, Wells T, Hutchinson P, Froste A. Health sciences cultural safety education in Australia, Canada, New Zealand, and the United States: a literature review. International Journal of Medical Education. 2018;9:271–285. doi: 10.5116/ijme.5bc7.21e2. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR89] Kutafina E, Laukamp D, Jonas SM. Wearable sensors in medical education: supporting hand hygiene training with a forearm EMG. Studies in Health Technology and Informatics. 2015;211:286–291. [PubMed] [Google Scholar]

[CR90] Kvedar J, Coye MJ, Everett W. Connected health: a review of technologies and strategies to improve patient care with telemedicine and telehealth. Health Affairs (Millwood) 2014;33(2):194–199. doi: 10.1377/hlthaff.2013.0992. [DOI] [PubMed] [Google Scholar]

[CR91] Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implemenation Science. 2010;5:69. doi: 10.1186/1748-5908-5-69. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR92] L'Hommedieu M, L'Hommedieu J, Begay C, Schenone A, Dimitropoulou L, Margolin G, Narayanan S, et al. Lessons learned: recommendations for implementing a longitudinal study using wearable and environmental sensors in a health care organization. Journal of Medical Internet Research mhealth uhealth. 2019;7(12):e13305. doi: 10.2196/13305. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR93] Li, H., Wu, J., Gao, Y., & Shi, Y. (2016). Examining individuals’ adoption of healthcare wearable devices: an empirical study from privacy calculus perspective. International Journal of Medical Informatics, 88, 8–17. http://www.ncbi.nlm.nih.gov/pubmed/26878757. [DOI] [PubMed]

[CR94] Lobelo F, Kelli HM, Tejedor SC, Pratt M, McConnell MV, Martin SS, Welk GJ. The wild wild west: a framework to integrate mhealth software applications and wearables to support physical activity assessment, counseling and interventions for cardiovascular disease risk reduction. Progress in Cardiovascular Disease. 2016;58(6):584–594. doi: 10.1016/j.pcad.2016.02.007. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR95] Loncar-Turukalo T, Zdravevski E, Silva J, Chouvarda I, Traijkovik V. Literature on wearable technology for connected health: scoping review of research trends, advances, and barriers. Journal of Medical Internet Research. 2019;21(9):e14017. doi: 10.2196/14017. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR96] Luxton DD, Kayl RA, Mishkind MC. mHealth data security: the need for HIPAA-compliant standardization. Telemedicine Journal and E-Health. 2012;18:284–288. doi: 10.1089/tmj.2011.0180. [DOI] [PubMed] [Google Scholar]

[CR97] Luxton DD. Artificial intelligence in behavioral health care. Boston, MA: Elsevier; 2016. [Google Scholar]

[CR98] Ma Y, Xu B, Bai Y, Sun G, Zhu R. Infer daily mood using mobile phone sensing. Ad Hoc Sensor Wireless Network. 2014;20:133–152. [Google Scholar]

[CR99] Mackintosh, K. A., Chappel, S. E., Salmon, J., Timperio, A., Ball, K., Brown, H., ... & Ridgers, K. D. (2019). Parental perspectives of a wearable activity tracker for children younger than 13 years: acceptability and usability study. Journal of Medical Internet Research mhealth uhealth,7(11), e13858. 10.2196/13858. [DOI] [PMC free article] [PubMed]

[CR100] Maddison, R., Cartledge, S., Rogerson, M., Goedhart, N. S., Ragbir Singh, T., Neil, C., ... & Ball, K. (2019). Usefulness of wearable cameras as a tool to enhance chronic disease self-management: scoping review. Journal of Medical Internet Research mhealth uhealth,7(1), e10371. 10.2196/10371. [DOI] [PMC free article] [PubMed]

[CR101] Maheu M., Drude, K., Hertlein, K., Lipschutz, R., Wall, K., Long, R., et al., & Hilty, D. M. (2019). An interdisciplinary framework for telebehavioral health competencies. Journal of Technology in Behavioral Science, 3(2), 108–40; correction 3(2):107. 10.1007/s41347-019-00113. [DOI]

[CR102] Maheu M, Drude K, Hertlein KM, Hilty DM. A framework for interprofessional telebehavioral health competencies: implementation and challenges moving forward. Academic Psychiatry. 2018;42(6):825–833. doi: 10.1007/s40596-018-0988-1. [DOI] [PubMed] [Google Scholar]

[CR103] Marcolino, M. S., Alkmim, M. B., Pessoa, C. G., Maia, J. X., & Cardoso, C. S. (2019). Development and implementation of a methodology for quality assessment of asynchronous teleconsultations. Telemedicine Journal and E-Health, 26(5). 10.1089/tmj.2019.0049. [DOI] [PubMed]

[CR104] Mermelstein H, Krupinski E, DeGuzman E, Rabinowitz T, Hilty DM. The application of technology to health: the evolution of telephone to telemedicine and telepsychiatry: a historical review and look at human factors. Journal of Technology in Behavioral Science. 2017;1(2):5–20. [Google Scholar]

[CR105] Metter, D. F., Woolley, F. R., Bradley, Y., Morrison, J., Phillips, W. T., Salman, U., ... & Dodd, G. D, 3rd (2006). Teaching radiology resident didactics using videoconferencing development of a videoconferencing network to increase the efficiency of radiology resident training. Academic Radiology,13(10), 1276–1285. 10.1016/j.acra.2006.06.023. [DOI] [PubMed]

[CR106] Mohr D, Zhang M, Schueller S. Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annual Review of Clinical Psychology. 2017;13:23–47. doi: 10.1146/annurev-clinpsy-032816-044949. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR107] Mostaghimi A, Olszewski AE, Bell SK, Roberts DH, Crotty BH. Erosion of digital professionalism during medical students’ core clinical clerkships. Journal of Medical Internet Research Medical Education. 2017;3(1):e9. doi: 10.2196/mededu.6879. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR108] Naslund JA, Aschbrenner KA, Bartels SJ. Wearable devices and smartphones for activity tracking among people with serious mental illness. Mental Health and Physical Activity. 2016;10:10–17. doi: 10.1016/j.mhpa.2016.02.001. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR109] National Academy of Sciences, Engineering, and Medicine (2020). Developing a patient-centered approach to optimizing veterans access to health care services. Retrieved from: https://www.nationalacademies.org/event/03-12-2020/developing-a-patient-centered-approach-to-optimizing-veterans-access-to-health-care-services-a-workshop.

[CR110] National Association of Social Workers. (2017). Association of social work boards, council on social work education, and clinical social work association, standards for technology in social work practice. Retrieved from: https://www.naswpress.org/publications/standards/technology.html.

[CR111] Office of Rural Health and Department of Veterans Affairs (2017). Employee education system, Veterans Health Administration Office of Care Coordination services. Cultural competence in telehealth clinics. Retrieved from: https://www.ruralhealth.va.gov/docs/cultural-competence-in-telehealth-clinics.pdf.

[CR112] Patel VL, Kaufman DR, Arocha JF. Emerging paradigms of cognition in medical decision-making. Journal of Biomedical Informatics. 2002;35(1):52–75. doi: 10.1016/s1532-0464(02)00009-6. [DOI] [PubMed] [Google Scholar]

[CR113] Piwek L, Ellis DA, Andrews S, Joinson A. The rise of consumer health wearables: promises and barriers. PLoS Medicine. 2016;13:e1001953. doi: 10.1371/journal.pmed.1001953. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR114] Proctor, E., Silmere, H., Raghavan, R., Raghavan, R., Hovmand, P., Aarons, G., ... & Hensley, M. (2010). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administrative Policy in Mental Health,38(2), 65–76. [DOI] [PMC free article] [PubMed]

[CR115] Ramkumar PN, Haeberle HS, Bloomfield MR, Schaffer JL, Kamath AT, Patterson BM, Krebs VE. Artificial intelligence and arthroplasty at a single institution: real-world applications of machine learning to big data, value-based care, mobile health, and remote patient monitoring. Journal of Arthroplasty. 2019;34(10):2204–2209. doi: 10.1016/j.arth.2019.06.018. [DOI] [PubMed] [Google Scholar]

[CR116] Ravizza A, De Maria C, Di Pietro L, Sternini F, Audenino AL, Bignardi C. Comprehensive review on current and future regulatory requirements on wearable sensors in preclinical and clinical testing. Frontiers in Bioengineering and Biotechnology. 2019;7:313. doi: 10.3389/fbioe.2019.00313. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR117] Ray G, Muhanna WA, Barney JB. Competing with IT: the role of shared IT-business understanding. Communications of the ACM. 2007;50(12):87–91. [Google Scholar]

[CR118] Rohani DA, Faurholt-Jepsen M, Kessing LV, Bardram JE. Correlations between objective behavioral features collected from mobile and wearable devices and depressive mood symptoms in patients with affective disorders: systematic review. Journal of Medical Internet Research mhealth uhealth. 2018;6(8):e165. doi: 10.2196/mhealth.9691. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR119] Roque F, Cechinel C, Weber TO, Lemos R, Villarroel R, Miranda D, Munoz R. Using depth cameras to detect patterns in oral presentations: a case study comparing two generations of computer engineering students. Sensors. 2019;19:3493. doi: 10.3390/s19163493. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR120] Royal College of Physicians and Surgeons Canada. (2015). CanMEDS physician competency framework. Retrieved from: http://canmeds.royalcollege.ca/uploads/en/framework/CanMEDS%202015%20Framework_EN_Reduced.pdf

[CR121] Russell MA, Gajos JM. Annual research review: ecological momentary assessment studies in child psychology and psychiatry. Journal of Child Psychology and Psychiatry. 2020;61(3):376–394. doi: 10.1111/jcpp.13204. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR122] Sano A, Phillips AJ, Yu AZ, McHill AW, Taylor S, Jaques N, Czeisler CA, Klerman EB, Picard RW. Recognizing academic performance, sleep quality, stress level, and mental health using personality traits, wearable sensors and mobile phones. International Conference on Wearable and Implantable Body Sensor Networks. 2015 doi: 10.1109/BSN.2015.7299420. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR123] Seko Y, Kidd S, Wiljer D, McKenzie K. Youth mental health interventions via mobile phones: a scoping review. Cyberpsycholology Behavioral and Social Networking. 2014;17(9):591–602. doi: 10.1089/cyber.2014.0078. [DOI] [PubMed] [Google Scholar]

[CR124] Seppälä, J., De Vita, I., Jämsä, T., Miettunen, J., Isohanni, M., Rubinsten, K., ... & Bulgheroni, M. (2019). Mobile phone and wearable sensor-based mhealth approaches for psychiatric disorders and symptoms: systematic review. Journal of Medical Internet Research Mental Health,6(2), e9819. 10.2196/mental.9819. [DOI] [PMC free article] [PubMed]

[CR125] Silva B, Rodrigues J, Diez I, Lopez-Coronado M, Saleem K. Mobile-health: a review of current state in 2015. Journal of Biomed Informatics. 2015;56:265–272. doi: 10.1016/j.jbi.2015.06.003. [DOI] [PubMed] [Google Scholar]

[CR126] Sockalingam S, Arena A, Serhal E, Mohri L, Alloo J, Crawford A. Building provincial mental health capacity in primary care: an evaluation of a project ECHO mental health program. Academic Psychiatry. 2018;42(4):451–457. doi: 10.1007/s40596-017-0735-z. [DOI] [PubMed] [Google Scholar]

[CR127] Solino-Fernandez D, Ding A, Bayro-Kaiser E, Ding EL. Willingness to adopt wearable devices with behavioral and economic incentives by health insurance wellness programs: results of a US cross-sectional survey with multiple consumer vignettes. BMC Public Health. 2019;19(1):1649. doi: 10.1186/s12889-019-7920-9. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR128] Statista. (2018). Connected wearable devices worldwide 2016–2021. Retrieved from: https://www.statista.com/statistics/487291/global-connected-wearable-devices/.

[CR129] Steinert, Y., Mann, K., Anderson, B., Barnett, B. M., Centeno, A., Naismith, L., ... & Dolmans, D. (2016). A systematic review of faculty development initiatives designed to enhance teaching effectiveness: a 10-year update: BEME Guide No. 40. Medical Teacher, 38(8), 769–786. 10.1080/0142159X.2016.1181851. [DOI] [PubMed]

[CR130] Steinhubl SR, Muse ED, Topol EJ. Can mobile health technologies transform health care? JAMA. 2013;310(22):2395–2396. doi: 10.1001/jama.2013.281078. [DOI] [PubMed] [Google Scholar]

[CR131] Sturmberg JP, O'Halloran D, Colagiuri R, Fernandez A, Lukersmith S, Torkfar G, Salvador-Carulla L. Health care frames - from Virchow to Obama and beyond: the changing frames in health care and their implications for patient care. Journal of Evaluation and Clinical Practice. 2014;20(6):1036–1044. doi: 10.1111/jep.12266. [DOI] [PubMed] [Google Scholar]

[CR132] Sunderji N, Crawford A, Jovanovic M. Telepsychiatry in graduate medical education: a narrative review. Academic Psychiatry. 2015;39(1):55–62. doi: 10.1007/s40596-014-0176-x. [DOI] [PubMed] [Google Scholar]

[CR133] Tachakra S, Wang XH, Istepanian RS, Song YH. Mobile e-health: the unwired evolution of telemedicine. Telemedicine Journal and E-Health. 2003;9(3):247–257. doi: 10.1089/153056203322502632. [DOI] [PubMed] [Google Scholar]

[CR134] Tekian A, Hodges BD, Roberts TE, Schuwirth L, Norcini J. Assessing competencies using milestones along the way. Medical Teacher. 2015;37(4):399–402. doi: 10.3109/0142159X.2014.993954. [DOI] [PubMed] [Google Scholar]

[CR135] Teshima J, Hodgins M, Boydell KM, Pignatiello A. Resident evaluation of a required telepsychiatry clinical experience. Academic Psychiatry. 2016;40(2):348–352. doi: 10.1007/s40596-015-0373-2. [DOI] [PubMed] [Google Scholar]

[CR136] Thompson WG, Kuhle CL, Koepp GA, McCrady-Spitzer SK, Levine JA. Go4Life exercise counseling, accelerometer feedback, and activity levels in older people. Archives of Gerontology and Geriatrics. 2014;58(3):314–319. doi: 10.1016/j.archger.2014.01.004. [DOI] [PubMed] [Google Scholar]

[CR137] Torous J, Roberts LW. The ethical use of mobile health technology in clinical psychiatry. Journal of Nervous and Mental Disorders. 2017;205(1):4–8. doi: 10.1097/nmd.0000000000000596. [DOI] [PubMed] [Google Scholar]

[CR138] Torous J, Chan S, Luo J, Boland R, Hilty D. Clinical informatics in psychiatric training: preparing today’s trainees for the already present future. Academic Psychiatry. 2017;42(5):694–697. doi: 10.1007/s40596-017-0811-4. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR139] Wahle F, Kowatsch T, Fleisch E, Rufer M, Weidt S. Mobile sensing and support for people with depression: a pilot trial in the wild. Journal of Medical Internet Research mhealth uhealth. 2016;4(3):e111. doi: 10.2196/mhealth.5960. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR140] Wang QE, Myers MD, Sundaram D. Digital natives and digital immigrants towards a model of digital fluency. Business and Information Systems Engineering. 2013;5(6):409–419. [Google Scholar]

[CR141] Winters, N., Langer, L., Nduku, P., Robson, J., O’Donovan, J., Maulik, P., ... & Nagraj, S. (2019). Using mobile technologies to support the training of community health workers in low-income and middle-income countries: mapping the evidence. BMJ Global Health,4(4), e001421. 10.1136/bmjgh-2019-001421. [DOI] [PMC free article] [PubMed]

[CR142] Winters N, Langer L, Geniets A. Scoping review assessing the evidence used to support the adoption of mobile health (mHealth) technologies for the education and training of community health workers (CHWs) in low-income and middle-income countries. British Medical Journal Open. 2018;8(7):e019827. doi: 10.1136/bmjopen-2017-019827. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR143] Witt DR, Kellogg RA, Snyder MP, Dunn J. Windows into human health through wearables data analytics. Current Opinion in Biomedical Engineering. 2019;9:28–46. doi: 10.1016/j.cobme.2019.01.001. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CR144] Yellowlees, P., Parish, M. B., González, Á., Chan, S., Hilty, D., Iosif, A., ... & Xiong, G. (2018). Asynchronous telepsychiatry: a component of stepped integrated care. Telemedicine Journal and E-Health,24(5), 375–378. [DOI] [PMC free article] [PubMed]

[CR145] Zalpuri I, Liu H, Stubbe D, Wrozek M, Hadsu J, Hilty DM. A competency-based framework for social media for trainees, faculty and others. Academic Psychiatry. 2018;42(6):808–817. doi: 10.1007/s40596-018-0983-6. [DOI] [PubMed] [Google Scholar]

Learning objective by learner level	Novice/advanced beginner	Competent/proficient	Advanced/expert
Learning and applying new wearable technology to practice
Learning objective* applied to wearable type	• Remember and understand wearable hardware, software, and sensors	• Apply wearable hardware, software, and sensors • Analyze impact of wearable type on patient care	• Understand, apply, and analyze wearables new to provider • Evaluate impact of wearable hardware, software, or sensors on patient outcomes
Teaching style	Didactic (e.g., classroom lecture style, grand rounds, continuing education training)	Didactic (e.g., classroom lecture style, grand rounds, continuing education training)	Didactic (e.g., classroom lecture style, grand rounds, continuing education training)
Competency domain(s)	• Medical knowledge • Patient care • Systems-based practice	• Medical knowledge • Patient care • Systems-based practice • Practice-based learning • Professionalism • Interpersonal and communication skills	• Patient care • Systems-based practice • Practice-based learning • Professionalism • Interpersonal and communication skills
Assessment methods	• Written test (multiple-choice/short answer)	• Written test (multiple-choice/short answer) • Oral presentation	• Oral presentation • Clinical outcomes • Chart review
Matching wearable to patient
Learning objective* applied to wearable type	• Understand and apply correct wearable type to clinical disorder	• Apply and evaluate impact of wearable type on patient care	• Evaluate the impact of new wearable type of patient care and outcomes • Evaluate impact of new wearable has on patient rapport
Teaching style	Applied learning with peers (e.g., “flipped” classroom; supervision, faculty; observing faculty, group observation, or co-Interviewing)	Applied learning with peers (e.g., “flipped” classroom; supervision, faculty; observing faculty, group observation, or co-interviewing)	Applied learning with peers (e.g., “flipped” classroom; supervision, faculty; observing faculty, group observation, or co-Interviewing)
Competency domain(s)	• Medical knowledge • Patient care • Systems-based practice	• Medical knowledge • Patient care • Systems-based practice • Practice-based learning	• Patient care • Systems-based practice • Interpersonal and Communicate Skill • Professionalism
Assessment methods	-Written test (multiple-choice/short answer) -Oral presentation	• Oral presentation • Clinical outcomes • Chart review • Formal evaluation (peer, supervisor, administration)	• Clinical outcomes • Chart review • Formal evaluation (peer, supervisor, administration)
Analyze and evaluate effectiveness of wearable
Learning objective* applied to wearable type	• Analyze and evaluate the effectiveness of wearable type on patient outcomes	• Analyze and evaluate the effectiveness of wearable type on patient outcomes • Evaluate impact use of wearable in practice	• Analyze and evaluate the effectiveness of wearable type on patient outcomes • Evaluate impact use of wearable in practice • Publish experience or clinical outcomes from wearable type and its impact on practice • Create new wearable hardware or software to fill a gap in patient care
Teaching style	Applied learning with peers (e.g., “flipped” classroom; supervision, faculty; observing faculty, group observation, or co-interviewing)	Professional self-evaluation and supervision	Professional self-evaluation and supervision
Competency domain(s)	• Patient care • Systems-based practice • Practice-based learning	• Patient care • Systems-based practice • Interpersonal and communication skill • Professionalism	• Patient care • Systems-based practice • Interpersonal and communication skill • Professionalism
Assessment methods	• Oral presentation	• Clinical outcomes • Chart review • Formal evaluation (peer, supervisor, administration)	-Clinical outcomes -Informal evaluation (peer-to-peer, student, administration)
Evaluate the impact of wearable on outcomes
Learning objective* applied to wearable type	Evaluate impact of wearable type based on patient feedback	• Evaluate impact of wearable type on patient outcomes across patients	• Evaluate impact of wearable type on patient outcomes in research • Evaluate impact of self-created new wearable type on patient experience and outcomes
Teaching style	Applied learning with peers (e.g., “flipped” classroom; supervision, faculty; observing faculty, group observation, or co-interviewing)	Quality improvement, evaluation, and research	Quality improvement, evaluation, and research
Competency domain(s)	• Patient care • Systems-based practice • Practice-based learning	• Patient care • System-based practice • Practice-based learning	• Patient care • System-based practice • Practice-based learning
Assessment methods	-Oral presentation	• Clinical outcomes • Chart review • Formal evaluation (administration)	• Clinical outcomes • Chart review • Formal evaluation (administration)
Teaching effectiveness
Learning objective* applied to wearable type	• Evaluate student or supervisee learning of wearable type	• Evaluate student or supervisee ability to communicate benefits and barriers of use of wearable type to others	• Evaluate student or supervisee ability to teach use of wearable type to others
Teaching style	Applied learning with peers (e.g., “flipped” classroom; supervision, faculty; observing faculty, group observation, or co-interviewing)	Teaching evaluation (e.g., student and peer feedback)	Teaching evaluation (e.g., student and peer feedback)
Competency domain(s)	• Medical knowledge • Practice-based learning • Interpersonal and communication skills	• Medical knowledge • System-based practice • Professionalism • Interpersonal and communication skills	• Medical knowledge • System-based practice • Professionalism • Interpersonal and communication skills
Assessment methods	• Oral presentation	• Written tests class average • Formal evaluation	• Written tests class average • Formal evaluation

PERMALINK

Sensor, Wearable, and Remote Patient Monitoring Competencies for Clinical Care and Training: Scoping Review

Donald M Hilty

Christina M Armstrong

Amanda Edwards-Stewart

Melanie T Gentry

David D Luxton

Elizabeth A Krupinski

Abstract

Introduction

Methods

Approach

Research Question

Identifying Relevant Studies

Study Selection

Fig. 1.

Data Charting

Analysis, Reporting, and the Meaning of Findings

Expert Opinions

Results

Expert Feedback

Table 1.

Table 3.

Literature

Sensor, Wearable, and Remote Patient Monitoring Technologies Evidence-Based

Technologies

Evidence-Based

Behavioral Health Profession and International Organization Position Statements and Guidelines on Telehealth and E-health

Table 2.

Shifting Perspectives for Patients, Clinicians, Teams, and Systems

Overview

Paradigm Shifts in Integrating Sensors and Wearables with In-person and Synchronous Care

Fig. 2.

Impact on Clinical, Technological, and Administrative Workflows

Health System and Institutional Approaches to Sensor and Wearable Competencies

Overview

Fig. 3.

Sensor, Wearable, and Remote Patient Monitoring Competencies

Teaching, Evaluation and Assessment: Clinical Care and Supervision in Training

Table 4.

Institutional Competencies

Discussion

Conclusions

Acknowledgments

Footnotes

Contributor Information

References

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases