Abstract
Digital health has been rapidly thrust into the forefront of care delivery. Poised to extend the clinician's reach, a new set of examination tools will redefine neurologic and neurosurgical care, serving as the basis for the digital neurologic examination. We describe its components and review specific technologies, which move beyond traditional video-based telemedicine encounters and include separate digital tools. A future suite of these clinical assessment technologies will blur the lines between history taking, examination, and remote monitoring. Prior to full-scale implementation, however, much more investigation is needed. Because of the nascent state of the technologies, researchers, clinicians, and developers should establish digital neurologic examination requirements in order to maximize its impact.
Keywords: Digital exam, Digital biomarkers, Wearables, Remote monitoring
Introduction
Digital health has been rapidly thrust into the forefront of care delivery [1, 2], which has been accelerated by the ongoing coronavirus disease 2019 (COVID-19) pandemic. Despite great potential, the bulk of digital health technologies have focused on wellness, not specific clinical indications, and with a largely unproven impact on health and disease [1, 2]. Technological advances and an increasing spotlight on clinical needs, however, have led to more specific intent and promise.
Poised to produce a new set of examination tools for clinicians, digital health clinical assessment technologies will redefine neurologic and neurosurgical care. They will serve as the basis for the digital neurologic examination. We describe the components of the prospective digital neurologic exam and review candidate technologies for each examination element. This new exam will advance telemedicine beyond basic telemedicine video-audio assessments. It will also advance remote monitoring beyond vital signs assessment. Despite the promise and existence of many digital health technologies relevant to neurological assessment, much work remains. The majority of these technologies are in the prototype, research, or conceptual phase. Thus, before entering the clinical sphere and becoming useful for patients, most require rigorous investigations in the populations and settings they promise to benefit. Thus, because of this nascent state the technologies, researchers, clinicians, and developers should establish requirements for the digital neurologic examination in order to maximize its impact. They should develop, test, and evaluate these tools for the patients, conditions, and environments in which the technologies will be deployed, including the patient home and work environments.
The neurological history and examination are the foundational tools of the effective neurologist and neurosurgeon. Examination instruments can be simple manual tools, like the reflex hammer, or complex machines like the ophthalmoscope. Needle electromyography (EMG) and nerve conduction studies are examples of additional complex tools widely touted as “extensions of the physical examination” [3]. Digital health neurologic tools are also poised to become extensions of the examination.
A digital toolbox (Table 1), as compared to the traditional neurologic examination version, may offer more quantitative, consistent, reliable, and reproducible results. This new approach to assessment and data collection comes with potential risk and costs. These include data overload, demand on data infrastructure, and greater difficulty in parsing signal and noise, among others. Unlike traditional methods, however, these new ones may produce high-frequency or continuous measurements. By collecting health data at times or environments (Table 2) outside traditional encounters, such as the clinic or inpatient ward, the approach may add value by effectively increasing access to care assessments. This allows patient assessment between, before, or after typical clinical encounters, such as in the home and on inpatient wards after clinical rounds. In other words, the digital neurology exam is relevant to all settings where patients exist.
Table 1.
Data capture concepts for the digital neurologic exam
| Feature | Description |
|---|---|
| Passive and active data collection | Patient-relevant data can be recorded without (passive) and with (active) patient participation. Ambient sensors in the home (e.g., video cameras) and wearables (e.g., wrist-based activity monitors) are examples that can generate passive data. A smartphone mobile app, which pings the patient to provide confirmation that a medication was ingested, represents active data entry |
| Frequent and continuous data collection | Asynchronous data: collected from patient at one time and location and reviewed by provider at another time and location Synchronous data: collected from patient and reviewed by provider at the same time (and either at the same or different locations) Asynchronous and synchronous can be recorded, analyzed, and interpreted, either during or between encounters while patient may be at home or work and in a clinic or hospital |
| Quantitative assessment | Quantitative data are produced by technologies that contribute to the digital neurologic exam, whereby patient-generated digital data allow assessments of nervous system functions, such as extremity movement and ambulation. Much in the way a clinician assesses multiple lines of data to reach a clinical assessment, multiple quantitative digital data sources can be fused to provide clinical analyses |
| Disease-specific assessments | Disease-relevant digital exam “modules” are tailored to the specific examination features of a disease. For ischemic stroke, these include blood pressure, heart rhythm, language, speech, motor, and gait assessments. For Parkinson disease, they include cognitive, behavioral, speech, motor, and gait assessments |
| Setting-specific assessments | Location-specific digital exam “modules” are also tailored to the needs and location of the patient and provider. For the clinic, automated retinal assessments using machine learning technology are deployed as digital kiosks. At home, patient retinal monitoring may require a separate set of retinal assessment technologies that interface with mobile phones |
Table 2.
Potential uses of the neurology digital exam in varied clinical environments
| Environment | Description |
|---|---|
| Clinic | Quantitative assessments that can be monitored over the patient course, such as voice quality with audio technology, tremor frequency with wearable accelerometers, and gait speed with activity trackers |
| Home and work | Remote monitoring providing frequent sampling of requisite patient-generated data in naturalistic settings such as blood pressure with advanced wearables, disease signs (e.g., limb movements and tremor) correlated with medication administration (via app-based self-reporting or digital pills), and seizure events with wearables or computer vision video capture |
| Wards and units | As with home and work, but tailored to the acute illness of the inpatient setting whereby providers spend limited time with patients and typically only capture vital signs between encounters or rely on subjective reports from nursing. Potentially useful inpatient neurology remote monitoring examples include seizure events with varied wearables (given unreliability of patient-generated button-pushing during continuous EEG monitoring), strength and activity monitoring through varied wearables, and speech and language assessments through audio technologies |
| Telemedicine | As above, but with real-time analysis and communication with the provider, the clinician can exploit the audio and video interface by using advanced speech, language, and motor computer technologies to both analyze current status and compare to prior states |
| Others | Most examples above apply to other environments, such as the operating room, emergency ward, during patient transfers, and during field-based care as is necessary in military, wilderness, and medic situations |
These digital data not only emulate the examination, but may also improve history taking. Thus, a future suite of these digital health technologies will blur the lines between history, examination, and remote monitoring. While activity and motion tracker output may recapitulate the gait examination, it may also provide details of the history, such as social excursion and wandering episode frequency in a dementia patient. Some of these new approaches, like the traditional exam, require active data collection in which someone must click a button or activate a technology in order to capture the data. Others, however, generate data passively, requiring no overt participation from the clinician, caregiver, or patient. Ultimately, the ability to combine the historical narrative, novel and traditional objective measures, and clinical acumen will improve the clinician's ability to prevent, detect, and manage disease [1].
The State of Digital Health Technologies
Digital health encompasses a wide range of uses and technologies that “use computing platforms, connectivity, software, and sensors for health care and related uses,” per the United States Food and Drug Administration (FDA) [4]. Current digital health technologies, while promising, largely operate in isolation and do not allow comprehensive neurological assessments or generally foster easy analysis by the clinician. If clinicians use varied individual technologies, each with separate proprietary interfaces and designs, they may become overwhelmed, as may the patient. Although clinical indications in neurology, neurosurgery, and psychiatry are commonly represented among digital health products [1], the technologies widely vary by level of technology readiness [5], development, verification, and validation. Further, the interoperability and usability of such technologies also varies widely and remains an important obstacle to widespread use and impact [6].
The regulation of digital health varies considerably, too, since digital health encompasses a wide range of technologies, applications, and use cases [4]. Pertinent to the digital neurologic exam, which implies assessment or monitoring technologies, digital health technologies include “software as a medical device” and medical devices. The regulatory landscape is changing, but the current requirements vary based on perceived clinical risk − technologies used for low-risk scenarios (e.g., smart watch activity tracker for exercise) do not require regulatory approval. Two additional pathways apply to such products: the FDA Software Precertification (Pre-Cert) Pilot Program essentially certifies technologies from a select list of pre-approved companies. Its Breakthrough Devices Program allows for fast-track approval of certain technologies.
Digital counterparts now exist for many elements of the neurologic exam (Table 3). Some aim to reproduce the physical exam in a digital form, such as transforming exam features from qualitative to quantitative. For example, pupilometers within virtual reality headsets capture pupillary size, which is comparable to traditional qualitative assessments performed at the bedside. Other digital approaches could offer entirely new capabilities such as language assessments using real-time analysis of low- and high-frequency word choices in patients with cognitive disorders.
An estimate of the technology readiness level for each element of the digital neurologic exam
a echnology readiness levels for digital health: the following rating scale has been adapted from NASA for clinical technology applications and use for digital health
| Technology readiness level | Description | Concept of implementation |
|---|---|---|
| 0 | Concept not yet explored | Basic research |
| 1 | Basic principles observed | |
| 2 | Technology concept formulated | |
| 3 | Experimental proof of concept | |
| 4 | Technology validated in lab | |
| 5 | Technology validated in relevant environment (clinically relevant environment in the case of key enabling technologies) | Clinical research |
| 6 | Technology demonstrated in relevant environment (clinically relevant environment in the case of key enabling technologies) | |
| 7 | System prototype demonstration in operational clinical environment | |
| 8 | Actual system completed and “patient ready” through test and demonstration (home, clinic, hospital) | Clinically ready |
| 9 | Actual system “patient ready” through successful real-world implementations | |
b Technology readiness levels for the digital neurologic exam: heat map of the estimated average technology readiness levels (colors shown in Table 3a) for the digital counterparts of the general and neurologic exams. Estimates were made by the authors, and do not represent a rigorous assessment of technology readiness levels for the technologies within each category − technologies within each category (row) are summarized in the body of the text in their corresponding sections
| Examination section | ||
|---|---|---|
| General | Vital signs | |
| General well-being and functional status | ||
| Cardiovascular | ||
| Respiratory | ||
| Mental status and behavior | ||
| Cognition | ||
| Affect and behavior | ||
| Activities of daily living | ||
| Cranial nerves | Olfaction | |
| Vision | ||
| Pupils | ||
| Eye and lid movement | ||
| Facial movement | ||
| Hearing and co-ordination | ||
| Speaking and swallowing | ||
| Head and neck movement | ||
| Tongue movement | ||
| Exocrine function | ||
| Motor | Strength | |
| Somatosensory function | Somatosensory function | |
| Reflexes | Reflexes | |
| Gait | Gait | |
c Digital counterparts of the general and neurologic exam with technology readiness levels: Part I − for each category, which represents multiple technologies and applications representing a range of technology readiness levels, an estimated average technology readiness level is provided for the entire category
| Exam feature | Technology | Technology readiness level |
|---|---|---|
| General | ||
| Vital signs | Varied wearables (headband, watch, wrist band, skin patch) | 7 |
| General well-being and functional status | App-based survey (patient-reported data) and varied wearables (position sensor, activity tracker) | 4 |
| Cardiovascular | Heart rate and rhythm: wearables (optical or mechanical pressure sensors); heart sounds: digital stethoscope; vascular patency: digital camera | 5 |
| Respiratory | Respiratory rate and rhythm: wearables (fabric-embedded sensors), digital stethoscope | 3 |
| Skin | Varied pathologies, including rashes and wounds: digital cameras | 4 |
| Mental status and behavior | ||
| Arousal/sleep | Wrist band actigraphy trackers, mattress-based and wearable sensors, EEG-embedded headbands | 5 |
| Cognition | Computer and app-based survey and assessments; audio and text-reading technologies: natural language processing software; internet-use tracking1 | 4 |
| Affect and behavior | Mental state: app-based survey (patient-reported data); audio technologies (vocal biomarkers); autonomic physiological state (e.g., anxiety biomarkers): varied wearables (as above) | 3 |
| Activities of daily living | App-based survey and assessments; location tracking, “smart home systems” (monitors, observatories) | 3 |
d Digital counterparts of the neurologic exam with technology readiness levels: Part II
| Technology readiness level | ||
|---|---|---|
| Cranial nerves | ||
| Olfaction | Mobile/web-based survey apps (algorithmic guidance on self-assessment) | 4 |
| Vision | Mobile/web-based apps (acuity, color, contrast, Amsler grid); digital fundoscopy | 4 |
| Pupils | Eye tracking (virtual/augmented reality headsets and fixed optical video sensors with computer vision software) | |
| Eye and lid movement | Virtual/augmented reality headsets, fixed optical video sensors (with computer vision software) | 3 |
| Facial movement | Facial recognition software (via optical/video sensors on mobile devices, computers, and smart mirrors) | 3 |
| Facial sensation | N/A | N/A |
| Hearing and co-ordination | Mobile auditory technologies; nystagmus/eye movements (see eye and lid movement section); posture, movement, and sway: varied wearables (accelerometer, position sensor) | 4 |
| Speaking and swallowing | Voice/cough analysis and swallow sound audio mobile and computer-based technologies | 3 |
| Head and neck movement | N/A | N/A |
| Tongue movement | N/A | N/A |
| Exocrine function | N/A | N/A |
| Motor function and co-ordination | ||
| Strength | Digital dynamometers | 3 |
| Movement, tremor, and co-ordination | Varied wearables (e.g., accelerometer, mouse/tablet tracking) | 4 |
| Somatosensory function | N/A | N/A |
| Reflexes | N/A | N/A |
| Gait | Activity trackers (e.g., smartband), insole shoe sensors, skin-mounted inertial wearables | 4 |
Internet use tracking: examples include sites visited (e.g., gambling) and content consumed and speed of data entry (e.g., to gauge mental ability trends).
While the clinician can test attention, memory, visual acuity, and color discrimination, a host of digital health apps can, too. Many exam elements can be captured through basic audio and video telemedicine technologies. These options span a wide array of technologies types from varied digital audio-video interfaces to machine learning-powered augmented reality systems.
Emerging wearables, for example, gather detailed information on patient motion, which offer novel means to assess limb movements, tremor, and gait. Beyond raw data capture, embedded software-powered analytics enhance perception in ways not possible with standard clinical encounters. As another example, digital capture of speech, keyboard typing, and limb movement can be transmitted to clinicians for review, but machine learning approaches may be better suited to digest such large data sets to better identify trends indicating early disease and clinical change [7, 8].
A host of diverse technologies are emerging to either enhance the virtual exam or collect pertinent health data previously unmeasurable outside clinics and hospitals. These technologies could complement the remote neurologist-guided examination, but also could be used by the patient for self-assessment and self-monitoring. The technologies span a wide array from wearable biosensors to machine learning-powered augmented reality systems.
The Digital General Exam
The following aims to review the suite of emerging or available technologies pertinent to the digital exam. They have varying degrees of clinical validation and varying relevance to different neurologic populations and settings. As with the traditional physical examination, digital tools would also be tailored to specific patient needs based on the clinical localization, suspicion, status, and disease.
The neurologic exam begins with a general examination, including an assessment of vital signs. Apart from activity trackers, wearables able to capture vital signs represent a leading digital health technology [1]. Every vital sign can be assessed, nearly continuously, through a wide array of sensor types and form factors, including headbands, watches, wrist bands, and skin patches [9]. The patient's general well-being, position, and behavior, along with other basic activities may be classified (lying, sitting, standing, walking, stationary biking, and sleep) by using a suite of integrated technologies producing varied patient digital data such as vital signs, step count, position, and posture [10]. As another example, digital biomarkers of activity appear to correlate with pain, which can be verified with patient-reported outcome measures [10].
Cardiovascular digital health approaches have had tremendous success with the advent of wearables to assess pulse and rhythm. Technologies using optical or mechanical pressure sensors can capture heart rate and rhythm information [12]. Such technologies are increasingly tested for their ability to detect arrhythmias such as atrial fibrillation through watch-embedded wrist sensors [12, 13, 14, 15, 16]. Digital stethoscopes can remotely transmit auscultatory information relevant to adventitious heart sounds such as murmurs, gallops, and rubs [17]. A smartphone camera lens placed on the finger outperformed a traditional physical examination (the Allen test) to confirm vascular (ulnar) patency, expanding the territory of remote vascular assessment [18].
The auscultatory portions of the digital respiratory exam may also be captured with a digital stethoscope. Novel form factors, such as low-profile fabric-embedded sensors, paired with machine learning algorithms aim to capture digital auscultatory to track specific pathologic states (e.g., congestive heart failure, asthma) that may produce respiratory crackles or wheezes [19, 20]. Beyond the capture of raw sound data to be evaluated by remote clinicians, such data may be assessed by machines for automated analysis and monitoring.
Optical sensors in smartphones transmit photographic information useful for the digital dermatologic exam. These data can feed machine learning algorithmic assessments such as for wounds, malignancy, and infection, such as erythema migrans with Lyme disease [21].
The Digital Neurologic Exam
Current technologies to assess arousal or level of consciousness primarily focus on sleep, but could be expanded for other purposes. Sleep-focused technological approaches include wrist band actigraphy trackers, mattress-based and wearable sensors to detect thoracic movements associated with respiration, and EEG-embedded headbands [22]. The data to support technologies to reliably classify sleep status and stage, and particularly actigraphy as a sole method, is limited [22, 23, 24, 25]. Very limited data exist to support such technologies for assessing arousal states in populations with delirium, dementia, or coma [26, 27].
A host of standard cognitive tests have been optimized for computer- and app-based mobile phone administration, which variably correlate with gold standard, lab-based testing. These may be self- or caregiver-administered in the same way a neurologist would perform the tests. These tests relate to standard cognitive assessment domains like attention, processing speed, language, memory, general knowledge, abstraction, judgment, insight, and reasoning using varied tests like Mini Mental State Examination, Stroop, Reaction Time, Trail Making, N-Back, and Memory-Word [27, 28, 29].
A suite of new tools exploits the digital interface to allow new cognitive testing methods. As an example, the Color-Shape Test (CST), which focuses on processing speed, was designed specifically as a digital health tool for tablets, and correlated with global performance measures from traditional assessments of cognitive function [30]. Natural language processing and machine learning software, which can ingest data from smartphone voice sensors and writing samples (text, email), can provide real-world assessments of language function. Furthermore, the nature by which patients interacting with the Web produces digital phenotypes that also reveal cognitive function, as has been demonstrated in Alzheimer and Parkinson patients [31]. Additionally, novel measures to detect wandering gathered through smartwatch-based location tracking assess cognition by proxy [32]. Smart home systems that track patients throughout the home also can infer cognitive function by classifying and monitoring activities of daily living [27]. Eye tracker and camera-based pupillometric measurements may indicate so-called cognitive load and the affective experience, an approach used by research psychologists [33, 34].
An array of digital biomarkers of disease may also provide valuable insights into patient affect and behavior. For example, vocal biomarkers from voice recordings provide both linguistic and paralinguistic features for which machine learning techniques may be used to identify and monitor depression [35]. Further, autonomic physiological (heart rate, respiratory rate, skin temperature) digital biomarkers, another exploratory area of digital health, collected from varied wearables may identify stress or anxiety levels [36].
Remote olfactory evaluations can be performed through an algorithmic self-assessment approach using an array of commonly available household items [37]. This approach has been repurposed for COVID-19 remote self-assessment.
The afferent visual system may be interrogated through myriad digital health technologies. Simple mobile apps used during the routine virtual visit or at home by the patient can monitor visual acuity (digital Snellen charts), color discrimination (digital color plates), contrast sensitivity, macular integrity (digital Amsler grid), and visual fields (digital perimetry) [38]. As with the traditional exam, all such components must be properly implemented, such as with standardized distance for testing and digital device font size. Many web- and smartphone-based tools were mainly studied in ophthalmic populations but could be repurposed and validated in neurologic and neuro-ophthalmic populations. These include handheld shape discrimination hyperacuity testing to allow self-monitoring for retinal disease patients [39, 40].
Smartphone-powered camera attachments can collect anterior ocular and fundoscopic data to be transmitted to the physician. These images can be analyzed with machine learning algorithms for diagnosis, grading, and even prognosis prediction for such diseases as diabetic retinopathy and age-related macular degeneration [41, 42, 43, 44, 45, 46].
Eye-tracking glasses or headsets with pupilometers can assess for efferent visual (like strabismus) problems and abnormalities of the pupils. These technologies have been used to assess children with strabismus and ocular motor palsies [47, 48, 49]. These and other forms of video-oculography may be useful as tools for acute presentations, for example, to detect ischemic stroke in new-onset vertigo [50].
The position of the lids and palpebral fissure width are amenable to automated camera-based assessment through facial recognition software [51]. Infrared sensors have also been developed to assess lid function in myasthenic patients by gleaning both palpebral fissure width and blink rate [52].
Facial movement, symmetry, and expressions may be assessed with facial recognition software. In the context of facial reanimation surgery after facial palsy, this technology has been used to provide automated palsy scale scoring [53]. As with many other technologies, the cameras and software intrinsic to this approach may be housed in a variety of forms, including desktop and mobile devices or even smart mirrors [54].
Hearing assessments have seen a rise in digital appraisal options. A host of remote auditory screening technologies use off-the-shelf in-ear on on-ear technologies. Some rival the utility of the traditional audiogram performed by audiologists using dedicated facilities and equipment. For example, a portable diagnostic audiometer to capture pure tone air conduction thresholds has been deployed for live telemedicine-based hearing assessments for patients in remote settings [55]. This approach appeared to match traditional audiometry performed in the traditional clinic-based audiogram booth [55].
Nystagmus can also be assessed with eye tracking technology to allow evaluation of the vestibular system [56]. Assessments of ocular fixation and movement may assist in the digital vestibular evaluation [57]. Varied wearables that monitor posture, movement, and sway allow balance and coordination assessments.
Speech may also be assessed with smartphone-based audio recording and voice analysis software [58]. Cough analysis is possible with varied sound recording technologies, some of which can passively collect background sounds from smartphones or environmental observatories in the patient's proximity [59].
Digital dynamometers quantify motor function such as at the shoulder or hand. They typically require a participant other than the patient, such as a clinician, to record measurements [60]. Wearable surface EMG sensor signals could serve as dynamometer surrogates for strength [61].
Accelerometer-based smartphone assessments of angular movement over time may also be used to track motor movements [62]. Body-worn sensors, which have been studied in rotator cuff surgery patients, can track arm movements over time to monitor recovery [63]. Similar approaches could be used to assess motor function and recovery in neurological populations, including stroke, demyelinating disease, and spinal cord injury.
Many investigators have explored digital motor assessments outside strength. Smartwatches and other wearables collecting data from the arms may allow assessments and monitoring of tremor, bradykinesia, or chorea such as in Parkinson disease and Huntington disease [7, 64, 65, 66]. Collecting data from computer mouse movements or tablet tapping may also provide information on tremor and motor function [31, 67].
Finally, activity trackers (e.g., smartbands), insole shoe sensors, and skin-mounted inertial wearable sensors have all been used to assess gait [58, 68, 69]. We are unaware of any published information on digital tools to assess the remaining cranial nerve systems (trigeminal sensory system; head, neck, and tongue movement; exocrine function), the somatosensory system, or reflexes.
Integration into Practice
There are 2 basic ways to capture meaningful digital data for neurologic patients. The first is to fully replicate the routine neurologic exam, gathering the same data but with digital health tools instead of standard ones. This brighter candle model improves efficiency, but may not fully exploit the technology's potential or produce a paradigm shift. The other way is to reimagine the neurologic exam using new tools to better match clinical needs. This is the electric bulb model.
The digital neurologic exam can represent both − the brighter candle model improves clinician examination efficiency by duplicating the current exam but using technology to gather the same information in new settings and times. The electric bulb model reimagines the clinician and patient roles by incorporating new and continuous physiologic data, behaviors, and concepts extending far beyond the capabilities of the traditional exam.
The digital neurology exam has immediate potential practical uses. Prior to full-scale implementation, however, much verification and validation are needed. Most such technologies are not ready for prime time (Table 3). Table 3 provides an estimate of the technology readiness level [5] for each element of the digital neurologic exam, whereby some technologies have been prototyped and researched in a lab setting, while others have been successfully demonstrated or validated in clinical environments.
Requirements for the digital exam should directly address clinical- and patient-oriented needs. For example, a requirements framework for the digital neurologic exam of a stroke patient would address blood pressure, heart rhythm, language, speech, motor function, and gait. Other logical groupings of digital exam components would mandate a separate list of requirements, such as the suite of technologies intended to capture the afferent and efferent neuro-ophthalmic exam, the motor exam, the gait exam, and so forth.
For each of the digital exam elements outlined above, the technology should be developed to meet specific clinically appropriate end-user requirements and tested against a reference standard where available [70]. Furthermore, the technology should be studied in the relevant population and operational environment, such as in patients with epilepsy at home or stroke patients in the hospital.
The near-term potential use cases are in the clinic, on the wards, and during telemedicine encounters. Within each of these environments, as is done with the traditional exam, the digital neurology exam should be tailored to the clinical environment and the specific clinical problem and patient being addressed.
In addition to validated condition-specific assessment technologies, more is needed. Even if the technology accurately captures a useful exam element like gait speed, the usefulness of any particular technology may diminish as more technologies are applied to the patient. A patient or clinician can be quickly overwhelmed with 10 technologies, each capturing a different exam element, which use different data visualization interfaces. When data arise from multiple technologies, they must be fused into sensible, usable assessments. For example, a global gait digital assessment could integrate gait speed, stride length, and quality that could be viewed in 3 separate platforms or integrated into one. A stroke patient benefits from ongoing assessment of blood pressure, heart rhythm, language, speech, motor function, and gait. Currently, this would require the use of multiple technologies from multiple vendors, which would be challenging to manage for the patient and clinician.
To achieve wide adoption, the following steps must occur, many of which have been outlined elsewhere [70, 71, 72, 73]. Neurologic digital health technology should focus on what should be done − not what can be done. More data from more technologies do not equate to better information that will improve neurological care. Thus, the essential requirements for neurological assessment should be established. A requirements-driven approach [70] is led by engineers working with clinicians, patients, and other key stakeholders including professional organizations like the American Neurological Association and American Academy of Neurology. Digital health organizations, such as Digital Medicine Society, Digital Therapeutics Alliance, HealthXL, and NODE. Health, may also help generate such requirements as should digital health and health technology companies.
One of the key current barriers to the digital neurologic exam is the compartmentalization of approaches and technologies. If the Parkinson disease patient requires periodic digital assessments that include cognition, behavior, speech, motor function, and gait, prescribing 5 or more separate technologies is likely to fail if this requires purchasing 5 different vendors that give different analysis and viewing platforms. Thus, the requirements must foster an integrated digital exam that is affordable, easy to use and interpret, and produces meaningful actions. Currently, no such approach exists, which represents a great opportunity.
Another barrier to practice integration is technology interoperability. To achieve such integration, a standard reference interoperability architecture is needed to allow data exchange among the varied technologies, and the electronic medical record. Although the architecture or its custodian does not yet exist, early versions such as the Medical Device Interoperability Reference Architecture (MDIRA) [74] could address digital health technologies, and a host of standard developers and stakeholders could produce and maintain it.
Next, the technologies themselves must be assessed. Extensive research is already occurring on them, but much more is required. Funding organizations, such as the National Institute of Neurological Disorders and Stroke within the USA, National Institutes of Health, or the Patient-Centered Outcomes Research Institute, could drive much of the required research investigation to improve technology-based neurological assessments. The investigations should focus on the effect on clinical outcome, cost, or care access, rather than whether or not the technologies can simply be implemented or mirror traditional assessments.
Hospital, provider, and health systems can also help lead this charge. Video-based telemedicine, remote monitoring, and virtual hospital programs will all benefit from the digital neurologic exam. As noted, requirements must be determined for specific patient populations in varied settings using these different, emerging approaches to care, all of which care systems can be leading.
Once these technologies have been proven effective and useful in research settings, clinicians and patients require guidance on which ones to use. Guidance could come from the FDA's Digital Health Center of Excellence and the aforementioned professional clinical and digital health organizations. Healthcare systems and industry could also provide guidance − Elektra Labs, for example, provides organizations guidance on choosing select digital health assessment technologies based on clinical or research needs. They and other allied companies could collectively recommend a suite of neurologic assessment technologies, tailored to patient needs, and integrate both with the patient's life (e.g., “smart home,” “smart clothing”) and the provider workflow (e.g., electronic health record interoperability).
If successful, Parkinson patients would come to the clinic with more than a verbal history and a paper journal. The home digital exam could gather cognitive testing, eye tracker measurements, digital voice recording, and limb movement assessments in the days, weeks, or months prior to the routine surveillance clinic visit. A suite of integrated technologies that captures all pertinent data for the particular patient and condition must be integrated together and with common analytics, visualization, and reporting platforms.
Like the tools of a traditional neurologic exam, no one technology assessment tool is a panacea. The goal is to produce the right mix of technologies and the data they produce. Otherwise, these technologies will produce an endless sea of data, potentially adding time and cost to an already overburdened care system. A thoughtful, requirements-driven approach to the digital neurologic exam is likely to improve and revolutionize neurological and neurosurgical care.
Conflict of Interest Statement
Dr. Adam B. Cohen consults for Pear Therapeutics and Thirty Madison, and has stock options in the later. Dr. Brian V. Nahed has founder shares in and serves as a consultant for REACT Neuro.
Funding Sources
This work required no funding source.
Author Contributions
A.B.C. and B.V.N. contributed to the preparation and writing of this manuscript. A.B.C. executed its conception and initial design.
References
- 1.Cohen AB, Dorsey ER, Mathews SC, Bates DW, Safavi K. A digital health industry cohort across the health continuum. NPJ Digit Med. 2020 May;3((1)):68–10. doi: 10.1038/s41746-020-0276-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Safavi K, Mathews SC, Bates DW, Dorsey ER, Cohen AB. Top-Funded Digital Health Companies And Their Impact On High-Burden, High-Cost Conditions. Health Aff (Millwood) 2019 Jan;38((1)):115–23. doi: 10.1377/hlthaff.2018.05081. [DOI] [PubMed] [Google Scholar]
- 3.Model Policy for Needle Electromyography and Nerve Conduction Studies American Association of Neuromuscular & Electrodiagnostic Medicine. 2011 Feb; Accessed on December 22, 2020 at https://www.aanem.org/getmedia/65934187-d91e-4336-9f3c-50522449e565/Model-Policy.pdf. [Google Scholar]
- 4.What is Digital Health? https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health.
- 5.Technology Readiness Level NASA. 2012 Oct 28; Accessed on February 1, 2021 at https://www.nasa.gov/directorates/heo/scan/engineering/technology/txt_accordion1.html. [Google Scholar]
- 6.Lehne M, Sass J, Essenwanger A, Schepers J, Thun S. Why digital medicine depends on interoperability. NPJ Digit Med. 2019 Aug;2((1)):79–5. doi: 10.1038/s41746-019-0158-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Lonini L, Dai A, Shawen N, Simuni T, Poon C, Shimanovich L, et al. Wearable sensors for Parkinson's disease: which data are worth collecting for training symptom detection models. NPJ Digit Med. 2018 Nov;1((1)):64–8. doi: 10.1038/s41746-018-0071-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Lahmiri S, Dawson DA, Shmuel A. Performance of machine learning methods in diagnosing Parkinson's disease based on dysphonia measures. Biomed Eng Lett. 2017 Oct;8((1)):29–39. doi: 10.1007/s13534-017-0051-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Piwek L, Ellis DA, Andrews S, Joinson A. The Rise of Consumer Health Wearables: promises and Barriers. PLoS Med. 2016 Feb;13((2)):e1001953. doi: 10.1371/journal.pmed.1001953. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Sen-Gupta E, Wright DE, Caccese JW, Wright JA, Jr, Jortberg E, Bhatkar V, et al. A Pivotal Study to Validate the Performance of a Novel Wearable Sensor and System for Biometric Monitoring in Clinical and Remote Environments. Digit Biomark. 2019 Mar;3((1)):1–13. doi: 10.1159/000493642. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Perraudin CG, Illiano VP, Calvo F, O'Hare E, Donnelly SC, Mullan RH, et al. Observational Study of a Wearable Sensor and Smartphone Application Supporting Unsupervised Exercises to Assess Pain and Stiffness. Digit Biomark. 2018 Oct;2((3)):106–25. doi: 10.1159/000493277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Walsh JA, 3rd, Topol EJ, Steinhubl SR. Novel wireless devices for cardiac monitoring. Circulation. 2014 Aug;130((7)):573–81. doi: 10.1161/CIRCULATIONAHA.114.009024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Reiffel JA, Verma A, Kowey PR, Halperin JL, Gersh BJ, Wachter R, et al. REVEAL AF Investigators Incidence of Previously Undiagnosed Atrial Fibrillation Using Insertable Cardiac Monitors in a High-Risk Population: the REVEAL AF Study. JAMA Cardiol. 2017 Oct;2((10)):1120–7. doi: 10.1001/jamacardio.2017.3180. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Perez MV, Mahaffey KW, Hedlin H, Rumsfeld JS, Garcia A, Ferris T, et al. Apple Heart Study Investigators Large-scale assessment of a smartwatch to identify atrial fibrillation. N Engl J Med. 2019 Nov;381((20)):1909–17. doi: 10.1056/NEJMoa1901183. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Bumgarner JM, Lambert CT, Hussein AA, Cantillon DJ, Baranowski B, Wolski K, et al. Smartwatch Algorithm for Automated Detection of Atrial Fibrillation. J Am Coll Cardiol. 2018 May;71((21)):2381–8. doi: 10.1016/j.jacc.2018.03.003. [DOI] [PubMed] [Google Scholar]
- 16.Dörr M, Nohturfft V, Brasier N, Bosshard E, Djurdjevic A, Gross S, et al. The WATCH AF Trial: SmartWATCHes for Detection of Atrial Fibrillation. JACC Clin Electrophysiol. 2019 Feb;5((2)):199–208. doi: 10.1016/j.jacep.2018.10.006. [DOI] [PubMed] [Google Scholar]
- 17.Lakhe A, Sodhi I, Warrier J, Sinha V. Development of digital stethoscope for telemedicine. J Med Eng Technol. 2016;40((1)):20–4. doi: 10.3109/03091902.2015.1116633. [DOI] [PubMed] [Google Scholar]
- 18.Di Santo P, Harnett DT, Simard T, Ramirez FD, Pourdjabbar A, Yousef A, et al. Photoplethysmography using a smartphone application for assessment of ulnar artery patency: a randomized clinical trial. CMAJ. 2018 Apr;190((13)):E380–8. doi: 10.1503/cmaj.170432. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.West E, McLane I, the DMTJO, 2019 Introducing Feelix, a digital stethoscope incorporating active noise control and automatic detection of lung sound abnormalities. J Acoustical Society of America. 2019;145((e1923)) [Google Scholar]
- 20.Sharma P, Newman K, Long CS, Gasiewski AJ, Barnes F. Use of Wavelet Transform to Detect Compensated and Decompensated Stages in the Congestive Heart Failure Patient. Biosensors (Basel) 2017 Sep;7((3)):40. doi: 10.3390/bios7030040. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Burlina PM, Joshi NJ, Ng E, Billings SD, Rebman AW, Aucott JN. Automated detection of erythema migrans and other confounding skin lesions via deep learning. Comput Biol Med. 2019 Feb;105:151–6. doi: 10.1016/j.compbiomed.2018.12.007. [DOI] [PubMed] [Google Scholar]
- 22.Bianchi MT. Sleep devices: wearables and nearables, informational and interventional, consumer and clinical. Metabolism. 2018 Jul;84:99–108. doi: 10.1016/j.metabol.2017.10.008. [DOI] [PubMed] [Google Scholar]
- 23.Lee J, Finkelstein J. Consumer sleep tracking devices: a critical review. Stud Health Technol Inform. 2015;210:458–60. [PubMed] [Google Scholar]
- 24.Sathyanarayana A, Joty S, Fernandez-Luque L, Ofli F, Srivastava J, Elmagarmid A, et al. Sleep Quality Prediction From Wearable Data Using Deep Learning. JMIR Mhealth Uhealth. 2016 Nov;4((4)):e125. doi: 10.2196/mhealth.6562. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Rosenberger ME, Buman MP, Haskell WL, McConnell MV, Carstensen LL. Twenty-four Hours of Sleep, Sedentary Behavior, and Physical Activity with Nine Wearable Devices. Med Sci Sports Exerc. 2016 Mar;48((3)):457–65. doi: 10.1249/MSS.0000000000000778. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Davoudi A, Manini TM, Bihorac A, Rashidi P. Role of Wearable Accelerometer Devices in Delirium Studies: A Systematic Review. Crit Care Explor. 2019 Sep;1((9)):e0027. doi: 10.1097/CCE.0000000000000027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Chinner A, Blane J, Lancaster C, Hinds C, Koychev I. Digital technologies for the assessment of cognition: a clinical review. Evid Based Ment Health. 2018 May;21((2)):67–71. doi: 10.1136/eb-2018-102890. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Tieges Z, Stíobhairt A, Scott K, Suchorab K, Weir A, Parks S, et al. Development of a smartphone application for the objective detection of attentional deficits in delirium. Int Psychogeriatr. 2015 Aug;27((8)):1251–62. doi: 10.1017/S1041610215000186. [DOI] [PubMed] [Google Scholar]
- 29.Jongstra S, Wijsman LW, Cachucho R, Hoevenaar-Blom MP, Mooijaart SP, Richard E. Cognitive Testing in People at Increased Risk of Dementia Using a Smartphone App: The iVitality Proof-of-Principle Study. JMIR Mhealth Uhealth. 2017;5((5)):e68. doi: 10.2196/mhealth.6939. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Brouillette RM, Foil H, Fontenot S, et al. Feasibility and validity of a smartphone based application for the assessment of cognitive function in the elderly. PLoS One. 2013 Jun;8((6)):e65925. doi: 10.1371/journal.pone.0065925. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.White RW, Doraiswamy PM, Horvitz E. Detecting neurodegenerative disorders from web search signals. NPJ Digit Med. 2018 Apr;1((1)):8–4. doi: 10.1038/s41746-018-0016-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Cachia C, Attard C, Montebello M. WanderRep: a reporting tool for caregivers of wandering persons with dementia. Xjenza. 2014;2((2)):2–7. [Google Scholar]
- 33.Palinko O, Kun AL, Shyrokov A, Heeman P. Estimating cognitive load using remote eye tracking in a driving simulator. Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications. 2010 Mar; Austin, Texas. [Google Scholar]
- 34.Kahneman D. Thinking, Fast and Slow. New York, New York: Farrar, Straus and Giroux; 2011. [Google Scholar]
- 35.Ozkanca Y, Öztürk MG, Ekmekci MN, Atkins DC, Demiroglu C, Ghomi RH. Depression Screening from Voice Samples of Patients Affected by Parkinson's Disease. Digit Biomark. 2019 May-Aug;3((2)):72–82. doi: 10.1159/000500354. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.van Fenema EM, Gal P, van de Griend MV, Jacobs GE, Cohen AF. A Pilot Study Evaluating the Physiological Parameters of Performance-Induced Stress in Undergraduate Music Students. Digit Biomark. 2018 Jan;1((2)):118–25. doi: 10.1159/000485469. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Snitz K, Perl O, Honigstein D, Secundo L, Ravia A, Yablonka A, et al. SmellSpace: An Odor-Based Social Network as a Platform for Collecting Olfactory Perceptual Data. Chem Senses. 2019 Apr;44((4)):267–78. doi: 10.1093/chemse/bjz014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Prea SM, Kong YX, Mehta A, He M, Crowston JG, Gupta V, et al. Six-month Longitudinal Comparison of a Portable Tablet Perimeter With the Humphrey Field Analyzer. Am J Ophthalmol. 2018 Jun;190:9–16. doi: 10.1016/j.ajo.2018.03.009. [DOI] [PubMed] [Google Scholar]
- 39.Wang YZ, He YG, Mitzel G, Zhang S, Bartlett M. Handheld shape discrimination hyperacuity test on a mobile device for remote monitoring of visual function in maculopathy. Invest Ophthalmol Vis Sci. 2013 Aug;54((8)):5497–505. doi: 10.1167/iovs.13-12037. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Chew EY, Clemons TE, Bressler SB, Elman MJ, Danis RP, Domalpally A, et al. AREDS2-HOME Study Research Group Randomized trial of a home monitoring system for early detection of choroidal neovascularization home monitoring of the Eye (HOME) study. Ophthalmology. 2014 Feb;121((2)):535–44. doi: 10.1016/j.ophtha.2013.10.027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Burlina P, Galdran A, Costa P, Cohen A, Campilho A. Computational Retinal Image Analysis. Academic Press; 2019. Artificial intelligence and deep learning in retinal image analysis; pp. pp. 379–404. [Google Scholar]
- 42.Winther C, Frisén L. Self-Testing of Vision in Age-Related Macula Degeneration: A Longitudinal Pilot Study Using a Smartphone-Based Rarebit Test. J Ophthalmol. 2015;2015((1)):285463–7. doi: 10.1155/2015/285463. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Yu SY, Yang JH, Kim Y, Kwak HW, Blumenkranz M. Reliability of Smartphone-Based Electronic Visual Acuity Testing: (Applications in Remote Monitoring and Clinical Research of Macular Pathology) Invest Ophthalmol Vis Sci. 2014;55((13)):5598–5598. [Google Scholar]
- 44.Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM. Automated Grading of Age-Related Macular Degeneration From Color Fundus Images Using Deep Convolutional Neural Networks. JAMA Ophthalmol. 2017 Nov;135((11)):1170–6. doi: 10.1001/jamaophthalmol.2017.3782. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Burlina P, Pacheco KD, Joshi N, Freund DE, Bressler NM. Comparing humans and deep learning performance for grading AMD: A study in using universal deep features and transfer learning for automated AMD analysis. Comput Biol Med. 2017 Mar;82:80–6. doi: 10.1016/j.compbiomed.2017.01.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Burlina PM, Joshi N, Pacheco KD, Freund DE, Kong J, Bressler NM. Use of Deep Learning for Detailed Severity Characterization and Estimation of 5-Year Risk Among Patients With Age-Related Macular Degeneration. JAMA Ophthalmol. 2018 Dec;136((12)):1359–66. doi: 10.1001/jamaophthalmol.2018.4118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Chen ZH, Fu H, Lo WL, Chi Z, Xu B. Eye-tracking-aided digital system for strabismus diagnosis. Healthc Technol Lett. 2018 Jan;5((1)):1–6. doi: 10.1049/htl.2016.0081. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Samadani U, Farooq S, Ritlop R, Warren F, Reyes M, Lamm E, et al. Detection of third and sixth cranial nerve palsies with a novel method for eye tracking while watching a short film clip. J Neurosurg. 2015 Mar;122((3)):707–20. doi: 10.3171/2014.10.JNS14762. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Chen Z, Fu H, Lo WL, Chi Z. Strabismus Recognition Using Eye-Tracking Data and Convolutional Neural Networks. J Healthc Eng. 2018 Apr;2018((4)):7692198–9. doi: 10.1155/2018/7692198. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Newman-Toker DE, Curthoys IS, Halmagyi GM. Diagnosing Stroke in Acute Vertigo: The HINTS Family of Eye Movement Tests and the Future of the “Eye ECG”. Semin Neurol. 2015 Oct;35((5)):506–21. doi: 10.1055/s-0035-1564298. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Ideta S, Ota Y, Yuki K, Noda M, Inoue M, Tsubota K. Evaluation of surgical outcomes for ptosis surgery by face recognition software. Asia Pac J Ophthalmol (Phila) 2015 Jan-Feb;4((1)):14–8. doi: 10.1097/APO.0000000000000045. [DOI] [PubMed] [Google Scholar]
- 52.Azri M, Young S, Lin H, Tan C, Yang Z. Diagnosis of Ocular Myasthenia Gravis by means of tracking eye parameters. Annu Int Conf IEEE Eng Med Biol Soc. 2014;2014:1460–4. doi: 10.1109/EMBC.2014.6943876. [DOI] [PubMed] [Google Scholar]
- 53.Romeo M, O'Reilly B, Robertson BF, Morley S. Validation of the Glasgow Facial Palsy Scale for the assessment of smile reanimation surgery in facial paralysis. Clin Otolaryngol. 2012 Jun;37((3)):181–7. doi: 10.1111/j.1749-4486.2012.02484.x. [DOI] [PubMed] [Google Scholar]
- 54.Miotto R, Danieletto M, Scelza JR, Kidd BA, Dudley JT. Reflecting health: smart mirrors for personalized medicine. NPJ Digit Med. 2018 Nov;1((1)):62–7. doi: 10.1038/s41746-018-0068-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Visagie A, Swanepoel W, Eikelboom RH. Accuracy of Remote Hearing Assessment in a Rural Community. Telemed J E Health. 2015 Nov;21((11)):930–7. doi: 10.1089/tmj.2014.0243. [DOI] [PubMed] [Google Scholar]
- 56.Giordano D, Pino C, Spampinato C, Di Pietro M, Reibaldi A. Eye tracker based method for quantitative analysis of pathological nystagmus. 24th International Symposium on Computer-Based Medical Systems. 2011 August 30; Bristol, England, United Kingdom. [Google Scholar]
- 57.Kumar D, Dutta A, Das A, Lahiri U. SmartEye Developing a Novel Eye Tracking System for Quantitative Assessment of Oculomotor Abnormalities. IEEE Trans Neural Syst Rehabil Eng. 24((10)):1051–9. doi: 10.1109/TNSRE.2016.2518222. [DOI] [PubMed] [Google Scholar]
- 58.Psaltos D, Chappie K, Karahanoglu FI, Chasse R, Demanuele C, Kelekar A, et al. Multimodal Wearable Sensors to Measure Gait and Voice. Digit Biomark. 2019 Oct;3((3)):133–44. doi: 10.1159/000503282. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Kvapilova L, Boza V, Dubec P, Majernik M, Bogar J, Jamison J, et al. Continuous Sound Collection Using Smartphones and Machine Learning to Measure Cough. Digit Biomark. 2019 Dec;3((3)):166–75. doi: 10.1159/000504666. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Cadogan A, Laslett M, Hing W, McNair P, Williams M. Reliability of a new hand-held dynamometer in measuring shoulder range of motion and strength. Man Ther. 2011 Feb;16((1)):97–101. doi: 10.1016/j.math.2010.05.005. [DOI] [PubMed] [Google Scholar]
- 61.Fanella G, De Salvo G, Frasca V, Calamani M, Onesti E, Ceccanti M, et al. Amyotrophic Lateral Sclerosis (ALS): telemedicine system for home care and patient monitoring. Clin Neurophysiol. 2019;130((1)):e9–10. [Google Scholar]
- 62.Bittel AJ, Elazzazi A, Bittel DC. Accuracy and Precision of an Accelerometer-Based Smartphone App Designed to Monitor and Record Angular Movement over Time. Telemed J E Health. 2016 Apr;22((4)):302–9. doi: 10.1089/tmj.2015.0063. [DOI] [PubMed] [Google Scholar]
- 63.Pichonnaz C, Duc C, Jolles BM, Aminian K, Bassin JP, Farron A. Alteration and recovery of arm usage in daily activities after rotator cuff surgery. J Shoulder Elbow Surg. 2015 Sep;24((9)):1346–52. doi: 10.1016/j.jse.2015.01.017. [DOI] [PubMed] [Google Scholar]
- 64.Gordon MF, Grachev ID, Mazeh I, Dolan Y, Reilmann R, Loupe PS, et al. Quantification of Motor Function in Huntington Disease Patients Using Wearable Sensor Devices. Digit Biomark. 2019 Sep;3((3)):103–15. doi: 10.1159/000502136. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Elm JJ, Daeschler M, Bataille L, Schneider R, Amara A, Espay AJ, et al. Feasibility and utility of a clinician dashboard from wearable and mobile application Parkinson's disease data. NPJ Digit Med. 2019 Sep;2((1)):95–6. doi: 10.1038/s41746-019-0169-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Kubben PL, Kuijf ML, Ackermans LP, Leentjens AF, Temel Y. TREMOR12: An Open-Source Mobile App for Tremor Quantification. Stereotact Funct Neurosurg. 2016;94((3)):182–6. doi: 10.1159/000446610. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67.Wissel BD, Mitsi G, Dwivedi AK, Papapetropoulos S, Larkin S, López Castellanos JR, et al. Tablet-Based Application for Objective Measurement of Motor Fluctuations in Parkinson Disease. Digit Biomark. 2018 Jan;1((2)):126–35. doi: 10.1159/000485468. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68.Sun R, Moon Y, McGinnis RS, Seagers K, Motl RW, Sheth N, et al. Assessment of Postural Sway in Individuals with Multiple Sclerosis Using a Novel Wearable Inertial Sensor. Digit Biomark. 2018 Jan;2((1)):1–10. doi: 10.1159/000485958. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.DasMahapatra P, Chiauzzi E, Bhalerao R, Rhodes J. Free-Living Physical Activity Monitoring in Adult US Patients with Multiple Sclerosis Using a Consumer Wearable Device. Digit Biomark. 2018 Apr;2((1)):47–63. doi: 10.1159/000488040. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 70.Mathews SC, McShea MJ, Hanley CL, Ravitz A, Labrique AB, Cohen AB. Digital health: a path to validation. NPJ Digit Med. 2019 May;2((1)):38–9. doi: 10.1038/s41746-019-0111-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 71.Wykes T, Schueller S. Why Reviewing Apps Is Not Enough: Transparency for Trust (T4T) Principles of Responsible Health App Marketplaces. J Med Internet Res. 2019;21(5):e12390. Cohen AB, Mathews SC, Dorsey ER, Bates DW, Safavi K. Direct-to-consumer digital health. Lancet Digit Health. 2020;2((4)):e163–5. doi: 10.2196/12390. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72.Cohen AB, Mathews SC, Dorsey ER, Bates DW, Safavi K. Direct-to-consumer digital health. Lancet Digit Health. 2020;2((4)):e163–e165. doi: 10.1016/S2589-7500(20)30057-1. [DOI] [PubMed] [Google Scholar]
- 73.Gordon WJ, Landman A, Zhang H, Bates DW. Beyond validation: getting health apps into clinical practice. NPJ Digit Med. 2020 Feb;3((1)):14–6. doi: 10.1038/s41746-019-0212-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 74.The Johns Hopkins Applied Physics Laboratory Medical Device Interoperability Reference Architecture. 2020. Apr, Accessed on December 21, 2020. https://secwww.jhuapl.edu/mdira.
