Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Dec 6.
Published in final edited form as: JAMA Ophthalmol. 2019 Aug 1;137(8):939–944. doi: 10.1001/jamaophthalmol.2019.1576

Evaluating New Ophthalmic Digital Devices for Safety and Effectiveness in the Context of Rapid Technological Development

Zachary M Bodnar 1, Ronald Schuchard 2, David Myung 3, Michelle E Tarver 4, Mark S Blumenkranz 5, Natalie A Afshari 6, Mark S Humayun 7, Christie Morse 8, Ken Nischal 9, Michael X Repka 10, Derek Sprunger 11, Michael Trese 12, Malvina B Eydelman 13
PMCID: PMC7196315  NIHMSID: NIHMS1577391  PMID: 31169870

Abstract

IMPORTANCE

The US Food and Drug Administration’s medical device regulatory pathway was initially conceived with hardware devices in mind. The emerging market for ophthalmic digital devices necessitates an evolution of this paradigm.

OBJECTIVES

To facilitate innovation in ophthalmic digital health with attention to safety and effectiveness.

EVIDENCE REVIEW

This article presents a summary of the presentations, discussions, and literature review that occurred during a joint Ophthalmic Digital Health workshop of the American Academy of Ophthalmology, the American Academy of Pediatrics, the American Association for Pediatric Ophthalmology and Strabismus, the American Society of Cataract and Refractive Surgery, the American Society of Retina Specialists, the Byers Eye Institute at Stanford and the US Food and Drug Administration.

FINDINGS

Criterion standards and expert graders are critically important in the evaluation of automated systems and telemedicine platforms. Training at all levels is important for the safe and effective operation of digital health devices. The risks associated with automation are substantially increased in rapidly progressive diseases. Cybersecurity and patient privacy warrant meticulous attention.

CONCLUSIONS AND RELEVANCE

With appropriate attention to safety and effectiveness, digital health technology could improve screening and treatment of ophthalmic diseases and improve access to care.

As computation has upended communication, financial transactions, manufacturing, media, and transportation, so too has it transformed ophthalmology with the introduction of innovations such as cross-sectional imaging, electronic medical records, and refractive surgery. The past decade has seen increasing miniaturization and the ascendency of mobile devices and distributed computing. In 2017, the percentage of Americans with smartphones reached 77% and the number of global internet users reached approximately 52% of the world’s population.1–3 With this increase in connectivity, patients are demanding improved access to their electronic health information and communication with health care practitioners. Such technology could help close gaps in health care access for rural and underserved areas. Artificial intelligence (AI) and machine learning techniques have experienced a renaissance as improved computational power has enabled the development of deep neural networks.4,5 This improvement enables highly accurate image classification, natural language processing, and image segmentation.6

The IQVIA Institute predicts the compound annual growth rate for the mobile health market to be 29%.7 Many mobile health developers are small organizations that are new to the medical device market, without experience navigating the regulatory pathway. As digital technologies shift the paradigm of health care delivery, regulation of those technologies evolves to continue to nurture an environment of innovation while maintaining the ability to establish effectiveness and safety. Regulators and developers consider how to evaluate software as a medical device, to preserve robust cybersecurity and patient privacy protections, at the same time maintaining the balance between automation, decision support and physician oversight in expert systems.

Ophthalmology and the ophthalmic medical device industry are evolving along with these developments. Systems for in-home monitoring of visual acuity, visual fields, and intraocular pressure are being developed.8–10 Mobile fundus cameras have the potential to expand the reach of ophthalmic telemedicine, and AI systems for diabetic retinopathy and visual field interpretation are improving the efficiency of screening.11–16

The US Food and Drug Administration (FDA), the American Academy of Ophthalmology, the American Academy of Pediatrics, the American Association for Pediatric Ophthalmology and Strabismus, the American Society of Cataract and Refractive Surgery, the American Society of Retina Specialists, and the Byers Eye Institute at Stanford hosted a public meeting on October 23, 2017, in the Washington, DC, metro area to foster innovation in ophthalmic digital health technologies. Ophthalmologists from all subspecialties, regulators from the FDA, and representatives from industry convened to discuss approaches to evaluate the safety and effectiveness of ophthalmic digital health technologies, including patient privacy, cybersecurity, telemedicine, AI, and mobile health.

Definition of Digital Health Technology

As defined by the Federal Food, Drug and Cosmetic Act, a medical device is an instrument, implant, or in vitro reagent that is (1) intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease, in humans or other animals; (2) which does not achieve its primary intended purposes through chemical action within or on the body and which is not dependent upon being metabolized.17

The International Medical Device Regulators Forum defines software as a medical device as “software intended to be used for 1 or more medical purposes that perform these purposes without being part of a hardware medical device.”18 For the purpose of this panel, ophthalmic digital health technology was defined to encompass software as a medical device, hardware devices with embedded software as a principal component (software in a medical device), diagnostic or decision support AI or other health data analytics, and ophthalmic telemedicine platforms. The FDA does not regulate the practice of telemedicine; however, it does regulate ophthalmic cameras and other devices that enable telemedicine (see https://www.fda.gov/MedicalDevices/DigitalHealth/default.htm for the most up-to-date guidance from the FDA regarding digital health).

FDA Regulation of Digital Health

The FDA strives to ensure that patients have timely and continued access to safe, effective, and high-quality medical devices.19 However, the regulatory process was originally formulated for hardware devices, which typically have long development cycles. Software iterations are much shorter. For software devices, physical risks such as toxic effects may not be applicable, but software-based devices can have more subtle and far-reaching implications for public health, particularly with regard to cybersecurity and privacy. For these reasons, federal regulators have made an effort to evaluate digital health technology with less focus on individual products and with greater collaboration with industry to promote safe and effective design, development and manufacturing processes.

Regulation of both hardware and software devices is based on risk. Risk is evaluated on the basis of functionality, independent of hardware or software platform. The 21st Century Cures Act revised the definition of a medical device set forth in the Federal Food, Drug, and Cosmetic Act to exclude health care administrative support software, software to encourage or maintain a healthy lifestyle, and electronic patient records.20 It also excludes medical device data systems, that is, software intended for transferring, storing, or displaying data and results.20 This eliminates or reduces the regulatory burden for some low-risk functionalities in recognition that the historical approach to hardware medical devices is less suited to the rapid-cycle development and validation used for software-based devices.21 The FDA is implementing pragmatic approaches, including the pre-certification program, that would allow lower-risk devices to be marketed after a streamlined premarket review.21

Safety and Effectiveness Considerations for Ophthalmic Telemedicine Systems

The panel strongly agreed that telemedicine is a cost-effective approach to improve timely access to care and early screening for diabetic retinopathy, retinopathy of prematurity (ROP), and glaucoma.22–26 A previous study27 supports the assertion that the diagnostic sensitivity and specificity for remote optic nerve head and visual field examinations is comparable to in-person examinations. Numerous analyses have shown that ROP screening using digital fundus photographs has extremely high sensitivity and specificity compared with binocular ophthalmoscopic bedside examinations for detecting referral-warranted (ie, type 2) ROP and treatment-warranted ROP.24,25 The English National Health Service Diabetic Eye Screening Programme has used telemedicine to help remove diabetic retinopathy as the leading cause of vision loss in working-age adults in the United Kingdom.28

A diagnostic error in the setting of telemedicine occurs if the diagnosis is misidentified or a failure to communicate the condition occurs, resulting in lack of follow up and treatment. Jani et al29 demonstrated that although a telemedicine program nearly doubled the rate of screening in rural and underserved patient populations, 60% of referred patients failed to complete the ophthalmology referral visit. This finding demonstrates an unmet public health need that might be addressed by patient education to ensure patient compliance. Certification and continuing medical education of physician and nonphysician readers is critical and may be necessary to minimize diagnostic error and ensure compliance.

The panelists discussed the importance of identifying and using a criterion standard for evaluating the effectiveness of telemedicine-based diagnoses. For example, the Early Treatment Diabetic Retinopathy Study (ETDRS) 7-field color stereo photograph is an established criterion standard for screening diabetic retinopathy, but it may provide less information than modern wide-field imaging modalities. Criterion standards for ophthalmic diagnostics may need to evolve with advances in digital health technology.

The panel recommended that operator variability and the setting in which the device will be used be considered when evaluating the reliability and repeatability of telemedicine-based measurements or diagnoses. A certification process, similar to that used for ophthalmic photographers, may be necessary. In addition, periodic auditing of graded images by a second reader may identify readers who are in need of remedial training. Multiple grading systems may exist for a given disease entity, and the choice of specific grading protocol matters less than consistent adherence to an agreed-upon system by readers for a given platform.

Panelists noted that established teleradiology platforms may provide some precedent for minimal display specifications, however the characteristics of ophthalmic images (eg, full color and/or stereoscopic) may be different from those of general radiologic images.30 The American Telemedicine Association has provided some guidance for display technologies with respect to diabetic retinopathy.31 It may make sense for poor-quality, ungradable images to result in referral by default, although this may result in increased cost owing to unnecessary in-person screening.

Telemedicine in Retinopathy of Prematurity

The panel suggested that image graders for ROP evaluation may need to be experienced ophthalmologists, because studies have shown that much less experienced doctors, including fellows in retina and pediatric ophthalmology specialties, have significantly poorer diagnostic accuracy when using the International Classification of ROP system (Table).32–36 The most common source of error was misidentification of plus disease (arterial tortuosity and venous dilation), and the problem was not limited to graduates of US training programs.35,37 Experts can be inconsistent in identifying plus disease. Computer-aided image analysis may improve the diagnosis of plus disease. Nevertheless, the panel agreed that, at present, oversight by skilled ophthalmologists is needed for the grading of ROP via telemedicine.

Table.

International Classification of Retinopathy of Prematuritya

Location of Disease Description
Zone I A circle centered on the optic disc with radius twice the distance from the optic disc to the fovea
Zone II The region within a circle centered on the optic disc with radius from the optic disc to the nasal ora serrata, excluding the area within zone I
Zone II The temporal crescent outside of zone II
Staging
 Stage 1 A demarcation line that separates the vascular from avascular retina
 Stage 2 A ridge separates the vascular from avascular retina
 Stage 3 Extraretinal fibrovascular proliferation
 Stage 4a Partial retinal detachment not involving the fovea
 Stage 4b Partial retinal detachment involving the fovea
 Stage 5 Total retinal detachment
Open in a new tab
a

Data from International Committee for the Classification of Retinopathy of Prematurity.32

Two-dimensional digital photographs are not adequate for recognition of advanced ROP with retinal detachment (stages 4 and 5) compared with in-person binocular indirect ophthalmoscopy.30 This finding supports increased frequency of telemedicine examinations for ROP and targeting high sensitivity for early stages of the disease to ensure that eyes at high risk for progression are flagged for in-person examinations before they progress to retinal detachment.3 Fluorescein angiography and multimodal imaging may increase diagnostic effectiveness for referral-warranted disease.38

Telemedicine in Diabetic Retinopathy

Although 2-dimensional digital color fundus photography can be effective for detecting and staging diabetic retinopathy, diabetic macular edema could go undetected without the concurrent use of optical coherence tomography.39 Surrogate markers such as exudates may be helpful, but they can be present in the absence of edema. The resolution of fundus cameras should be sufficient to detect microaneurysms. Wide-field imaging may reduce the rate of ungradable images. The American Telemedicine Association has published some guidance on equipment specifications and personnel qualifications and clinical validation.31

Cybersecurity, Interoperability, and Patient Privacy

Device interoperability, the ability for systems and devices to safely exchange and use information, is critical to maximize the potential of digital health technologies. Data exchange is integral to creating a “learning health system” as proposed by the Institute of Medicine, in which real-world data from day-to-day clinical interactions are fed back into the scientific discovery process.40 This would streamline the postmarket surveillance of treatments for small cohorts of patients with rare diseases, so long as the intervention and outcomes data are transparent. The use of established digital communication standards for exchanging health data, such as Digital Imaging and Communications in Medicine (DICOM) and Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR), is encouraged to facilitate interoperability between devices of different manufacturers.37,41 However, adherence to standards alone does not ensure interoperability.

To protect patient privacy and safety, data must be exchanged in a cryptographically secure channel when using wireless or network communication. Most of the panel agreed that cloud storage is more secure than on-device storage for clinical data and personal health information. Platform developers are encouraged to use established cloud service providers rather than try to develop their own in-house security framework because the established companies have superior infrastructure, including physical security measures for data warehouses. Many cloud storage providers are certified as Health Insurance Portability and Accountability Act of 1996–compliant in the United States, and the United Kingdom has recently produced guidance for cloud-based patient data storage.42

Points of entry are the weaknesses of any cybersecurity system. In particular, user error, such as poor password utilization and nonadherence to protocol are the most common security lapses. For this reason, panel members advocated enforcing sufficiently complex password requirements (to prevent dictionary and brute-force attacks) and 2-factor authentication.43 Basic user training regarding phishing scams and social engineering is also important.

Compartmentalization was proposed as a means to protect data privacy and security. Users of digital health systems should have authorization only for information necessary to perform the tasks necessary for their roles, and, if possible, should work with deidentified data. Retinal photographs could someday be used as authentication credentials, making databases of such images high-value targets for intrusion (biometric devices for iris recognition are already available on some smartphones). Systems developers may wish to enlist cybersecurity firms to perform periodic penetration testing to ensure maximal data protection.

In many states, the patients’ location is considered the site of practice, and the laws related to the practice of telemedicine vary by state. To comply with state licensure laws, proprietors of telemedicine platforms may wish to leverage the mobile device localization services. However, improper use or disclosure of such data may infringe on users’ privacy.

Safety and Effectiveness Considerations Regarding Automated Diagnosis vs Diagnostic Aids

Some participants felt that automation is most appropriately implemented in the primary care setting rather than in the specialist’s office. However, the level of autonomy may also change who bears the liability for medical error (operator vs device maker). The panel concluded that automation may be more appropriate for slowly progressive diseases, such as glaucoma and diabetic retinopathy, than for a rapidly progressive disease such as ROP, wherein a missed diagnosis can lead to blindness within days. In such a setting, digital diagnostic aids could be useful because they may increase physician detection by highlighting subtle findings or other disease processes. Automation may be necessary to identify inflection points in high-volume data generated by frequent home monitoring.

Although human image graders may be biased or susceptible to human performance factors such as fatigue, automated processes allow for continuous access and consistency. In addition, the instantaneous reporting capability of automated systems could decrease latency and increase throughput and quality. However, there is a risk that automated systems designed for screening a specific disease may miss evidence of other disease processes where human graders might not. Systems that highlight areas of concern within the images using heat maps may increase confidence that the underlying AI is identifying actual disease markers rather than artifactual bias from the training data.44

As with telemedicine systems in general, it is important to evaluate automatic systems against a criterion standard. Image-grading systems should be evaluated against multiple expert graders. Poor-quality or ungradable images should be included when validating these systems to ensure that adequate sensitivity is maintained in real-world settings and when image quality is degraded by the optics of an individual patient’s eye (eg, owing to media opacity). Training sets for machine learning systems must adequately represent the intended screening population to avoid bias. Some guidelines for implementation are provided by the American Telemedicine Association regarding diabetic retinopathy.31

Safety and Effectiveness Considerations for Ophthalmic Digital Devices in Differing Use Settings

High-frequency at-home monitoring may result in information overload, and physicians cannot be expected to act on every individual measurement. This concern could be mitigated by analysis of data trends. Recent evidence shows that the use of electronic medical records by ophthalmologists decreases physician productivity.45,46 This finding suggests that meticulous user interface design and attention to physician workflow patterns are important to avoid compounding the effect of digital data overload. It should not be assumed that more frequent monitoring results in better outcomes without clinical evidence, or that home monitoring can replace physician visits.

Panelists who have experience with existing systems for home-monitoring of age-related macular degeneration, as well as other diseases, reported that there can be great variability between at-home measurements and that analytical strategies to separate signal from noise may be necessary. Sources of variability include the patient operator learning curve and environmental factors such as variable background luminance, as well as diurnal variation. By detecting and accounting for variability in environmental factors, the systems may be made more reliable. They could also flag unreliable measurements.

The value of home-monitoring devices must be conveyed clearly via their user experience and design to incentivize patients to use them. Devices that frequently interrupt users, whether patients or providers, may be overly burdensome and counterproductive. Many ophthalmic diseases affect individuals older than 65 years, and mobile platform–based devices and other digital systems may not be well designed for older users who are less proficient with these technologies. The panel recommended that mechanisms be in place to prevent unsafe consumer self-management and ensure treatment for certain high-risk entities such as refractive amblyopia.

Safeguards for Mitigating Risks With Ophthalmic Digital Health Devices

The panel recommended incorporation of built-in software and hardware safety checks by ophthalmic digital health developers. Several panel members advocated leveraging the agile development practices used by many software developers to evaluate safety and effectiveness continuously throughout the device development cycle (Figure).47 Panelists advocated checks for safety and effectiveness at the level of unit and integration testing. The panel also recommended using methods to limit intended users, such as password authentication, and mechanisms to restrict installation of software to approved devices. Developers of medical device software that work in conjunction with hardware add-ons, consumer hardware devices (such as smartphones), or both should know that the hardware components may need to be evaluated according to established standards for optical radiation safety (eg, American National Standards Institute [ANSI] Z80–36) and electromagnetic interference and compatibility (eg, International Electrotechnical Commission [IEC] 60601) as well as be manufactured under a quality system (eg, International Organization for Standardization [ISO] 13485) with attention to risk management guidelines (eg, ISO 14971) if they are to be used as medical device platforms.

Figure.

Figure.

Open in a new tab

Device Development Cycle

This illustration shows multiple places to incorporate safety checks in an agile development framework.

The panel members agreed that training modules and tutorials on the proper operation of ophthalmic digital health devices are important both for clinician users and patient users. Periodic recertification may be necessary to ensure continued safety and quality of data collection. As with traditional medical devices, clear labeling for patient use is an important safety precaution.

Summary and Next Steps

The Ophthalmic Digital Health Workshop brought together regulators, health care practitioners, researchers, medical device firms, mobile application developers, patients, and other stakeholders to discuss ways to ensure safety and effectiveness of ophthalmic digital health devices. The importance of validating telemedicine platforms and automated systems against a criterion standard and multiple experts was advocated, as was the importance of training of testers, patient users, image graders, and physician users. The need for physician oversight of digital systems when monitoring rapidly progressive or high-risk diseases was highlighted. Methods to maintain cybersecurity and patient privacy, including de-identification, 2-factor authentication and cloud storage were discussed. Ophthalmic digital health devices should be designed with the user and the setting of intended use in mind. Labeling and training modules can help to promote safe usage. As digital health device developments and utilization increase, the FDA will continue efforts to encourage and cultivate further innovation in ophthalmic digital health.

Key Points.

Question

How could ophthalmic digital devices be evaluated for safety and effectiveness to streamline innovation?

Findings

This Special Communication from a workshop on ophthalmic digital devices comprising regulators, health care practitioners, researchers, medical device firms, mobile application developers, patients, and other stakeholders concluded that because of the risks associated with automation are substantially increased in rapidly progressive diseases, criterion standards and expert graders are critically important in the evaluation of automated systems and telemedicine platforms. Training at all levels is important for the safe and effective operation of digital health devices, and to promote cybersecurity and protect patient privacy.

Meaning

With appropriate validation against criterion standards, physician oversight, and robust cybersecurity protocols, digital health technology could improve screening and treatment of ophthalmic diseases and improve access to care.

Acknowledgments

Conflict of Interest Disclosures: Dr Myung reported holding a patent for ophthalmic imaging adapters for smartphones. Dr Blumenkranz reported being an equity holder and a member of the board of directors of Verana Health (Formerly DigiSight Technologies) outside the submitted work and holding a patent to a method for measurement of visual function on a smart device issued. Dr Afshari reported stock ownership in Alpine Biotherapeutics, Aescula Tech, and Trefoil Biotherapeutics; grants R01EY029166-01 (Application of RNA-targeting Cas9 to Fuchs dystrophy), R01EY028983-01 (FGF Signaling in lacrimal gland homeostasis, regeneration and disease), and R01EY025090 (Limbal Stem Cell Fate and Corneal Specific Enhancers) from the National Institutes of Health outside the submitted work. Dr Nischal reported receiving an honorarium from Carl Zeiss Meditech Inc outside the submitted work. Dr Repka reported receiving grants from American Academy of Ophthalmology outside the submitted work. Dr Trese reported other support from Phoenix Technology. No other disclosures were reported.

Funding/Support: Funding for this workshop, including travel for speakers and panelists, was provided by the US Food and Drug Administration, the American Academy of Ophthalmology, the American Academy of Pediatrics, the American Association for Pediatric Ophthalmology and Strabismus, the American Society of Cataract and Refractive Surgery, the American Society of Retina Specialists, and the Byers Eye Institute at Stanford and via in-person and online attendee registration fees.

Role of the Funder/Sponsor: Each of sponsoring organizations was responsible for the design and conduct of the workshop; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.

Contributor Information

Zachary M. Bodnar, Department of Ophthalmology, Byers Eye Institute, Stanford University, Palo Alto, California.

Ronald Schuchard, Center for Devices and Radiological Health, Division of Ophthalmic and ENT Devices, US Food and Drug Administration, Silver Spring, Maryland.

David Myung, Department of Ophthalmology, Byers Eye Institute, Stanford University, Palo Alto, California.

Michelle E. Tarver, Center for Devices and Radiological Health, Division of Ophthalmic and ENT Devices, US Food and Drug Administration, Silver Spring, Maryland.

Mark S. Blumenkranz, Department of Ophthalmology, Byers Eye Institute, Stanford University, Palo Alto, California.

Natalie A. Afshari, Cornea and Refractive Surgery, FDA Committee, American Society of Cataract and Refractive Surgery, Fairfax, Virginia.

Mark S. Humayun, American Society of Retinal Specialists, Chicago, Illinois.

Christie Morse, American Association for Pediatric Ophthalmology and Strabismus, San Francisco, California.

Ken Nischal, Section on Ophthalmology, American Academy of Pediatrics, Itasca, Illinois.

Michael X. Repka, American Academy of Ophthalmology, San Francisco, California.

Derek Sprunger, American Association for Pediatric Ophthalmology and Strabismus, San Francisco, California.

Michael Trese, American Academy of Ophthalmology, San Francisco, California.

Malvina B. Eydelman, Center for Devices and Radiological Health, Division of Ophthalmic and ENT Devices, US Food and Drug Administration, Silver Spring, Maryland.

REFERENCES

  • 1.Pew Research Center. Mobile Fact Sheet. https://www.pewinternet.org/fact-sheet/mobile/. Published February 5, 2018. Accessed April 27, 2019.
  • 2.Stats IW. The Internet Big Picture: Internet World Stats, 2017. https://www.internetworldstats.com/stats.htm. Accessed April 27, 2019.
  • 3.Patel SN, Singh R, Jonas KE, et al. for the i-ROP Research Consortium. Inconsistencies in the diagnosis of aggressive posterior retinopathy of prematurity. J VitreoRetinal Dis 2017;1(3):181–186. doi: 10.1177/2474126417691848 [DOI] [Google Scholar]
  • 4.Winston PH. Lecture 12b: Deep Neural Nets. MIT OpenCourseWare. https://www.youtube.com/watch?v=VrMHA3yX_QI. Accessed April 27, 2019.
  • 5.Winston PH. Lecture 12a: Neural Nets. MIT OpenCourseWare. https://www.youtube.com/watch?v=uXt8qF2Zzfo. Accessed April 27, 2019.
  • 6.Tan K-H, Lim B. The artificial intelligence renaissance: deep learning and the road to human-level machine intelligence. APSIPA Trans Signal Inf Process. 2018;7:e6. Published online July 23, 2018. Accessed April 27, 2019. doi: 10.1017/ATSIP.2018.6 [DOI] [Google Scholar]
  • 7.State L The Rise of mHealth Apps: A Market Snapshot. https://liquid-state.com/mhealth-appsmarket-snapshot/. Published March 26, 2018. Accessed April 27, 2019.
  • 8.Ittoop SM, SooHoo JR, Seibold LK, Mansouri K, Kahook MY. Systematic Review of Current Devices for 24-h Intraocular Pressure Monitoring. Adv Ther 2016;33(10):1679–1690. doi: 10.1007/s12325-016-0388-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Chew EY, Clemons TE, Bressler SB, et al. ; AREDS2-HOME Study Research Group. Randomized trial of a home monitoring system for early detection of choroidal neovascularization home monitoring of the Eye (HOME) study. Ophthalmology. 2014;121(2):535–544. doi: 10.1016/j.ophtha.2013.10.027 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Anderson AJ, Bedggood PA, George Kong YX, Martin KR, Vingrys AJ. Can home monitoring allow earlier detection of rapid visual field progression in glaucoma? Ophthalmology. 2017;124(12):1735–1742. doi: 10.1016/j.ophtha.2017.06.028 [DOI] [PubMed] [Google Scholar]
  • 11.Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016;316(22):2402–2410. doi: 10.1001/jama.2016.17216 [DOI] [PubMed] [Google Scholar]
  • 12.Roychowdhury S, Koozekanani DD, Parhi KK. DREAM: diabetic retinopathy analysis using machine learning. IEEE J Biomed Health Inform 2014;18(5):1717–1728. doi: 10.1109/JBHI.2013.2294635 [DOI] [PubMed] [Google Scholar]
  • 13.Goldbaum MH, Sample PA, White H, et al. Interpretation of automated perimetry for glaucoma by neural network. Invest Ophthalmol Vis Sci 1994;35(9):3362–3373. [PubMed] [Google Scholar]
  • 14.Brigatti L, Hoffman D, Caprioli J. Neural networks to identify glaucoma with structural and functional measurements. Am J Ophthalmol 1996; 121(5):511–521. doi: 10.1016/S0002-9394(14)75425-X [DOI] [PubMed] [Google Scholar]
  • 15.Brigatti L, Nouri-Mahdavi K, Weitzman M, Caprioli J. Automatic detection of glaucomatous visual field progression with neural networks. Arch Ophthalmol 1997;115(6):725–728. doi: 10.1001/archopht.1997.01100150727005 [DOI] [PubMed] [Google Scholar]
  • 16.Bengtsson B, Bizios D, Heijl A. Effects of input data on the performance of a neural network in distinguishing normal and glaucomatous visual fields. Invest Ophthalmol Vis Sci 2005;46(10): 3730–3736. doi: 10.1167/iovs.05-0175 [DOI] [PubMed] [Google Scholar]
  • 17.Food US and Administration Drug. Federal Food Drug and Cosmetic Act https://www.fda.gov/regulatory-information/laws-enforced-fda/federalfood-drug-and-cosmetic-act-fdc-act. Updated March 29, 2018. Accessed April 27, 2019.
  • 18.International Medical Device Regulators Forum. Software as a Medical Device (SaMD): Key Definitions. 2013. http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-131209-samd-keydefinitions-140901.pdf. Published December 9, 2013. Accessed April 27, 2019.
  • 19.US Food and Drug Administration. About the Center for Devices and Radiological Health. https://www.fda.gov/aboutfda/centersoffices/officeofmedicalproductsandtobacco/cdrh/. Updated March 13, 2019. Accessed April 27, 2019.
  • 20.21st Century Cures Act. Pub L No. 114–255, 130 Stat 1033. https://www.govinfo.gov/content/pkg/PLAW-114publ255/pdf/PLAW-114publ255.pdf. Accessed April 27, 2019.
  • 21.US Food and Drug Administration. Digital Health Innovation Action Plan. https://www.fda.gov/media/106331/download. Published 2017. Accessed April 27, 2019.
  • 22.Thomas S, Hodge W, Malvankar-Mehta M. The cost-effectiveness analysis of teleglaucoma screening device. PLoS One. 2015;10(9):e0137913. doi: 10.1371/journal.pone.0137913 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Verma S, Arora S, Kassam F, Edwards MC, Damji KF. Northern Alberta remote teleglaucoma program: clinical outcomes and patient disposition. Can J Ophthalmol 2014;49(2):135–140. doi: 10.1016/j.jcjo.2013.11.005 [DOI] [PubMed] [Google Scholar]
  • 24.Wang SK, Callaway NF, Wallenstein MB, Henderson MT, Leng T, Moshfeghi DM. SUNDROP: six years of screening for retinopathy of prematurity with telemedicine. Can J Ophthalmol 2015;50(2):101–106. doi: 10.1016/j.jcjo.2014.11.005 [DOI] [PubMed] [Google Scholar]
  • 25.Chiang MF, Melia M, Buffenn AN, et al. Detection of clinically significant retinopathy of prematurity using wide-angle digital retinal photography: a report by the American Academy of Ophthalmology. Ophthalmology 2012;119(6):1272–1280. doi: 10.1016/j.ophtha.2012.01.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Biten H, Redd TK, Moleta C, et al. ; Imaging & Informatics in Retinopathy of Prematurity (ROP) Research Consortium. Diagnostic accuracy of ophthalmoscopy vs telemedicine in examinations for retinopathy of prematurity. JAMA Ophthalmol 2018;136(5):498–504. doi: 10.1001/jamaophthalmol.2018.0649 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Thomas SM, Jeyaraman MM, Hodge WG, Hutnik C, Costella J, Malvankar-Mehta MS. The effectiveness of teleglaucoma versus in-patient examination for glaucoma screening: a systematic review and meta-analysis [published correction appears in PLoS One. 2015;10(3):e0118688]. PLoS One. 2014;9(12):e113779. doi: 10.1371/journal.pone.0113779 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Scanlon PH. The English National Screening Programme for diabetic retinopathy 2003–2016. Acta Diabetol 2017;54(6):515–525. doi: 10.1007/s00592-017-0974-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Jani PD, Forbes L, Choudhury A, Preisser JS, Viera AJ, Garg S. Evaluation of diabetic retinal screening and factors for ophthalmology referral in a telemedicine network. JAMA Ophthalmol 2017; 135(7):706–714. doi: 10.1001/jamaophthalmol.2017.1150 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Patel SN, Singh R, Jonas KE, et al. ; Imaging and Informatics for Retinopathy of Prematurity Research Consortium. Telemedical diagnosis of stage 4 stage 5 retinopathy of prematurity. Ophthalmol Retina 2018;2(1):59–64. doi: 10.1016/j.oret.2017.04.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Li HK, Horton M, Bursell SE, et al. ; American Telemedicine Association Diabetic Retinopathy Telehealth Practice Recommendations Working Group. Telehealth practice recommendations for diabetic retinopathy, second edition. Telemed J E Health 2011;17(10):814–837. doi: 10.1089/tmj.2011.0075 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.International Committee for the Classification of Retinopathy of Prematurity. The International Classification of Retinopathy of Prematurity revisited. Arch Ophthalmol 2005;123(7):991–999. doi: 10.1001/archopht.123.7.991 [DOI] [PubMed] [Google Scholar]
  • 33.Myung JS, Paul Chan RV, Espiritu MJ, et al. Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training. J AAPOS 2011;15 (6):573–578. doi: 10.1016/j.jaapos.2011.06.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Paul Chan RV, Williams SL, Yonekawa Y, Weissgold DJ, Lee TC, Chiang MF. Accuracy of retinopathy of prematurity diagnosis by retinal fellows. Retina 2010;30(6):958–965. doi: 10.1097/IAE.0b013e3181c9696a [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Swamy L, Patel S, Jonas KE, et al. Characterization of errors in retinopathy of prematurity (ROP) diagnosis by ophthalmology residents. J AAPOS 2016;20(4):e44. doi: 10.1016/j.jaapos.2016.07.168 [DOI] [Google Scholar]
  • 36.Kang KB, Swamy L, Patel SN, et al. Characterization of errors in retinopathy of prematurity (ROP) diagnosis by international ophthalmology residents [ARVO abstract]. Invest Ophthalmol Vis Sci 2016;57(12). [Google Scholar]
  • 37.Health Level Seven International website. 2018. https://www.hl7.org/. Accessed April 27, 2019
  • 38.Klufas MA, Patel SN, Ryan MC, et al. Influence of fluorescein angiography on the diagnosis and management of retinopathy of prematurity. Ophthalmology. 2015;122(8):1601–1608. doi: 10.1016/j.ophtha.2015.04.023 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Sanborn GE, Wroblewski JJ. Evaluation of a combination digital retinal camera with spectral-domain optical coherence tomography (SD-OCT) that might be used for the screening of diabetic retinopathy with telemedicine: a pilot study. J Diabetes Complications. 2018;32(11):1046–1050. doi: 10.1016/j.jdiacomp.2018.08.010 [DOI] [PubMed] [Google Scholar]
  • 40.Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: National Academies Press; 2013. [PubMed] [Google Scholar]
  • 41.National Electrical Manufacturers Association. DICOM Standard: Current Edition. https://www.dicomstandard.org/current/. Accessed April 27, 2019.
  • 42.NHS Digital. NHS and social care data: off-shoring and the use of public cloud services.https://digital.nhs.uk/data-and-information/looking-after-information/data-security-andinformation-governance/nhs-and-social-care-dataoff-shoring-and-the-use-of-public-cloud-services. Accessed April 27, 2019.
  • 43.Grassi P, Fenton J, Newton E, et al. Digital Identity Guidelines—Authentication and Lifecycle Management. Washington, DC: United States Department of Commerce National Institute of Standards and Technology; 2017. doi: 10.6028/NIST.SP.800-63b [DOI] [Google Scholar]
  • 44.Gargeya R, Leng T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology. 2017;124(7):962–969. doi: 10.1016/j.ophtha.2017.02.008 [DOI] [PubMed] [Google Scholar]
  • 45.Redd TK, Read-Brown S, Choi D, Yackel TR, Tu DC, Chiang MF. Electronic health record impact on productivity and efficiency in an academic pediatric ophthalmology practice. J AAPOS 2014;18(6):584–589. doi: 10.1016/j.jaapos.2014.08.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Lam JG, Lee BS, Chen PP. The effect of electronic health records adoption on patient visit volume at an academic ophthalmology department. BMC Health Serv Res 2016;16:7. doi: 10.1186/s12913-015-1255-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Fitzgerald B, Stol K-J, O’Sullivan R, O’Brien D. Scaling Agile Methods to Regulated Environments: An Industry Case Study. San Francisco, CA: Software Engineering in Practice; 2013. [Google Scholar]

RESOURCES