Multimodal Eye Imaging, Retina Characteristics, and Psychological Assessment Dataset

Abstract

The eyes provide insights into psychology, potentially offering a distinctive perspective for psychological health profiles. However, there exist a notable deficiency in datasets that simultaneously encompass eye features and psychological assessments. To address this gap, our study presents a dataset that included Fundus Photography, Psychological Assessment, Retina Characteristics, and Multimodal Imaging (FPRM). FPRM dataset comprise fundus images at different wavelengths (548 nm and 605 nm), image of oxygen saturation for the retina and 8 specific retinal vessels, videos of retinal blood flow and pupillary light reflex, along with 61 items of multimodal quantitative measurement from 384 participants. Additionally, it features psychological assessments across five dimensions (geriatric depression, generalized anxiety disorder, insomnia, activities of daily living, and deterioration), accompanied by fundus photographs and 6 items of retina characteristics from 1683 participants. FPRM dataset is the first to integrate multimodal ophthalmic data and psychological assessments, not only advancing the development of machine learning applications but also facilitating in-depth research into the relationship between eye health and psychological health profiles.

Background & Summary

The deterioration of vision markedly diminishes quality of life by preventing individuals from engaging in daily activities, impairing their connection with others, and resulting in a shortfall of independence. The alterations in lifestyle, coupled with the ensuing sense of loss, frequently precipitate depression, anxiety, and profound loneliness^1–5. To delve deeper into the relationship between eye health and psychological health profiles, there is an urgent need for datasets that concurrently include psychological assessments and eye imaging data.

However, fundus photographs currently employed in clinical practice, while essential for structural imaging modality, often overlook momentous details by focusing merely on retinal structural abnormalities for diagnosing diseases. Certain non-structural eye indicators, such as eye movements and pupil dilation, have emerged as novel markers in in neurological and psychological health profiles⁶. The involvement of the cerebral cortex and numerous neural pathways in eye fixation or gaze suggests that⁶, suggesting that variations in eye movements and pupil dilation could imply psychological health profiles. Additionally, the high demand of retina for oxygen links its oxygen saturation directly with conditions such as diabetes, hypertension, and Alzheimer’s disease, positioning the retinal oxygen saturation as a potential symbol for disease assessment^7–11. To harness these insights, our dataset utilizes the advanced multimodal eye imaging and analysis system, designed to concurrently capture images and videos of oxygen saturation, eye movement, pupillary light reflex, and blood flow, along with exhaustive quantitative data^7,12,13, aiming to provide a more holistic exploration of the link between eye imaging and psychological health profiles.

As the inaugural publicly accessible dataset, we supplied a Fundus Photography, Psychological Assessment, Retina Characteristics, and Multimodal Imaging (FPRM) Dataset, which comprised 3361 fundus photographs from 1683 participants, fundus images at different wavelengths (548 nm and 605 nm), image of oxygen saturation for the entire retina and 8 specific retinal vessels, videos of retinal blood flow and pupillary light reflex, alongside 61 items of multimodal quantitative measurement from 384 participants. Additionally, it includes psychological assessments across 5 dimensions (geriatric depression, generalized anxiety disorder, insomnia, activities of daily living, and deterioration). Notably, our dataset includes 6 items of retina lesion features from fundus photographs, emphasizing that identifying these characteristics at the subclinical stage enhance early intervention and lead to better outcomes^14,15. Moreover, each fundus photograph in the dataset was evaluated and labeled according to four quality criteria (brightness, reflection, veil, and sharpness). We are confident that the FPRM dataset will empower researchers to delve further into the connection between the eye health and psychological health profiles, and further advance machine learning through its comprehensive multimodal data.

Methods

Data collection

To proactively tackle mental health issues and needs among the elderly, an elderly psychological care campaign was initiated by Jinshan Mental Health Center. The project was integrated with health examinations for the elderly. We deployed fundus photography machines and multimodal machines collecting eye information while gathering psychological data. This study received approval from the National Clinical Research Center for Ophthalmic Diseases and the Ethics Committee of Shanghai Jinshan Mental Health Center. Communities called on participants aged over 55 to attend community hospitals for physical examinations and simultaneous mental health screenings. The participants consented to the open publication of the data. The demographics of the participants were shown in the uploaded Excel file. Sex, age, height, weight, body mass index, and hypertension status were provided.

Fundus photography, psychological assessment, retina characteristics, and multimodal Imaging (FPRM) dataset encompassed multimodal imaging data and quantitative measurement, fundus photography, psychological assessments, and retina characteristics. Fundus photography of both eyes was collected from 1683 participants in three hospitals, with a comprehensive assessment of retina characteristics. Besides, 1683 participants completed psychological questionnaires, yielding psychological-related data, and 384 participants underwent multimodal examinations, resulting in multimodal imaging data (Fig. 1).

Fig. 1.

Flow chart of FPRM dataset.

Acquisition of fundus photographs

Photographers were unaware of participant information and received training on adhering to standardized operating procedures. During the capturing process, patients were in a seated position, with the lower jaw placed on the lower jaw support. Fundus photographs were collected (Smartscope PRO, Optomed Oy, Oulu, Finland^16,17). All images were collected by experienced ophthalmologists (Fig. 1).

Image processing procedure

For the same participant’s one eye, we captured multiple fundus photographs. After capturing, the fundus photographs were uploaded to a dedicated reading center for information exchange. Initially, a preliminary quality check was performed on the raw image data at the reading center. To enhance dataset usability, only the best photo for the same participant’s one eye was retained. Subsequently, three professionally trained ophthalmologists assessed the retina characteristics based on the retained photo. To ensure dataset quality, we evaluated image quality in terms of brightness, reflection, veil, and sharpness. The quality assessment task was carried out by three retinal specialists, each with more than 5 years of experience (Fig. 1). Detailed descriptions were provided below.

Assessment of fundus photographs quality

To ensure the suitability of dataset for various research applications, we assessed image quality based on four aspects (brightness, reflection, veil, and sharpness). The quality assessment task involved 3 ophthalmic experts (each with > 5 years of experience) evaluating 3361 fundus photographs using a monitor positioned at approximately 30 cm viewing distance. Any disagreements were adjudicated by an ophthalmic expert with over 10 years of experience. Prior to the task, we developed a universal quality grading scale (Table 1). The scale comprised four parts, collectively forming a four-digit binary number, such as 0011, 0101, 1001, and 1100. Each set of four digits represented the quality of one image, with a value of 1 indicating alignment with the quality assessment scale, and 0 indicating misalignment.

Table 1.

Quality assessment scale of fundus photography.

Items	Principle description
Brightness	0: The image has overexposure, underexposure, or unclear color; 1: The image has good brightness and color.
Reflection	0: The image has reflection; 1: The image has not reflection.
Veil	0: The part of image is hidden; 1: The image is uncovered.
Sharpness	0: The image is misty; 1: The image is clear and not blurry.

High quality images are utilized for analyzing the relationship between fundus photographs and deterioration, activities of daily living, insomnia, generalized anxiety disorder, and geriatric depression, or for co-training machine learning for the assessment of 20,118 retina characteristics, or for exploring the association between fundus photographs and multimodal imaging. Poor quality images are frequently encountered in clinical practice, so including these images provides a systematic approach to simulate the real world and examine the efficacy of machine learning under different image qualities.

Assessment of retina characteristics

The assessment of retina characteristics was independently conducted by three professionally trained ophthalmologists. Any disputes were resolved through consensus. We developed a standardized assessment inventory, and each professionally trained ophthalmologist assessed all participants based on the standardized assessment inventory (Supplement Table 1). The final results consisted of 6 items, with each result of items represented by 0 or 1. For ease of analysis, they collectively formed a 6-bit binary number. Each set of 6 bits represented a feature of a photo, where the 1 indicated an event occurrence, and 0 indicated non-occurrence. The current diagnostic process was quite rigorous, with almost all retina characteristics being determined. This dataset provided fundus photos of both healthy individuals and those with various abnormal retina characteristics, serving as material for machine learning.

The assessment of retina characteristics included microaneurysm, intraretinal hemorrhage, hard seepage, drusen, cotton wool spot, and macular edema.

Acquisition of multimodal data

Description of multimodal machine

Multimodal imaging data was performed by the Multimodal Eye Functional Imaging and Analysis System (Discovery Model E, SysEye Technology LTD., Chongqing, China), a novel integration of retinal multispectral imaging (MSI), retinal oximetry, and laser speckle contrast imaging (LSCI), employed for functional imaging of retinal vessels, enabling early detection or monitoring of functional changes¹³. This method has been utilized in published studies^7,12,18. We integrated structural and functional retinal imaging techniques, conducting fundus color imaging under illumination from light-emitting diodes (LEDs) at wavelengths of 470 nm, 550 nm, and 600 nm. Retinal oxygen saturation measurements were performed under 550 nm and 600 nm LED illumination. Employing 850 nm laser diode illumination at 82 frames per second, blood flow was evaluated using LSCI. LSCI enables visualization of retinal vessels without the requirement for exogenous contrast agents and provides time-resolved blood flow information. The retinal oxygen saturation is calculated based on differences in tissue absorption spectral characteristics and spectrophotometric measurement techniques, which are directly correlated with optical density values. Analysis of dual-wavelength images, including registration, vessel segmentation, and optical density ratio calculation, provides information on the oxygen saturation distribution of retinal arterioles, venules, and capillaries. We created an oxygen concentration map of retinal vessels and quantitatively analyzed the oxygen saturation of 4 arteries and 4 veins adjacent to the optic disc (Fig. 1). All information was collected from right eye.

Data collection of multimodal imaging data

We collected information from multiple modalities (61 items), including blood oxygen levels in the eye, eye movement, pupillary light reflex, and blood flow. Additionally, to facilitate public analysis of this data, we also provide information generated during the calculation process (Supplement Table 2).

Blood oxygen levels in the eye

Retinal images were taken of participants at wavelengths of 548 and 605 nm, and optical density (OD) values were computed from the grayscale values within and surrounding the blood vessels. The SO₂ concentration was then calculated from the OD values. Finally, the oxygen saturation was calculated based on the SO₂ concentration.

Average blood oxygen saturation of vessel 1–8 in the right eye and average blood oxygen saturation of vessel 1–8 in the left eye.

Eye movement

First, we captured video of the eyes of participants. A threshold-based segmentation approach was used to detect the centers of the pupils and corneal reflection points in each video frame. Variations in the distance between the pupil center and corneal reflection point indicate changes in the gaze of the participants. Utilizing the curve of distance changes between the pupil center and corneal reflection point, we computed data related to eye movements.

Ratio of right eye movement speed to task target speed, eye motion latency binocularly, average velocity of the 1–3 eye movement binocularly, ratio of left eye movement speed to task target speed.

Pupillary light reflex

Initially, we recorded videos of the eyes of participants. A threshold-based pupil segmentation algorithm was employed to segment the pupils in each video frame. Elliptical fitting of the segmented pupil boundaries yielded the major axis lengths and central coordinates of the pupils at various times. Subsequently, the data on the major axis lengths of the pupil at various times provided the baseline pupil size, the initiation point of constriction, and the minimum diameter of the pupil, thus yielding the data of pupillary light reflex.

Pupil diameter at rest, minimum pupil diameter after light stimulation, pupillary constriction amplitude, ratio of pupillary constriction amplitude to baseline pupil diameter, time from onset of light stimulation to initiation of pupil constriction, duration of pupillary constriction, pupil recovery time from maximum constriction to reach 75% of baseline diameter, average speed of pupillary constriction, maximum rate of pupillary constriction, and average speed of pupillary dilation.

Blood flow

We used Laser Speckle Contrast Imaging (LSCI) to assess retinal blood flow, and LSCI is a well-established technique for analyzing blood flow dynamics^19–21. LSCI offers the benefits of rapid execution and a broad field of view. When a laser illuminates a rough surface, the resulting high interference leads to a speckle phenomenon. As the scatterers, such as red blood cells, move, the speckle pattern appears blurred. This blurred speckle, or speckle pattern image (SPI), captures information from the moving red blood cells and can be recorded by a camera within a given exposure time. Goodman quantified the degree of blur as the laser speckle contrast, thus computing the data of blood flow.

Then, resistivity index, skew, falling rate, raising rate, blow out score, blow out time, acceleration time index, flow acceleration index, blood flow volume/minutes, systole ratio, diastole time, systole time, and heart rate were performed.

The technology simplifies clinical workflows, enabling the swift acquisition of structural MSI images, retinal oxygen measurements, and LSCI blood flow information without the need for patients to move between instruments. All captures were conducted by experienced physicians. This dataset expedited research on diagnosing, monitoring, and elucidating disease mechanisms in the eyes using MSI, retinal oximetry, and LSCI, offering a novel perspective for machine learning.

Acquisition of psychological assessment

Participants filled out the questionnaire in a bright room, and they were seated in a comfortable chair, which contributed to a comfortable environment. All questionnaires were collected by experienced doctors to ensure the accuracy and reliability of the responses, encouraging to obtain genuine answers from participants, providing credible data for the study (Fig. 1). Psychological assessment included geriatric depression evaluated by Geriatric Depression Scales (GDS)²², generalized anxiety disorder evaluated by generalized anxiety disorder (GAD) scales²³, insomnia evaluated by Athens Insomnia Scale (AIS)²⁴, activities of daily living evaluated by activities of daily living (ADL) scales²⁵, and deterioration evaluated by Global Deterioration Scale²⁶. The detailed description of the questionnaires is as follows.

Geriatric depression

GDS-15 is a depression scale for the elderly, consisting of only 15 items. It primarily assesses the respondent’s depressive state in the past week, including 15 questions related to mood, reduced activity, withdrawal, negative evaluation, and more. GDS-5 is a brief version of GDS-15, suitable for a quick screening of depressive states. The specific contents of the scale are detailed in the Supplement tables 3, 4. We employed GDS-5 for initial screening, and those with abnormal results proceed to evaluate with GDS-15. The initial screening consisted of 5 questions. A score of 0 indicated no significant depression risk, while a score between 1–5 continued the evaluation with GDS-15. The depression status was determined based on the GDS-15 score, where 0–4 score indicated no depressive symptoms, 5–9 score indicated mild depression, and 10–15 score indicated depression.

Generalized anxiety disorder

GAD-7 is a screening tool developed for generalized anxiety disorder, consisting of 7 items related to anxiety disorders. It assesses the frequency of symptoms related to the past two weeks. Each item uses a 0–3 score and four-level scoring method, with higher total scores indicating greater severity. GAD-2 is a preliminary screening tool based on the GAD-7 scale, containing only the first two items of the GAD-7 scale. We performed GAD-2 for initial screening, and those with abnormal results proceed to evaluate with GAD-7. For GAD-2, a score of 0–2 represented a normal state, and further evaluation with GAD-7 was not needed. The score of 3–6 required continued assessment with GAD-7, and anxiety status was determined based on the GAD-7 score, where 0–4 indicated no anxiety symptoms, 5–9 indicated mild anxiety, and 10-21 indicated anxiety. The specific contents of the scale were detailed in the Supplement tables 5, 6.

Insomnia

The AIS-8 scale is primarily employed to aid in assessing insomnia, comprising 8 items, with each item graded from 0 to 3 for severity. In this survey, AIS-3 was utilized for the initial screening of insomnia, and for those with abnormal initial screening, the complete AIS scale was administered. The classification criterion for our survey was 0 score, signifying no sleep problems. If this criterion was met, the complete AIS scale evaluation was not pursued; otherwise, there may be sleep issues, and the complete AIS evaluation was continued. The complete AIS has a total score range of 0–24, categorized as 0–3 score for no sleep problems, 4–6 score for suspicious sleep problems, and 7–21 score for the presence of sleep problems. The specific contents of the scale were detailed in the Supplement tables 7, 8.

Activities of daily living

The ADL scale was developed in 1969 by Lawton and Brody in the United States, aimed at assessing the daily living abilities of the elderly. This scale is typically administered through interviews with knowledgeable individuals. ADL comprises 14 items, including two parts: 6 items for physical self-maintenance scale (toilet, feeding, dressing, grooming, physical ambulation, and bathing) and 8 items for instrumental activities of daily living scale (ability to use telephone, shopping, food preparation, housekeeping, laundry, mode of transportation, responsibility for own medications, and ability to handle finances). The ADL scale was to be administered by interviewing knowledgeable individuals. Based on the information provided by informants about the elderly individual, an appropriate score was selected. A score of 1 indicated complete independence, 2 implied some difficulty but still manageable, 3 signified being inability to independently complete the task, requiring assistance, and 4 denoted the impossibility of performing the task. A score of 1 on a specific item was deemed normal, while scores from 2 to 4 indicated a decline in function. The minimum total score was 14, and the maximum was 56. A total score less than 16 indicated complete normalcy. If there were 2 or more items with a score of ≥ 3, or the total score was ≥ 22, it suggested functional impairment. The specific contents of the scale were detailed in the Supplement table 9.

Deterioration

For individuals at risk of cognitive impairment, physicians should further assess the Global Deterioration Scale to evaluate the severity of cognitive decline. The scale is divided into seven levels, each with detailed descriptive definitions (Supplement table 10).

Data Records

The FPRM dataset has been uploaded to Synapse in zip format under a data usage agreement²⁷. Detailed description is supplied in Table 2. The decompressed files consiste of 1 first-level folders, named “Images”, and 5 Microsoft Office Excel lists, named “Demographics of the FPRM participants.xlsx”, “FPRM quality assessment.xlsx”, “Multimodal imaging data.xlsx”, “Assessment of retina characteristics.xlsx”, and “Psychological assessment.xlsx”, respectively.

Table 2.

The description of data records.

First-level	Second-level	Third-level	Fourth-level	Fifth-level	Six-level
Images folder	Multimodal imaging data folder	Participant ID folder	fundus_imaging folder	548 folder (n = 384)	fundus image (JPG)
				605 folder (n = 384)	fundus image (JPG)
			oxygen_saturation folder	all_retina_oxygen_saturation folder (n = 384)	image of retinal oxygen saturation (JPG)
				eight_vessels_oxygen_saturation folder (n = 384)	illustrative image of the selected 8 vessels around the optic disc (JPG)
			retina_bloodflow folder	video of retinal blood flow (mp4)
			pupillary_light_reflex	video of pupillary light reflex (mp4)
	Fundus photographs folder		fundus photographs of right eye (jpg) (n = 1683)
			fundus photographs of left eye (jpg) (n = 1678)
Demographics of the FPRM participants.xlsx
FPRM quality assessment.xlsx
Multimodal imaging data.xlsx
Assessment of retina characteristics.xlsx
Psychological assessment.xlsx

In the “Images” first-level folder, there are 2 second-level folders named “Multimodal imaging data” and “Fundus photographs”.

Demographics of the FPRM participants

In the file “Demographics of the FPRM participants.xlsx”, there are 7 columns, named ID, Sex, Age (years), Height (cm), Weight (kg), Body Mass Index, and Hypertension (yes, 1; no, 0) from left to right.

Fundus photographs and retinal characteristics

Imaging

In the “Fundus photographs” second-level folder, there are 1683 third-level folders named after participant IDs. Each third-level folder contain fundus photographs of right eye and left eye.

Retinal characteristics

In the file “Assessment of retina characteristics.xlsx”, there are 12 columns, named ID, Left-Macular edema, Left-Microaneurysm, Left-Intraretinal hemorrhage, Left-Hard seepage, Left-Drusen, Left-Cotton wool spot, Right-Macular edema, Right -Microaneurysm, Right -Intraretinal hemorrhage, Right -Hard seepage, Right -Drusen, Right -Cotton wool spot from left to right.

Quality assessment

In the file “FPRM quality assessment.xlsx”, there are 11 columns, named ID, Sex, Age (years), Left-Brightness, Left-Reflection, Left-Veil, Left-Sharpness, Right-Brightness, Right-Reflection, Right-Veil, and Right-Sharpness from left to right.

Multimodal data

Imaging

In the second-level folder “Multimodal imaging data”, there are 384 third-level folders named after participant IDs. Each third-level folder contain 4 fourth-level folders named “oxygen_saturation”, “fundus_imaging”, “pupillary_light_reflex”, and “retina_bloodflow”, respectively. In the “oxygen_saturation” fourth-level folder, there are two fifth-level folders named “all_retina_oxygen_saturation” and “eight_vessels_oxygen_saturation”. The fifth-level folder “all_retina_oxygen_saturation” contains an image of retinal oxygen saturation, and the image is named after the participants’ ID.jpg. Within the fifth-level folder “eight_vessels_oxygen_saturation,” there is an illustrative image of the selected 8 vessels around the optic disc, named after the participants’ ID.jpg. In the “fundus_imaging” fourth-level folder, there are two fifth-level folders named “548” and “605”. The fifth-level folder “548” contains a fundus image named after the participants’ ID.jpg. The fifth-level folder “605” contains another fundus image named after the participants’ ID.jpg. In the “retina_bloodflow” fourth-level folder, there is a video of retinal blood flow named after the participants’ ID.mp4. In the “ pupillary_light_reflex “ fourth-level folder, there is a video of pupillary light reflex named after the participants’ ID.mp4.

Multimodal quantitative measurement

In the file “Multimodal imaging data.xlsx”, there are 62 columns, named ID, Average blood oxygen saturation of vessel 1 in the right eye, Average blood oxygen saturation of vessel 2 in the right eye, Average blood oxygen saturation of vessel 3 in the right eye, Average blood oxygen saturation of vessel 4 in the right eye, Average blood oxygen saturation of vessel 5 in the right eye, Average blood oxygen saturation of vessel 6 in the right eye, Average blood oxygen saturation of vessel 7 in the right eye, Average blood oxygen saturation of vessel 8 in the right eye, Heart rate, Systole time, Diastole time, Systole ratio, Blood flow volume/minutes, Flow acceleration index, Acceleration time index, Blow out time, Blow out score, Raising rate, Falling rate, Skew, Resistivity index, Ratio of right eye movement speed to task target speed, Eye motion latency in the right eye, Average velocity of the first eye movement in the right eye, Average velocity of the second eye movement in the right eye, Average velocity of the third eye movement in the right eye, Ratio of left eye movement speed to task target speed, Eye motion latency in the left eye, Average velocity of the first eye movement in the left eye, Average velocity of the second eye movement in the left eye, Average velocity of the third eye movement in the left eye, Pupil diameter at rest, Minimum pupil diameter after light stimulation, Pupillary constriction amplitude, Ratio of pupillary constriction amplitude to baseline pupil diameter, Time from onset of light stimulation to initiation of pupil constriction, Duration of pupillary constriction, Pupil recovery time from maximum constriction to reach 75% of baseline diameter, Average speed of pupillary constriction, Maximum rate of pupillary constriction, and Average speed of pupillary dilation, Grayscale values of the right eye image at wavelength 548 nm, The logarithmic reflectance values of the right eye image at wavelength 548 nm (with vessels), The logarithmic reflectance values of the right eye image at wavelength 548 nm (without vessels), Grayscale values of the right eye image at wavelength 605 nm, The logarithmic reflectance values of the right eye image at wavelength 605 nm (with vessels), The logarithmic reflectance values of the right eye image at wavelength 605 nm (without vessels), Grayscale values of the left eye image at wavelength 548 nm, The logarithmic reflectance values of the left eye image at wavelength 548 nm (with vessels), The logarithmic reflectance values of the left eye image at wavelength 548 nm (without vessels), Grayscale values of the left eye image at wavelength 605 nm, The logarithmic reflectance values of the left eye image at wavelength 605 nm (with vessels), The logarithmic reflectance values of the left eye image at wavelength 605 nm (without vessels), Average blood oxygen saturation of vessel 1 in the left eye, Average blood oxygen saturation of vessel 2 in the left eye, Average blood oxygen saturation of vessel 3 in the left eye, Average blood oxygen saturation of vessel 4 in the left eye, Average blood oxygen saturation of vessel 5 in the left eye, Average blood oxygen saturation of vessel 6 in the left eye, Average blood oxygen saturation of vessel 7 in the left eye, Average blood oxygen saturation of vessel 8 in the left eye from left to right.

Psychological assessment

In the file “Psychological assessment.xlsx”, there are 6 columns, named ID, Geriatric depression score, Generalized anxiety disorder score, Insomnia score, Activities of daily living score, and Deterioration grade from left to right.

Technical Validation

Image quality

There were a total of 1678 fundus photographs for the left eye, with 59 having poor brightness, 738 exhibiting varying degrees of reflection, mostly mild, 489 having veil, and 484 lacking sufficient sharpness. For right eye fundus photographs, there were 1683 in total, with 66 having poor brightness, 730 showing reflection, 506 having veil, and 412 lacking sufficient sharpness. Brightness was machine-generated, with minimal influence from the subjects. Veil, insufficient sharpness, and reflection may result from lower cooperation of the elderly during photography. Multimodal images showed good quality in various aspects. Including some low-quality images was for better simulating clinical data in the real world and objectively testing the effectiveness of machine learning.

Consistency in assessing retina characteristics

To generate a sufficiently reliable dataset, three professionally trained ophthalmologists read fundus images at the reading center, and any disputes were resolved through consensus. In this study, we used Fleiss Kappa coefficients to assess the consistency of the results²⁸. The Fleiss Kappa coefficient of “Microaneurysm”, “Intraretinal hemorrhage”, “Hard seepage”, “Drusen”, “Cotton wool spot”, and “Macular edema” was 0.863, 0.867, 0.823, 0.735, 0.753, and, 0.839, indicating the perfect agreement.

Usage Notes

The dataset is under a data usage agreement at Synapse due to the sensitivity of the dataset. The data acquisition process is as follows. Firstly, have a certified user Synapse account. Secondly, send a copy of the completed Data Usage Agreement along with Synapse account information to corresponding author email. Data access will be provided to all users that agree to the terms and conditions of the data usage agreement. The FPRM dataset is the first dataset to simultaneously provide psychological diseases, fundus photographs, and manually identified data for retina characteristics. In addition, multimodal imaging is a technique that provided highly detailed images and data of various microstructures in the eye. The FPRM dataset is the first to provide multimodal imaging data with up to 61 types of data, offering new insights into exploring eye and the psychological diseases, and providing novel input data for machine learning. The entire dataset can be downloaded from the link²⁷. Users should cite this dataset in their research and acknowledge contribution of the dataset to their studies. Data Usage Agreement should be signed and followed and before use data. Data Usage Agreement is provided in the supplementary information.

Author contributions

Huixun Jia, Xiaodong Sun, and Qiushi Ren designed the project. Guanran Zhang, Yanlin Qu, Yanping Zhang, Jiayi Tang, Chunyan Wang, Haogui Yin, Xiaoping Yao, Ting Shen and Gengshi Liang executed the project. Guanran Zhang and Yanlin Qu wrote the manuscript. All authors checked and approved the manuscript.

Code availability

No novel code was used in the construction of the FPRM dataset.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41597-024-03690-6.