Quantitative biomechanical analysis in validating a video-based model to remotely assess physical frailty: a potential solution to telehealth and globalized remote-patient monitoring

Abstract

Assessing physical frailty (PF) is vital for early risk detection, tailored interventions, preventive care, and efficient healthcare planning. However, traditional PF assessments are often impractical, requiring clinic visits and significant resources. We introduce a video-based frailty meter (vFM) that utilizes machine learning (ML) to assess PF indicators from a 20 s exercise, facilitating remote and efficient healthcare planning. This study validates the vFM against a sensor-based frailty meter (sFM) through elbow flexion and extension exercises recorded via webcam and video conferencing app. We developed the vFM using Google’s MediaPipe ML model to track elbow motion during a 20 s elbow flexion and extension exercise, recorded via a standard webcam. To validate vFM, 65 participants aged 20–85 performed the exercise under single-task and dual-task conditions, the latter including counting backward from a random two-digit number. We analyzed elbow angular velocity to extract frailty indicators—slowness, weakness, rigidity, exhaustion, and unsteadiness—and compared these with sFM results using intraclass correlation coefficient analysis and Bland–Altman plots. The vFM results demonstrated high precision (0.00–7.14%) and low bias (0.00–0.09%), showing excellent agreement with sFM outcomes (ICC(2,1): 0.973–0.999), unaffected by clothing color or environmental factors. The vFM offers a quick, accurate method for remote PF assessment, surpassing previous video-based frailty assessments in accuracy and environmental robustness, particularly in estimating elbow motion as a surrogate for the 'rigidity' phenotype. This innovation simplifies PF assessments for telehealth applications, promising advancements in preventive care and healthcare planning without the need for sensors or specialized infrastructure.

Introduction

Frailty, a clinical condition characterized by increased vulnerability due to age-related decline in physiological systems, has gained significant attention in the medical, scientific, and public health communities [1, 2]. Physical frailty (PF), though it can affect any age group, is more prevalent in older adults [3]. Studies have shown that PF in the elderly is linked to increased mortality, surgical complications, longer hospital stays, and poor discharge outcomes [4, 5], making its assessment crucial for predicting outcomes and tracking post-intervention recovery in this demographic.

Although a systematic review identified more than 20 different methods for PF assessment [6], two of the most popular methods are frailty phenotype (PF) by Fried et al. [7] and frailty index (FI) by Rockwood et al. [8]. The PF focuses on five physical components: slowness, weakness, exhaustion, inactivity, and shrinking, whereas FI quantifies age-related health deficits by calculating the ratio of present to potential deficits. Numerous tools based on the PF and FI have been developed for PF screening in research and clinical settings [9]. Existing PF assessment tools primarily assess slowness through walking tests and evaluate weakness using specialized equipment like a dynamometer for handgrip strength, with measurements obtained through subjective surveys and objective assessments, including grip maximum voluntary contraction and a 4.5-m walk test [10–12]. The requirement for gait analysis particularly limits their suitability for home, remote, or unsupervised settings due to the associated fall risk, especially among frail individuals [13]. Additionally, the practicality of these methods is often constrained by the time needed (ranging from 15 to 20 min) to collect questionnaire data and perform gait and balance tests, the need for specific equipment (e.g., a dynamometer for grip strength testing), and space requirements for gait assessment [14]. Moreover, conventional methods may not accommodate individuals with mobility restrictions, such as amputees or those who are wheelchair-bound, who may not necessarily be frail, further underscoring the need for innovative approaches in PF assessment [15, 16].

The 20 s Upper Frailty Meter (UFM) assessment can be another useful tool for measuring PF and cognitive declines using a wrist-worn sensor (i.e., tri-axial gyroscope) [10, 11, 17]. This sensor-based frailty meter (sFM) assesses PF by measuring elbow angular velocity during a 20 s elbow flexion/extension task, generating a Frailty Index (FI) from zero to one, where higher values indicate increased frailty severity. sFM’s validation against the Fried Frailty Phenotypes Criteria in 117 older adults showed strong correlation (r = 0.67–0.68) and effect size (d = 0.99–1.38) between sFM parameters and Fried phenotypes [18]. Further studies demonstrated sFM's significant high agreement with a modified frailty index (r = 0.72) in bedbound geriatric patients [19], predictive capability in post-hospital discharge outcomes [20], and effectiveness in identifying cognitive impairment in older adults through a dual-task variant [17]. sFM’s utility extends to clinical studies in areas like vascular surgery, Alzheimer’s screening, and pulmonary disease [21, 22]. Its design is particularly beneficial for frail older adults or patients with limited mobility, as it can be administered while sitting or lying down, offering a safer alternative to traditional gait-based tests [23].

Despite the sFM's simplicity, its dependence on a wrist-sensor restricts its use in telemedicine and remote patient monitoring. This is due to potential sensor malfunctions, increased system complexity, and the necessity for users to be adept at charging the device, pairing it with a mobile device, and adjusting preset parameters [24–26]. To alleviate clinical costs and the burden of human error, video-based assessment may be a viable alternative for marker/sensor-based motion analysis, which could have a wide range of applications [27]. To enable practical PF assessment in remote settings, this study introduces a video-based frailty meter (vFM) as an alternative to sensor usage. We suggest employing MediaPipe, a novel AI (artificial intelligence) model developed by Google [28]. In our study, we developed our model utilizing MediaPipe due to its real-time processing capabilities. This methodology allows for rapid ML inference and processing on standard hardware, presenting a significant advantage over alternatives like OpenPose [28]. MediaPipe stands out for its enhanced accuracy and speed in ML inference, requiring no advanced hardware, which broadens its applicability across various high-accuracy and efficiency-demanding tasks [29]. Furthermore, MediaPipe streamlines the analysis by removing the need for any calibration or adjustments [30]. We hypothesize that the use of MediaPipe can provide a more accurate PF assessment, surpassing previous video-based frailty assessments in terms of accuracy and environmental robustness. This improvement is expected in the Frailty Index and phenotype parameters such as flexion time, power, range of motion, power reduction, and flexion/extension variability in both single and dual-tasks.

Materials and methods

Study population

To validate vFM against sFM, we recruited adults ranging from young to older age groups, specifically those aged 20 to 85 years, to ensure our study sample was diverse and representative, covering a spectrum from healthy and robust to frail individuals. Participants were excluded from the study if they were unable to walk with or without walking aids, were receiving hospice care, had a history of upper extremity injuries or surgeries, had major hearing or visual impairments, were unable to fluently read or speak English, or had severe cognitive impairment (MOCA score < 11). The Baylor College of Medicine Institutional Review Board approved the study protocol (protocol number: H-43917). The methods used were in accordance with the relevant guidelines and regulations, and the Helsinki Declaration.

Protocol for frailty meter assessment: single-task and dual-task assessments

The PF test was carried out according to the validated frailty meter assessment protocol [18]. To ensure unobstructed hand movements, participants were positioned either standing or sitting on a chair without armrests. The test required participants to repetitively flex and extend their dominant arm as quickly as possible for a duration of 20 s. We administered the upper extremity test under two distinct conditions. In the single-task condition, participants performed the exercise without any cognitive distractions. Conversely, the dual-task condition added the challenge of counting backward aloud from a randomly selected two-digit number, thereby incorporating a cognitive distractive task. (Fig. 1).

Fig. 1.

The 20-s upper frailty meter assessment: Single-task (left) and dual-task (right)

To mitigate the risk of fatigue, there was a mandatory rest period of at least 5 min between the two testing conditions. During the test, participants' elbow movements were recorded laterally using a standard smartphone or tablet camera capturing their sagittal plane motion. In both test scenarios, participants wore the sFM device, which includes a wrist-worn gyroscope sensor (BioSensics LLC, Newton, MA, USA), to evaluate PF and associated phenotypes. This method of assessment is in line with approaches detailed in prior literature references [10, 11].

Video data analysis: 20 s angular velocity and FM parameters

For the extraction of elbow kinematics from video data, we utilized Google's MediaPipe deep learning framework [28]. At the heart of this framework lies the Detector module, which uses lightweight convolutional neural networks (CNNs). These CNNs are characterized by a 5 × 5 kernel size, followed by pointwise convolution and the use of the Rectified Linear Units (ReLU) activation function. The subsequent Estimator module pinpoints 33 distinct landmarks on the human body, as illustrated in Fig. 2. We utilized diverse camera sources in our study, resulting in video qualities ranging from 480p to Full HD (1920 × 1080 pixels). In our research, to accurately estimate elbow rotational velocity, we concentrated on three key landmarks: the shoulder, elbow, and wrist.

Fig. 2.

This figure illustrates the dictionary of the BlazePose deep learning model’s 33 landmarks on human body [28]

Then we calculated the angle between the three pivotal landmarks in all video frames. Based on the angle signal, the elbow’s angular velocity was determined by differentiating the angle derived from the video, as depicted in Fig. 3A, B. To determine the onset of movement and select a 20-s interval of elbow movements to extract frailty-related features of interest, we employed the zero-crossing technique in conjunction with a series of biomechanical rules described in detail in our previous publication [11]. This technique enables the detection of the onset of movement while extracting the true flexion–extension signal from a noisy elbow angular velocity signal. This angular velocity provided the essential pre-processed data for identifying specific frailty phenotypes, which were instrumental in calculating the frailty index, both in video and sensor data (Fig. 3C). For instance, the range of motion was calculated from the angular velocity by integrating it over time. The Frailty Index is quantified on a scale from 0 to 1, where higher values indicate increasing severity in PF, as detailed in prior study [11] (Eq. 1). The fundamental definitions of the frailty index and related phenotypes are outlined in Table 1.

$$\begin{array}{cc}\hfill \text{Frailty}\text{index}& =-1.7357\times {10}^{-3}\times \left(\text{Range}\text{of}\text{motion}\right)-1.2026\times {10}^{-3}\hfill \\ \hfill & \times \left(\text{Power}\text{reduction}\right)+0.36848\times {10}^{-3}\times \left(\text{Flexion}\text{time}\right)-0.49396\hfill \\ \hfill & \times \left(\text{Flexion}\text{time}\text{variability}\right)+0.48974\times \left(\text{Extension}\text{time}\text{variablity}\right)\hfill \\ \hfill & +0.24495\hfill \\ \hfill \end{array}$$ 1

Fig. 3.

An illustration of the computational workflow of our method to evaluate physical frailty (PF) phenotypes, derived from 2D video and sensor recording of a 20 s upper extremity frailty meter assessment. A Participants are asked to fully extend and flex their elbow repetitively for 20 s. For the video-based frailty meter (vFM), a standard webcam records participants from a sagittal view, while for the sensor-based frailty meter (sFM), participants wear a wrist-worn inertial sensor. The 20 s elbow flexion–extension test is first performed without any cognitive distraction, termed the single-task condition. After a resting break, the test is repeated with participants counting backward aloud from a random number, termed the dual-task condition. B Using the MediaPipe machine learning (ML) model, elbow velocity during the 20 s exercise is estimated. C From this elbow velocity signal, another model estimates kinematic parameters of interest to determine digital biomarkers associated with different frailty phenotypes and the frailty index

Table 1.

Definitions of frailty index and related phenotypes regarding representative parameters

Frailty phenotype	Parameter	Definition [11]
Frailty index	Frailty index (No unit, n.u.)	A scale from 0 to 1 indicates the severity of physical frailty, and a higher value indicates a greater degree of frailty
Slowness	Flexion time (sec)	Duration of flexion (3B.1)
Weakness	Power (deg2/s3) × 103	Product of the angular acceleration and range of angular velocity
Rigidity	Range of motion (deg)	Range of flexion and extension motion
Exhaustion	Power reduction (%)	Difference for percentage of average power between the first and last 10 s of angular velocity phase (3B.2)
Unsteadiness	Flexion time variability (n.u.)	Coefficient of variance (CV) for flexion time (Fig. 3B.3)
	Extension time variability (n.u.)	CV for extension time (3B.4)

To synchronize video and sensor data, we standardized both to a 25 Hz sampling rate, down-sampling the video to match the sensor. This rate, based on the Nyquist theorem [31], sufficiently captures the average elbow flexion–extension cycle of 1.25 Hz in our dataset. We did precise signal onset detection and analysis focused on the 'steady-state' phase, excluding initial and terminal acceleration phases. Data smoothing and noise removal were achieved using a Butterworth low-pass filter of 5th order with a 2.5 Hz cut-off frequency. This cut-off frequency was chosen based on the empirical observation that it effectively filters out high-frequency noise while retaining the integrity of the elbow movement signal for our study population. This approach effectively reduces noise while preserving data integrity. All coding was done in Python version 3.10.

Statistical analysis

All statistical analyses were performed by using SPSS for Windows (version 28.0, SPSS Inc., Chicago, IL). For the validity test, an intraclass correlation coefficient (ICC (2,1); two-way random single measures) analysis was used to assess validity of 20-s time series angular velocity and FM parameters both single-task and dual-task.

ICC ranges from 0 to 1, with values interpreted as follows: below 0.5 (poor), 0.5–0.75 (moderate), 0.75–0.9 (good), and above 0.9 (excellent) [32]. A pairwise T-test was used to compare the FM phenotypes between vFM and sFM. The limit of agreements (LOA) were calculated using Bland–Altman plots to show the differences between vFM and sFM data to determine data bias (i.e., mean of difference) and precision (i.e., standard deviation of difference) between two systems [33]. Based on the range of sFM values, a percentage of bias and precision values was calculated (Eq. 2). A significant level was set at p < 0.05.

$${\text{Bias}}_{\%}=\left(\frac{\text{Mean}\text{of}\text{difference}\text{between}\text{vFM}\text{and}\text{sFM}}{\text{Range}\text{of}\text{parameters}{\left(\text{Max}-\text{min}\right)}_{\text{sFM}}}\right)\times 100$$

$${\text{Precision}}_{\%}=\left(\frac{\text{Standard}\text{deviation}\text{of}\text{difference}\text{between}\text{vFM}\text{and}\text{sFM}}{\text{Range}\text{of}\text{parameters}{\left(\text{Max}-\text{min}\right)}_{\text{sFM}}}\right)\times 100$$ 2

Results

Among the 70 participants, 5 participants were excluded from the analyses; two were excluded due to severe cognitive impairment (MOCA < 11) and three participants were excluded because they did not correctly perform the 20 s UFM assessments (n = 3). Consequently, total of 65 participants successfully completed the single-task and dual-task both vFM and sFM assessments (Feasibility = 95.6%). The details demographics and clinical characteristics are shown in the Table 2.

Table 2.

Demographics and clinical characteristics

Variables	All participants (n = 65)
Demographics
Age (years)	56.03 ± 18.70
BMI (kg/m2)	28.64 ± 6.11
Gender (Female, n, %)	49 (75.4%)
Race (n, %)
White	24 (36.9%)
Black	30 (46.2%)
Asian	11 (16.9%)
Ethnicity (non-Hispanic, n, %)	61 (93.8%)
Dominant hand (n, %)
Left	8 (12.3%)
Right	57 (87.7%)
Clinical characteristics (n, %)
MOCA (score)	24.66 ± 4.24
Diagnosis cognitive impairment by physician	11 (16.9%)
Musculoskeletal problems except upper extremity	11 (16.9%)
Depression	7 (10.8%)
Diabetes	11 (16.9%)
History of fall in last year	10 (15.4%)
Using walking assistance	7 (10.8%)

Mean ± (std)

BMI: Body mass index. MOCA: Montreal Cognitive Assessment

In the validity analysis, the 20-s angular velocity of vFM indicated excellent data agreement with sFM for both the single-task (averaged ICC(2,1) = 0.983 ± 0.021, range: 0.887–0.999) and dual-task (averaged ICC(2.1) = 0.980 ± 0.019, range: 0.914–0.998) (Supplementary Table 1). In addition, the Bland–Altman plots in a representative case showed bias for the angular velocity between two systems were low bias and high precision in both single-task (mean of difference: 0.09%; std of difference: 1.06%) and dual-task (mean of difference: 0.07%; std of difference: 0.79%). All values were within a 95% LOA from the mean difference between sFM and vFM (Fig. 4).

Fig. 4.

Bland–Altman plots describing the agreement for 20-s time series angular velocity between vFM and sFM: A representative case for a single-task (ST), and b dual-task (DT)

As for the FM parameters, the vFM frailty index and related parameters including flexion time, power, range of motion, and variability for flexion and extension times indicated excellent agreement with sFM (single-task: averaged ICC(2,1) = 0.995 ± 0.004, range: 0.990–0.999; dual-task: averaged ICC(2,1) = 0.988 ± 0.009; range: 0.976–0.999). As well as there were no significant differences between vFM and sFM in both single-task and dual-task outcomes (p > 0.05) (Table 3). The Bland–Altman plots showed low bias and high precision for all FM parameters in both single-task (mean of difference: 0.00–0.75%; std of difference: 0.00–6.14%) and dual-task (mean of difference: 0.09–1.05%; std of difference: 1.16–7.40%), and most of parameters were within a 95% LOA from the mean difference between sFM and vFM, and the proportion of measurements outside of 95% LOA was 3.08–7.69% (Fig. 5).

Table 3.

Comparisons between vFM and sFM frailty index and related parameters for single-task and dual-task assessments

Variables	vFM (n = 65)	sFM (n = 65)	p-value: Pairwise t-test	ICC(2,1)
ST parameters
Frailty index (n.u.)	0.17 ± 0.07	0.17 ± 0.07	0.341	0.998**
Slowness: Flexion time (sec)	0.43 ± 0.15	0.43 ± 0.15	0.691	0.999**
Weakness: Power (deg2/s3) × 103	116.92 ± 97.77	117.25 ± 96.59	0.822	0.996**
Rigidity: Range of motion (deg)	131.53 ± 19.57	131.72 ± 18.36	0.615	0.994**
Exhaustion: Power reduction (%)	− 0.66 ± 16.22	− 0.19 ± 16.15	0.250	0.990**
Unsteadiness:
Flexion time variability (n.u.)	0.09 ± 0.04	0.09 ± 0.04	0.450	0.973**
Extension time variability (n.u.)	0.09 ± 0.05	0.09 ± 0.04	0.651	0.987**
DT parameters
Frailty index (n.u.)	0.19 ± 0.09	0.19 ± 0.08	0.452	0.984**
Slowness: Flexion time (sec)	0.49 ± 0.18	0.49 ± 0.18	0.441	0.999**
Weakness: Power (deg2/s3) × 103	91.12 ± 70.92	91.90 ± 72.93	0.670	0.990**
Rigidity: Range of motion (deg)	133.24 ± 19.27	133.32 ± 17.96	0.858	0.991**
Exhaustion: Power reduction (%)	− 3.87 ± 19.34	− 4.64 ± 16.75	0.260	0.976**
Unsteadiness:
Flexion time variability (n.u.)	0.13 ± 0.07	0.13 ± 0.07	0.395	0.976**
Extension time variability (n.u.)	0.13 ± 0.08	0.13 ± 0.08	0.693	0.978**

Mean ± std. ICC: Intraclass coefficient.

n.u.: No unit. vFM: Video-based frailty meter. sFM: Sensor-based frailty meter.

ICC(2,1) and pair-wised t-test were performed to validate frailty phenotypes between vFM and sFM.

** is a p < 0.01

Fig. 5.

Bland–Altman plots describing the agreement between the vFM and sFM frailty index and related parameters in both single-task (ST) and dual-task (DT)

Discussion

This study proposed a practical and rapid approach to assess physical frailty using a video-based solution with a standard 2D video camera. This solution utilized Google's MediaPipe deep learning framework and biomechanical modeling of the upper extremity to estimate elbow rotation speed during a 20-s repetitive elbow flexion–extension exercise. By further analyzing the pattern of elbow velocity, we measured the frailty index and various frailty phenotypes, including slowness, weakness, exhaustion, rigidity, and unsteadiness. The new procedure, termed vFM, demonstrated excellent agreement and high reliability under both single-task and dual-task conditions compared with the previously validated sFM. Furthermore, unlike sFM, vFM does not require any sensor, thereby facilitating its use for various applications, including remote assessment of physical function using video-conferencing platforms, integration into telemedicine platforms, or offline data collection using a standard smartphone camera.

Recent studies, such as Schwenk et al. [34], Zhou et al. [35] and Park et al. [36], have explored the use of wearable sensors for frailty assessment, focusing primarily on analyzing daily gait, balance, and functional performance tests. Despite their innovation, these methods are resource-intensive, requiring sensors and time-consuming gait and balance tests, making them impractical and costly for in-home or remote assessments. Other studies like those by Razjouyan et al. [37] and Park et al. [38], have proposed remote patient monitoring solutions using wearable sensors to monitor patterns of daily physical activities. However, the necessity for prolonged monitoring to accurately identify frailty and its phenotypes restricts their utility in telemedicine, which demands quick and efficient consultations. The sFM emerged as a rapid assessment tool, delivering results in 20 s and adaptable across settings. Yet, its reliance on wearable sensor technology poses significant challenges for telemedicine, including potential technical malfunctions, complex setup requirements, and user difficulties with device management. These limitations underscore the imperative for simpler, more user-friendly assessment methods that reduce dependency on advanced wearables.

In exploring alternatives, video-based frailty assessment has shown promise. Zahiri et al. [24] validated a video-based solution against sFM, utilizing the OpenPose model to assess frailty indices and phenotypes. However, this model’s complexity and advanced computer hardware (e.g., advanced CPUs/GPUs) demands limit its suitability for everyday settings such as mobile phones which usually does not have advanced hardware. Moreover, its agreement with sFM in detecting frailty indices ranged from poor to very strong correlations (i.e., r = 0.12–0.99), in particular they showed poor correlation in unsteadiness variables such as flexion time variability (r = 0.23) and extension time variability (r = 0.12), and strong correlation in power reduction, range of motion, and frailty index (r = 0.63–0.75) [24], which reflects some phenotypes may be not correctly evaluate through this model compared to our outcomes (i.e., ICC(2,1) values beyond 0.9). Further studies [26, 39] introduced video-based assessments using deep neural networks and color wristband tracking to monitor motion. Yet, this approach’s reliance on color-band regions is hindered by sensitivity to lighting variations and similar-colored objects in the background. Optical-flow-based corrections were attempted for pose estimation refinement but were susceptible to errors from environmental changes and noisy motions. Additionally, these studies reported agreement rates with sFM ICC(2,1) ranging from 0.65 to 0.72 in angles of flexion and extension [24] and range of motion (ICC(2,1) = 0.68) [25], lower than our proposed model, particularly in estimating elbow motion for rigidity phenotype assessment [25, 26].

Our vFM findings indicate a high level of reliability and consistency with the sFM evidenced by an ICC(2,1) ranging between 0.973 and 0.999. The model showcased high precision (0.00–7.40%) with minimal bias (0.00–0.09%) outperforming existing video-based frailty assessments. Our model also does not affect the race factors such as different skin colors, as shown by our subgroup ICC(2,1) analysis results for the 20 s angular velocity in both single-task and dual-task, which indicated excellent agreement (i.e., White (n = 24): 0.986–0.987; Black (n = 30): 0.972–0.977; Asian (n = 11): 0.988–0.989). Additionally, the duration for our vFM analysis varied from 15 to 30 s per trial, depending on video quality. This adaptability to different video standards ensures efficient processing, with an average analysis time of about 25 s per trial. Such a timeframe supports real-time or near-real-time evaluations, essential for telehealth applications.

The COVID-19 pandemic has underscored the challenges of conducting in-person measurements in hospitals or laboratories, thereby emphasizing the importance of remote-based assessments [40]. They significantly reduce organizational challenges such as scheduling conflicts, experimenter bias, and costs associated with travel, laboratory use, and personnel [41]. The 20-s sFM assessment simplifies evaluating PF status without additional surveys or traditional assessments such as the Fried Frailty Phenotypes Criteria, showing moderate to strong correlations with conventional frailty parameters [10–12]. The vFM protocol proposed in this study allows for the estimation of frailty using a standard camera, such as those found in smartphones, smart tablets, or webcams. This facilitates its integration into decentralized clinical trial settings, whether in smaller clinics or at homes. This approach enhances biomedical research scalability, participant inclusivity, and access by removing traditional trial barriers [41], reaching underrepresented or hard-to-reach groups through platforms like video conferencing, websites, and mobile applications, allowing for global use without necessitating physical presence.

Despite promising results and a high level of agreement with the validated sFM, our study encounters a few methodological limitations. Firstly, while our method exhibits high concordance with the validated sFM, further validation is necessary to ascertain its capacity to accurately classify different frailty statuses, including robust, pre-frail, and frail [11]. Secondly, it requires validation against more established physical frailty assessments, such as the frailty phenotypes introduced by Fried and colleagues [7]. Additionally, there are a few technical challenges that future studies could address. For instance, our proposed model based on MediaPipe algorithm estimates positions based on landmarks on the subject’s body segments (e.g., shoulder, elbow, wrist) [28]. If a body segment is not clearly captured during video recording—a situation that may frequently occur during telemedicine consultations—the algorithm might fail to accurately estimate elbow motion. Moreover, the accuracy of the measurement may be affected by the type of clothing worn; for example, loose long sleeves could impede the model’s ability to track elbow motion precisely. Despite these limitations, our study presents a practical, objective, and rapid frailty assessment method that is adaptable across various settings, including telemedicine. This flexibility may facilitate the integration of routine clinical frailty assessments and support their inclusion in large clinical trials, potentially enhancing the use of decentralized clinical trial applications. This approach paves the way for broader adoption and integration of frailty assessments in clinical and research settings, underscoring the potential for significant advancements in patient care and study design.

Conclusions

In this study, we introduced an objective, rapid, and practical method for assessing physical frailty and key frailty phenotypes, such as slowness, weakness, rigidity, exhaustion, and unsteadiness, through a 20-s repetitive elbow flexion–extension exercise protocol. The cornerstone of our innovation lies in the application of a deep learning algorithm, utilizing MediaPipe for the swift tracking of upper extremity motions and the real-time extraction of frailty metrics and phenotypes, rendering this approach suitable for efficient telemedicine consultations. A significant advancement is the capability to ascertain frailty through 2D video recordings using a standard camera, offering a distinct advantage over sensor-based methodologies. Furthermore, our video-based analysis method overcomes the limitations associated with previous video-based frailty assessments, which were often affected by environmental variables such as lighting, background color, and the presence of extraneous objects, as well as their inability to precise and fast measurement of the kinematics of upper extremities. Despite demonstrating promising concordance with validated sensor-based frailty metrics, the proposed video-based solution necessitates further validation in future studies to verify its clinical applicability, including its effectiveness in identifying frail individuals and predicting adverse health events.

Author contributions

MDR, ML and BN conceived and designed research; AM performed experiments; MDR, ML, and JB analyzed data; MDR developed all video and sensor analysis and deep learning codes; MKY and MEK supported to recruit participants; MDR, ML, JB, SB, AO, GC, and BN interpreted results of experiments; MDR and ML prepared figures; MDR, ML, JB, SB, and AM drafted manuscript; MDR, MKY, MEK, ML, JB, SB, AO, GC, and BN edited and revised manuscript; All authors contributed to the article and approved the submitted version.

Funding

Research reported in this publication was supported by The National Institute on Aging of the National Institutes of Health under award number R44-AG061951-02 and Precision Aging Network under award number U19AG065169. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Declarations

Conflict of interest

B.N. serves as a consultant for BioSensics LLC. However, his consultancy role was not related to the scope of this study. He is also a co-inventor of the sensor-based frailty meter, which is protected by a patent held by the University of Arizona. B.N. did not participate in the data analysis for this study. No additional potential conflicts of interest have been reported by other authors.

Ethical approval

The studies involving human participants were reviewed and approved by the Institutional Review Board for Human Subject Research for Baylor College of Medicine and Affiliated Hospitals (BCM IRB number: H-43917). This study was carried out in accordance with the declaration of Helsinki.

Consent to participate and publication

The patients/participants provided their written informed consent to participate in this study and publication of the manuscript. The informed consent was obtained from all participants and/or their legal guardians.