Skin Carotenoid Score Level and Race Affect Intradevice Repeatability of Veggie Meter^® at a Single Time Point

Abstract

Objective:

To compare the reliability of 2 methods (3 scan average vs average of scan 2 and 3) of skin carotenoid score (SCS) measurement and identify participant-level factors affecting intradevice repeatability of Veggie Meter^® (VM).

Design:

Cross-sectional study in Illinois.

Participants:

The sample (N = 587) included about 36% children, 27% adolescents, and 32% adults; 67% were White, 14% Asian, 10% Black or African American, 8% mixed race, 1% Native Hawaiian, and 0.4% American Indian; 92% were non-Hispanic, and 53% female.

Variable Measured:

Three consecutive SCS readings using 2 devices. Self-reported or parent-reported demographic information.

Analysis:

Paired sample t-tests compared the coefficient of variation of the 3-scan average SCS with the coefficient of variation of the second and third-scan average SCS. We conducted sign tests and generated Bland-Altman plots to determine the type of errors. Kruskal-Wallis and Mann-Whitney tests assessed if repeatability coefficients of the 3 consecutive SCS readings were different across age, race, ethnicity, and SCS quartiles, where repeatability coefficients = within-subject SD × √2 × 1.96.

Results:

Averaging the second and third scans of SCS produced 1.2% higher reliability in VM 1 and 0.7% higher reliability in VM 2 than averaging all 3 scans (t_{VM 1} = −7.1, P < 0.001; t_{VM 2} = −3.1, P = 0.002). Repeatability coefficients were significantly higher among participants with higher SCS than lower SCS, in both devices (H_{VM 1} = 30.2, P < 0.001; H_{VM 2} = 9.7, P = 0.02). Pairwise comparisons showed that repeatability coefficients were significantly lower for Black or African American participants than White (mean SCS difference = 22.3, P = 0.001) and Asian participants (mean SCS difference = 23.7, P = 0.005) in one device (VM 1).

Conclusion and Implications:

Findings support using the average of the second and third scans as a more reliable method for assessing SCS. Reporting device-specific repeatability coefficients for the VM may enhance the interpretability of SCS, particularly for individuals with low SCS and for Black or African American participants.

INTRODUCTION

Pressure-mediated reflection spectroscopy-based device, the Veggie Meter^® (VM), offers an emerging and validated method to objectively assess fruit and vegetable consumption in adults and children by measuring skin carotenoid score (SCS).¹ Fruits and vegetables, specifically the dark green and colored varieties, are the predominant sources of dietary carotenoids in humans.² After ingestion, carotenoids circulate in the body and accumulate in tissues, such as the liver, eyes, and different skin areas, with the highest levels in the forehead and palms.³ Previous research has validated the VM to measure SCS against blood carotenoids to approximate dietary carotenoid intake.^1,4 For example, there are validation studies examining the association between VM-assessed SCS and plasma or serum carotenoids in newborns, with a correlation of ρ = 0.71 (P < 0.001,⁵ in children (r = 0.53, P < 0.001),⁶ and in adult populations (r = 0.71, P < 0.001).^1,7 While validation studies are emerging, there are still relatively few investigations into device reliability and repeatability. This is especially important because the VM manufacturer guidelines from Longevity Inc. recommend triplicating SCS readings, either by averaging 3 single scans or using the 3 scan method to improve the consistency in SCS output.^8,9 In addition, intervention studies track SCS over multiple time points. Currently, there is evidence for varying levels of reliability and repeatability in VM devices.^8,10 Therefore, to accurately assess changes in participants’ SCS over time, it is important first to quantify the variation in SCS at a single time point, determine whether the error is random or systematic, and identify the most reliable data collection method.

Reliability and repeatability are critical psychometric characteristics for estimating the variations in results during repeated measures^11,12 (Supplementary Table). Reliability refers to the magnitude of error between repeated readings when the data are intended to differentiate between subjects or groups of subjects.¹² High reliability for a method allows researchers to differentiate between people or groups of people. For example, if we have a choice of 2 data collection methods to measure an outcome in a clinical trial or observational study, using the method with higher reliability will give greater statistical power to detect differences between intervention and control groups, for a given sample size.^11,12 In contrast, repeatability is defined as the variation in readings when the same device is used to measure SCS multiple times, on the same subject, and under the identical conditions.^11,12 High repeatability for a method ensures consistency over repeated data collection timepoints; thereby, repeatability will help researchers to compare and analyze the within-person changes in a participant’s SCS from baseline to postintervention and follow-up.

As we mentioned above, the VM manufacturer guidelines from Longevity Inc. recommend triplicate SCS readings instead of a single scan to improve the consistency in SCS measurement.^8,9 Per manufacturer guidelines, in the 3 scan mode, the device prompts the user to insert and remove the finger 3 times, then automatically computes the average of these readings. Alternatively, users can perform separate single scans and manually calculate the average score.^9,13 However, the manufacturer offers no additional guidance on enhancing reliability beyond these instructions. Two research studies compared the reliability of the single scan method with the 3 scan average method¹⁰ and of the 3 scan average method with the 2 scan average method (scan 2 and scan 3).¹⁴ Obana et al.¹⁰ reported that the single scan method had a higher coefficient of variation (% CV = 9.14) and hence, lower reliability than the 3 scan method (% CV = 4.81). Jilcott Pitts and colleagues (2018)¹⁴ reported that the 3 scan average method had lower reliability (% CV = 6.8%) than averaging scan 2 and scan 3 (% CV = 5.6%) (Supplementary Table). To our knowledge, these findings have not been explored in any additional published study. To address the intradevice SCS variation, Obana et al (2023)¹⁰ further suggested using corrective regression equations with device-specific constants. However, these regression equations are suitable for correcting systematic errors and do not account for errors generated because of participant-level factors and random errors, such as minor inconsistencies in finger placement, device warm-up effects, or unpredictable device fluctuations. Thus, further research is needed to better understand the types of errors and to determine which method offers higher reliability across multiple data collection timepoints in VM.

Overall, previous studies reported high intradevice repeatability for VM in terms of Pearson correlation coefficients (r = 0.96–0.97), intraclass correlation coefficients (ICC) (0.99),¹⁰ and relative SD (3.4% to 4.1%).⁸ Pearson’s correlation coefficients and ICC are expressed in scale format, and higher coefficients mean better repeatability. The relative SD is a percentage, and a higher relative SD means lower repeatability. Neither of these statistics is in the same units as the measured variable (e.g., SCS). Therefore, they have limited clinical applicability beyond highlighting the psychometric rigor of the method.¹⁵ The repeatability coefficient is another statistical method to assess device repeatability, which allows researchers to quantify the SCS variation in the same scale and units as the VM output (i.e., differences in SCS).¹¹ However, no previous studies have reported repeatability coefficients for VM, nor factors related to device repeatability, such as demographics or SCS.

Participants’ SCS levels affect the intradevice repeatability of VM.⁸ For example, Ermakov et al (2018)⁸ reported that the device repeatability decreased with lower SCS levels because of increased noise and reduced signal for the device. Moreover, researchers are recommended to use the ring finger because measuring SCS on different hands and fingers can cause variation in scores, as tissue homogeneity and skin properties may affect device repeatability.¹³ Anecdotal evidence suggests that younger children have smaller finger sizes, which may not fully cover the lens area of the VM device, potentially affecting the device’s ability to accurately capture SCS in this population. In addition, age-related differences in skin texture and carotenoid metabolism may further contribute to intradevice variability, although findings on how SCS varies across age groups are mixed.¹⁶ Finally, intradevice repeatability of VM has not been previously assessed in the context of participants’ race and ethnicity. However, limited evidence suggests higher SCS in non-White adults than in White adults (187.7 ± 77.7 vs 142.4 ± 67.5, P < 0.001),^16,17 though it remains unclear whether these small differences in SCS would affect device repeatability. Considering the gap in the literature, this study investigated whether SCS levels at a single time point, participants’ age, and race/ethnicity contribute to the VM intradevice repeatability (Supplementary Table).

This study aimed to evaluate the intradevice reliability of 2 SCS data collection methods by comparing the coefficient of variation (% CV) from 3 scans vs 2 scans, to identify the type of error present in 3 consecutive scans, and to identify participant-level factors affecting intradevice repeatability of VM. We hypothesized that the method for the 2 scan average would have higher reliability compared with the average of all 3 scans, based on prior research findings.¹⁴ We also hypothesized that the error between consecutive readings would be random across the 3 scans. For intradevice repeatability, we hypothesized that the repeatability would be lower for participants with lower SCS, higher for older participants than children, and device repeatability would not be different between different racial/ethnic groups of participants. These findings will fill in the knowledge gap about intradevice reliability and repeatability of VM devices, which is important for tracking SCS over multiple data points and for making informed decisions about reporting the average of the second and third scans or the recommended average scores of all 3 scans.^9,13

METHODS

This study used secondary data from multiple research projects conducted between 2021 and 2024 and each with unique objectives.^18–21 In this study, we only used baseline data from the participants, and no postintervention data were included in the analyses. All data collection occurred within the laboratory setting. The deidentified datasets were merged to create a final dataset for this study, which included demographic variables and SCS data. The original research project procedures underwent a full board review and were approved by the University of Illinois Institutional Review Board and adhered to the Declaration of Helsinki (Institutional Review Board nos. 18305, 21611, 20012, 19811). All participants read and signed informed consent according to the respective Institutional Review Board protocols of each project. Parents/guardians provided written consent for all children, and we obtained written assent from all children except the toddlers, before collecting data.^18–21

Participants

For the purposes of this study, we extracted demographic variables, including participants’ age, race, ethnicity, and household income. We collected children’s demographic characteristics from a parent/guardian. Adult participants responded to their own demographic characteristics using a survey. We collected survey data either online using Qualtrics or using paper-based surveys. The participants were subdivided into 5 age groups, as follows: children (aged 1–9 years), preadolescents and adolescents (aged 10–18 years), young adults (aged 19–24 years), adults (aged 25–64 years), and older adults (≥ 65 years). Participants reported their own race or ethnicity, and parents reported this information for their children. Race response options included: American Indian, Alaska Native, Asian, Black or African American, Native Hawaiian, Pacific Islander, White, and an open-ended option for the participant to add any other races not listed. Ethnicity options included: Hispanic, non-Hispanic, and an open-ended option for participants to include other ethnicities not listed.

Skin Carotenoid Measurement

We assessed SCS using 2 VM devices (VM 1 and VM 2) following a standardized methodology across all research projects. We scanned the ring finger of the participant’s non-dominant hand after cleaning with a 70% isopropyl alcohol wipe.¹³ Participants placed a clean, dry ring finger on the convex lens of the VM for the assessment. We used the single scan mode to record 3 individual scores in quick succession for each participant. However, we did not time the scans. We also documented the hand used (left or right). We instructed participants to avoid eating foods that could temporarily stain the skin, such as Cheetos or Takis, on the day of testing.¹³ We did not provide any guidance about self-tanning lotion use, and use of such products was not an exclusion criterion. Data collection for VM 1 occurred between June 2022 to December 2024, whereas VM 2 data collection took place from March 2018 to June 2022.

Veggie Meter^® device specifications and calibration protocol.

The model numbers for the 2 VM devices were VM 1: VM200251 and VM 2: 617W0042. The operating instructions for VM 1 were dated June 15, 2022, and for VM 2, February 27, 2018, which can be assumed as the device manufacturing date.⁹ After purchasing the devices, the researchers did not send any of them to the proprietary company for calibration during the original study periods. On the basis of the manual, both devices measured SCS on a scale of 0–800. The devices were calibrated to the device-specific dark/white maximum intensity references within 1–2 hours of the data collection, following manufacturer recommendations. The dark/white reference values for calibration were different per the manufacturer’s instructions. VM 1 did not require a warm-up period and was calibrated to a maximal intensity of approximately 200 ± 200 units for the dark stick and 37,000 to 47,000 units for the white stick. In contrast, VM 2 required 5 minutes of warm-up time before calibration and was calibrated to a maximal intensity of approximately 9,900 ± 500 units for the dark stick and 40,000 to 55,000 units for the white stick. Apart from the warm-up period, the rest of the procedure for calibration was the same for both devices: the dark reference stick was inserted such that the black felt material was facing the lens and positioned a few millimeters above the lens. After positioning, the dark reference calibration was performed twice. Next, the white reference stick was inserted such that the indentation on the white plastic fit snugly over the convex lens. The white reference calibration was then performed twice.

Statistical Analyses

We checked SCS values for outliers and found that all scores fell within the expected range of 0–800, with no extreme or implausible values detected. We conducted the Shapiro-Wilk and Kolmogorov-Smirnov tests to assess the normality of the residuals. Both tests, along with histograms, indicated that the SCS were not normally distributed. As a result, we chose to use nonparametric statistical tests for the subsequent analyses. We conducted an ICC analysis to assess the repeatability of SCS obtained from the 2 VM devices. The repeatability coefficients of the 3 SCS values were calculated using the following formula¹¹: repeatability coefficient = within-subject SD × √2 × 1.96. Here, higher repeatability coefficients mean lower repeatability, poorer consistency, and higher variability in SCS during multiple data collection points.

We calculated % CV (CV = SD/mean × 100) for the 3 scans and scan 2 and scan 3 as an intradevice reliability assessment. Two paired sample t-tests were conducted to compare the % CV between the 3 scan averages vs the scan 2 and scan 3 averages for each device.

We compared mean SCS values from scan 1, scan 2, and scan 3 using pairwise Wilcoxon signed rank tests and adjusted for multiple comparisons with Bonferroni correction. To examine whether the error between 2 consecutive scans was random or systematic, we calculated the difference between the first and second scans (Δ1 = scan 1 − scan 2) and between the second and third scans (Δ2 = scan 2 − scan 3). We then conducted sign tests to assess if there was a significant tendency for these differences to follow a consistent direction (positive vs negative) for a participant, indicating systematic error. In addition, we calculated the relative difference for Δ1 and Δ2 by taking the absolute difference between the 2 scans, dividing it by their average, and multiplying by 100 to express it as a percentage. This allowed us to interpret the magnitude of error in SCS relative to typical scan values. In addition, we prepared 4 Bland-Altman plots that visually interpret agreements and identify systematic error for Δ1 and Δ2 for both devices. We took a stepwise approach to track error patterns in consecutive readings. Examining consecutive differences allows for the detection of systematic shifts in error direction or random fluctuations between scans. In contrast, a direct comparison between scan 1 and scan 3 would only capture the overall difference without providing insight into how errors evolve between repeated readings, making it less informative.

To examine differences in intradevice repeatability across the 2 VM devices for SCS levels, age, and race, we conducted 3 different Kruskal-Wallis tests with the repeatability coefficient as the dependent variable and SCS quartiles, age, race, and ethnicity as the independent variables, respectively. Frequencies for ethnic subgroups were low; hence, for the inferential statistics, we recoded the ethnicity into 2 groups, Hispanic and non-Hispanic. To examine differences in intradevice repeatability across VM devices for ethnicity (Hispanic vs nonHispanic), we conducted a Mann-Whitney test with the repeatability coefficient as the dependent variable and ethnicity as the independent variable. For each device, race categories with fewer than 5 participants (Native Hawaiian and American Indian/Alaska Native) and age groups with fewer than 5 participants (young adults and older adults) were excluded from subgroup analyses to meet assumptions for the Kruskal-Wallis test and ensure adequate group sizes for reliable statistical comparisons.²² We conducted a partial correlation analysis between the repeatability coefficient for VM 1 with racial groups (White, Asian, Black or African American, and mixed), while controlling for quartiles of SCS.

For the Kruskal-Wallis test statistics, the Benjamini-Hochberg correction was applied for multiple analyses, and test statistics with P < 0.04 were considered statistically significant. For the sign tests, P < 0.05 was considered significant. All statistical analyses were conducted using SPSS (version 29.0.2.0, IBM, 2022)

RESULTS

Demographic Characteristics

Table 1 shows participants’ demographic characteristics. The final sample for this study included a total of 587 participants, with 35.6% children, 26.7% adolescents, 5.5% young adults, 30.3% adults, and 1.9% older adults. The racial composition was predominantly White (66.6%), followed by Asian (14.1%), Black or African American (10.1%), Mixed (7.6%), Native Hawaiian (1.1%), and American Indian/Alaska Native (.4%). Ethnically, 91.7% were non-Hispanic. About 53.0% were female and 41.9% were male. The average SCS was 287 ± 97.1. Most times, we measured SCS from the participant’s left hand (70.5%). Household income distribution showed that 49.5% of participants had an income above $90,000 (Table 1).

Table 1.

Demographic Characteristics of the Participants and Descriptive Statistics for Skin Carotenoid Score and Participant’s Weight Status^a

	Devices Used to Measure Skin Carotenoid Scoreb
Variables	VM 1 Device (n = 357)	VM 2 Device (n = 230)	Total (n = 587)
Age, y, mean ± SD	23.2 ± 17	15.6 ± 13.2	20.2 ± 16
Age, n (%)
Children	107 (30)	102 (44.3)	209 (35.6)
Adolescents	79 (22.1)	78 (33.9)	157 (26.7)
Young adults	30 (8.4)	2 (.9)	32 (5.5)
Adults	130 (36.4)	48 (20.9)	178 (30.3)
Older adults	11 (3.1)	0	11 (1.9)
Race, n (%)
White	228 (67.7)	121 (64.7)	349 (66.6)
American Indian/Alaska Native	2 (0.6)	–	2 (0.4)
Asian	56 (16.6)	18 (9.6)	74 (14.1)
Black or African American	34 (10.1)	19 (10.2)	53 (10.1)
Native Hawaiian	0	6 (3.2)	6 (1.1)
Mixed	17 (5)	23 (12.3)	40 (7.6)
Ethnicity, n (%)
Hispanic, White	27 (7.6)	3 (1.3)	30 (5.1)
Hispanic, Black, or African American	5 (1.4)	0	5 (.9)
Hispanic, others	8 (2.3)	6 (2.6)	14 (2.3)
Non-Hispanic	317 (88.8)	221 (95.2)	538 (91.7)
Sex, n (%)
Female	204 (57.1)	107 (46.5)	311 (53)
Male	142 (39.8)	104 (45.2)	246 (41.9)
Average skin carotenoid score, mean ± SD	283.6 ± 95.2	292.5 ± 100.0	287 ± 97.1
Scan 1	279.6 ± 96.0	294.2 ± 104.2	285.3 ± 99.4
Scan 2	284.9 ± 96.2	291.5 ± 102.5	287.5 ± 98.7
Scan 3	286.2 ± 98.0	291.7 ± 98.8	288.4 ± 98.3
Hand used, n (%)
Left	270 (75.6)	144 (62.6)	414 (70.5)
Right	87 (24.4)	86 (37.4)	173 (29.5)
Household income, $, n (%)
< 30,000	39 (11.4)	20 (10.9)	59 (11.2)
31,000–60,000	68 (19.7)	37 (20)	105 (19.9)
61,000–90,000	57 (16.6)	46 (24.8)	103 (19.5)
> 90,000	179 (52.1)	79 (42.7)	261 (49.5)

^aTotals may not sum to 100% because of missing values;

^bWe measured skin carotenoid scores using 2 different Veggie Meter^® devices: VM 1 and VM 2

Overall Intradevice Repeatability and Reliability

Reliability (% CV) over 3 scans for VM 1 was 6.7% and for VM 2 was 6.1%. The ICC indicated excellent repeatability for both VM devices, with an ICC of 0.98 for both VM 1 and VM 2. The average repeatability coefficient was 49.7 ± 31.3 for VM 1 and 45.3 ± 47.04 for VM 2, reflecting a slightly lower variability and higher repeatability in multiple scans for VM 2.

Comparing Intradevice Reliability for Averaging All 3 Scans With the Average of the Second and Third Scans

The % CV over 3 repeated scans for VM 1 was 6.7% and for VM 2 was 6.1%. The % CVs for the second and third scans were 5.5% for VM 1 and 5.4% for VM 2, respectively. Therefore, the % CV for the scan 2 and scan 3 was 1.2% lower in VM 1 and 0.7% lower in VM 2 than considering all 3 scans, indicating higher reliability of the average of scan 2 and scan 3 over the average of all 3 scans (t_{VM 1} = −7.1, P < 0.001; t_{VM 2} = −3.1, P = 0.002).

Type of Error Within Individual Repeated Scans

For both devices, the mean SCS from scan 1 was significantly different from scan 2 (Z_VM1 = −0.2, P = 0.001; Z_VM2 = −0.2, P = 0.02) and from scan 3 (Z_VM1 = −2.9, P = 0.01; Z_VM2 = 0.2, P = 0.02). There were no significant differences between scan 2 and scan 3 for either device (Z_VM1 = 0.1, P = 1.00; Z_VM2 = 0, P = 1.00).

The means for Δ1 (Δ1 = scan 1 − scan 2) were −5.3 ± 28.8 for VM 1 and 2.7 ± 28.5 for VM 2, corresponding to relative differences of 1.9% and 1.02%, respectively. The means for Δ2 (Δ2 = scan 2 − scan 3) were −1.3 ± 28.4 for VM 1 and −0.2 ± 34.6 for VM 2, corresponding to.5 and.6%, respectively. The sign test revealed that for both devices, scan 1 was significantly more likely to be lower than scan 2 (S_VM1 = 2.8, P = 0.006; S_{VM 2} = −2.3, P = 0.02), rather than higher or equal. In contrast, no significant differences were observed between scan 2 and scan 3 for either device (S_VM1 = −1.3, P = 0.2; S_{VM 2} = −0.14, P = 0.9). The Bland-Altman plots for scan 1 and scan 2 for both VM devices show data points scattered outside the limits of agreement (Figures A and B). For scan 2 and scan 3, the data points in the Bland-Altman plots exhibited a more uniform distribution around the mean difference line, with fewer outliers falling outside the limits of agreement (Figures C and D). Each plot includes horizontal reference lines for the mean difference (bold black line) and the limits of agreement (mean ± 1.96 × SD) (red dotted lines). Points falling within the limits of agreement indicate good agreement between the 2 scores. Thus, Figures A and B indicate the presence of systematic error in the first 2 scans, and descriptively, the magnitude of the error was different between VM 1 and VM 2.

Figure.

Bland-Altman Plots for 2 Veggie Meter^® (VM 1 and VM 2) devices. This figure shows the agreement between scan 1 and scan 2, and between scan 2 and scan 3, for measuring skin carotenoid scores in 2 different Veggie Meter^® devices: VM 1 and VM 2. Each plot includes horizontal reference lines for the mean difference [bold black line] and the limits of agreement (mean ± 1.96 × SD) (red dotted lines). Points falling within the limits of agreement indicate good agreement between the 2 scans.

Factors Affecting Intradevice Repeatability of Veggie Meter^®

Table 2 shows the mean rank values, Mann-Whitney, and Kruskal-Wallis test results exploring how intradevice repeatability may be affected by SCS levels, age, race, and ethnicity. For both devices, there were significant overall differences in the repeatability coefficient across the quartiles of SCS (H_{VM 1} = 30.2, P < 0.001; H_{VM 2} = 9.7, P = 0.02) (Table 2). For VM 1, intradevice repeatability coefficients for quartile 4 (i.e., those with the highest SCS scores) were significantly lower compared with the other 3 quartiles. The mean differences for repeatability coefficients were as follows: quartile 1 vs quartile 4 (25.3, P < 0.001), quartile 2 vs quartile 4 (25.7, P < 0.001), and quartile 3 vs quartile 4 (17.9, P < 0.001). For VM 2, the main effect was significant (H_{VM 2} = 9.7, P = 0.02), but pairwise comparisons between the quartiles did not show any significant mean differences. Overall, Table 2 demonstrates that the repeatability of the device is higher with higher SCS levels.

Table 2.

Participant-level Factors Affecting Intradevice Repeatability Coefficients for Skin Carotenoid Scores Measured Using 2 Veggie Meter^® Devices (VM 1 and VM 2)

	VM 1 Device (n = 357)		VM 2 Device (n = 230)
Factors	N, Mean Rank	Test Statistic, P	N, Mean Rank	Test Statistic, P
Age		5.3, 0.3		7.3, 0.06
Children	107, 172.3		102, 128.3
Adolescents	79, 174.8		78, 102.0
Young adults	30, 177.2		–
Adults	130, 181.9		48, 110.3
Older adults	11, 246.0		–
Race		21.4, < 0.001*		3.9, 0.4
Whiteb	228, 176.7		121, 97.6
Asianc	56, 180.9		18, 72.2
Black or African Americanb,c	34, 95.8		19, 99.0
Native Hawaiian	–		6, 88.0
Mixed	17, 171.2		23, 89.4
Ethnicity		−0.3, 0.8		−0.3, 0.8
Hispanic	40, 170.4		9, 88.7
Non-Hispanic	309, 175.6		177, 93.7
Quartiles		30.2, < 0.001*		9.7, 0.02*
Firstd	88, 157.7		58, 108.0
Seconde	98, 154.4		50, 98.4
Thirdf	93, 180.2		53, 115.1
Fourthd,e,f	78, 232.4		69, 134.5

^aThe test statistics are calculated using the Mann-Whitney test for Ethnicity and the Kruskal-Wallis test for age, race, and Quartiles in SPSS (version 29.02.0);

^b,cBlack participants had significantly lower repeatability for VM 1;

^d,e,fFourth quartile participants had significantly higher repeatability than the other 3 quartiles for VM 1. For VM 2, the pairwise differences were nonsignificant;

^*Statistically significant after applying Benjamini-Hochberg correction with a 5% False Discovery Rate

Notes: Cells are blank for age and race subgroups with fewer than 5 participants, as these sample sizes were insufficient for inclusion in the analysis

Neither device showed significant differences in repeatability coefficients across age and ethnic groups (Table 2). For VM 1, intradevice repeatability coefficients for SCS differed significantly across racial groups (H = 21.5, P < 0.001). Specifically, the repeatability coefficient was significantly higher in Black or African American participants than in White participants (n_Black = 34, n_White = 228; mean difference = 22.26; P = 0.001) and Asian participants (n_Black = 34, n_Asian = 56; mean difference = 23.70; P = 0.005). This means that the VM 1 repeatability was lower in Black or African American participants compared with White and Asian participants. Pairwise comparisons of all other racial groups showed no significant differences. Partial correlation between the repeatability coefficient for VM 1 with racial groups, whereas controlling for quartiles of SCS, was significant (r₋₁ = −0.21, P < 0.001). Intradevice repeatability coefficients for VM 2 were not different across the racial groups.

DISCUSSION

This study fills the current research gap by assessing the intradevice reliability and repeatability of 2 VM devices across participants with different SCS levels, ages, races, and ethnicities. Both devices had high ICCs (0.98), which is comparable to the average ICC of 0.99 from 4 VM devices in Obana et al (2023).¹⁰ In addition, the % CVs for all 3 scans in this study were 6.7% and 6.1%, which are also comparable with that from previous studies, which ranged from 4.8% to 6.8%.^8,10,14 Even though both devices showed excellent agreement via ICC values and comparable reliability with previous research studies (% CVs), the average repeatability coefficients showed a variation of about 47 units of SCS. The 47 units of SCS represent a large amount of variation, considering previously conducted randomized controlled trials reported about 45 units⁴ to 62.3 units⁷ of change in SCS after 3 weeks of carotenoid-rich juice intervention.^4,7

For repeated scans within individuals, our findings indicated the presence of systematic error for the first 2 scans, as scan 1 was more frequently lower than scan 2, rather than higher or equal, across both VM devices. However, this pattern was not observed between scan 2 and scan 3. One possible explanation for the significantly lower scan 1 values may be the warm-up effect commonly observed in optical devices, where the initial scan of SCS may be influenced by subtle calibration shifts or stabilization of the light sensor.²³ In addition, physiological factors, such as initial changes in skin perfusion or temperature on first contact with the device, may contribute to lower readings during the first scan. Thus, reporting the second and third-scan average score along with the 3-scan average score for VM would be helpful to interpret study findings. This is further supported by Jilcott Pitts et al (2018)¹⁴ and these findings report significantly higher reliability (lower value of % CV) for scan 2 and scan 3 compared with all 3 scans, as well as the higher agreement and repeatability of SCS for scan 2 and scan 3 in Bland-Altman plots. Finally, no systematic error or directional bias was observed for repeated scans between participants, suggesting that corrective regression equations may not be an effective way to address intradevice errors of the VM during multiple data collection points.

Similar to Ermakov et al (2018)⁸, who reported higher device repeatability at higher SCS levels because of reduced noise in the data signal, we also found greater repeatability and consequently lower variation for both devices at higher SCS levels. This is important because participants with low SCS, as well as those undergoing a dietary intervention with only marginal improvements in diet quality, may not have the changes in their SCS accurately detected because of low device repeatability. In addition, one of the VM devices (VM 1) in this study demonstrated lower repeatability for the 34 Black or African American participants than White and Asian participants, whereas the other device (VM 2) did not exhibit this pattern. This discrepancy may relate to several factors. First, lower average SCS values could partially explain the lower repeatability, as pairwise comparisons showed Black or African American participants (241.8) had significantly lower SCS than Asian participants (330.1) but similar SCS to White participants (276) for VM1. To clarify, partial correlation analyses controlling for SCS still showed an independent association between lower repeatability and Black or African American racial identity (r₋₁ = −0.21, P < 0.001), suggesting the presence of contributing factors beyond SCS levels. Second, although the VM device is designed to account for skin melanin,⁸ older devices may be more susceptible to the residual skin melanin effects. Finally, the smaller sample size of Black or African American participants in the VM 2 group (n = 19) may have limited our ability to detect differences. Thus, other unmeasured factors may potentially influence device performance, particularly in older VM devices that require future research.

This study has several limitations. We used a secondary dataset that included baseline data from several research projects. The participants were not the same for the 2 VM devices, which may introduce variability in the results. Thus, this study only reports intradevice repeatability and reliability and does not compare the 2 devices in any instances. The skin color and skin melanin were not measured, thus we could not report if those are related to device reliability and repeatability. However, Ermakov et al (2018)⁸ reported that skin melanin does not have a significant effect on VM’s repeatability. Data were not normally distributed, hence nonparametric statistical tests were conducted for all research questions.²⁴ In addition, the subgroup analyses for age, race, and ethnicity were impacted by the low number of participants within some groups. This may have also limited the statistical power of these analyses and may affect the generalizability of the findings, particularly for underrepresented groups.

Among the strengths, this study offered novel findings, replicated previous study findings with a larger sample size, and captured important factors affecting device repeatability, such as baseline SCS levels and racial differences, which will be valuable for informing future research and improving the applicability of skin carotenoid measurements across diverse populations. We used nonparametric methods that do not assume normality and equal variances and are robust against outliers in the data.^22,24

IMPLICATIONS FOR RESEARCH, PRACTICE, AND POLICY

This study suggests that averaging the second and third scans may result in more reliable SCS measurements in VM than averaging all 3 scans. Although we could not compare the 2 devices, descriptively, the newer VM device (VM 2) had higher reliability and repeatability than the older VM device (VM 1), highlighting the importance of returning older devices to the manufacturer for calibration and servicing. In addition, VM device repeatability may be lower for participants with lower SCS, as well as for Black or African American participants. Researchers may calculate their device-specific repeatability coefficients and compare them with the observed changes to account for variation in interpreting findings correctly. Studies involving participants with low SCS may benefit from sample stratification, such as categorizing participants into high vs low SCS groups, and the use of additional standard dietary assessment methods alongside the VM. However, further research involving a larger, racially and ethnically diverse sample and cross-device calibration is needed to fully understand the findings before considering a change to the current recommendation of averaging all 3 scans.

Supplementary Material

SUPPLEMENTARY DATA

Supplementary data related to this article can be found at https://doi.org/10.1016/j.jneb.2025.11.003.