Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Jul 27.
Published in final edited form as: AIDS. 2021 Jun 1;35(7):1051–1060. doi: 10.1097/QAD.0000000000002852

Agreement between self-reported and objective measures of sleep in people with HIV and lifestyle-similar HIV-negative individuals

Davide De Francesco 1, Caroline A Sabin 1, Alan Winston 2, Patrick W G Mallon 3, Jane Anderson 4, Marta Boffito 5, Nicki D Doyle 2, Lewis Haddow 1,6, Frank A Post 7, Jaime H Vera 8, Memory Sachikonye 9, Susan Redline 10,11,12, Ken M Kunisaki 13,14
PMCID: PMC9328164  NIHMSID: NIHMS1819730  PMID: 33635846

Abstract

Objectives:

To evaluate the agreement between self-reported sleep measures and insomnia with objectively-measured sleep parameters in people with HIV (PWH) and HIV-negative individuals.

Design:

Cross-sectional analysis of PWH and lifestyle-similar HIV-negative individuals.

Methods:

Self-reported measures included time spent in bed, sleep onset latency and a validated insomnia questionnaire. Objective measures were assessed via 7-days/nights of actigraphy data to determine average and intra-individual variability of several sleep measures (including time spent in bed and onset latency). Spearman’s correlation coefficient and Cohen’s κ were used to assess agreement between self-reported and actigraphy-assessed measures. Associations between insomnia and actigraphy-assessed sleep parameters were evaluated using partial least-square discriminant analysis (PLS-DA).

Results:

We found fair correlation between self-reported and actigraphy-assessed time spent in bed in 342 PWH (rs=0.46) and 119 HIV-negative individuals (rs=0.48). Among PWH, the correlation did not differ by age, education, depressive symptoms and self-reported insomnia (all p>0.05), but was stronger in males (p=0.05) and in those with BMI ≥25 kg/m2 (p<0.001). Agreement between self-reported and actigraphy-assessed sleep onset latency was poor in both PWH (κ=0.002, p=0.49) and HIV-negative individuals (κ=0.009, p=0.65). According to PLS-DA, self-reported insomnia most strongly correlated with intra-individual variability of sleep duration, movement index and efficiency.

Conclusions:

We report poor-to-fair agreement between self-reported and actigraphy-assessed sleep measures in PWH. Insomnia symptoms correlated with regularity of sleep duration, quality and efficiency. These findings highlight the importance of both patient-reported and objective measures of daily sleep variation, for better understanding sleep disorders in PWH.

Keywords: HIV, sleep, insomnia, actigraphy, agreement

Introduction

People with HIV (PWH) often report poor sleep quality and various sleep disturbances, such as insomnia, fatigue and daytime sleepiness [15]. Factors potentially contributing to poor sleep in PWH include behavioral and psychosocial factors [4,6], inflammation and neuronal damage induced by HIV [7], and adverse effects of antiretroviral drugs and concomitant medications [8]. As poor sleep health has been shown to be associated with many negative mental and physical outcomes, including depression [9], cardiovascular disease [10], cancer, cognitive function [11] and mortality [12], sleep has become an increasing concern for PWH.

Whilst the majority of studies that have investigated sleep disturbances and their consequences in PWH have relied on self-reported questionnaires, data regarding objective measures of sleep in populations of PWH are sparse. Objective methods to measure sleep, such as polysomnography [13] and actigraphy devices, are available. In particular, wrist-worn actigraphy devices offer an objective way to assess sleep patterns over multiple days. Although these methods have been increasingly used in sleep research focusing on the general population [14], similar studies among PWH are lacking. Furthermore, given the potential implications of a lack of agreement between subjective and objective measures of sleep in the interpretation of findings from epidemiological studies, few studies have investigated the agreement between self-reported sleep habits and actigraphy-assessed sleep measures in populations of people without HIV [1522]. Results of these studies have shown a varying degree of correlation for self-reported and actigraphy-assessed sleep duration (or time spent in bed) and sleep onset latency. Correlation coefficients for sleep duration ranged from 0.28 and 0.60 [15,17,1922], with self-assessed sleep duration often overestimating the actigraphy-assessed measure [15,17,1921]. Recently, a study of 52 PWH reported a correlation of 0.66 between self-reported and actigraphy-assessed total sleep time, with self-report overestimating the actigraphy measure by approximately one hour [23].

Importantly, recent studies have also shown how the extent of agreement between self-reported and actigraphy-assessed sleep measures may be affected by socio-demographic and clinical characteristics such as age [15,19], gender [16,19], ethnicity [15,21], level of education [15], depressive symptoms [16,21], poor sleep quality and symptoms of insomnia [15,16,19,21,22]. Populations of PWH in resource-rich settings tend to differ from the general population with regards to most of these factors; therefore the agreement between self-reported and actigraphy-assessed sleep measures in PWH may differ substantially from that reported in studies of people without HIV.

Our aims were to thoroughly evaluate the correlation and agreement between self-reported and actigraphy-assessed time spent in bed and sleep onset latency in a larger cohort of PWH and HIV-negative individuals and to investigate the association between actigraphy-assessed sleep parameters and self-reported symptoms of insomnia in PWH.

Methods

Study participants and procedures

PWH and HIV-negative individuals were recruited into a nested sub-study to investigate sleep patterns from the Pharmacokinetic and Clinical Observations in People Over Fifty (POPPY) study [24,25]. Briefly, a group of PWH aged ≥50 years were recruited from eight HIV outpatient clinics in UK and Ireland into the POPPY study. An additional group of PWH aged between 18 and 50 years and a group of HIV-negative individuals aged ≥50 years were recruited from the same HIV clinics and affiliated sexual health centres and were frequency matched on gender, ethnicity, sexual orientation and location (in or out of London) to the group of PWH aged ≥50 years.

A subset of POPPY participants from each study group was recruited to take part in this sub-study and underwent a single study visit between March 2017 and July 2018 followed by in-home procedures including a daily sleep diary and actigraph measurements, and an additional visit to return the devices and the completed diaries [26]. Additional inclusion criteria for this sub-study were: ability to wear a wrist actigraph for a week and to adhere to study procedures (according to the investigator’s judgement). All participants provided written informed consent and the study was approved by the UK Health Research Authority & Research Ethics Committee (number 16/LO/2175) and local ethics committee and/or institutional review boards.

At the study visit, information regarding antiretroviral therapy, sleep patterns, use of sleep medication, and sleep questionnaires to evaluate insomnia, fatigue and sleep-related quality of life were collected via self-report. Depressive symptoms were evaluated using the Patient Health Questionnaire-9 (PHQ-9) and scores ≥10 were considered indicative of moderate/severe depressive symptoms [27].

For the present analysis, only participants with at least 5 days/nights’ worth of valid actigraphy data and with completed sleep questionnaire data were included.

Sleep questionnaires

Self-reported time spent in bed and sleep onset latency were collected via questionnaire at the study visit. Participants were asked when they usually go to bed and wake up on weekdays and weekends separately, and how long it usually takes them to fall asleep (possible options: <5 minutes, 5 to 20 minutes, 20 to 60 minutes, >60 minutes). Weekday and weekend time spent in bed was calculated from reported bed and wake up times. Average self-reported time spent in bed was calculated as a weighted average of weekday and weekend time spent in bed (2/7×weekend + 5/7×weekday).

Participants also completed the Insomnia Severity Index questionnaire [28] to evaluate insomnia [29]. Scores range from 0 (no symptoms) to 28 (severe symptoms) and self-reported insomnia was defined as a score ≥15, as recommended by the developers [28].

Actigraphy measurements

A wrist-worn actigraph device (ActiGraph wGT3X-BT®, ActiGraph Corporation) was used to record high resolution movement data, used to estimate physical activity and sleep/wake periods. Participants were instructed to wear the device continuously from the end of study visit until the time of return (at least 7 days later) unless contraindicated in order to avoid damage to the device. In addition, daily sleep diaries were completed at home by study participants describing timing of sleep, nocturnal awakenings, daytime napping and interruptions in the use of the actigraph device.

Actigraphy data were scored centrally, blinded to HIV status, after review of the raw actigraphy data and data obtained from sleep diaries. Daily sleep measures described in Supplementary Table were obtained and, for each participant, the average (i.e. mean) and the intra-individual variability (i.e. standard deviation: SD) of these measures across the 7-day observation period were obtained.

Statistical analysis

Spearman’s correlation coefficient (rs) was used to evaluate the correlation between self-reported and actigraphy-assessed time spent in bed, separately in PWH and HIV-negative individuals and, within PWH, separately according to groups defined by age, gender, ethnicity, education, BMI, depressive symptoms, current alcohol consumption, recreational drug use, work schedule, use of antiretroviral drugs with potential adverse effects impacting on sleep according to current European guidelines (i.e. efavirenz, rilpivirine or integrase strand transfer inhibitors) [30], use of sleep medication and self-reported clinical diagnosis of insomnia. Correlations in different groups were compared using the z-test after Fisher’s transformation [31]. Agreement between self-reported and actigraphy-assessed time spent in bed was assessed using the Bland–Altman plot [32]. The agreement between self-reported and actigraphy-assessed sleep onset latency was evaluated, separately in PWH and HIV-negative individuals, using the weighted Cohen’s κ statistics [33] and interpreted following Landis and Koch [34] guidelines.

Among PWH, Wilcoxon rank-sum test was used to test the univariable association between self-reported insomnia (ISI≥15) and actigraphy-assessed sleep measures one-at-the-time. Partial least-square discriminant analysis (PLS-DA) was used to explore these associations simultaneously. PLS-DA identifies linear combinations of an input set of variables (i.e. actigraphy-assessed sleep measures) that best discriminate individuals based on an outcome variable (i.e. self-reported insomnia vs. no self-reported insomnia). Weights assigned to each sleep measure to define this linear combination were used to identify those sleep measures with the strongest association with self-reported insomnia whilst simultaneously considering all sleep measures.

All analyses were performed using R v3.5.1, with p-values <0.05 considered as statistically significant.

Results

Participant characteristics

A total of 342 PWH and 119 HIV-negative individuals completed sleep questionnaires and had at least 5 days/nights’ worth of actigraphy data (Table 1). PWH were predominantly males (86.6%), of white ethnicity (88.6%) with a median (interquartile range: IQR) age of 57 (52, 62) years. Most PWH were on antiretroviral therapy (91.8%) with an HIV-RNA <40 copies/mL (97.1%). Compared to PWH, HIV-negative individuals were older (p<0.001, as expected due to study design) and less likely to be male (67.2%, p<0.001) and men having sex with men (52.9%, p<0.001).

Table 1:

Socio-demographic, lifestyle, clinical and HIV-related characteristics of study participants at study visit.

Median (IQR) or n (%) PWH (n=342) HIV-negative (n=119) p-value

Male gender 296 (86.6%) 80 (67.2%) <0.001
Age [years] 57 (52, 62) 61 (57, 66) <0.001
White ethnicity 303 (88.6%) 109 (91.6%) 0.36
MSM/homosexual 271 (79.2%) 63 (52.9%) <0.001
Years of education 16 (12, 18) 16 (13, 18) 0.82
BMI [Kg/cm2] 25.4 (23.5, 28.7) 25.8 (23.6, 29.7) 0.17
Work schedule 0.007
 Day shift 121 (35.4%) 60 (51.7%)
 Other/Irregular shift 57 (16.7%) 13 (11.2%)
 Retired/Don’t work 164 (48.0%) 43 (37.1%)
Current alcohol use 279 (81.6%) 109 (91.6%) 0.01
Current smoking 91 (26.7%) 18 (15.1%) 0.01
Current recreational drugs 92 (26.9%) 17 (14.3%) 0.005
Ever injected drugs 32 (9.4%) 2 (1.7%) 0.004
Depressive symptoms 39 (12.4%) 4 (3.4%) 0.006
Use of sleep medication 31 (9.1%) 2 (1.7%) 0.006
Clinical insomnia 76 (22.9%) 7 (5.9%) <0.001
Current CD4+ count [cells/μL] 630 (483, 831) N/A N/A
Nadir CD4+ count [cells/μL] 190 (87, 290) N/A N/A
Years since HIV diagnosis 17.7 (10.8, 24.5) N/A N/A
On antiretroviral treatment 314 (91.8%) N/A N/A
On Efavirenz 53 (16.9%) N/A N/A
On Rilpivirine 35 (11.2%) N/A N/A
On INSTI* 72 (22.9%) N/A N/A
HIV RNA <40 copies/mL 330 (97.1%) N/A N/A
Sleep duration (actigraphy) [h] 7.0 (6.3, 7.6) 7.2 (6.7, 7.6) 0.12
Time spent in bed
 Self-report [h] 8.0 (7.1, 8.8) 7.9 (7.2, 8.5) 0.19
 Actigraphy [h] 8.1 (7.4, 8.7) 8.1 (7.5, 8.7) 0.75

Note: MSM: men who have sex with men; INSTI: integrase strand transfer inhibitor; p-values obtained were obtained from χ2, Fisher’s exact and Wilcoxon rank‐sum tests as appropriate.

*

29, 30 and 13 PWH were on raltegravir, dolutegravir and elvitegravir, respectively.

The prevalence of self-reported insomnia was significantly higher in PWH (22.9%) compared to HIV-negative individuals (5.9%, p<0.001). Moderate/severe depressive symptoms and use of sleep medication were also more prevalent among PWH (12.4% and 9.1%, respectively) compared to HIV-negative individuals (3.4% and 1.7%, respectively, both p’s=0.006). Among PWH, median (IQR) self-reported and actigraphy-assessed time spent in bed was 8.0 (7.1, 8.8) and 8.1 (7.4, 8.7) hours, respectively, and did not differ significantly from those observed among HIV-negative individuals [7.9 (7.2, 8.5) and 8.1 (7.5, 8.7) hours, respectively; p=0.19 and p=0.75]. Median (IQR) actigraphy-assessed sleep duration was 7.0 (6.3, 7.6) and 7.2 (6.7, 7.6) hours in PWH and HIV-negative individuals, respectively (p=0.12).

Time spent in bed

The correlation between self-reported and actigraphy-assessed time spent in bed was fair in both PWH [rs (95% CI) = 0.46 (0.37, 0.54)] and HIV-negative individuals [rs (95% CI) = 0.48 (0.33, 0.61)], and did not differ between the two groups (p=0.79). Bland-Altman plots of the differences between self-reported and actigraphy assessed time spent in bed and their average, separately in PWH and HIV-negative individuals, are presented in Figure 1. Among PWH, the mean (95% CI) difference was −0.1 (−0.2, 0.1) hours indicating no systematic overestimation or underestimation of self-report compared to actigraphy-assessed time spent in bed (p=0.22). The correlation of the difference between the two measures of time spent in bed with their average was poor [rs (95%) = 0.18 (0.08, 0.28)] but significant (p<0.001), suggesting self-report may modestly underestimate time in bed in PWH with shorter sleep duration but overestimate it in those with longer sleep durations. The magnitude of disagreement did not vary by average time spent in bed (p-value of the F test for heteroscedasticity=0.67).

Figure 1:

Figure 1:

Bland-Altman plot of self-reported and actigraphy-assessed time spent in bed among PWH (n=342) and HIV-negative individuals (n=119). Lower and upper limits of agreement (95% CI) among PWH were −2.8 (−2.5, −3.0) and 2.6 (2.3, 2.8) hours, respectively. Limits of agreement (95% CI) among HIV-negative individuals were −2.3 (−2.0, −2.6) and 1.7 (1.4, 2.0) hours.

Among HIV-negative individuals, the mean (95% CI) difference between self-reported and actigraphy-assessed time spent in bed was negative, −0.3 (−0.5, −0.1) hours, suggesting self-reported time in bed was systematically shorter than actigraphy-assessed time in bed (p<0.001). There was a fair and significant correlation between the difference between self-reported and actigraphy-assessed time spent in bed and their average [rs (95%) = 0.21 (0.03, 0.37), p=0.03], with no evidence of heteroscedasticity (p=0.44).

Of note, the correlation between self-reported time spent in bed and actigraphy-assessed sleep duration was 0.36 (0.27, 0.45) in PWH and 0.39 (0.23, 0.54) in HIV-negative individuals, and did not differ between groups (p=0.74). Self-reported time spent in bed was longer than actigraphy-assessed sleep duration in both PWH and HIV-negative individuals by a mean (95% CI) of 1.0 (0.9, 1.2) and 0.7 (0.5, 0.9) hours (Supplementary Material).

Stratified analyses among PWH showed no difference in the overestimation or underestimation of self-reported compared to actigraphy-assessed time spent in bed according to gender, age, BMI, alcohol consumption, recreational drug use, efavirenz, rilpivirine or integrase strand transfer inhibitors use, use of sleep medication and self-reported insomnia (Table 2). Self-report appeared to overestimate time spent in bed in PWH with <12 years of education (difference=0.3 hours) but underestimate in those with ≥12 years of education (difference=−0.2 hours, p=0.03). Similarly, overestimation/underestimation patterns seemed to differ according to ethnicity (p=0.11), depressive symptoms (p=0.08) and work schedule (p=0.04).

Table 2:

Difference and correlation (rs) between self-reported and actigraphy-assessed time spent in bed (hours), according to participant characteristics of PWH. P-values to test whether differences and correlations varied according to participant characteristics are also reported.

Variable Difference [h]
(95% CI)
p-value1 rs (95% CI) p-value2 Interaction p-value

Gender 0.21 0.05
 Male 0.2 (−0.3, 0.7) 0.50 (0.41, 0.58) <0.001
 Female −0.1 (−0.3, 0.0) 0.18 (−0.11, 0.45) 0.22
Age 0.54 0.73
 18–50 years −0.1 (−0.3, 0.1) 0.45 (0.34, 0.55) <0.001
 ≥50 years 0.0 (−0.3, 0.2) 0.48 (0.31, 0.62) <0.001
Ethnicity 0.11 0.11
 Black-African 0.4 (−0.3, 1.0) 0.26 (−0.07, 0.53) 0.12
 White −0.1 (−0.3, 0.0) 0.50 (0.41, 0.52) <0.001
Years of education 0.03 0.43
 <12 years 0.3 (−0.1, 0.7) 0.59 (0.37, 0.74) <0.001
 ≥12 years −0.2 (−0.3, 0.0) 0.42 (0.32, 0.51) <0.001
BMI 0.42 <0.001
 <25 kg/m2 −0.2 (−0.3, 0.0) 0.65 (0.55, 0.73) <0.001
 ≥25 kg/m2 0.0 (−0.3, 0.2) 0.29 (0.15, 0.42) <0.001
Depressive symptoms 0.08 0.15
 No −0.2 (−0.3, 0.0) 0.48 (0.38, 0.56) <0.001
 Yes 0.3 (−0.2, 0.9) 0.38 (0.07, 0.62) 0.02
Alcohol consumption 0.22 0.16
 No 0.1 (−0.3, 0.5) 0.32 (0.08, 0.53) 0.01
 Yes −0.1 (−0.3, 0.0) 0.48 (0.39, 0.57) <0.001
Recreational drug use 0.40 0.04
 No −0.1 (−0.2, 0.1) 0.51 (0.41, 0.59) <0.001
 Yes −0.2 (−0.5, 0.1) 0.28 (0.08, 0.46) 0.007
Work schedule 0.04
 Other/Irregular shift −0.4 (−0.8, 0.0) 0.48 (0.24, 0.66) <0.001 *0.89
 Retired/Don’t work 0.1 (−0.1, 0.3) 0.42 (0.29, 0.54) <0.001 *0.42
 Day shift −0.4 (−0.2, 0.0) 0.50 (0.35, 0.62) <0.001
Use of sleep medication 0.24 0.13
 No −0.1 (−0.2, 0.1) 0.48 (0.39, 0.57) <0.001
 Yes −0.4 (−0.9, 0.2) 0.22 (−0.14, 0.53) 0.22
On EFV, RPV or INSTI 0.69 0.20
 No −0.1 (−0.3, 0.1) 0.53 (0.40, 0.63) <0.001
 Yes −0.1 (−0.3, 0.2) 0.41 (0.27, 0.53) <0.001
Self-reported insomnia 0.80 0.99
 No −0.1 (−0.2, 0.1) 0.44 (0.33, 0.53) <0.001
 Yes −0.1 (−0.5, 0.3) 0.47 (0.27, 0.63) <0.001

Note:

p-value1: testing whether the difference between self-reported and actigraphy-assessed time spent in bed varied according to the characteristic (from t-test or ANOVA, as appropriate)

p-value2: testing the significance of the correlation between self-reported and actigraphy-assessed time spent in bed; Interaction p-value: testing whether the correlation varied according to the characteristic.

*

: p-value comparing the correlation to the one in PWH on ‘Day shift’ work schedule

EFV: Efavirenz; RPV: Rilpivirine; INSTI: integrase strand transfer inhibitor

The correlation between self-reported and actigraphy-assessed time spent in bed did not differ by age, education, depressive symptoms, alcohol consumption, work schedule, use of efavirenz, rilpivirine or integrase strand transfer inhibitors, and self-reported insomnia (Table 2). The correlation appeared stronger in males compared to females (rs= 0.50 vs. 0.18, p=0.05), in those with low compared to high BMI (rs= 0.65 vs. 0.29, p<0.001), and in those not reporting use of recreation drugs compared to those reporting it (rs= 0.51 vs. 0.28, p=0.04).

Sleep onset latency

Self-reported sleep onset latency was <5 minutes for 61 (17.8%) PWH, 5–20 minutes for 145 (42.4%) PWH, 20–60 minutes for 93 (27.2%) PWH and >60 minutes for 43 (12.6%) PWH. On the other hand, actigraphy-assessed onset latency was 5–20 minutes for the majority of PWH (n=330, 96.5%), with only 3 (0.9%) PWH with 20–60 minutes onset latency and none (0.0%) with onset latency >60 minutes. Agreement between self-reported and actigraphy-assessed onset latency was poor [weighted κ (95% CI)=0.002 (−0.02, 0.03)] and was no better than chance (p=0.88, Table 3).

Table 3:

Agreement between self-reported and actigraphy-assessed sleep onset latency in PWH and HIV-negative individuals.

PWH (n=342) – κ (95% CI) = 0.002 (−0.02, 0.03)

Self-assessed onset latency Actigraphy-assessed onset latency
<5 minutes 5–20 minutes 20–60 minutes >60 minutes

<5 minutes 2 (0.6%) 58 (17.0%) 1 (0.3%) 0 (0.0%)
5–20 minutes 4 (1.2%) 140 (40.9%) 1 (0.3%) 0 (0.0%)
20–60 minutes 2 (0.6%) 90 (26.3%) 1 (0.3%) 0 (0.0%)
>60 minutes 1 (0.3%) 42 (12.3%) 0 (0.0%) 0 (0.0%)

HIV-negative individuals (n=119) – κ (95% CI) = 0.009 (−0.03, 0.05)

Self-assessed onset latency Actigraphy-assessed onset latency
<5 minutes 5–20 minutes 20–60 minutes >60 minutes

<5 minutes 1 (0.8%) 40 (33.6%) 0 (0.0%) 0 (0.0%)
5–20 minutes 1 (0.8%) 54 (45.4%) 0 (0.0%) 0 (0.0%)
20–60 minutes 0 (0.0%) 19 (16.0%) 0 (0.0%) 0 (0.0%)
>60 minutes 0 (0.0%) 4 (3.4%) 0 (0.0%) 0 (0.0%)

Compared to PWH, HIV-negative individuals had shorter self-reported but not actigraphy-assessed onset latency (p<0.001 and p=0.49, respectively). Similarly to PWH, the agreement between self-reported and actigraphy-assessed onset latency was no better than chance [weighted κ (95% CI)=0.009 (−0.03, 0.05), p=0.65].

Associations between actigraphy-assessed measures and self-reported insomnia

According to univariable analysis, PWH with self-reported insomnia had later average sleep onset (p=0.04), mid-point (p=0.05) and out-of-bed time (p=0.03), lower maintenance efficiency (p=0.06), greater movement index (p=0.04), and longer nocturnal awakenings (p=0.05) than PWH without insomnia (Table 4). The night-to-night variability of several actigraphy-assessed measures appeared to be greater in PWH with insomnia compared to those without insomnia; these include sleep mid-point (p=0.03), out-of-bed time (p=0.003), sleep duration (p<0.001), WASO (p=0.003), maintenance efficiency (p=0.002), movement index (p=0.001) and mean length of nocturnal awakenings (p=0.009).

Table 4:

Median (IQR) of each sleep measure in PWH with and without insomnia with the PLS-DA coefficient to predict self-reported clinical insomnia.

Median (IQR) PWH without insomnia (n=256) PWH with insomnia (n=76) p-value PLS-DA coeff.

Average sleep onset time [clock time] 23:03 (21:21, 00:52) 23:28 (21:53, 1:33) 0.04 0.21
SD of sleep onset time [minutes] 48 (30, 77) 56 (34, 86) 0.09 0.05
Average sleep mid-point [clock time] 3:54 (3:02, 4:40) 4:02 (3:26, 4:59) 0.05 0.17
SD of sleep mid-point [minutes] 41 (28, 63) 50 (37, 65) 0.03 0.04
Average out-of-bed time [clock time] 7:52 (7:00, 8:42) 8:02 (7:25, 9:22) 0.03 0.24
SD of out-of-bed time [minutes] 52 (36, 77) 65 (44, 92) 0.003 0.20
Average onset latency [minutes] 7 (6, 9) 7 (6, 9) 0.40 0.14
SD of onset latency [minutes] 3 (2, 5) 4 (2, 7) 0.10 0.17
Average duration [hours] 7.1 (6.4, 7.6) 6.9 (6.2, 7.7) 0.44 −0.12
SD of duration [minutes] 54 (38, 75) 81 (45, 96) <0.001 0.41
Average WASO [minutes] 53 (39, 74) 59 (46, 78) 0.15 0.14
SD of WASO [minutes] 18 (13, 28) 24 (16, 36) 0.003 0.27
Average maintenance efficiency [%] 89.0 (84.8, 91.6) 87.3 (82.9, 90.5) 0.06 −0.19
SD of maintenance efficiency [%] 3.6 (2.4, 4.7) 4.2 (2.8, 6.3) 0.002 0.35
Average movement index [%] 17.1 (14.0, 22.3) 18.8 (16.0, 23.5) 0.04 0.16
SD of movement index [%] 3.4 (2.5, 4.9) 4.3 (3.1, 6.8) 0.001 0.40
Average fragmentation index [%] 12.7 (9.7, 16.1) 13.1 (8.9, 16.5) 0.73 0.13
SD of fragmentation index [%] 8.2 (6.4, 10.1) 7.9 (6.8, 9.7) 0.74 0.20
Average number of awakenings 19 (14, 23) 19 (14, 23) 0.81 −0.01
SD of number of awakenings 5 (4, 7) 6 (4, 8) 0.24 0.05
Average mean length of awakenings [min] 3.0 (2.4, 3.7) 3.3 (2.6, 3.7) 0.05 0.15
SD of mean length of awakenings [min] 0.8 (0.6, 1.1) 1.0 (0.6, 1.4) 0.009 0.28

According to PLS-DA, the actigraphy measures with the greatest ability to discriminate between PWH with and without insomnia were the night-to-night variability of sleep duration (PLS-DA coefficient=0.41), movement index (0.40), maintenance efficiency (0.35), mean length of nocturnal awakenings (0.28) and WASO (0.27), and the average out-of-bed time (0.24).

Discussion

We found a fair correlation between self-reported and actigraphy-assessed time spent in bed in PWH, similar to that observed in HIV-negative individuals with similar lifestyles. On the other hand, the agreement between self-reported and actigraphy-assessed sleep onset latency was poor and no better than chance in both PWH and HIV-negative individuals. Self-reported symptoms of insomnia were more prevalent in PWH compared to HIV-negative individuals. While differences in sleep timing were observed, the factors that appeared to most strongly discriminate those with and without insomnia related to actigraphy measures relating to regularity of sleep, rather than sleep duration or efficiency.

Overall, among PWH, there was no systematic overestimation or underestimation of self-report compared to objective actigraphy with regards to time spent in bed. Within HIV-negative individuals, self-report appeared to underestimate time spent in bed determined by actigraphy by approximately 20 minutes. Prior studies have generally found that self-assessed sleep duration overestimates actigraphy-assessed sleep duration [15,17,1921], including among PWH [23]. Differences in findings may be due to the different questions used to determine self-assessed sleep duration or time spent in bed. In fact, self-reported time spent in bed was significantly longer than actigraphy-assessed sleep duration. This has also been demonstrated by Jackson et al. [22] where questions about the habitual number of hours of sleep or about the habitual bed and wake times (as used in our study) yielded different patterns of underestimation or overestimation of self-report versus objective sleep measures.

Yet, the correlation between self-reported and actigraphy-assessed time spent in bed observed in both PWH and HIV-negative individuals is in line with that reported by previous studies conducted in diverse populations of people with and without HIV [15,17,1923], and we found no difference between the two groups.

A few factors appeared to influence the concordance and/or correlation between self-reported and actigraphy-assessed time spent in bed in PWH. The correlation between self-reported and actigraphy-assessed time spent in bed was stronger in males, in PWH of white ethnicity, in those with a BMI ≥25 kg/m2 and in those not reporting use of recreational drugs or sleep medication, compared to their respective counterparts. Moreover, overestimation by self-report was seen in PWH of black-African ethnicity, those with fewer years of formal education and those reporting depressive symptoms as opposed to the underestimation seen in the respective counterparts of PWH. Similar findings have been previously reported for ethnicity [15,21], depressive symptoms [21], education [15], BMI [15] and sleep medication use [16]. Differences in agreement between objective and subjective sleep measures may be explained by different extents of sleep misperception and ability in estimating habitual sleep times (potentially due to greater night-to-night sleep irregularity and difficulties in estimating sleep latency), tiredness and fatigue experienced as a consequence of depressive symptoms or obesity and leading to underestimation of sleep, negative reporting style or heightened somatic sensitivity. The potential implications of this differential agreement between sleep measures may be important: if associations between sleep duration and health among PWH are investigated using self-reported sleep duration, conclusions may not be entirely due to actual sleep duration depending on the socio-demographic characteristics and prevalence of depressive symptoms in the population under study.

Agreement between self-reported and actigraphy-assessed sleep onset latency was poor, regardless of HIV-status. In line with previous studies in HIV-negative populations [20,35,36], self-report overestimated objectively-measured latency especially in PWH, for whom self-assessed, but not actigraphy-assessed, sleep onset latency was significantly longer than in HIV-negative individuals. Actigraphy has been shown to underestimate sleep onset latency compared to polysomnography, the ‘gold standard’ for measuring sleep [37,38], and may be subject to low accuracy in discriminating between wakefulness and sleep when there is lack of movement [39]. On the other hand, self-report often overestimates sleep onset latency also when compared to polysomnography [40,41], suggesting a genuine tendency for individuals to over-report sleep onset latency.

Whilst PWH appear more likely to report more severe symptoms of insomnia compared to HIV-negative individuals, the most traditional sleep parameters such as sleep duration, onset latency, efficiency and wake after sleep onset, as objectively measured by actigraphy, poorly correlate with subjective symptoms of insomnia in PWH. This finding is consistent with previous studies [36,42], suggesting perceived sleep quality poorly reflect aspects of sleep that are traditionally thought to be the most important. Adding to previous studies, we report self-assessed symptoms of insomnia among PWH are more strongly correlated with objectively-measured regularity of sleep duration and quality. Night-to-night regularity of sleep patterns has traditionally been overlooked when investigating the associations between sleep and health outcomes. Only more recently, studies have started to shed light on the role of sleep regularity on different health outcomes [4345]. In particular, irregular sleep patterns and consequent circadian disruptions have been shown to be associated with higher incidence of multiple metabolic disturbances [4345]. As cardiometabolic diseases represent an important focus of care for PWH, our findings highlight the importance of regular night-to-night sleep patterns with respects to the perceived quality of sleep and health of PWH, supporting the development of sleep hygiene interventions that promote sleep regularity.

To our knowledge, this is the first study to report results of both self-reported sleep questionnaires and objective measures of sleep over several days among PWH, with a sample size allowing us to stratify by several factors. Nevertheless, the number of HIV-negative individuals with self-reported insomnia was too small to conduct meaningful analysis to determine whether objective measures of sleep associated with self-reported insomnia varied from those observed in PWH. Moreover, our study has some limitations. First, although our actigraphy measures were assessed by research-grade devices with standardized central scoring supplemented by nightly sleep logs, it is still subject to potential misinterpretations of periods of activity and inactivity so that movement during sleep may be detected as wake time and lack of movement in the absence of sleep may be recorded as sleep. However, compared to the usual ‘gold standard’ of polysomnography, actigraphy allows recording of habitual sleep in one’s usual environment, over several days/nights. Secondly, we administered a single retrospective question about sleep time, then recorded actigraphy over the subsequent seven days and nights. Although we think it is unlikely that sleep patterns would have changed abruptly, we cannot eliminate the possibility that sleep patterns changed between the questionnaire administration and the actigraphy data collection. Thirdly, age differences between PWH and HIV-negative individuals may have introduced bias when comparing the agreement between self-reported and actigraphy-assessed measures across the two groups. However, we found no difference across age groups in the concordance and correlation between self-reported and actigraphy-assessed measures among PWH. Finally, our study was not able to determine how agreement between self-reported and actigraphy-assessed sleep might be affected by other unmeasured factors. We accounted for use of alcohol and recreational drugs and shift work, but did not have measures of other lifestyle factors such as physical activity or dietary behaviors.

Conclusions

In this large study investigating sleep quality in PWH using both subjective and objective measures of sleep, we found fair correlations between self-reported and actigraphy-assessed sleep duration, but poor agreement when it comes to estimate sleep onset latency. The associations between subjective and objective measures of sleep in PWH did not appear to differ to those seen in HIV-negative individuals with similar lifestyles. Importantly, subjective symptoms of insomnia reported by PWH seem to reflect high night-to-night variability in sleep duration, quality and efficiency. Variability in nightly sleep patterns has not been previously studied in PWH and should be a priority for future insomnia research in PWH. Our findings highlight the importance of both patient-reported symptoms and objective sleep measures for better understanding the psychological and physiological determinants of sleep disturbances in PWH.

Supplementary Material

Supplementary material

Supplementary Table: list of daily sleep measures obtained from actigraphy devices over the 7 days/nights observational period

Supplementary Figure: Bland-Altman plot of self-reported time spent in bed and actigraphy-assessed sleep duration among PWH (n=342) and HIV-negative individuals (n=119)

Acknowledgements

POPPY-Sleep Management Team: Ken Kunisaki, Susan Redline, Alan Winston, Caroline Sabin, Paddy Mallon, Nicki Doyle, Amalia Ndoutoumou. POPPY-Sleep Sleep Reading Centre Team: Emily Kaplan, Dan Mobley, Michael Rueschman, Michelle Reid (Brigham and Women’s Hospital, Boston/USA). POPPY-Sleep methodology/statistics/analysis: Caroline Sabin, Davide De Francesco, Mike Rueschman. POPPY-Sleep Sites and Trials Unit: Caldecot Centre, King’s College Hospital (Frank Post, Beatriz Santana Suárez, Lucy Campbell); Department of Infection and Population Health, University College London (Lewis Haddow, Michelle Beynon, Abigail Severn, Anna-Lena Salz, Hinal Lukha); Elton John Centre, Brighton and Sussex University Hospital (Jaime Vera, Rebecca Gleig, Sarah Kirk); HIV Molecular Research Group, School of Medicine, University College Dublin (Paddy Mallon, Alan Macken, Sumesh Babu, Aoife McDermott); Homerton Sexual Health Services, Homerton University Hospital (Jane Anderson, Sambasivarao Pelluri, Anna Price); Imperial Clinical Trials Unit, Imperial College London (Amalia Ndoutoumou, Daphne Babalis); St. Mary’s Hospital London, Imperial College Healthcare NHS Trust (Alan Winston, Felix Dransfield); St Stephen’s Centre, Chelsea and Westminster Hospital (Marta Boffito, Michelle Anthonipillai, Peter Fernando, Shane Hardwick, Chido Chiwome, Candida Fernandez, Ana Milinkovic). POPPY-Sleep Funders: The POPPY-Sleep sub study is funded by the US National Heart, Lung, and Blood Institute (R01 HL131049); the main POPPY study is funded from investigator initiated grants from BMS, Gilead Sciences, Janssen, MSD and ViiV Healthcare. We acknowledge the use of the NIHR/Wellcome Trust Clinical Research Facility at King’s College Hospital. The research is also supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Imperial College Healthcare NHS Trust and Imperial College London.

The Insomnia Severity Index questionnaire is licensed by Mapi Research Trust (Lyon, France) and a license agreement was completed for use in this study.

Funding:

This study is funded by the US National Heart, Lung, and Blood Institute (R01 HL131049); by investigator initiated grants from BMS, Gilead Sciences, Janssen, MSD and ViiV Healthcare; and by the National Institute for Health Research (NIHR) Biomedical Research Centre.

Footnotes

Conflicts of interest

The views presented here are those of the authors and do not reflect the views of the US Government, US Department of Veterans Affairs, or the authors’ affiliated institutions.

References

  • 1.Rubinstein ML, Selwyn PA. High prevalence of insomnia in an outpatient population with HIV infection. Journal of Acquired Immune Deficiency Syndromes and Human Retrovirology 1998; 19:260–265. [DOI] [PubMed] [Google Scholar]
  • 2.Salahuddin N, Barroso J, Leserman J, Harmon JL, Pence BW. Daytime sleepiness, nighttime sleep quality, stressful life events, and HIV-related fatigue. Journal of the Association of Nurses in AIDS Care 2009; 20(1):6–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Lee KA, Gay C, Portillo CJ, Coggins T, Davis H, Pullinger CR, et al. Types of sleep problems in adults living with HIV/AIDS. Journal of Clinical Sleep Medicine 2012; 8(01):67–75. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Crum-Cianflone NF, Roediger MP, Moore DJ, Hale B, Weintrob A, Ganesan A, et al. Prevalence and factors associated with sleep disturbances among early-treated HIV-infected persons. Clinical infectious diseases 2012; 54(10):1485–1494. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Allavena C, Guimard T, Billaud E, De la Tullaye S, Reliquet V, Pineau S, et al. Prevalence and risk factors of sleep disturbance in a large HIV-infected adult population. AIDS and Behavior 2016; 20(2):339–344. [DOI] [PubMed] [Google Scholar]
  • 6.Taibi DM. Sleep disturbances in persons living with HIV. Journal of the Association of Nurses in AIDS Care 2013; 24(1):S72–S85. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Brigham EP, Patil SP, Jacobson LP, Margolick JB, Godfrey R, Johnson J, et al. Association between systemic inflammation and obstructive sleep apnea in men with or at risk for HIV infection. Antiviral therapy 2014; 19(8):725. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Huang X, Li H, Meyers K, Xia W, Meng Z, Li C, et al. Burden of sleep disturbances and associated risk factors: A cross-sectional survey among HIV-infected persons on antiretroviral therapy across China. Scientific reports 2017; 7(1):3657. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Adrien J Neurobiological bases for the relation between sleep and depression. Sleep medicine reviews 2002; 6(5):341–351. [PubMed] [Google Scholar]
  • 10.Malhotra A, Loscalzo J. Sleep and cardiovascular disease: an overview. Progress in cardiovascular diseases 2009; 51(4):279. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Lo JC, Groeger JA, Cheng GH, Dijk D-J, Chee MW. Self-reported sleep duration and cognitive performance in older adults: a systematic review and meta-analysis. Sleep medicine 2016; 17:87–98. [DOI] [PubMed] [Google Scholar]
  • 12.Irwin MR. Why sleep is important for health: a psychoneuroimmunology perspective. Annual review of psychology 2015; 66:143–172. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Kushida CA, Littner MR, Morgenthaler T, Alessi CA, Bailey D, Coleman J Jr, et al. Practice parameters for the indications for polysomnography and related procedures: an update for 2005. Sleep 2005; 28(4):499–523. [DOI] [PubMed] [Google Scholar]
  • 14.Sadeh A The role and validity of actigraphy in sleep medicine: an update. Sleep medicine reviews 2011; 15(4):259–267. [DOI] [PubMed] [Google Scholar]
  • 15.Lauderdale DS, Knutson KL, Yan LL, Liu K, Rathouz PJ. Sleep duration: how well do self-reports reflect objective measures? The CARDIA Sleep Study. Epidemiology (Cambridge, Mass) 2008; 19(6):838. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Van Den Berg JF, Van Rooij FJ, Vos H, Tulen JH, Hofman A, Miedema HM, et al. Disagreement between subjective and actigraphic measures of sleep duration in a population-based study of elderly persons. Journal of sleep research 2008; 17(3):295–302. [DOI] [PubMed] [Google Scholar]
  • 17.Patel SR, Blackwell T, Ancoli-Israel S, Stone KL, Osteoporotic Fractures in Men-Mr OSRG. Sleep characteristics of self-reported long sleepers. Sleep 2012; 35(5):641–648. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Girschik J, Fritschi L, Heyworth J, Waters F. Validation of self-reported sleep against actigraphy. Journal of epidemiology 2012; 22(5):462–468. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Cespedes EM, Hu FB, Redline S, Rosner B, Alcantara C, Cai J, et al. Comparison of self-reported sleep duration with actigraphy: results from the Hispanic Community Health Study/Study of Latinos Sueño Ancillary Study. American journal of epidemiology 2016; 183(6):561–573. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Campanini MZ, Lopez-Garcia E, Rodríguez-Artalejo F, González AD, Andrade SM, Mesas AE. Agreement between sleep diary and actigraphy in a highly educated Brazilian population. Sleep medicine 2017; 35:27–34. [DOI] [PubMed] [Google Scholar]
  • 21.Matthews KA, Patel SR, Pantesco EJ, Buysse DJ, Kamarck TW, Lee L, et al. Similarities and differences in estimates of sleep duration by polysomnography, actigraphy, diary, and self-reported habitual sleep in a community sample. Sleep Health 2018; 4(1):96–103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Jackson CL, Ward JB, Johnson DA, Sims M, Wilson J, Redline S. Concordance between self-reported and actigraphy-assessed sleep duration among African-American adults: findings from the Jackson Heart Sleep Study. Sleep 2020; 43(3):zsz246. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Campbell LM, Kohli M, Lee EE, Kaufmann CN, Higgins M, Delgadillo JD, et al. Objective and subjective sleep measures are associated with neurocognition in aging adults with and without HIV. The Clinical Neuropsychologist 2020:1–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Bagkeris E, Burgess L, Mallon PW, Post FA, Boffito M, Sachikonye M, et al. Cohort profile: The Pharmacokinetic and clinical Observations in PeoPle over fiftY (POPPY) study. International Journal of Epidemiology 2018:dyy072–dyy072. [DOI] [PubMed] [Google Scholar]
  • 25.Sabin CA, Harding R, Doyle N, Redline S, De Francesco D, Mallon PW, et al. Associations between widespread pain and sleep quality in people with HIV. JAIDS Journal of Acquired Immune Deficiency Syndromes 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Kunisaki KM, De Francesco D, Sabin CA, Winston A, Mallon PW, Anderson J, et al. Sleep Disorders in HIV: A Substudy of the Pharmacokinetics and Clinical Observations in People Over Fifty (POPPY) Study. In: Open Forum Infectious Diseases; 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Kroenke K, Spitzer RL, Williams JBW. The PHQ-9. Journal of General Internal Medicine 2001; 16(9):606–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Bastien CH, Vallières A, Morin CM. Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep medicine 2001; 2(4):297–307. [DOI] [PubMed] [Google Scholar]
  • 29.Morin CM, Belleville G, Bélanger L, Ivers H. The Insomnia Severity Index: psychometric indicators to detect insomnia cases and evaluate treatment response. Sleep 2011; 34(5):601–608. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.EACS. Guidelines Version 10.0 November 2019. In; 2019.
  • 31.Fisher RA. Statistical methods for research workers. In: Breakthroughs in statistics: Springer; 1992. pp. 66–70. [Google Scholar]
  • 32.Bland JM, Altman DG. Measuring agreement in method comparison studies. Statistical methods in medical research 1999; 8(2):135–160. [DOI] [PubMed] [Google Scholar]
  • 33.Cohen J Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 1968; 70(4):213. [DOI] [PubMed] [Google Scholar]
  • 34.Landis JR, Koch GG. The measurement of observer agreement for categorical data. biometrics 1977:159–174. [PubMed] [Google Scholar]
  • 35.Lockley SW, Skene DJ, Arendt J. Comparison between subjective and actigraphic measurement of sleep and sleep rhythms. Journal of Sleep Research 1999; 8(3):175–183. [DOI] [PubMed] [Google Scholar]
  • 36.Landry GJ, Best JR, Liu-Ambrose T. Measuring sleep quality in older adults: a comparison using subjective and objective methods. Frontiers in aging neuroscience 2015; 7:166. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.McCall C, McCall WV. Comparison of actigraphy with polysomnography and sleep logs in depressed insomniacs. Journal of sleep research 2012; 21(1):122–127. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Cellini N, Buman MP, McDevitt EA, Ricker AA, Mednick SC. Direct comparison of two actigraphy devices with polysomnographically recorded naps in healthy young adults. Chronobiology international 2013; 30(5):691–698. [DOI] [PubMed] [Google Scholar]
  • 39.Paquet J, Kawinska A, Carrier J. Wake detection capacity of actigraphy during sleep. Sleep 2007; 30(10):1362–1369. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Baker FC, Maloney S, Driver HS. A comparison of subjective estimates of sleep with objective polysomnographic data in healthy men and women. Journal of psychosomatic research 1999; 47(4):335–341. [DOI] [PubMed] [Google Scholar]
  • 41.Silva GE, Goodwin JL, Sherrill DL, Arnold JL, Bootzin RR, Smith T, et al. Relationship between reported and measured sleep times: the sleep heart health study (SHHS). Journal of Clinical Sleep Medicine 2007; 3(06):622–630. [PMC free article] [PubMed] [Google Scholar]
  • 42.Hoang HTX, Molassiotis A, Chan CW, Nguyen TH. New-onset insomnia among cancer patients undergoing chemotherapy: prevalence, risk factors, and its correlation with other symptoms. Sleep and Breathing 2019:1–11. [DOI] [PubMed] [Google Scholar]
  • 43.Sohail S, Yu L, Bennett DA, Buchman AS, Lim ASP. Irregular 24-hour activity rhythms and the metabolic syndrome in older adults. Chronobiology International 2015; 32(6):802–813. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Lunsford-Avery JR, Engelhard MM, Navar AM, Kollins SH. Validation of the Sleep Regularity Index in Older Adults and Associations with Cardiometabolic Risk. Scientific Reports 2018; 8(1):14158. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Huang T, Redline S. Cross-sectional and Prospective Associations of Actigraphy-Assessed Sleep Regularity With Metabolic Abnormalities: The Multi-Ethnic Study of Atherosclerosis. Diabetes Care 2019; 42(8):1422–1429. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary material

Supplementary Table: list of daily sleep measures obtained from actigraphy devices over the 7 days/nights observational period

Supplementary Figure: Bland-Altman plot of self-reported time spent in bed and actigraphy-assessed sleep duration among PWH (n=342) and HIV-negative individuals (n=119)

RESOURCES