Abstract
Inaccurate self-reported data on medication exposure lead to less reliable study findings. From 2013 to 2015, we assessed the validity of information on medication use collected via a mailed medication inventory among 223 Women's Health Initiative participants who were members of a health-care delivery system. Self-reported information on medication use was compared with pharmacy records for statins, calcium channel blockers, β-blockers, and bisphosphonates. We assessed sensitivity, specificity, and positive predictive value (PPV) for current medication use. We assessed agreement on duration of use (<2, 2, 3, 4, or ≥5 years) by means of the weighted κ statistic. The mean age of participants was 77 years. Statins, β-blockers, and calcium channel blockers were each reported by over 15% of women, and bisphosphonates were reported by 4.5%. Compared with pharmacy records, the sensitivity, specificity, and PPV for self-reported use of statins, β-blockers, and calcium channel blockers were all 95% or greater. The sensitivity and PPV for bisphosphonate use were both 80% (95% confidence interval: 44, 97), and specificity was 99% (95% confidence interval: 97, 100). The κ statistic for duration of use was 0.87 or greater for all 4 medication classes. Compared with pharmacy records, self-reported information on current medication use and duration of use collected via mailed medication inventory among older women had almost perfect agreement for use of statins, β-blockers, and calcium channel blockers.
Keywords: data collection, medications, questionnaires, pharmacy records, self-reporting, validity
Medication use is commonly assessed by self-report in research studies, but its accuracy is limited by participant recall error (1, 2). Inaccurate self-reported data can lead to exposure misclassification and, thereby, to less reliable study findings. Validity studies can estimate misclassification in self-reported data and help determine whether study results may be biased. Pharmacy and medical records are often used as a “gold standard” for validity studies of self-reported medication use, and pharmacy records may be more complete than medical records (1, 2). Increased age is associated with diminished accuracy of self-reported events (1, 2), which makes validity studies among older women an important research topic.
Existing validity studies have demonstrated good agreement between self-reported medication use and pharmacy or medical records among older women, but most studies used interviewer-administered forms or asked about specific medications (1, 3–8). In particular, the accuracy of data on use of statin and antihypertensive medications collected via interviewer-administrated forms as compared with pharmacy records has been established among older women (3), but the accuracy of data collected via self-administered mailed forms has not been studied. Additionally, good accuracy of self-reported bisphosphonate use as compared with pharmacy records was found among chronic glucocorticoid users, but it has not been studied in a general population of older women (9).
The accuracy of data from self-administered mailed forms may differ from the accuracy of data derived from interviewer-administered forms and forms that query about specific medications. Thus, we conducted a validity study to compare self-reported medication use data collected via a mailed medication inventory with pharmacy records among older Women's Health Initiative (WHI) participants for 4 classes of chronically used medications: statins, calcium channel blockers, β-blockers, and bisphosphonates.
METHODS
Women's Health Initiative
The WHI is an ongoing research study with the primary aim of developing strategies that reduce disease morbidity and mortality in older women. Between 1993 and 1998, WHI investigators recruited 161,808 postmenopausal women aged 50–79 years at enrollment to participate in randomized clinical trials and an observational study, conducted between 1993 and 2005. The clinical trials included studies of hormone therapy with estrogen alone and with estrogen plus progestin, dietary modification, and calcium and vitamin D supplementation (10). Beginning in 2005, a total of 115,407 women agreed to continue participation as part of the WHI Extension Studies.
The WHI study design and methods have been described in detail elsewhere (11–13). In 2008–2009, a current medication inventory was administered by mail to all active WHI participants (14).
Group Health Cooperative
Subjects for the present study were WHI participants who had also been enrollees in the Group Health Cooperative (Seattle, Washington) for at least 5 years at the completion of the 2008–2009 medication inventory. Group Health (GH) is an integrated health-care delivery system that provides comprehensive health care to approximately 600,000 residents of Washington State and Idaho. GH's electronic pharmacy database was created in 1977 and has been used in numerous observational studies (15). In a study of seniors who were GH enrollees, 91% of enrollees with a drug benefit and 78% of enrollees without a drug benefit reported that they filled their prescriptions exclusively at GH-owned pharmacies (16). In 2009, 84% of female GH enrollees aged 65 years or older had a drug benefit.
Study participants and recruitment
We identified 580 WHI participants who completed the 2008–2009 medication inventory, were enrolled in WHI at the Seattle clinic site, and reported having insurance through a health maintenance organization. We mailed letters to these women asking them if they were members of GH and, if so, asking permission to use existing WHI self-report data and GH pharmacy records for the validity study. Several Seattle health clinics offer health maintenance organization insurance. We asked women to respond only if they were GH enrollees. A total of 278 (48%) consented to participate, 130 (22%) responded that they were not GH enrollees, and 172 (30%) did not respond. Of the 278 women who consented, 223 were actually GH enrollees for at least 5 years at the 2008–2009 medication inventory date and were eligible for the study. For these 223 women, we analyzed self-reported medication use as compared with GH pharmacy records for 4 classes of chronically used medications: statins, calcium channel blockers, β-blockers, and bisphosphonates.
The validity study was conducted from 2013 to 2015. The Institutional Review Office of the Fred Hutchinson Cancer Research Center (Seattle, Washington) approved the protocol.
Exposure classification
In WHI, duration of medication exposure was self-reported on the 2008–2009 medication inventory, which was mailed to participants, self-administered, and returned to WHI investigators by mail. Instructions on the medication inventory indicated that telephone assistance was available if needed, but only 1 of the 223 validity study participants completed the form by telephone. Current medication use was defined by the answer to the question, “Are you currently taking any medications that require a prescription from a doctor or health care provider?” The form instructed participants who answered “yes” to gather all of their current prescription medications. The form included 1 example prescription label with a completed entry. Participants wrote down the name, strength, type (tablet, capsule, etc.), and duration of use (<1 month, 1–12 months, a continuous number of years) of each prescription medication. Medications were categorized into medication classes using Medi-Span (Medi-Span Inc., Indianapolis, Indiana), a pharmaceutical reference database. We defined women as users of a medication class if they reported using any medication in that class. We categorized self-reported duration of use from the medication inventory into groups (<2, 2, 3, 4, and ≥5 years).
The reference date for medication exposure was the date on which the participant completed the 2008–1009 WHI medication inventory. For this analysis, we used GH pharmacy records for the period 5 years before the reference date through 60 days after the reference date. From the GH data, we defined the duration of medication use as the reference date minus the date of the first pharmacy prescription during a period of continuous use that included the reference date. We defined a period of use as continuous if there was at least 1 prescription for a medication class during a 60-day period after the run-out date of the last prescription. We calculated the run-out date as the prescription date plus the number of days’ worth of medication supplied in the prescription. We defined a woman as a true current user of a medication class if the period of continuous use for that medication class included the reference date. We categorized duration of use into groups (<2, 2, 3, 4, and ≥5 years).
Covariates
Self-reported characteristics of interest included age, race, educational level, income level, marital status, and self-rated general health status (11, 17). We used the most recent value collected in WHI on or before the medication inventory date.
Statistical analysis
We determined the characteristics of validity study participants and WHI participants not enrolled in the validity study. Accuracy of self-reported medication use reported on the 2008–2009 medication inventory was examined using pharmacy records as the gold standard for the 4 medication classes (statins, calcium channel blockers, β-blockers, and bisphosphonates). We assessed sensitivity (proportion of positives in the pharmacy records that were also reported on the medication inventory), specificity (proportion of negatives in the pharmacy records that were also reported as negatives on the medication inventory), and positive predictive value (PPV; proportion of positives reported by the medication inventory that were verified by pharmacy records) for each medication class. We assessed the accuracy of self-reported duration of medication use (<2, 2, 3, 4, or ≥5 years) using the weighted κ statistic. The κ statistic measures the amount of interrater agreement beyond agreement expected by chance alone, and the weighted κ statistic gives partial credit for close agreement between individual answers in 2 sets of ordered categorical variables (18–20). Confidence intervals for the κ statistic were bias-adjusted using 1,000 naive bootstrap repetitions (21–23). The number of women with missing information on duration of use ranged from 0 to 6 for the 4 medication classes; these women were excluded from calculation of the κ statistic for that medication class. We also conducted a logistic regression analysis to assess participant characteristics as predictors of the accuracy of self-reported data. For each medication class, we fitted 2 multivariate logistic regression models: one to examine age, educational level, marital status, and a general health rating of “fair” or “poor” as predictors of disagreement between self-reported medication use and pharmacy records and one to examine age, educational level, and marital status as predictors of disagreement between duration of self-reported medication use and duration from pharmacy records. All statistical tests were 2-tailed (α = 0.05), and all analyses were conducted with Stata statistical software, version 13 (StataCorp LP, College Station, Texas).
RESULTS
Descriptive characteristics
Validity study participants (n = 223) were more likely to be white and had higher educational attainment compared with WHI participants not in the validity study (n = 97,225; Table 1). Self-reported bisphosphonate use was less common among validity study participants.
Table 1.
Characteristics of 223 Validity Study Participants and 97,225 Participants Not in the Validity Study Who Completed the 2008–2009 Medication Inventory, Women's Health Initiative, 2013–2015
| Characteristic | WHI Validity Study Participants (n = 223) |
Other WHI Study Participants (n = 97,225) |
||
|---|---|---|---|---|
| No. | % | No. | % | |
| Age, yearsa | 77.4 (6.4) | 75.4 (6.6) | ||
| White/Caucasian race | 211 | 94.6 | 84,833 | 87.3 |
| Educational attainmentb | ||||
| High school diploma/GED or less | 26 | 11.7 | 18,413 | 18.9 |
| Schooling after high school | 71 | 31.8 | 35,462 | 36.5 |
| College degree or higher | 124 | 55.6 | 42,709 | 43.9 |
| Marital statusb | ||||
| Never married | 5 | 2.2 | 3,970 | 4.1 |
| Married/marriage-like relationship | 166 | 74.4 | 65,022 | 66.9 |
| Divorced/separated/widowed | 52 | 23.3 | 27,875 | 28.7 |
| Annual income, dollarsb | ||||
| <20,000 | 13 | 5.8 | 10,810 | 11.1 |
| 20,000–34,999 | 43 | 19.3 | 20,556 | 21.1 |
| 35,000–49,999 | 68 | 30.5 | 19,554 | 20.1 |
| ≥50,000 | 94 | 42.2 | 40,945 | 42.1 |
| General self-rated health of “fair” or “poor” | 14 | 6.3 | 9,261 | 9.5 |
| Self-reported medication use | ||||
| Statin | 80 | 35.9 | 38,764 | 39.9 |
| β-blocker | 66 | 29.6 | 26,398 | 27.2 |
| Calcium channel blocker | 35 | 15.7 | 17,305 | 17.8 |
| Bisphosphonate | 10 | 4.5 | 17,075 | 17.6 |
Abbreviations: GED, General Educational Development; WHI, Women's Health Initiative.
a Values are expressed as mean (standard deviation).
b Numbers for subgroups do not sum to the total because of missing data.
Accuracy of self-reported medication use
Self-reported current medication use was over 15% for statins, β-blockers, and calcium channel blockers and was 4.5% for bisphosphonates. The sensitivity, specificity, and PPV were all 95% or greater for statins, β-blockers, and calcium channel blockers (Table 2). For bisphosphonates, the sensitivity and PPV were 80% and the specificity was 99%. The weighted κ statistic comparing self-reported duration of medication use with true duration of use showed almost perfect agreement for all 4 medication classes; κ values ranged from 0.87 to 0.96. In the multivariate-adjusted logistic regression analysis, none of the participant characteristics were predictive of accuracy of self-reported medication use or self-reported duration of medication use for any medication class.
Table 2.
Accuracy of Self-Reported Data on Current Medication Use As Compared With Pharmacy Records for Statins, β-Blockers, Calcium Channel Blockers, and Bisphosphonates Among 223 Oldera Postmenopausal Women, Women's Health Initiative, 2013–2015
| Medication Class | Accuracy of Self-Report of Current Medication Use |
Accuracy of Self-Report of Duration of Current Medication Use |
||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No. of Participants |
Sensitivity |
Specificity |
PPV |
|||||||||
| Self-Report Yes, Pharmacy Yes | Self-Report Yes, Pharmacy No | Self-Report No, Pharmacy Yes | Self-Report No, Pharmacy No | |||||||||
| % | 95% CI | % | 95% CI | % | 95% CI | κ Statisticb | 95% CI | |||||
| Statins | 79 | 4 | 1 | 139 | 95 | 88, 99 | 99 | 96, 100 | 99 | 93, 100 | 0.96 | 0.92, 0.98 |
| β-blockers | 65 | 2 | 1 | 155 | 97 | 90, 100 | 99 | 96, 100 | 98 | 92, 100 | 0.87 | 0.73, 0.96 |
| Calcium channel blockers | 35 | 0 | 0 | 188 | 100 | 90, 100 | 100 | 98, 100 | 100 | 90, 100 | 0.89 | 0.75, 0.97 |
| Bisphosphonates | 8 | 2 | 2 | 211 | 80 | 44, 97 | 99 | 97, 100 | 80 | 44, 97 | 0.95 | 0.67, 1.00 |
Abbreviations: CI, confidence interval; PPV, positive predictive value.
a Aged 50–79 years at enrollment.
b Weighted bias-adjusted κ statistic for duration of medication use (<2, 2, 3, 4, or ≥5 years). Participants with incomplete data on duration of use were excluded from calculation of the κ statistic for that medication class (6 from statins, 6 from β-blockers, 3 from calcium channel blockers, and 0 from bisphosphonates).
DISCUSSION
Our study, carried out among 223 older WHI participants, suggested that a mailed medication inventory is an excellent source of medication exposure data for chronically used medications. Compared with pharmacy records, we found near-perfect sensitivity and PPV for self-reported use of statins, β-blockers, and calcium channel blockers and 80% sensitivity and PPV for bisphosphonates. Specificity of self-reported medication use was nearly perfect, and the κ statistic for self-reported duration of medication use showed near-perfect agreement for all 4 drug classes.
We are not aware of any other studies examining the validity of a self-administered mailed medication inventory that asked participants to report all current medications without prompting them for specific medications. Our results are similar to those of Boudreau et al. (3), who used an interviewer-administered form that prompted respondents for specific medications and found near-perfect agreement between self-reported data and pharmacy records for recent (within 6 months) use of statins and antihypertensive drugs among older members of GH. Compared with the findings of Boudreau et al. (3), current self-reported medication use collected via a mailed medication inventory was as accurate as data collected by an interviewer. Using mailed study forms can provide substantial economic savings compared with interviewer-administered forms, especially for studies as large as the WHI, which collected a mailed medication inventory form from over 97,000 women. It is notable that the WHI medication inventory asked participants to refer to their prescription labels while completing the form. This technique could be utilized in other studies that collect self-reported data on current medication use.
Advanced age has been associated with decreased accuracy of self-reported events (2), and validity studies of self-reported medication use have found associations between older age and decreased recall accuracy for medication use (1, 9). Our study found no association between increased age and recall accuracy. However, our study had limited power to detect differences by age.
There were several limitations to our study. The study sample may not have been representative of the general population of older women, because 95% of the subjects were white, and 56% had a college degree or higher educational attainment. The relatively high proportion of whites and the relatively high educational attainment in the sample studied here reflect the characteristics of the Seattle population in this age group (24) and the WHI's targeted recruitment of members of minority groups at other WHI clinic sites (11, 12). Additionally, our findings should not be generalized to past medication use, infrequently used medications, or less socially acceptable medications. Furthermore, we had few bisphosphonate users, which may have limited the accuracy of estimates related to bisphosphonates. As with any validity study using pharmacy records, disagreement between self-reported medication use and pharmacy records may have been due to subjects’ not taking a medication after filling a pharmacy prescription or filling prescriptions at pharmacies other than the pharmacies included in the validity study.
In conclusion, in this population of older women, a mailed medication inventory appears to be a highly accurate means of assessing current use and duration of use of 4 classes of chronically used medications. Our findings are important for epidemiologic research, because self-reported medication data from the WHI are used in numerous analyses of medication associations.
ACKNOWLEDGMENTS
Author affiliations: Department of Epidemiology, School of Public Health, University of Washington, Seattle, Washington (Rebecca L. Drieling, Andrea Z. LaCroix, Shirley A. A. Beresford, Denise M. Boudreau, Susan R. Heckbert); Department of Biostatistics, School of Public Health, University of Washington, Seattle, Washington (Charles Kooperberg); Divisions of Epidemiology and Family Medicine, Department of Family Medicine and Public Health, School of Medicine, University of California, San Diego, San Diego, California (Andrea Z. LaCroix); Fred Hutchinson Cancer Research Center, Seattle, Washington (Andrea Z. LaCroix, Shirley A. A. Beresford, Charles Kooperberg); Group Health Research Institute, Group Health Cooperative, Seattle, Washington (Andrea Z. LaCroix, Denise M. Boudreau, Susan R. Heckbert); and Department of Pharmacy, School of Pharmacy, University of Washington, Seattle, Washington (Denise M. Boudreau, Susan R. Heckbert).
This research was partially supported by the NCI Biobehavioral Cancer Prevention and Control Training Program at the University of Washington (grant R25CA092408). The Women's Health Initiative (WHI) programs are funded by the National Heart, Lung, and Blood Institute through contracts HHSN268201100046C, HHSN268201100001C, HHSN268201100002C, HHSN268201100003C, and HHSN268201100004C.
The WHI Investigators—Program Office: Jacques Rossouw, Shari Ludlam, Dale Burwen, Joan McGowan, Leslie Ford, and Nancy Geller (National Heart, Lung, and Blood Institute, Bethesda, Maryland); Clinical Coordinating Center: Garnet Anderson, Ross Prentice, Andrea LaCroix, and Charles Kooperberg (Fred Hutchinson Cancer Research Center, Seattle, Washington); principal investigators and academic centers: JoAnn E. Manson (Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts); Barbara V. Howard (MedStar Health Research Institute/Howard University, Washington, DC); Marcia L. Stefanick (Stanford Prevention Research Center, Stanford, California); Rebecca Jackson (The Ohio State University, Columbus, Ohio); Cynthia A. Thomson (University of Arizona, Tucson/Phoenix, Arizona); Jean Wactawski-Wende (State University of New York at Buffalo, Buffalo, New York); Marian Limacher (University of Florida, Gainesville/Jacksonville, Florida); Robert Wallace (University of Iowa, Iowa City/Davenport, Iowa); Lewis Kuller (University of Pittsburgh, Pittsburgh, Pennsylvania); and Sally Shumaker (Wake Forest University School of Medicine, Winston-Salem, North Carolina).
A.Z.L. serves on scientific advisory boards for AMGen, Inc. (Thousand Oaks, California) and Sermonix Pharmaceuticals LLC (Columbus, Ohio) and served as a one-time consultant for Pfizer, Inc. (New York, New York) in the past 12 months.
REFERENCES
- 1.West SL, Savitz DA, Koch G et al. Recall accuracy for prescription medications: self-report compared with database information. Am J Epidemiol. 1995;14210:1103–1112. [DOI] [PubMed] [Google Scholar]
- 2.White E, Armstrong BK, Saracci R. Principles of Exposure Measurement in Epidemiology: Collecting, Evaluating, and Improving Measures of Disease Risk Factors. 2nd ed New York, NY: Oxford University Press; 2008. [Google Scholar]
- 3.Boudreau DM, Daling JR, Malone KE et al. A validation study of patient interview data and pharmacy records for antihypertensive, statin, and antidepressant medication use among older women. Am J Epidemiol. 2004;1593:308–317. [DOI] [PubMed] [Google Scholar]
- 4.Pit SW, Byles JE, Cockburn J. Accuracy of telephone self-report of drug use in older people and agreement with pharmaceutical claims data. Drugs Aging. 2008;251:71–80. [DOI] [PubMed] [Google Scholar]
- 5.Solomon DH, Stedman M, Licari A et al. Agreement between patient report and medical record review for medications used for rheumatoid arthritis: the accuracy of self-reported medication information in patient registries. Arthritis Rheum. 2007;572:234–239. [DOI] [PubMed] [Google Scholar]
- 6.Cotterchio M, Kreiger N, Darlington G et al. Comparison of self-reported and physician-reported antidepressant medication use. Ann Epidemiol. 1999;95:283–289. [DOI] [PubMed] [Google Scholar]
- 7.Persson I, Bergkvist L, Adami HO. Reliability of women's histories of climacteric oestrogen treatment assessed by prescription forms. Int J Epidemiol. 1987;162:222–228. [DOI] [PubMed] [Google Scholar]
- 8.Paganini-Hill A, Ross RK. Reliability of recall of drug usage and other health-related information. Am J Epidemiol. 1982;1161:114–122. [DOI] [PubMed] [Google Scholar]
- 9.Curtis JR, Westfall AO, Allison J et al. Agreement and validity of pharmacy data versus self-report for use of osteoporosis medications among chronic glucocorticoid users. Pharmacoepidemiol Drug Saf. 2006;1510:710–718. [DOI] [PubMed] [Google Scholar]
- 10.Rossouw JE, Anderson GL, Prentice RL et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial. JAMA. 2002;2883:321–333. [DOI] [PubMed] [Google Scholar]
- 11.Women's Health Initiative Study Group. Design of the Women's Health Initiative clinical trial and observational study. Control Clin Trials. 1998;191:61–109. [DOI] [PubMed] [Google Scholar]
- 12.Anderson GL, Manson J, Wallace R et al. Implementation of the Women's Health Initiative study design. Ann Epidemiol. 2003;13(9 suppl):S5–S17. [DOI] [PubMed] [Google Scholar]
- 13.Curb JD, McTiernan A, Heckbert SR et al. Outcomes ascertainment and adjudication methods in the Women's Health Initiative. Ann Epidemiol. 2003;13(9 suppl):S122–S128. [DOI] [PubMed] [Google Scholar]
- 14.WHI Forms. F153 v1—Medication and Supplement Inventory. http://www.whi.org/researchers/studydoc/SitePages/Forms.aspx Published July 1, 2009. Updated July 1, 2015. Accessed July 14, 2015.
- 15.Saunders K. Group Health Cooperative of Puget Sound. In: Strom BL, ed. Pharmacoepidemiology. 3rd ed Chichester, United Kingdom: John Wiley & Sons Ltd.; 2000:247–262. [Google Scholar]
- 16.Boudreau DM, Doescher MP, Jackson JE et al. Impact of healthcare delivery system on where HMO-enrolled seniors purchase medications. Ann Pharmacother. 2004;38(7-8):1317–1318. [DOI] [PubMed] [Google Scholar]
- 17.Ware JE Jr, Gandek B. Overview of the SF-36 Health Survey and the International Quality of Life Assessment (IQOLA) Project. J Clin Epidemiol. 1998;5111:903–912. [DOI] [PubMed] [Google Scholar]
- 18.Fleiss JL, Levin BA, Paik MC. Statistical Methods for Rates and Proportions. 3rd ed Hoboken, NJ: John Wiley & Sons, Inc.; 2003. [Google Scholar]
- 19.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;331:159–174. [PubMed] [Google Scholar]
- 20.Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;704:213–220. [DOI] [PubMed] [Google Scholar]
- 21.Efron B, Tibshirani R. An Introduction to the Bootstrap. New York, NY: Chapman & Hall, Inc.; 1993. [Google Scholar]
- 22.Lee J, Fung KP. Confidence interval of the kappa coefficient by bootstrap resampling. Psychiatry Res. 1993;491:97–98. [DOI] [PubMed] [Google Scholar]
- 23.Cantor A. Sample-size calculations for Cohen's kappa. Psychol Methods. 1996;12:150–153. [Google Scholar]
- 24.Ryan CL, Siebens J. Educational Attainment in the United States: 2009. Washington, DC: Bureau of the Census, Economics and Statistics Administration, US Department of Commerce; 2012. http://www.census.gov.edgekey-staging.net/content/dam/Census/library/publications/2012/demo/p20-566.pdf Updated February 1, 2012. Accessed November 22, 2015. [Google Scholar]