Abstract
Objective
Evaluate the association between thyroid disease and use of insecticides, herbicides, fumigants/fungicides in male applicators in the Agricultural Health Study.
Methods
We examined the association between use of 50 specific pesticides and self-reported hypothyroidism, hyperthyroidism, and ‘other’ thyroid disease among 22,246 male pesticide applicators.
Results
There was increased odds of hypothyroidism with ever-use of the herbicides 2,4-D, 2,4,5-T, 2,4,5-TP, alachlor, dicamba, and petroleum oil. Hypothyroidism was also associated with ever-use of eight insecticides: organochlorines chlordane, DDT, heptachlor, lindane, and toxaphene; organophosphates diazinon and malathion; and the carbamate carbofuran. Exposure-response analysis showed increasing odds with increasing level of exposure for the herbicides alachlor and 2,4-D, and the insecticides aldrin, chlordane, DDT, lindane, and parathion.
Conclusions
There is an association between hypothyroidism and specific herbicides and insecticides in male applicators, similar to previous results for spouses.
Keywords: Endocrine Disruptor, Hypothyroidism, Organochlorines, Pesticides, Thyroxine
Introduction
Thyroid disease is common with a prevalence of approximately 7.5% in the U.S. population. This may be an underestimate because some studies have reported an additional 9% of adults with unidentified clinical or subclinical disease (1-3). Known risk factors for thyroid disease, such as age, genetic predisposition, and female gender (4) are non-modifiable, making the identification of modifiable risk factors for thyroid disease all the more important. Thyroid disease is less common in men than women, and has been less studied, so there is a paucity of data regarding risk factors in men.
Some pesticides have been reported to act as endocrine disruptors with effects on many systems, including the thyroid (5-7). Studies focusing on the effect of pesticides on circulating thyroid stimulating hormone (TSH) or thyroid hormone levels, triiodothyronine (T3) and thyroxine (T4), have found conflicting results(8-10). Mechanisms for thyroid disruption may include interference at the level of the hypothalamic pituitary thyroid axis, inhibition of iodine intake in the thyroid gland by the sodium-iodide symporter, increased synthesis of uridine diphosphate glucoronyltransferase altered binding of the transport proteins thyroid binding globulin and transthyretin resulting in increased excretion of thyroid hormone, decreased cellular uptake of thyroid hormone, alteration in the expression of deiodinases, altered transcriptional activity of thyroid hormone receptors, and up or down regulated expression of thyroid hormone regulated genes (11-13). However, not all pesticides have been classified as endocrine disruptors, and those that have do not all have the same effect on thyroid function or have the same mechanism of action.
We have previously reported associations between hypothyroidism and use of organochlorine insecticides, the fungicides, maneb/mancozeb and benomyl, and the herbicide paraquat in female spouses of private pesticide applicators in the Agricultural Health Study (AHS)(14). Thyroid disease is in general more common in women, and it is unknown whether a similar association between pesticide use and hypothyroidism exists in men. The male pesticide applicators in the AHS generally use a larger number of pesticides and often apply larger amounts of individual pesticides than the female spouses who are usually not the primary applicators (15). Accordingly more information on pesticide use such as frequency and duration were collected for the licensed applicators. In this study, we sought to evaluate the relationship between pesticide use and self-reported thyroid disease (hypothyroidism, hyperthyroidism and ‘other’ thyroid disease) in the male pesticide applicators in the AHS.
Materials and Methods
Study Population and Exposure Assessment
The AHS is a prospective study of commercial and private pesticide applicators and spouses in Iowa and North Carolina (16). The institutional review boards of the National Institutes of Health (Bethesda, Maryland) and their contractors approved the study. Informed consent was indicated by the return of enrollment questionnaires and participation in follow-up interviews.
Pesticide applicators applying for certification to use restricted-use pesticides were recruited between 1993 and 1997. Pesticide applicators were asked to complete two self-administered questionnaires, the first at the licensing site (enrollment) and the second a more detailed take-home questionnaire completed within 1-3 months of enrollment (Phase I; http://www.aghealth.org/questionnaires.html). In the enrollment questionnaire, the private pesticide applicators reported ever-use of 50 itemized pesticides and provided detailed information on frequency of use (days per year) and years applied for 22 of the pesticides. Demographics, education, smoking history, years worked or lived on a farm were assessed at enrollment. In the take-home questionnaire, completed by 44% of the enrolled applicators, information was collected on body mass index and on detailed patterns of use for the remaining 28 pesticides, including the organochlorine insecticides. Characteristics of applicators that did and did not complete the take-home questionnaire were very similar (17). We evaluated two exposure metrics: 1) ever-use and 2) intensity-weighted cumulative days of use, calculated by multiplying lifetime days of use (days/year used × years applied) by a weighting algorithm based on application methods, mixing habits, repairs and use of personal protective equipment (18).
Thyroid Disease
Private applicators were re-contacted by telephone for follow-up interviews in Phase II (1999-2003) and in Phase III (2005-2010), when they provided information on thyroid disease. Self-reported, lifetime history of physician-diagnosed thyroid disease was based on information collected in Phase II and/or Phase III interviews. In both, the applicators’ answer to “Has a doctor or other health professional ever told you that you had thyroid disease or thyroid problems?” was used to define thyroid disease. Prevalent thyroid disease was classified into three mutually exclusive groups as described previously (14): hypothyroid, hyperthyroid, and ‘other’. Participants were considered to have hypothyroidism if they answered yes to at least one of the following questions in Phase II or III: Were you ever told you have an underactive thyroid? Was this due to thyroiditis, Hashimoto’s, or an autoimmune disease? Was there some other cause identified for underactive thyroid disease? A diagnosis of hyperthyroidism was established if the participant answered yes to any of the following questions in Phase II or III: Were you ever told you had an overactive thyroid? Was it due to Grave’s disease? Was it due to thyrotoxicosis? Was there some other cause of overactive thyroid that was identified? Participants were considered to have ‘other’ thyroid disease if they did not report hyperthyroidism or hypothyroidism but reported any one of the following - an enlarged thyroid, thyroid nodules, or goiter- or were told they had some type of thyroid problem or thyroid disease. Additionally, a small number (n=19) of participants who answered yes to having both hypothyroidism and hyperthyroidism in Phase II and/or Phase III (i.e. hyperthyroid in phase II and hypothyroid in phase III or hypothyroid in phase II and hyperthyroid in phase III) were added to the “other” thyroid disease category. Thyroid disease was not reported at baseline and information on age of diagnosis was incomplete at follow-up, so our analysis considered lifetime history of thyroid disease rather than thyroid disease incident since enrollment.
Statistical Analyses
This analysis was restricted to male private pesticide applicators who provided complete data on ever-use of 50 individual pesticides at enrollment (35,505), information on thyroid disease in phase II or phase III (22,854) and complete data on all covariates. The final analysis was on the 22,246 applicators with complete data for pesticides, thyroid disease and covariates (Figure 1).
We examined the association between ever-use of 50 pesticides and the occurrence of any thyroid disease (hypothyroidism, hyperthyroidism, or “other” versus controls with no thyroid disease) using polytomous regression. The base model was formulated using a directed acyclic graphic (DAG) causal framework approach to incorporate the causal relations among independent and dependent variables. The DAG identified age at enrollment (≤ 38, 39-46, 47-57, ≥ 58 years), education (≤ high school, > high school), and body mass index (BMI, kg/m2) at enrollment (< 25, 25-29.9, ≥ 30 kg/m2) to be the minimum sufficient adjustment set (MSAS) of covariates for the base model. Smoking (never, past, current) and years lived or worked on a farm (> 30 years, ≤ 30 years) were considered but not included. Because there were few associations between pesticide use and hyperthyroidism or “other” thyroid diseases, we focused additional analyses on risk of pesticide use on hypothyroid disease using logistic regression in models including the same covariates.
Exposure-response was assessed based on intensity-weighted cumulative days of use. To create the exposure levels, we split the exposure distribution for each pesticide at the median value of the intensity-adjusted cumulative days of use among users and created a three level variable: no exposure (zero days of use), low exposure (≤ median), and high exposure (> median), and coded them respectively as 0,1, 2. Exposure-response was assessed using the Cochran-Armitage test for trend (p< 0.05 significant). Exposure-response models were not constructed for fungicides and fumigants because use of these chemicals was limited.
Pesticide applicators typically use more than one pesticide; hence the potential confounding effects of use of correlated pesticides was evaluated. Spearman correlation coefficients were calculated for all 2-way combinations of pesticides significantly associated with hypothyroid disease (p < 0.05) in single pesticide models. We then assessed risk for hypothyroid disease in models including pairs of moderately correlated pesticides (r ≥ 0.3) and compared the resulting pesticide parameter estimates to those from models including only one pesticide. All analysis results were conducted using SAS software (version 9.2; SAS Institute, Inc., Cary, North Carolina).
Results
Population Demographics
We had complete information on all base model covariates, use of pesticides (fumigants, fungicides, herbicides and insecticides) and thyroid disease for 22,246 male private pesticide applicators in the AHS (Figure 1). A total of 461 (2%) applicators reported hypothyroidism, 175 (1%) applicators reported hyperthyroidism, and 283 (1%) reported “other” thyroid disease (19 applicators with both overactive and underactive thyroid disease and 264 reporting goiter, enlarged thyroid, thyroid nodules, or unspecified thyroid disease). The population consisted predominantly of white men from Iowa (Table 1) with a mean age (SD) of 45.6 ± 12 years and mean BMI (SD) of 27.6 ± 4.0 kg/m2 at enrollment. Compared with participants without thyroid disease, those in each thyroid disease category were more likely to be older. Individuals reporting hypothyroidism had a higher BMI than those in other categories. Though smoking status and years lived or worked on a farm appeared to be associated with thyroid disease (p<0.0001), these associations went away after adjustment for the covariates age, education and BMI.
Table 1.
characteristic | No Thyroid Disease (n=21,327; 96%) |
Hyperthyroid (n=175; 1%) |
Hypothyroid (n=461; 2%) |
Other Thyroid (n=283; 1%) |
||||
---|---|---|---|---|---|---|---|---|
No. | % | No. | % | No. | % | No. | % | |
State | ||||||||
North Carolina | 5,841 | 27.4 | 66 | 37.7 | 124 | 26.9 | 83 | 29.3 |
Iowa | 15,486 | 72.6 | 109 | 62.3 | 337 | 73.1 | 200 | 70.7 |
Age, years | ||||||||
≤ 38 | 6,613 | 31.0 | 25 | 14.3 | 62 | 13.5 | 41 | 14.5 |
39-46 | 5,618 | 26.3 | 46 | 26.3 | 109 | 23.6 | 59 | 20.8 |
47-57 | 5,253 | 24.6 | 59 | 33.7 | 137 | 29.7 | 80 | 28.3 |
≥ 58 | 3,843 | 18.0 | 45 | 25.7 | 153 | 33.2 | 103 | 36.4 |
Body mass index, kg/m2 | ||||||||
< 25 | 5,414 | 25.4 | 39 | 22.3 | 86 | 18.7 | 61 | 21.6 |
25-29.9 | 10,917 | 51.2 | 94 | 53.7 | 250 | 54.2 | 154 | 54.4 |
≥ 30 | 4,996 | 23.4 | 42 | 24.0 | 125 | 27.1 | 68 | 24.0 |
Racea | ||||||||
White | 21,018 | 98.6 | 171 | 97.7 | 459 | 99.6 | 281 | 99.3 |
Nonwhite | 289 | 1.4 | 4 | 2.3 | 2 | 0.4 | 2 | 0.7 |
Education | ||||||||
≤ High School | 11,169 | 52.4 | 85 | 48.6 | 228 | 49.5 | 172 | 60.8 |
> High School | 10,158 | 47.6 | 90 | 51.4 | 233 | 50.5 | 111 | 39.2 |
Smoking Status | ||||||||
Never | 12,318 | 57.8 | 90 | 51.4 | 249 | 54.0 | 156 | 55.1 |
Past | 2,863 | 13.4 | 22 | 12.6 | 45 | 9.8 | 21 | 7.4 |
Current | 6,146 | 28.8 | 63 | 36.0 | 167 | 36.2 | 106 | 37.5 |
Worked/Lived on Farm, yearsb | ||||||||
≤ 30 | 2,410 | 22.7 | 19 | 19.2 | 45 | 17.7 | 22 | 14.5 |
> 30 | 8,220 | 77.3 | 80 | 80.8 | 210 | 82.3 | 130 | 85.5 |
Missing for 21 subjects without thyroid disease.
Missing for 10,697 subjects without thyroid disease, 76 subjects with hyperthyroidism, 206 with hypothyroidism, and 131 subjects with ‘other’ thyroid disease.
Hypothyroid Disease
Similar results were seen using polytomous and logistic regression strategies for hypothyroid disease (polytomous regression data not shown). The fungicide metalaxyl was associated with decreased odds of hypothyroidism (Table 2). None of the fumigants was associated with thyroid disease (Table 2).
Table 2.
No thyroid disease n =21,327; 96% Ever-Usec (%) |
Hypothyroid n = 461; 2% Ever-Use (%) |
OR (95%) | |
---|---|---|---|
Fumigants | |||
Brom-O-Gas | 2,752 (12.9) | 51 (11.1) | 0.81(0.6,1.09) |
Aluminum phosphide | 1,044 (4.9) | 21 (4.6) | 0.92(0.59,1.44) |
Carbon tetrachloride/Carbon Disulfide |
1,089 (5.1) | 37 (8.0) | 1.19(0.84,1.68) |
Ethylene dibromide | 680 (3.2) | 17 (3.7) | 1.02(0.62,1.67) |
Fungicides | |||
Benomyl | 1,848 (8.7) | 49 (10.6) | 1.13(0.83,1.52) |
Chlorothalonil | 1,388 (6.5) | 27 (5.9) | 0.89(0.6,1.32) |
Captan | 2,516(11.8) | 68(14.8) | 1.27(0.98,1.66) |
Maneb/Mancozeb | 1,789(8.4) | 39 (8.5) | 0.9(0.64,1.25) |
Metalaxyl | 4,380 (20.5) | 66 (14.3) | 0.66(0.5,0.85)* |
Ziram | 309 (1.4) | b | 0.57(0.21,1.53) |
Abbreviations: CI, confidence interval; OR, odds ratio.
Odds ratios were calculated using logistic regression and were adjusted for BMI (<25, 25-29.9, >=30), age at enrollment (≤ 38, 39-46, 47-57, ≥ 58 years), and education (≤ high school, > high school). oRs are relative to 1.0 for the reference category of unexposed.
Cell with count <5.
Ever-Use at time of enrollment.
The herbicides 2,4-D, 2,4,5-T, 2,4,5-TP, alachlor, dicamba, and petroleum oil were all associated with an increased odds of hypothyroidism (Table 3). Eight insecticides were also associated with increased odds of hypothyroidism: the organochlorines chlordane, DDT, heptachlor, lindane, and toxaphene; the organophosphates diazinon and malathion; and the carbamate carbofuran (Table 4).
Table 3.
Herbicide | No Thyroid Disease n =21,327; 96 % Ever-useb (%) |
Hypothyroid n = 461; 2 % Ever-use (%) |
OR (95%) |
---|---|---|---|
2 4-D | 16,809(78.8) | 392(85) | 1.35(1.04,1.76)* |
2,4,5-T | 4,498(21.1) | 153(33.2) | 1.38(1.12,1.69)* |
2,4,5-T P | 1,890(8.9) | 67(14.5) | 1.39(1.06,1.82)* |
Alachlor | 11,641(54.5) | 284(61.6) | 1.24(1.02,1.5)* |
Atrazine | 15,920(74.6) | 347(75.3) | 0.99(0.8,1.22) |
Butylate | 7,190(33.7) | 160(34.7) | 0.98(0.81,1.19) |
Chlorimuron-ethyl | 8,264(38.7) | 151(32.8) | 0.85(0.7,1.04) |
Cyanazine | 9,450(44.3) | 222(48.2) | 1.13(0.94,1.36) |
Dicamba | 11,685(54.8) | 289(62.7) | 1.37(1.13,1.66)* |
EPTC | 4,520(21.2) | 110(23.9) | 1.19(0.96,1.48) |
Glyphosate | 16,320(76.5) | 366(79.4) | 1.18(0.94,1.49) |
Imazethapyr | 9,955(46.7) | 200(43.4) | 0.95(0.79,1.15) |
Metolachlor | 10,315(48) | 234(50.8) | 1.14(0.95,1.38) |
Metribuzin | 10,359(48.6) | 236(51.2) | 1.07(0.89,1.29) |
Paraquat | 4,652(21.8) | 114(24.7) | 1.11(0.9,1.38) |
Pendimethalin | 9,524(44.7) | 183(39.7) | 0.86(0.71,1.04) |
Petroleum Oil | 10,595(49.7) | 257(55.7) | 1.23(1.02,1.48)* |
Trifluralin | 11,727(55) | 269(58.4) | 1.11(0.92,1.34) |
Abbreviations: CI, confidence interval; OR, odds ratio; EPTC, s-ethyl dipropylcarbamothioate; 2,4-D,2,4-dichlorophenoxyacetic acid; 2,4,5-T, 2,4,5-trichlorophenoxyacetic acid; 2,4,5-TP, 2,4,5-trichlorophenoxy-propionic acid.
Odds ratios were calculated using logistic regression and were adjusted for BMI (<25, 25-29.9, >=30), age at enrollment (≤ 38, 39-46, 47-57, ≥58 years), and education (≤ high school, > high school). Relative to 1.0 for the reference category of unexposed.
Ever-use at time of enrollment.
p<0.05
Table 4.
Insecticide | No thyroid disease n =21,327; 96% Ever-useb (%) |
Hypothyroid n = 461; 2% Ever-use (%) |
OR (95%) |
---|---|---|---|
Organochlorines | |||
Aldrin | 4,042(19.0) | 128(27.8) | 1.09(0.88,1.36) |
Chlordane | 5,351(25.1) | 176(38.2) | 1.36(1.12,1.66)* |
DDT | 4,863(22.8) | 172(37.3) | 1.25(1,1.56)* |
Dieldrin | 1,470(6.9) | 49(10.6) | 1.04(0.76,1.42) |
Heptachlor | 3,470(16.3) | 123(26.7) | 1.3(1.04,1.62)* |
Lindane | 4,387(20.6) | 133(28.9) | 1.35(1.1,1.66)* |
Toxaphene | 2,903(13.6) | 102(22.1) | 1.35(1.07,1.7)* |
Organophosphates | |||
Chlorpyrifos | 9,290(43.6) | 209(45.3) | 1.12(0.93,1.34) |
Coumaphos | 1,861(8.7) | 52(11.3) | 1.24(0.93,1.67) |
Diazinon | 6,636(31.1) | 179(38.8) | 1.24(1.02,1.5)* |
Dichlorvos | 2,414(11.3) | 69(15.0) | 1.26(0.97,1.64) |
Fonofos | 4,899(23.0) | 117(25.4) | 1.08(0.87,1.33) |
Malathion | 15,261(71.6) | 362(78.5) | 1.29(1.03,1.62)* |
Parathion | 2,996(14.0) | 86(18.7) | 1.2(0.94,1.52) |
Phorate | 7,285(34.2) | 174(37.7) | 1.05(0.87,1.27) |
Terbufos | 8,733(40.9) | 188(40.8) | 1.01(0.83,1.22) |
Trichlorfon | 121(0.6) | 6(1.3) | 2.19(0.95,5.03) |
Carbamates | |||
Aldicarb | 2,052(9.6) | 42(9.1) | 0.99(0.72,1.37) |
Carbaryl | 11,652(54.6) | 277(60.1) | 1.08(0.89,1.31) |
Carbofuran | 5,941(27.9) | 170(36.9) | 1.31(1.08,1.59)* |
Other | |||
Permethrin (for crops) | 2,895(13.6) | 68(14.8) | 1.19(0.92,1.55) |
Permethrin (for animals) | 3,161(14.8) | 70(15.2) | 1.14(0.88,1.48) |
Abbreviations: CI, confidence interval; OR, odds ratio.
Odds ratios were calculated using logistic regression and were adjusted for BMI (<25, 25-29.9, >=30), age at enrollment (≤ 38, 39-46, 47-57, ≥58 years), and education (≤ high school, > high school). ORs relative to 1.0 for the reference category of unexposed.
Ever-use at time of enrollment
p<0.05
Results from exposure-response analyses using the intensity-weighted exposure measure are shown in Table 5. Only pesticides with increasing trends are shown. A statistically significant monotonic exposure-response trend was seen for the herbicides 2,4-D and alachlor, the organochlorine insecticides aldrin, chlordane, DDT and lindane, and the organophosphate parathion (Table 5). There were some pesticides with borderline statistical significance in trend including the herbicides cyanazine and EPTC (Table 5). Additionally, some pesticides showed an association at low exposure, but not high exposure, including the organochlorine insecticide toxaphene, the organophosphate insecticide diazinon, and the carbamate insecticide carbofuran (Table 5).
Table 5.
Pesticide | Dose | No Thyroid Diseaseb n =21,883; 96% No. (%) |
Hypothyroidb n = 473; 2% No. (%) |
OR (95%) | P-value trend |
---|---|---|---|---|---|
Herbicides | |||||
2, 4-D | ref | 4,514 (21.7) | 69 (15.2) | 1.0 | 0.025 |
low | 8,119(39.0) | 177(39.1) | 1.34(1.01,1.78) | ||
high | 8,208(39.4) | 207(45.7) | 1.4(1.06,1.85) | ||
Alachlor | ref | 9,619 (46.1) | 176 (39.0) | 1.0 | 0.018 |
low | 5,626(27.0) | 131(28.9) | 1.19(0.95,1.5) | ||
high | 5,614(26.9) | 144(32.0) | 1.3(1.04,1.62) | ||
Cyanazine | ref | 11,815 (56.2) | 238 (52.2) | 1.0 | 0.10 |
low | 4,605(21.9) | 104(22.8) | 1.08(0.85,1.36) | ||
high | 4,599(21.9) | 114(25.0) | 1.2(0.95,1.5) | ||
EPTC | ref | 16,738 (79.2) | 350 (76.6) | 1.0 | 0.112 |
low | 2,184(10.3) | 53(11.6) | 1.18(0.88,1.58) | ||
high | 2,200(10.4) | 54(11.8) | 1.22(0.91,1.63) | ||
Insecticides
Organochlorines |
|||||
Aldrin | ref | 7,966 (81.5) | 163 (66.5) | 1.0 | 0.020 |
low | 890(9.1) | 38(15.5) | 1.35(0.93,1.97) | ||
high | 914(9.4) | 44(18.0) | 1.43(1,2.05) | ||
Chlordane | ref | 7,308 (79.2) | 143 (65.3) | 1.0 | 0.021 |
low | 950(10.3) | 33(15.1) | 1.33(0.9,1.97) | ||
high | 968(10.5) | 43(19.6) | 1.53(1.07,2.2) | ||
DDT | ref | 7,505 (78.5) | 134 (58.0) | 1.0 | 0.021 |
low | 1,025(10.7) | 48(20.8) | 1.53(1.05,2.23) | ||
high | 1,025(10.7) | 49(21.2) | 1.53(1.05,2.23) | ||
Lindane | ref | 7,717 (83.3) | 158 (74.9) | 1.0 | 0.029 |
low | 773(8.3) | 26(12.3) | 1.41(0.92,2.16) | ||
high | 775(8.4) | 27 (12.8) | 1.45(0.96,2.21) | ||
Toxaphene | ref | 8,856 (88.6) | 186 (80.2) | 1.0 | 0.072 |
low | 565(5.7) | 25(10.8) | 1.63(1.06,2.51) | ||
high | 570(5.7) | 21(9.1) | 1.3(0.81,2.07) | ||
Organophosphates | |||||
Diazinon | ref | 6,747 (77.0) | 138 (67.6) | 1.0 | 0.183 |
low | 1,004(11.5) | 37(18.1) | 1.59(1.09,2.31) | ||
high | 1,015(11.6) | 29(14.2) | 1.18(0.78,1.78) | ||
Malathion | ref | 2,241 (24.8) | 35 (15.6) | 1.0 | 0.083 |
low | 3,377(37.3) | 92(40.9) | 1.63(1.09,2.42) | ||
high | 3,430(37.9) | 98(43.6) | 1.54(1.04,2.28) | ||
Parathion | ref | 8,826 (92.3) | 192 (86.1) | 1.0 | 0.007 |
low | 367(3.8) | 13(5.8) | 1.38(0.78,2.46) | ||
high | 368(3.8) | 18(8.1) | 1.9(1.15,3.13) | ||
Carbamates | |||||
Carbofuran | ref | 15,322 (72.7) | 290 (64.6) | 1.0 | 0.072 |
low | 2,868(13.6) | 86(19.2 | 1.37(1.07,1.75) | ||
high | 2,889(13.7) | 73(16.3) | 1.16(0.89,1.51) |
Abbreviations: CI, confidence interval; OR, odds ratio; 2,4-D,2,4-dichlorophenoxyacetic acid; EPTC, s-ethyl dipropylcarbamothioate.
Odds ratios were calculated using logistic regression and were adjusted for BMI (<25, 25-29.9, >=30), age at enrollment (≤ 38, 39-46, 47-57, ≥58 years), and education (≤ high school, > high school). The intensity-adjusted cumulative day use was classified into three levels: no exposure (zero day use), low exposure (no larger than the median day use of the applicants with a non-zero day use, and high exposure (larger than the median day use of the applicants with a non-zero day use).
Total number of subjects available. Sample sizes varied across analyses due to missing data.
We considered the influence of correlated pesticides for those statistically significantly associated with hypothyroid disease (metalaxyl, 2,4-D, 2,4,5-T, 2,4,5-TP, alachlor, dicamba, petroleum oil, chlordane, DDT, heptachlor, lindane, toxaphene, diazinon, malathion, and carbofuran). Eleven pairs of pesticides were moderately correlated with correlation coefficients > 0.3 (Table 6). When each pair of moderately correlated pesticides was simultaneously added to the base model, associations between dicamba, 2,4,5-T, chlordane, and toxaphene and hypothyroid disease remained statistically significant and associations with the other pesticides and hypothyroid disease were reduced < 10% (Table 6).
Table 6.
Correlated Pair |
Pesticide | Correlation Coefficient |
Hypothyroid OR (95%) |
---|---|---|---|
1 | Dicamba 2,4- D |
0.32 | 1.31(1.07, 1.60)* 1.22(0.93, 1.60) |
2 | 2,4,5- TP 2,4,5 -T |
0.50 | 1.17(0.86, 1.59) 1.30(1.03, 1.65)* |
3 | 2,4,5- T Chlordane |
0.31 | 1.29(1.04, 1.59)* 1.28(1.04, 1.57)* |
4 | 2,4,5 -T DDT |
0.35 | 1.33(1.07, 1.64)* 1.15(0.91, 1.45) |
5 | 2,4,5 -T Heptachlor |
0.33 | 1.31(1.06, 1.63)* 1.19(0.94, 1.5) |
6 | Diazinon Chlordane |
0.34 | 1.14(0.93, 1.40) 1.30(1.05, 1.61)* |
7 | Chlordane DDT |
0.39 | 1.31(1.06, 1.62)* 1.14(0.90, 1.44) |
8 | Chlordane Heptachlor |
0.32 | 1.30(1.06, 1.60)* 1.20(0.95, 1.51) |
9 | Chlordane Toxaphene |
0.36 | 1.29(1.04, 1.59)* 1.22(0.96, 1.56) |
10 | DDT Heptachlor |
0.37 | 1.18(0.94, 1.49) 1.24(0.98, 1.56) |
11 | DDT Toxaphene |
0.35 | 1.17(0.93, 1.47) 1.28(1.01, 1.63)* |
Abbreviations: CI, confidence interval; OR, odds ratio; 2,4-D,2,4-dichlorophenoxyacetic acid; 2,4,5-T, 2,4,5-trichlorophenoxyacetic acid; 2,4,5-TP, 2,4,5-trichlorophenoxy-propionic acid.
Odds ratios were calculated using logistic regression and were adjusted for BMI (<25, 25-29.9, >=30), age at enrollment (≤ 38, 39-46, 47-57, ≥58 years), and education (≤ high school, > high school). Relative to 1.0 for the reference category of unexposed to both pesticides.
p<0.05
Hyperthyroid Disease
None of the fungicides and fumigants was associated with hyperthyroid disease (data not shown). However, the herbicides 2,4,5-TP (OR = 0.46, 95% CI = 0.23, 0.90) and butylate (OR = 0.69, 95% CI = 0.49, 0.98) as well as the insecticide carbofuran (OR = 0.69, 95% CI = 0.49. 0.98) were associated with a decreased odds of hyperthyroidism.
Other Thyroid Disease
None of the fungicides or fumigants was associated with “other” thyroid disease (data not shown). Although the herbicide EPTC (OR =1.37; 95% CI =1.03,1.82) and the insecticides fonofos (OR =1.31; 95% CI =1.00,1.73), terbufos (OR =1.48; 95% CI =1.16,1.89), and carbofuran (OR =1.51; 95% CI =1.19,1.93) were associated with an increased odds of “other” thyroid disease, we did not evaluate this further because “other thyroid disease” is such a heterogeneous group.
Discussion
These results for male private pesticide applicators from the AHS further support the hypothesis that pesticides play a role in the etiology of thyroid disease. To our knowledge, no studies have reported an increased risk of thyroid disease specifically in men exposed to herbicides and insecticides, likely due to the low prevalence of thyroid disease in men. In this study, we found the prevalence of thyroid disease in male private applicators to be 4%, with 2% having hypothyroidism. The Colorado Thyroid Disease Prevalence Study reported 3-16% of men with age-dependent thyroid dysfunction, with a prevalence of 5% for men between the ages of 45-54(1), similar to the average age of 45.6 years for the men in our study. A Brazilian study also reported similar results with a prevalence of hypothyroidism in men of 5.4%(19). However, we relied on self-report of thyroid disease to obtain the prevalence, which may result in misclassification of thyroid disease and underestimate the true prevalence in the AHS.
Pesticides classified as organochlorine insecticides and chlorinated herbicides were most consistently associated with hypothyroid disease in this cohort. Five organochlorine insecticides were independently associated with increased odds of hypothyroid disease in male pesticide applicators, chlordane, DDT, heptachlor, lindane and toxaphene. Only chlordane, lindane and toxaphene were independently associated as they remained significant in models including pairs of correlated pesticides. Previously we found an association between ever-use of organochlorine insecticides as a group and hypothyroidism in female spouses in the AHS, but only a suggestive association specifically for chlordane and hypothyroidism(14). Possible reasons for this observation include i) a larger percentage of men were exposed to chlordane than women (38.2%,176 men versus 5.2%, 58 women), ii) male applicators may have been exposed to higher levels or organochlorines than female spouses, and/or iii) male applicators may have an organochlorine exposure of longer duration.
Organochlorine insecticides are lipophilic and are environmentally and biologically persistent. Even though many have been banned from use in the United States for decades, many persons are still found to have detectable levels of these compounds or their metabolites. Organochlorine exposure and abnormal thyroid function have been reported in both animals and humans (20) (21) (22). There have also been human observational studies of reduced circulating free and total T4 levels with exposures to specific organochlorines or other halogenated compounds including polychlorinated biphenyl (PCB) congeners, hexachlorobenzene, DDE, DDT, and derivatives and metabolites of these compounds(11, 13, 23, 24).
The biological mechanism(s) underlying the changes in thyroid hormones and histologic changes in the thyroid are not completely understood. However, several mechanisms have been suggested including interference with thyroid hormone binding proteins TBG and transthyretin and induction of hepatic enzyme uridine diphosphate glucoronyltransferase resulting in increased clearance of thyroid hormone. There are conflicting results about effects of pesticides and other halogenated compounds on the thyroid hormone receptor(7). Rats exposed to the breakdown product of the organochlorine DDT, p,p -DDE, had significantly reduced free T4, reduced transthyretin, and elevated concentrations of the thyroid hormone receptor in the hypothalamus. Additionally, concentrations of the hepatic enzymes CYP1A1, CYP2B1, and uridine diphosphate glucuronyltransferase were also increased, suggesting DDE results in hepatic enzyme induction(25). Exposure to environmental pollutants may also contribute to a rise in thyroid autoimmunity. Increased thyroid volume and hypoechogenicity on thyroid ultrasound, higher levels of thyroid stimulating hormone (TSH), and an increased prevalence of thyroid peroxidase antibodies have been noted in persons exposed to a mixture of the organochlorines polychlorinated biphenyls (PCBs), 2,2′-bis(4-chlorophenyl)-1,1-dichloroethylene (p,p’-DDE), hexachlorobenzene (HCB), and dioxins and furans(10). However, it is important to note that not all organochlorine insecticides have been reported to be thyroid hormone disruptors, suggesting that this may not be a class effect. In this study, only 8 of the 22 chlorinated pesticides and 5 of the 7 organochlorine insecticides evaluated were associated with hypothyroidism. There may be effects specific to a certain compound or synergism when a combination of compounds are used that are not seen when an individual chemical is used alone. There have also been human and animal studies of chronic organophosphate and organochlorine insecticide exposure have also shown both an increase in TSH and T4, and a decrease in T3 and T4 (26, 27).
Dicamba, 2,4-D, 2,4,5-T and 2,4,5-TP are all phenoxy herbicides. Additionally, both 2,4-D and 2,4,5-T are components of Agent Orange, and it is known that 2,4,5-T was contaminated by the dioxin 2,3,7,8,-tetrachlorodibenzo-p-dioxin (TCDD)(28). In addition, 2,4,5-TP has been reported to be frequently contaminated with TCDD. TCDD exposure was associated with increased TSH levels in Air force veterans involved in spraying Agent Orange from 1962-1971(29). It is possible that the observed association between 2,4,5-T and 2,4,5-TP and hypothyroidism in our study is an effect of dioxin exposure. There is less data for 2,4-D and hypothyroidism, However, in rat studies, 2,4-D has been reported to be an endocrine disruptor with effects on the pituitary resulting in decreased levels of T3 and T4 and hypothyroidism (30).
We evaluated exposure-response relationships with hypothyroidism for the insecticides and herbicides considered in this study, even those for which ever-use was not associated with hypothyroidism, because we did not want to exclude a pesticide that may be associated with hypothyroidism only with high exposure. Not all of the pesticides noted to be significant in the ever-use analysis showed evidence of an exposure-response relationship. Alachlor and 2,4-D were the only herbicides that had a significant exposure-response trend. The organochlorine insecticides chlordane, DDT and lindane also showed an exposure response as did aldrin and parathion, although none of these was associated with hypothyroidism in the ever-use analysis. Further research is necessary to understand these trends and it will also be important to explore this non-monotonic relationship in other thyroid diseases specifically hyperthyroidism in future studies.
Strengths of this study include the large sample size and detailed information on pesticide use that allowed us to explore exposure-response relationships and account for correlated pesticide use. Limitations include the potential for recall bias affecting exposure estimates, reliance on self-reported disease, and possible selection bias due to high drop-out rates. We feel that recall bias is unlikely for three reasons. First, exposure was ascertained years before outcome information was ascertained. Second, individuals were asked about many pesticides at enrollment and many outcomes at follow-up. If recall bias were present, it is unlikely that we would have findings that were specific to individual chemicals, or outcomes. Finally, there is no strong literature regarding pesticides and thyroid disease, so it is unlikely that participants would have been aware of suspected pesticide-thyroid disease associations. There is also the potential for misclassification of thyroid disease subtypes by patients. However, we do not feel this is a major concern, given only 19 people reported a different thyroid disease state in the two follow up questionnaires (hypothyroidism and hyperthyroidism); the remainder were consistent in their responses. Because of the way the data was obtained, we were unable to evaluate incident disease alone. It is possible that by evaluating prevalent thyroid disease that we may underestimate associations with pesticides, if we included cases that occurred prior to pesticide use. Lastly, there is the possibility of selection bias due to a large number of applicators who did not complete all study questionnaires. Previous studies with the AHS have also shown few differences between responders and non-responders for the take-home (17) and the Phase II questionnaires (31). There were only slight differences between those who completed all questionnaires and those who only completed the enrollment questionnaire. Those who completed all questionnaires were slightly older (mean age = 46 vs. 44 years), and more likely to have education past high school (47% vs. 41%) compared with those who only completed the enrollment questionnaire. Most importantly, there was no difference in their medical history at enrollment.
Not all pesticides in the same functional or chemical class have a similar biological effect, so using summary variables for all pesticides in a class can dilute the effects of specific chemicals. A strength of the AHS is the ability to evaluate specific chemicals, so we have not explored associations using summary class variables. We have insufficient power to address chemical interactions for individual pesticides. More importantly, we are unable to evaluate whether multiple pesticides were applied at the same time or applied separately at different times. Additionally, confounding by more than one pesticide and potential interactions between co-applied pesticides need to be considered. Many applicators apply more than one pesticide at a time and we are not able to separate out which pesticides were coincidentally applied versus those that were applied alone or at different times. Given that we evaluated 50 different pesticides with 3 different thyroid outcomes, a chance association is always a possibility. However, given the evidence of exposure-response for some chemicals, as well as consistency with other studies, we feel that chance alone does not explain our findings.
Interestingly, when we evaluated DDT in the group > 58 years of age, those who would have had an opportunity to apply DDT before it was banned, our results were attenuated. Although the odd ratio was > 1 (OR= 1.1), it was no longer statistically significant (95% CI= 0.9, 1.6). It is possible that the apparent association in the main analysis was due to the small number of cases or uncontrolled confounding by age. It is also important to consider that because DDT and its metabolites are persistent pesticides, non-applicators of DDT could have also been exposed resulting biasing results toward the null.
Conclusion
We found statistically significant associations between specific herbicides and insecticides and hypothyroid disease in male pesticide applicators. These associations were consistent with previous findings from their spouses showing an association between ever use of organochlorine insecticides and hypothyroidism, although the present analysis identified a greater number of specific chemicals associated with hypothyroid disease. Our results support other evidence on the potential endocrine disrupting role of specific organochlorine insecticides and other pesticides and require further investigation to determine their mechanism of action.
Acknowledgements
The authors thank Stuart Long for preparing the dataset for this analysis. We used AHS datasets with the following data release numbers for this study: Phase 1: p1rel0906.00, Phase 2: p2rel0907.00, Phase 3: p3rel0901.00, General Data Files: rel0905.00.
This work was supported in part by the Intramural Research Program of the NIH, National Institute of Environmental Health Sciences (Z01-ES049030), and National Cancer Institute (Z01-CP-1-119).
List of Abbreviations
- (AHS)
Agricultural Health Study
- (T3)
triiodothyronine
- (T4)
thyroxine
- (TSH)
thyroid stimulating hormone
Footnotes
Conflicts of Interest and Source of Funding:
None of the authors have any conflicts of interest to disclose.
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Canaris G, Manowitz N, Mayor G, Ridgway E. The Colorado thyroid disease prevalence study. Arch Intern Med. 2000;160(4):526–34. doi: 10.1001/archinte.160.4.526. [DOI] [PubMed] [Google Scholar]
- 2.Hollowell J, Staehling N, Flanders W, Hannon W, Gunter E, Spencer C, Braverman L. Serum TSH, T(4), and thyroid antibodies in the United States population (1988 to 1994): National Health and Nutrition Examination Survey (NHANES III) J Clin Endocrinol Metab. 2002;87(2):489–99. doi: 10.1210/jcem.87.2.8182. [DOI] [PubMed] [Google Scholar]
- 3.Vanderpump M, Tunbridge W. The epidemiology of thyroid disease. In: Braverman L, Utiger R, editors. The Thyroid. 9th Edition Lippincott-Raven; Philadelphia: 1996. pp. 474–82. [Google Scholar]
- 4.Strieder TG, Prummel MF, Tijssen JG, Endert E, Wiersinga WM. Risk factors for and prevalence of thyroid disorders in a cross-sectional study among healthy female relatives of patients with autoimmune thyroid disease. Clin Endocrinol (Oxf) 2003 Sep;59(3):396–401. doi: 10.1046/j.1365-2265.2003.01862.x. [DOI] [PubMed] [Google Scholar]
- 5.Patrick L. Thyroid disruption: mechanism and clinical implications in human health. Altern Med Rev. 2009 Dec;14(4):326–46. [PubMed] [Google Scholar]
- 6.Langer P. The impacts of organochlorines and other persistent pollutants on thyroid and metabolic health. Front Neuroendocrinol. 2010 Oct;31(4):497–518. doi: 10.1016/j.yfrne.2010.08.001. [DOI] [PubMed] [Google Scholar]
- 7.Diamanti-Kandarakis E, Bourguignon JP, Giudice LC, Hauser R, Prins GS, Soto AM, Zoeller RT, Gore AC. Endocrine-disrupting chemicals: an Endocrine Society scientific statement. Endocr Rev. 2009 Jun;30(4):293–342. doi: 10.1210/er.2009-0002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Brucker-Davis F. Effects of environmental synthetic chemicals on thyroid function. Thyroid. 1998;8(9):827–56. doi: 10.1089/thy.1998.8.827. [DOI] [PubMed] [Google Scholar]
- 9.Toft G, Flyvbierg A, Bonde J. Thyroid function in Danish greenhouse workers. Environ Health. 2006;6(5):32. doi: 10.1186/1476-069X-5-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Radikova Z, Tajtakova M, Kocan A, Trnovec T, Sebokova E, Klimes I, Langer P. Possible effects of environmental nitrates and toxic organochlorines on human thyroid in highly polluted areas in Slovakia. Thyroid. 2008 Mar;18(3):353–62. doi: 10.1089/thy.2007.0182. [DOI] [PubMed] [Google Scholar]
- 11.Zoeller R. Environmental chemicals impacting the thyroid: targets and consequences. Thyroid. 2007;17(9):811–7. doi: 10.1089/thy.2007.0107. [DOI] [PubMed] [Google Scholar]
- 12.Jugan ML, Levi Y, Blondeau JP. Endocrine disruptors and thyroid hormone physiology. Biochem Pharmacol. 2010 Apr 1;79(7):939–47. doi: 10.1016/j.bcp.2009.11.006. [DOI] [PubMed] [Google Scholar]
- 13.Boas M, Feldt-Rasmussen U, Skakkebaek N, Main K. Environmental chemicals and thyroid function. Eur J Endocrinol. 2006;154(5):599–611. doi: 10.1530/eje.1.02128. [DOI] [PubMed] [Google Scholar]
- 14.Goldner WS, Sandler DP, Yu F, Hoppin JA, Kamel F, Levan TD. Pesticide use and thyroid disease among women in the Agricultural Health Study. Am J Epidemiol. 2010 Feb 15;171(4):455–64. doi: 10.1093/aje/kwp404. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kirrane EF, Hoppin JA, Umbach DM, Samanic C, Sandler DP. Patterns of pesticide use and their determinants among wives of farmer pesticide applicators in the Agricultural Health Study. J Occup Environ Med. 2004 Aug;46(8):856–65. doi: 10.1097/01.jom.0000135521.15169.3e. [DOI] [PubMed] [Google Scholar]
- 16.Alavania M, Sandler D, McMaster S, Zahm S, McDonnell C, Lynch C, Pennybacker M, Rothman N, Dosemeci M, Bond A, Blair A. The Agricultural Health Study. Environ Health Perspect. 1996;104(4):362–9. doi: 10.1289/ehp.96104362. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Tarone RE, Alavanja MC, Zahm SH, Lubin JH, Sandler DP, McMaster SB, Rothman N, Blair A. The Agricultural Health Study: factors affecting completion and return of self-administered questionnaires in a large prospective cohort study of pesticide applicators. Am J Ind Med. 1997 Feb;31(2):233–42. doi: 10.1002/(sici)1097-0274(199702)31:2<233::aid-ajim13>3.0.co;2-2. [DOI] [PubMed] [Google Scholar]
- 18.Dosemeci M, Alavanja MC, Rowland AS, Mage D, Zahm SH, Rothman N, Lubin JH, Hoppin JA, Sandler DP, Blair A. A quantitative approach for estimating exposure to pesticides in the Agricultural Health Study. Ann Occup Hyg. 2002 Mar;46(2):245–60. doi: 10.1093/annhyg/mef011. [DOI] [PubMed] [Google Scholar]
- 19.Bensenor IM, Goulart AC, Lotufo PA, Menezes PR, Scazufca M. Prevalence of thyroid disorders among older people: results from the Sao Paulo Ageing & Health Study. Cad Saude Publica. Jan;27(1):155–61. doi: 10.1590/s0102-311x2011000100016. [DOI] [PubMed] [Google Scholar]
- 20.Wade MG, Parent S, Finnson KW, Foster W, Younglai E, McMahon A, Cyr DG, Hughes C. Thyroid toxicity due to subchronic exposure to a complex mixture of 16 organochlorines, lead, and cadmium. Toxicol Sci. 2002 Jun;67(2):207, 18. doi: 10.1093/toxsci/67.2.207. [DOI] [PubMed] [Google Scholar]
- 21.Kirkegaard M, Sonne C, Dietz R, Letcher RJ, Jensen AL, Hansen SS, Jenssen BM, Grandjean P. Alterations in thyroid hormone status in Greenland sledge dogs exposed to whale blubber contaminated with organohalogen compounds. Ecotoxicol Environ Saf. 2011 Jan;74(1):157–63. doi: 10.1016/j.ecoenv.2010.08.040. [DOI] [PubMed] [Google Scholar]
- 22.Das K, Vossen A, Tolley K, Vikingsson G, Thron K, Muller G, Baumgartner W, Siebert U. Interfollicular fibrosis in the thyroid of the harbour porpoise: an endocrine disruption? Arch Environ Contam Toxicol. 2006 Nov;51(4):720–9. doi: 10.1007/s00244-005-0098-4. [DOI] [PubMed] [Google Scholar]
- 23.Chevrier J, Eskenazi B, Holland N, Bradman A, Barr DB. Effects of exposure to polychlorinated biphenyls and organochlorine pesticides on thyroid function during pregnancy. Am J Epidemiol. 2008 Aug 1;168(3):298–310. doi: 10.1093/aje/kwn136. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Koopman-Esseboom C, Weisglas-Kuperus N, de Ridder MA, Van der Paauw CG, Tuinstra LG, Sauer PJ. Effects of polychlorinated biphenyl/dioxin exposure and feeding type on infants’ mental and psychomotor development. Pediatrics. 1996 May;97(5):700–6. [PubMed] [Google Scholar]
- 25.Liu C, Shi Y, Li H, Wang Y, Yang K. p,p’-DDE Disturbs the Homeostasis of Thyroid Hormones via Thyroid Hormone Receptors, Transthyretin, and Hepatic Enzymes. Horm Metab Res. 2011 Jun;43(6):391–6. doi: 10.1055/s-0031-1277135. [DOI] [PubMed] [Google Scholar]
- 26.Lacasana M, Lopez-Flores I, Rodriguez-Barranco M, Aguilar-Garduno C, Blanco-Munoz J, Perez-Mendez O, Gamboa R, Bassol S, Cebrian ME. Association between organophosphate pesticides exposure and thyroid hormones in floriculture workers. Toxicol Appl Pharmacol. 2010 Feb 15;243(1):19–26. doi: 10.1016/j.taap.2009.11.008. [DOI] [PubMed] [Google Scholar]
- 27.Rawlings NC, Cook SJ, Waldbillig D. Effects of the pesticides carbofuran, chlorpyrifos, dimethoate, lindane, triallate, trifluralin, 2,4-D, and pentachlorophenol on the metabolic endocrine and reproductive endocrine system in ewes. J Toxicol Environ Health A. 1998 May 8;54(1):21–36. doi: 10.1080/009841098159006. [DOI] [PubMed] [Google Scholar]
- 28.Schecter A, Quynh HT, Pavuk M, Papke O, Malisch R, Constable JD. Food as a source of dioxin exposure in the residents of Bien Hoa City, Vietnam. J Occup Environ Med. 2003 Aug;45(8):781–8. doi: 10.1097/01.jom.0000085786.50835.71. [DOI] [PubMed] [Google Scholar]
- 29.Pavuk M, Schecter AJ, Akhtar FZ, Michalek JE. Serum 2,3,7,8-tetrachlorodibenzo-p-dioxin (TCDD) levels and thyroid function in Air Force veterans of the Vietnam War. Ann Epidemiol. 2003 May;13(5):335–43. doi: 10.1016/s1047-2797(02)00422-2. [DOI] [PubMed] [Google Scholar]
- 30.Stoker T, Kaydos E, Jeffay S, Cooper R. Effect of 2,4-D Exposure on Pubertal Development and Thyroid Function in the Male Wistar RatBiology of Reproduction. 2007;77:75. [Google Scholar]
- 31.Montgomery MP, Kamel F, Hoppin JA, Beane Freeman LE, Alavanja MC, Sandler DP. Effects of self-reported health conditions and pesticide exposures on probability of follow-up in a prospective cohort study. Am J Ind Med. May;53(5):486–96. doi: 10.1002/ajim.20789. [DOI] [PMC free article] [PubMed] [Google Scholar]