Testosterone Imposters: An Analysis of Popular Online Testosterone Boosting Supplements

Abstract

Introduction:

Testosterone boosting supplements (T-Boosters) are prominently featured on Amazon.com with numerous dedicated pages and claims that they “naturally” increase testosterone levels.

Aim:

To evaluate the highest rated and frequently reviewed T-Boosters on Amazon.com to facilitate patient counseling regarding marketing myths, T-Booster formulations, and evidence for efficacy and safety.

Methods:

The Amazon marketplace was queried using the keywords “testosterone” + “booster” with default search settings and ranking items based on relevance. The top 5 T-Boosters identified on July 22, 2018 were reviewed based on price, ratings, reviews, manufacturer details, and ingredients. Consumer reviews were categorized using core themes in the Androgen Deficiency in the Aging Male (ADAM) questionnaire as a proxy to understand T-Booster efficacy and reanalyzed following filtration of untrustworthy comments using ReviewMeta.com, a proprietary Amazon customer review analysis software.

Outcomes:

Quantitative and qualitative evaluation of T-Boosters on Amazon.com.

Results:

The top 5 T-Boosters had an average ± SD of 2,761 ± 5,112 reviews and a rating of 4.56 ± 0.25 stars. Nineteen unique ingredients were identified across these T-Boosters, and literature review revealed 191 studies involving the 10 most common ingredients, of which 19% involved human subjects, 53% animal models, 15% in vitro studies and 12% case reports or review articles. Amongst 37 human studies, 30% observed an increase in T levels, 3% a decrease, 46% no effect, and 22% were indeterminate. Analysis of top customer reviews from the first two pages of reviews for each supplement revealed differences in ADAM score before and after ReviewMeta.com filtration. After filtration, there was a 91% decrease in users reporting increased libido, a 59% decrease in reports of increased energy, a 93% decrease in reports of improved strength/endurance, a 60% decrease in reports of improved erections, an elimination of reports of improved work performance, a 67% decrease in reports of improved sleep, and an 89% decrease in reports of improved sports ability.

Strengths/Limitations:

Strengths include the novel approach to assess consumers’ perceptions and satisfaction of T-Boosters, and summary information that clinicians can provide patients. Limitations include selection bias, a small number of supplements analyzed and proprietary nature of Amazon review analysis software.

Conclusions:

T-Boosters are easily available online. Our investigation revealed that limited human studies have evaluated T-Boosters, resulting in no definitive findings of efficacy. In the absence of additional human studies, patients should be cautioned before considering T-Boosters, given the availability of highly effective Food and Drug Administration approved therapies.

Introduction

Hypogonadism is a clinical syndrome that is characterized by low serum testosterone (T) levels and the presence of symptoms including low libido, fatigue and sexual dysfunction, among others[1–3]. The incidence of hypogonadism increases with age and affects approximately 4-5 million men in the United States[4].

Testosterone therapy (TTh) is often used to treat symptomatic hypogonadal men. The use of TTh nearly tripled between 2001 and 2011, rendering therapies for hypogonadism one of the most rapidly growing pharmaceutical product categories[5]. However, following this period of initial growth, T prescriptions significantly declined between 2013 and 2016[6]. Several factors contributed to this decline, including a 2015 Food and Drug Administration (FDA) safety announcement attributing a potentially increased cardiovascular risk to TTh[7]. These concerns were compounded by fears that TTh increases prostate cancer risk. Socioeconomic forces also influenced the downward trend in TTh use. Le et al. noted that reimbursement and insurance coverage for TTh lack transparency and hinder patients from receiving appropriate care for hypogonadism[8]. The authors concluded that regulatory and structural barriers were in part forcing patients to actively employ online marketplaces to research and procure alternatives to prescription testosterone.

Recent work by Cui et al. demonstrated that patients are increasingly seeking herbal and nutrient based supplements to address hypogonadal symptoms[9,10]. These supplements are typically marketed as Testosterone Boosters (T-Boosters). T-boosters are not regulated by the FDA and lack a standardized composition, being composed of a myriad of ingredients. Despite this variability, prominent ingredients in many T-Boosters include horny goat weed, saw palmetto and fenugreek. The laxity of FDA supplement regulation is well established, and has enabled T-Boosters to proliferate without rigorous quality control, ingredient standardization or verification of touted benefits [10]. T-Booster marketing proclaims that these products can naturally improve T levels while ensuring lower cost and comparable efficacy to FDA-approved therapies such as testosterone. The dangers of these marketing myths are evidenced by a string of recent case reports highlighting adverse reactions following T-Booster use among impressionable populations, including adolescents and health enthusiasts[11–13].

The internet is increasingly becoming the default location for the acquisition of health and nutritional supplements[14,15]. This consumer trend parallels the average American’s growing preference for electronic commerce (e-commerce) over physical brick and mortar stores[14,16]. Amazon.com dominates the e-commerce market with over 49% market share[15]. Amazon continues to legitimize its aspirations to be a health marketplace as evidenced by the company’s recent acquisition of PillPack, an online pharmacy, as well as its establishment of a partnership with Berkshire Hathaway and JPMorgan Chase & Co to pursue healthcare-related activities. T-Boosters are prominently featured on Amazon with numerous dedicated pages and claims that they can “naturally” increase T. The presence of T-Boosters on the Amazon marketplace ensures high levels of visibility and accessibility to consumers[14].

Social commerce encourages consumers to comment, share and “promote” products via online reviews. As a result, customers buying products, including T-Boosters, often utilize user reviews to inform and validate product efficacy. Amazon product reviews have come under increasing scrutiny due to predatory habits by vendors, fake reviews generated by ‘collusive spammers’, and paid reviews[17–24]. The significance of these fake product reviews is heightened by the fact that patients place high levels of trust in health-related information displayed online[25].

Physicians should be aware of the most common T-Boosters, given their easy obtainability and high consumer interest. The present study evaluates the most highly rated and frequently reviewed T-Boosters on Amazon.com to facilitate patient counseling regarding marketing myths, T-Booster formulations and evidence for efficacy behind these supplements. The aims of this study were to answer the following clinical questions: 1) what is the underlying evidence for efficacy of T Booster ingredients and 2) are T Booster product reviews from real customers or just a form of digital marketing?

Methods

Identification of The Top 5 T-Boosters

T-booster supplements were identified by querying the Amazon.com website using the keyword “testosterone booster” (Figure 1). The 5 highest-ranked supplements on Amazon using default search settings which sort and present featured items, on July 22^nd, 2018, were selected for further examination. Supplement information, ingredients, and comments from the initial 2 available pages of reviews were collected and stored offline to ensure consistency in data analysis given the dynamic and routinely updated online marketplace.

Figure 1:

Overview of Investigative Method Employed In this Study

Assurance of Organic Search Results

A new Amazon account was created to minimize bias from user profiling techniques including third party tracking and browser cookies[26]. Bias was further mitigated by using browser-based private mode features for all online data acquisition. Additional efforts to ensure integrity of search results involved installing uBlock Origin, a third-party advertising and tracking blocker (Quebec, Canada https://github.com/gorhill).

Analysis of the 10 Most Common Ingredients

Supplement ingredients were aggregated and catalogued for each supplement, and the 10 most frequently utilized ingredients across the set of supplements were identified. A PubMed search was performed for the 10 most common ingredients in conjunction with the keyword ‘testosterone.’ A systematic review was undertaken to classify articles based on study population, including human studies, animal studies, in vitro studies, and other studies (case reports, review articles). Identified studies were reviewed to assess the impact of each on T levels.

Analysis of User Comments

All available Amazon reviews from the first two product review pages were aggregated for each product. The Androgen Deficiency in the Aging Male (ADAM) questionnaire (Supplementary Figure 1), a validated tool for identifying and classifying hypogonadal symptoms, and assessing response to T boosting therapies, was employed to analyze reviews for T boosting efficacy[27]. Two different raters (A.B., A.S.) coded all aggregated comments using the ADAM questionnaire. A detailed example of an Amazon comment qualitatively analyzed via ADAM is diagrammed in Figure 2. Inter-rater reliability was determined using the kappa statistic[28]. A third rater (N.T.) determined the final ADAM scoring of a comment if a difference between raters was observed.

Figure 2:

Breakdown of Qualitative Analysis of Amazon Reviews Using Androgen Deficiency in the Aging Male (ADAM) Questionnaire

ReviewMeta Screen

A proprietary web-based online review analyzing service, ReviewMeta, was used to analyze the integrity of review comments for each supplement. ReviewMeta analyzes products hosted on the Amazon marketplace using 11 different filters including 1) Suspicious Reviewers, 2) Reviewer Ease, 3) Rating Trend, 4) Unverified Purchases, 5) Word Count Comparison, 6) Phrase Repetition, 7) Overlapping Review History, 8) Reviewer Participation, 9) Brand Repeats, 10) Incentivized Reviews and 11) Deleted Reviews. ReviewMeta identifies the percentage of overall comments that were deemed suspect. The ReviewMeta algorithm also assigns a score of Pass, Warn or Fail under each of the 11 domains to provide more insight into patterns underlying suspicious reviews. After the filtering process, ReviewMeta produces a ranked list of the most trustworthy comments. The number of failed categories for each supplement was correlated to the percentage of eliminated untrustworthy comments using Pearson correlation. All trustworthy comments produced by ReviewMeta were analyzed using the ADAM questionnaire as detailed above.

Results

Testosterone Supplements and Ingredients

The top 5 testosterone supplements on Amazon on July 22, 2018, using the search terms described in Methods, were 1) Prime Labs Men's Testosterone Booster (Prime Labs), 2) Dr. Martin’s Extra Strength Herbal Blend Testosterone Booster (Extra Strength), 3) Iron Brothers Supplements Testosterone Booster (Iron Brothers), 4) Prometheus Wellness Pro-T Premium Testosterone Booster (Pro-T) and 5) Invictus Labs’ Alpha Boost Testosterone Booster (Alpha Boost). Supplement brand names, manufacturer details, costs per unit and ingredient profiles are presented in Table 1. The top 5 T-Boosters had an average ± SD of 2,761 ± 5,112 reviews (Range 6-11,158). Products in this cohort had an average rating of 4.6 ± 0.25 stars (Range 4.5-5).

Table 1:

Overview of Top 5 T Booster Supplement Based on Cost and Ingredient Profile

Supplement information		Ingrediants
Name	Manufacturer	Number of ingredients	Cost ($) per pill	Ashwagand ha root extract	Bioperine	Boron	Boron Citrate	Calcium	Diindoly m ethane (DIIM)	Fenugree k extract	Horney goat weed	Maca root powder	Nettle extract	Orchic substance	Sarsaparil la	Saw palemett o extract	Tongkat Ali extract	Tribulus Terrestris extract	Vitamin B3	Vitamin D	Wild Yam extract	Zinc
Prime Labs	Prime labs	7	$0.33			x					x		x	x	x	x	x
Alpha boost	Invictus labs	9	$0.32			x		x			x		x	x	x	x	x				x
Extra strength	Dr. Martin's nutrition	9	$0.20			x		x			x		x	x	x	x	x				x
Iron brothers	Iron Brothers supplements	12	$0.27	x	x		x		x	x	x	x					x	x	x	x		x
Pro-T	Prometheus Wellness LLC	12	$0.28	x	x		x		x	x	x	x					x	x	x	x		x

All 5 T-Boosters were sold by unique U.S.-based manufacturers and referred to as being produced in ‘FDA Registered’ facilities. Three of the 5 T-Boosters did not report quantities of individual ingredients, but rather only provided a list of all ingredients. A total of 19 unique ingredients were identified across the group of supplements; each supplement consisted of an average of 9.8 ± 2.2 ingredients (Range 7-12). Boron, horny goat weed, and tongkat ali extract were present in all 5 supplements.

Despite being marketed under different names, several T-Boosters, namely Alpha Boost and Extra Strength as well as Pro-T and Iron Brothers all had the same ingredients, including quantities of each ingredient. The average cost per bottle was $20.16 ± $5.34 (Range $11.99-$25.23). The average suggested serving size was 2.4 pills (Range: 2-3 pills). The average cost per pill was $0.28 ± $0.05 (Range $0.19-$0.33). One of the 5 supplements received a designation as an Amazon Choice purchase.

Ingredient Analysis

A PubMed analysis as detailed in the methods was undertaken for the 10 most common ingredients used across all supplements and is presented in Table 2. We assessed how many studies examining each of these ingredients have been published and grouped these based on in vivo human and animal studies, in vitro studies, and other types of studies. Our search, which included the ingredient as well as the term “testosterone”, yielded 191 studies across the 10 most common ingredients, of which 19% (37/191) involved human subjects, 53% (103/191) animal models, 15% (28/191) in vitro studies and 12% (23/191) other types of studies such as case reports or review articles. There was no uniform distribution of articles among the 10 most common supplements. An average of 19.1 ± 15.0 articles was identified per ingredient (Range 7 to 54). The most extensively studied ingredient was saw palmetto extract, with 53 articles including 10 human studies. Boron and maca root powder both had 28 articles each. Among the 10 most common ingredients, fenugreek had the highest ratio of human to total studies at 64% (7/11).

Table 2:

Overview of Systematic PubMed Literature Review for 10 Most Common T Booster Ingredients

Ingredients	# of supplements included	Inclusion in PubMed searchable studies					Impact on Testosterone Levels in Human Studies
		Human Studies	Animal Studies	In vitro studies	Other	Total	Positive Effect	Negative Effect	No effect	Indeterminate
Tongkat Ali Extract	5	3	15	5	4	27	2	0	1	0
Horny Goat Weed	5	0	8	0	0	8	0	0	0	0
Saw Palmetto Extract	3	10	23	15	6	54	2	0	6	2
Boron	3	9	11	1	7	28	2	0	6	1
Nettle Extract	3	1	4	1	1	7	0	0	1	0
Maca Root Powder	2	4	20	2	2	28	0	1	3	0
Ashwagandha Root Extract	2	1	9	1	2	13	1	0	0	0
Fenugreek	2	7	3	0	1	11	4	0	0	3
Bioperine	2	1	5	2	0	8	0	0	0	1
DIIM	2	1	5	1	0	7	0	0	0	1
	Total	37	103	28	23	191	11	1	17	8

Among the 37 human studies, 30% (11/37) observed an increase in T levels, 3% (1/37) a decrease, 46% (17/37) no effect and 22% (8/37) were indeterminate. Accurate comparisons between the 37 human studies were not possible given non-uniform supplementation protocols and sampling timelines.

Analysis of Supplement Comments Prior to ReviewMeta Filtration

User comments on Amazon were used as a proxy to understand the benefits and efficacy of T-Boosters. All reviews from the top 2 pages of comments were collected for each supplement, resulting in a total of 65 reviews, and analyzed using the ADAM questionnaire as described in methods. The kappa statistic for inter-rater reliability was computed to be 0.79, indicating substantial agreement among our raters prior to a third rater addressing any discrepancies[29]. We found that among the assembled comments, more than 40% of users reported improvements in energy and strength/endurance. In addition, 17% of users reported increased libido, and 29% increased sports activity. No users reported improvements in enjoyment of life, happiness level, or height.

Analysis of Supplement Comments Following ReviewMeta Filtration

All supplements were queried using the proprietary ReviewMeta service as detailed in the methods to analyze malicious review trends and eliminate untrustworthy comments. As previously described, the algorithm utilizes 11 categories, such as unverified purchases, incentivized reviews, and phrase repetitions and determines whether reviews of a product “pass,” “warn” or “fail” under each domain. We correlated the number of “fail” results with the number of comments that were removed and found a positive correlation (R = 0.875, p-value = 0.05) (Supplementary Figure 2). The more “fail” results that a supplement had, the more likely that it had questionable reviews. A total of 13,806 reviews existed among the 5 products and of these, 66.6% were considered to be untrustworthy using ReviewMeta filtration and excluded from analysis.

After filtration, a total of 43, user comments were re-evaluated using the ADAM questionnaire to semi-quantitatively determine the effects of individual T-Boosters on quality of life. The kappa statistic was calculated to be 0.68, indicating substantial inter-rater agreement prior to resolution of disagreements by a third rater[29]. After filtering supplement reviews using the ReviewMeta service, a 91% decrease in users reporting increased libido, a 59% decrease in reports of increased energy, a 93% decrease in reports of improved strength/endurance, a 60% decrease in reports of increased erections, a 100% decrease in reports of improved work performance, a 67% decrease in reports of improved sleep, and an 89% decrease in reports of improved sports ability were observed (Figure 3). No consumers reported improvement in enjoyment of life, height or happiness.

Figure 3:

Comparison of Androgen Deficiency in Aging Male (ADAM) Comments Between Pre and Post ReviewMeta Filter Analysis

Discussion

E-commerce is an increasingly popular option for consumers in researching and purchasing products[14,16]. Online marketplaces like Amazon.com are capitalizing on this trend by reducing complicated decision making for consumers using algorithms that individualize product recommendations[30–32]. Research into the impact of these technologies on consumer behavior within the health space is still emerging. De Frietas et al. found that pharmaceutical companies leverage online consumption dynamics to increase consumer awareness and desire for healthcare products[25]. De Frietas went on establish that online consumers are especially vulnerable to digital pharmaceutical marketing as a result of 5 factors: 1) internet dependence, 2) excessive trust in the veracity of online information, 3) lack of awareness of pharmaceutical company influence, 4) social isolation and 5) detail fixation[25]. These vulnerabilities are relevant to marketing and interest surrounding hypogonadism. Layton et al. demonstrated that an increase in direct-to-consumer television advertising about hypogonadism across 75 market areas was associated with an increase in testosterone testing, new therapy initiation and therapy without adequate testing[33]. Mintzes supported these conclusions by drawing attention to how pharmaceutical brands employ techniques such as ghostwriting or fake reviews to disingenuously promote testosterone boosting therapies [34]. The growing role of the internet, combined with the receptiveness of many men to pursuing therapies that influence testosterone levels, emphasizes the importance of understanding T-Boosters hosted on Amazon.com.

In the present study, the top 5 T-Boosters on Amazon.com on July 22, 2018 were identified and found to be produced by 5 unique U.S. manufacturers. Several of these manufacturers did not report the exact quantities of active ingredients in their supplements, providing only lists of ingredients. Patients should be counseled regarding the minimal standardization of ingredients and concerns regarding supplement manufacturing processes[35–38]. Physicians can point to case reports demonstrating that T-Booster supplements have included banned substances, including steroids, resulting in adverse side effects during use[11,12]. While the problem of contamination is not exclusive to T-Boosters, the use of these products among competitive athletes who routinely undergo testing for banned substances underscores the need for transparency in labeling the full spectrum of ingredients found in these supplements[36,39,40].

We also noted that despite a variety of marketing approaches among these 5 supplements, several shared identical ingredient profiles. This raises questions as to the authenticity of both on-bottle marketing as well as reviews touting the benefits of these products. Furthermore, all of these supplements were marketed as being produced in ‘FDA Registered Facilities.’ Patients should be counseled that the FDA does not formally review and approve supplements in the U.S., and therefore such claims should be viewed with caution. We identified that T-Boosters are relatively inexpensive compared to TTh, though clear evidence of efficacy, as well as adverse events associated with use, are often lacking. Physicians and policymakers should be aware that increased transparency about insurance reimbursements and low-cost compounded drugs for conventional TTh can potentially help alleviate the drive of patients seeking alternative therapies for hypogonadism.

A review of the literature supporting T-Boosters underscores the paucity of data available on the impact of T-Booster ingredients on serum testosterone levels. We observed that only 19% of the studies examining the ten most common supplement ingredients were conducted in human subjects. Even among these human studies, most ingredients were either indeterminate or ineffective at actually raising testosterone levels. However, accurate comparisons between the 37 human studies were not possible given non-uniform supplementation protocols and sampling timelines. Patients should be counseled regarding the limited amount of data investigating the effects of these supplements on both testosterone levels and the male reproductive system more globally. As such, patients should be informed that several of the studies we encountered did not formally examine testosterone levels but rather utilized corollaries of improvement in serum testosterone levels such as subjective improvements in libido or strength. The lack of uniform data on supplementation protocols draws into question claims made in online customer reviews that 2-3 T-Booster pills per day can lead to significant improvements in testosterone levels.

Nearly 66.6% of reviews among the top 5 supplements were filtered out by the ReviewMeta.com algorithm. The filtration of most reviews leads us to question the validity of the comments affiliated with each supplement. The positive linear correlation between the number of “fails” detected by ReviewMeta supports the efficacy of the ReviewMeta service and leads to doubt regarding the accuracy of these product reviews. This skepticism is compounded by notable differences between pre- and post-screen ADAM comment analysis. Given that comments were overall more muted in their praise for these supplements following ReviewMeta filtration substantiates our claim that the first two pages of reviews should not overly influence a consumer’s decision making. Furthermore, many of the product reviewers failed to explicitly describe the supplementation regimen that they followed.

The variability in these reviews demonstrates that physicians should counsel patients to temper their expectations about taking these products to boost serum testosterone levels, and even with regards to subjective improvements in symptoms related to testosterone deficiency. We observed that nearly 30% of product reviews continued to report improvements in energy following identification of trustworthy reviews. The fact that many reviews after filtration continued to indicate product efficacy across the ‘energy’ ADAM category provides insight into consumer’s subjective sense of product efficacy. Understanding patients’ overall satisfaction with these products can enable healthcare providers to target future counseling efforts.

There are several limitations of this study that should be acknowledged. First, this analysis was limited to 5 T-Boosting supplements that were identified as the top hits on a single search. Numerous additional T-Boosters are available on Amazon.com, but at the time of our search, did not rank highly enough to be considered for this work, though they may be relevant in the overall discussion of T-Boosters. Despite the small sample size of this study, we believe our selection of the top 5 supplements is appropriate given that customers often do not browse past the first few products. Furthermore, the small sample size does not impact our overall finding of numerous disingenuous reviews associated with each product, and we believe this trend is likely to be even more prevalent among lower ranked products. Second, we acknowledge that the proprietary nature of ReviewMeta can draw criticism. While we recognize that more work is required in validating ReviewMeta findings that facilitate the correlation between customer reviews and quantifiable effects of these supplements on hypogonadal symptoms, there are no other comparable services. During our analysis, we discovered that the Iron Brothers supplement that we had initially identified as a top 5 supplement, had all of its comments deleted due to an internal Amazon quality control check. This supports our claim that many of the reviews touting benefits of certain T-Boosters may be false or artificially manufactured.

Conclusion

T-Boosters are easily available online and used by many fitness enthusiasts. Physicians should be aware of T-Booster contents and their efficacy to better counsel patients. Though marketing, sometimes disguised as consumer reviews on online product pages, would lead consumers to believe otherwise, evidence that rigorously supports a positive impact of these products on testosterone levels and hypogonadal symptoms is lacking. Our review revealed that only limited, flawed human studies have evaluated efficacy of some of the most common ingredients in these supplements, with no definitive findings. In the absence of more definitive human data, patients should be cautioned before considering the use of T-Boosters, particularly given the availability of highly effective drug therapies.

Supplementary Material

2

Supplementary Figure 2: Linear Regression of Relationship Between Number of Failed ReviewMeta Filters and Percentage of Comments Filtered Post ReviewMeta Filter The Pearson Correlation Coefficient was calculated to be 0.875 indicating a positive relationship. The associated p-value was computed using a t-distribution

4

Supplementary Figure 1: Questions Included Within the Androgen Deficiency In the Aging Male (ADAM) Questionnaire