What Information Can the Lay Public Find About Osteoporosis Treatment? A Descriptive Study Coding the Content and Quality of Bisphosphonate Information on the Internet

Abstract

Purpose:

The US Preventative Services Task Force recommends osteoporosis screening and treatment with bisphosphonates in high-risk populations. However, bisphosphonate use among individuals with osteoporosis remains low. The content and quality of information from outside sources may influence individuals’ bisphosphonate decisions. Therefore, we sought to assess the content and quality of osteoporosis treatment information available to the public by conducting an internet search and coding available bisphosphonate information.

Methods:

Eleven search terms about osteoporosis and bisphosphonates were entered into four search engines. Two raters assessed websites for information about bisphosphonates, whether and how benefits and side effects were described and quantified, contraindications, and dosing instructions. Coders also assessed website interface and slant/balance of information.

Results:

1473 websites were identified. 227 websites met inclusion criteria and were coded. The most common bisphosphonate benefit described was prevention of bone density loss (77.1% of websites). The most common side effects described were gastrointestinal problems (66.1%) and jaw osteonecrosis (58.6%). Most websites did not quantify bisphosphonate benefits (78.0%) or side effects (82.4%). Complementary/integrative health websites (p < .001) and pharmaceutical litigation websites (p < .001) were more often slanted against taking bisphosphonates, compared to all websites coded. General medical knowledge websites were more balanced than other websites (p = .023).

Conclusions:

The quality of bisphosphonate information on the internet varies substantially. Providers counseling patients about osteoporosis treatment should inquire about patients’ baseline bisphosphonate knowledge. Providers can complement accurate information and address potential bisphosphonate misconceptions.

Despite its effectiveness, bisphosphonate use for osteoporosis is low. We assessed bisphosphonate information on the internet and found the most commonly listed benefits/risks were bone density loss, gastrointestinal issues, and jaw necrosis, that risk quantification was rare, and information quality varied. Findings underscore the importance of clinical communication about bisphosphonates.

Introduction

An estimated 12.3 million older adults in the US will develop osteoporosis by 2020.[1] Osteoporosis is defined by reduced bone density and microarchitectural deterioration of bone, and can lead to increased bone frailty and fracture risk.[2] Hip fractures are one of the more serious consequences of osteoporosis, and can result in declines in functional status, activities of daily living, independence, and psychological and cognitive health.[3] Vertebral fractures and kyphosis (forward curvature of the spine) are also common consequences of osteoporosis, and can result in chronic pain and other physical challenges.[4]

The US Preventive Services Task Force, American College of Physicians, the National Osteoporosis Foundation, and other national and international organizations recommend osteoporosis screening and treatment with bisphosphonates for high-risk populations (e.g., postmenopausal women at increased fracture risk; women 65 years of age and older)[5–7]. Bisphosphonates have been shown to prevent further loss in bone density[8] and significantly reduce vertebral and non-vertebral fractures in women with osteoporosis.[9–11] Although bisphosphonate use far exceeds use of other pharmacological treatments for osteoporosis (e.g., hormone therapy, Raloxifene, Calcitonin)[12], acceptance of and adherence to therapy has declined in recent years.[13][14] Approximately half of patients prescribed bisphosphonates will discontinue their medication within one year.[15,16] Patients cite many reasons for hesitating to begin or continue bisphosphonates. For example, some worry about rare but serious side effects, like osteonecrosis (death of bone tissue due to a lack of blood supply) of the jaw or atypical femoral fracture.[17] These side effects only occur in about 1 in 10,000 patients[18], but the seriousness of these adverse effects can be alarming for some.[9] Others find oral bisphosphonate dosing instructions (e.g. take on an empty stomach with a full glass of water; remain upright for 30 minutes) challenging to follow.[17]

The content and quality of information from outside sources may also influence individuals’ bisphosphonate decisions. Information about bisphosphonates can be used to supplement patient-provider communication and can support informed decision-making about medication use for older adults with osteoporosis.[14,17,19] Older adults use the internet frequently for health-related searches.[20,21] Searches for bisphosphonate information are common, and may influence opinions about these medications. For example, after a news story about potential adverse effects of bisphosphonates, there was a corresponding spike in internet searches for Fosamax, a brand-name bisphosphonate.[14]

Little is known about the content and quality of consumer-oriented bisphosphonate information available on the internet. This study aimed to assess the content and quality of online bisphosphonate information by systematically evaluating the quality of information available to the lay public. Results could be used to inform providers about high-quality bisphosphonate resources available on the internet. Results can also identify potential missing or inaccurate information available on the internet that providers can use to improve their communication to patients.

Methods

Search Procedure

During the summer and fall of 2018, we performed searches with four search engines (Google, Yahoo, Ask, and Bing) using eleven search terms: “Osteoporosis”, “Osteoporosis treatment”, “Osteoporosis medication”, “Should I take Boniva?”, “Should I take Reclast?”, “Should I take Fosamax?”, “Should I take Actonel?”, “Boniva”, Reclast”, “Actonel”, and “Fosamax”. Fosamax, Boniva, Actonel, and Reclast were selected as search terms due to the availability of these drugs to treat osteoporosis, the recognizability of the drugs by laypeople, their length of time on the market, [14] the frequency with which individuals conduct internet searches for brand names of medications, [22] and layperson preference and trust for brand name medications over generics.[23] Websites were selected from the top results, defined as appearing within the first three pages of search results from each of the four search engines. Research indicates that about 91% of internet searchers do not look beyond the first page of search results.[24] Web searches were conducted with no stored browsing history to avoid personalized or targeted results.

Websites were eligible for inclusion if they contained educational information about osteoporosis prevention or treatment, and were intended for the public. We excluded: paid advertisements, duplicate websites, non-U.S. websites, message boards, sites with only definitions (e.g., an online dictionary), scholarly articles, news articles, prescription package inserts, and sites that restrict content to registered users or were inaccessible. We coded content from the primary webpage located in a search, as well as content within one click of the primary page (i.e., within two clicks of the search engine result). For example, coders assessed content on the primary page, and could also click on content linked from the primary page to secondary pages. Webpages could have included many links to secondary pages available within one click of the primary page, and all content on both primary and secondary pages were coded. Coders did not review advertisement content on any page, nor did they review content linked outside of the secondary pages in order to mimic search and navigation skills of typical internet users.[24]

Data Analysis

We created a website coding instrument to characterize the quality and content of online information about bisphosphonates accessible to the public. The research team included behavioral scientists with health communication and decision-making expertise (LNF and MCP), a rheumatologist with expertise in decision-making and osteoporosis (LF), and an occupational therapist with expertise with aging adults and fall prevention (SS). The instrument was based on a thorough review of the published literature on bisphosphonates including evidence summaries available via uptodate.com (a clinical decision support resource), the Agency for Healthcare Research and Quality (AHRQ), and the International Osteoporosis Foundation (IOF). Items were created to assess the quality of website information based on international standards for information that can help support individuals’ preference-sensitive health decisions[25].

Two authors (LNF and MCP) drafted the first version of the coding instrument. The draft instrument was reviewed (SS, LF, MR), and iteratively revised to ensure consistency with the literature and study goals. The final coding instrument can be found in Appendix A. Screening items initially assessed whether the websites provided information on the prevention or treatment of osteoporosis. If the website contained information about any treatment for osteoporosis, the content was evaluated to determine if it included information about benefits and possible side effects of using bisphosphonates, whether and how benefits and possible side effects were quantified (e.g. percentage, frequency, odds ratio, etc.), and contraindications and dosing instructions/requirements of bisphosphonates. We separated side effect information into those with strong evidence (detailed on uptodate.com, the AHRQ, and IOF websites, with large studies consistently indicating these side effects are associated with bisphosphonates), mixed evidence (those with conflicting data about whether they are side effects of bisphosphonates), or side effects with limited evidence (as described by smaller studies, or clinical perceptions, without compelling data supporting them).

Quantification refers to how benefits or side effects were presented numerically (including percentages, odds ratios or relative ratios, 1 in X format, frequencies, use of a time scale, and visual diagrams/charts, or other visual aids; percentages and frequencies are the most transparent methods of communicating risks and benefits, and visual displays can aid understanding)[26]. The instrument also included one item that assessed the coder’s impression of the website interface user friendliness and one item about the balance of information presented about treatment (including whether the rater found the website slanted toward taking bisphosphonates, toward not taking bisphosphonates, or balanced). Balance was rated based on websites’ bisphosphonate content and their emphasis on benefits versus risks of bisphosphonate use. For instance, websites that only detailed potential side effects, but not benefits of medications, were rated as slanted against bisphosphonates. Two items from the Ottawa Acceptability Questionnaire[27] were used to rate whether websites included enough information for older adults to make treatment decisions about osteoporosis. Descriptive information such as the type of organization that hosted each website, the author of each website’s content, use of scholarly references, year of last website update, and number of clicks per website that were needed to obtain treatment information outlined in the scoring instrument were recorded.

The first 20 websites were independently scored by the two raters (LNF, MR). Cohen’s kappa was calculated for website content, and interrater reliability was strong (kappa = .87). Coders discussed inconsistent codes and revised the codebook when necessary to ensure coding consistency. If consensus could not be reached on particular items, a third rater (MCP) was consulted and the codebook revised. The remaining websites were then coded by one of the two raters (LNF, MR) using the revised codebook. Coders communicated throughout the process to ensure fidelity to the instrument and goals of the project. Any minor questions or disagreements regarding coding were resolved by consulting the principal investigator (MCP), and larger questions were resolved by consulting the entire study team. We present frequencies, proportions, and cross tabulations using chi-square tests to describe our findings. All analyses were conducted in IBM SPSS Statistics 23. Institutional Review Board approval was not needed for this project as no human subjects were involved as participants.

Results

Search results

Figure 1 displays reasons for exclusion and a flow diagram. A total of 1473 websites appeared within the first three pages of search results using the eleven search terms across four search engines. 1212 websites were excluded because they were either clearly marked as search engine advertisements (n=178), were duplicates of one another (n=809) or met at least one other exclusion criteria (n=225). The remaining 261 websites initially appeared to include osteoporosis information and/or prevention or treatment information. Upon further review, 34 of these sites did not include information about osteoporosis prevention or treatment, and therefore were not coded. The final sample included 227 websites that were fully coded for the content and quality of bisphosphonate information.

Figure 1.

Inclusion and exclusion flow diagram

* Advertisements that were not indicated as such in search engine results, but were found to advertise a for-profit product, with no educational content, upon further review.

Website transparency: Hosts, date of updates, and author qualifications

The websites were hosted by a variety of types of organizations, with the most common being medical knowledge websites (e.g. WebMD; 33.04%). See Table 1 for website host types with corresponding data. The year of last website update ranged from 1997 to 2018. 119 websites (52.4%) were updated in 2018, 28 (12.3%) in 2017, and 14 (6.2%) in 2016. 15 (6.6%) did not include a date of last update, and the remaining 51 (22.5%) were updated between 1997 and 2015. Coders used a mean of 2.78 clicks to locate the information outlined in the coding instrument. Of the 227 websites, 77 (33.9%) listed the author of the content, 61 (26.9%) indicated the author’s qualifications (e.g. medical or professional degree), and 51 (22.5%) listed the author’s affiliation (e.g. affiliation with a university or medical organization). 32 websites (14.1%) included in-text citations and 64 websites (28.2%) included a scholarly reference list.

Table 1.

Website hosts of coded websites (N=227)

	N	%
Medical knowledge websites	75	33.04%
Non-profit organizations	23	10.13%
Hospitals/Medical centers	22	9.69%
General knowledge websites	20	8.81%
Pharmaceutical litigation websites	18	7.93%
Government websites	17	7.49%
Complementary and integrative health sites	15	6.61%
Healthcare-focused organizations	12	5.29%
Academic institutions	8	3.52%
Pharmaceutical companies	6	2.64%
Blogs	3	1.32%
Personal websites	3	1.32%
Entertainment or media organizations	2	0.88%
Other	2	0.88%
Aging-focused organizations	1	0.44%

Quality of websites bisphosphonate information

Raters subjectively indicated that 76 websites (33.5%) had enough information available for older adults to make an informed decision about taking bisphosphonate medications. For example, websites that included both pros and cons of bisphosphonates, including accurate information about benefits and side effects, were rated as having enough information to make an informed decision. Many websites omitted important information such as reducing fracture risk, increasing risk of osteonecrosis of the jaw or atypical femoral fracture, and contraindications (Tables 2–4). On a 1 to 5 scale of user friendliness of the website, 117 (51.5%) of websites were rated as user friendly (score of 5).

Table 2.

Descriptive statistics of benefits and possible side effects on coded websites (N =227)

Evidence-supported bisphosphonate benefits		N (%) out of total number of coded websites	N (%) numerically quantifying benefits or risks, out of total number of coded websites
	Prevents bone density loss	175 (77.1%)	17 (7.5%)
	Decreases risk of breaking any bone	112 (49.3%)	26 (11.5%)
	Decreases risk of hip fracture	61 (26.9%)	23 (10.1%)
	Decreases risk of wrist fracture	21 (9.3%)	13 (5.7%)
	Decreases risk of spinal fracture	68 (30%)	28 (12.3%)
	Decreases risk of kyphosis	4 (1.8%)	3 (1.3%)
Type of benefit quantification
(not mutually exclusive)	Percentages	40 (17.6%)	–
	Odds ratios or relative ratios	12 (5.3%)	–
	1 in X	5 (2.2%)	–
	Frequencies (e.g., X out of 100)	4 (1.8%)	–
	Time scale (e.g., benefit over X years)	24 (10.6%)	–
	Visual diagrams	7 (3.1%)	–
	Other visuals	4 (1.8%)	–
Evidence-supported bisphosphonate side effects
	Osteonecrosis of jaw	133 (58.6%)	26 (11.5%)
	Atypical femoral fracture	99 (43.6%)	23 (10.1%)
	Stomach or gastrointestinal complaints	150 (66.1%)	11 (4.8%)
	Acute phase reaction symptoms	55 (24.7%)	4 (1.8%)
Type of side effect quantification
	Percentages	23 (10.1%)	–
	Odds ratios or relative ratios	2 (0.9%)	–
	1 in X	13 (5.7%)	–
	Frequencies	6 (2.6%)	–
	Time scale	12 (5.3%)	–
	Visual diagrams	6 (2.6%)	–
	Other visuals	6 (2.6%)	–
Mixed-evidence bisphosphonate side effects
	Esophageal cancer	22 (9.7%)	–
	Atrial fibrillation	49 (21.6%)	–
	Acute kidney injury/renal failure	29 (12.8%)	–
	Acute phase reaction (in the context of oral meds)	22 (9.7%)	–
	Musculoskeletal pain (in the context of oral meds/outside of acute phase reaction symptoms)	105 (46.3%)	–
	Other various side effects	83 (36.6%)	–
At least one benefit and one side effect mentioned on website		166 (73.1%)	27 (11.89%)

Table 4.

Descriptive statistics of dosing instructions or requirements on coded websites (N =227)

Dosing instructions		N (%) out of total number of coded websites
Oral	Take on empty stomach	89 (39.2%)
	Take with glass of water	91 (40.1%)
	Do not eat, drink, take other meds for 30+ minutes	88 (38.8%)
	Remain upright for 30+ minutes	103 (45.4%)
IV	Hydrate prior to infusion	22 (9.9%)
	Infusion administered by medical professional/at medical office	35 (15.7%)
	Administration of infusion requires specified amount of time (Ibandronate: 15–30 s; Zoledronate: 30+ m)	20 (9%)

Content of websites’ bisphosphonate benefit and side effect information

Table 2 displays descriptive statistics of websites’ display of bisphosphonate benefits and possible side effects that are supported by strong evidence. It also shows how benefits and risks were quantified, if at all. Table 2 also displays descriptive information about websites that include possible bisphosphonate side effects with mixed or limited evidence. The most common benefits described (in 175, 77.1% of websites) were the prevention of bone density loss, followed by decreasing one’s risk of fracture (described in 112, 49.3% of websites). Websites most commonly used percentages as a way to express evidence-supported benefits (described in 40, 17.6% of all websites). However, the majority (78%) of the 227 websites did not quantify any bisphosphonate benefits.

Websites most commonly reported side effects with strong evidence relating to stomach or gastrointestinal complaints and osteonecrosis of the jaw, followed by atypical femoral fracture. Symptoms of an acute phase reaction for IV bisphosphonates were mentioned on about a quarter of websites. 187 of the 227 websites (82.4%) did not quantify any of the possible bisphosphonate side effects. When they did include numeric quantification values, websites most commonly used percentages to convey risk of possible side effects (in 23 websites, 10.1%). The most common side effects with mixed evidence listed on coded websites were musculoskeletal pain (105, 46.3% of sites), followed by atrial fibrillation (49, 21.6%)[28].

We also examined the balance of how benefits of bisphosphonates were presented compared with possible side effects. Most (166, 73.1%) listed both benefits and side effects of bisphosphonates, and almost all of those (153, 92.2%) equally emphasized or displayed the benefits and side effects in-text (through use of the same font, bolding, underlining, and formatting). Only a few (27, 11.89%) websites quantified both the benefits and side effects; most of those 27 (17, 63%) used the same type of quantification for benefits as for side effects (e.g., both used percents, both used odds ratios). More than half (135, 59.5%) of the websites were subjectively rated as balanced in their presentation of information about bisphosphonates. However, about a quarter (58, 25.6%) were described as slanted against taking bisphosphonates and 34 websites (15%) were described as slanted in favor of taking bisphosphonates. For these subjective ratings, when individually comparing each to all other website types, complementary and integrative health websites (χ² (2, 227) =46.80, p < .001) and pharmaceutical company litigation sites (χ² (2, 227) = 28.36, p < .001) were more commonly rated as slanted against taking bisphosphonates. General medical knowledge websites were more commonly rated as presenting balanced information (χ² (2, 227) =7.56, p = .023), compared to all websites coded. For example, Medline websites were rated as balanced and rated as having sufficient information for individuals to make an informed choice about bisphosphonates (https://medlineplus.gov/ency/article/000360.htm, https://medlineplus.gov/ency/patientinstructions/000502.htm). Drugs.com contained a higher quality website (https://www.drugs.com/health-guide/osteoporosis.html), compared to other websites coded that were hosted by private organizations. Websites of higher quality such as these often included details like: causes or risk factors for osteoporosis, symptoms, method of bone density assessment and diagnosis, prevention and treatment options (e.g. lifestyle change, various classes/types of medications). Treatment options were often prominently featured on these webpages and one click to a subpage detailed further information about how bisphosphonates (or other pharmaceuticals) worked to treat osteoporosis, with similar emphasis on medication benefits, side effects, and dosing instructions. These websites were simple to navigate, may have included bulleted or easy to read information, and did not attempt to mislead the reader with inaccurate information.

Contraindications and dosing instructions

Rates pertaining to websites’ inclusion of information about contraindications and dosing instructions or requirements are referenced in Tables 3 and 4, respectively. Kidney impairment was the most common contraindication listed on websites. The most common dosing instruction referenced was the need to remain in an upright position for at least 30 minutes following oral bisphosphonate administration. About a third of all websites (74, 33%) included all four oral dosing instructions. For IV bisphosphonates, the most common requirement referenced on websites was that administration must occur at/from a healthcare professional, although this was only explicitly stated on 35 sites (15.7%). 9 websites (4%) included all three IV dosing requirements.

Table 3.

Descriptive statistics of contraindications on coded websites (N = 227)

Contraindications		N (%) out of total number of coded websites
Oral only
	Past bariatric surgery	6 (2.6%)
	Chronic kidney disease	54 (23.8%)
	Vitamin D deficiency	15 (6.6%)
IV only
	Hypocalcemia	28 (12.6%)
	Renal impairment	31 (13.7%)

Discussion

This study explored the content and quality of bisphosphonate information on the internet through coding websites with information about osteoporosis treatment. We found that the content and quality of bisphosphonate information on the internet varies widely. Prevention of bone density loss, gastrointestinal problems, and osteonecrosis of the jaw are most commonly featured as benefits/side effects of bisphosphonates. Quantification of benefits and side effects was rare. These findings highlight the importance of informative, high quality medical information on the internet, especially for older adults and their family members.

Importantly, both benefits of bisphosphonates and possible side effects were commonly described on available websites. However, most websites focused on stomach or gastrointestinal problems when discussing possible bisphosphonate side effects. This side effect is not necessarily supported by strong evidence. Research suggests that there is little increase in incidence of stomach problems in older adults that take oral bisphosphonates if administered as dosing instructions direct[29]. The frequent discussion of this side effect may reflect fears of common, daily side effects, rather than rare but more serious side effects. Similarly, more than a third of websites mentioned specific oral bisphosphonate dosing instructions and requirements, which require forethought and planning around timing of the bisphosphonates with other medications, with meals, and with liquid intake. Osteonecrosis of the jaw was mentioned slightly less frequently, but the uniqueness of this particular side effect may be alarming to older adults.

In addition, quantification of benefits and side effects was uncommon across websites. This finding is concerning given that older adults may not be able to find bisphosphonate information with numeric values comparing benefits to potential risks that helps them make an informed, preference-concordant decision. When websites included benefit and risk information numerically, though, they quantified it using percentages, which are more transparent than other forms of risk presentation.[30] However, few personalized this information using risk calculators so users could assess their own benefit/risk ratio.

Over half of websites presented information about bisphosphonates that raters perceived as balanced (in terms of benefits and risks of medications), but about a quarter of websites presented information that was slanted against taking bisphosphonates. Slanted information was more commonly associated with complementary and integrative health websites and pharmaceutical litigation websites. These sites arose in a number of searches and together accounted for 14–15% of websites coded. Websites also varied in when they were last updated, although most were updated in the past three years.

These findings should be interpreted in the context of several study limitations. Evidence supporting benefits and side effects of osteoporosis medications is evolving, and internet sites sometimes become unavailable over time. We coded data on websites available in the summer and fall of 2018 about benefits and side effects. We also categorized information as supported by strong evidence, or mixed or limited evidence. Some evidence may become stronger over time with new data. In addition, because websites may be available to the lay public even after they are no longer actively updated by hosts, clear information about an update policy and date of last update is essential to help consumers evaluate the information. Finally, we used a navigation strategy and search terms that older adults might commonly use when searching for information about osteoporosis treatment, including brand names of medications. However, we conducted a sensitivity check by searching for the generic medication names; no new websites were identified using these terms.

This study also has several unique strengths. The current investigation builds on past studies that examined the content and quality of online osteoporosis treatment information. These investigations focused either on osteoporosis alone (rather than medications)[31], on nutrition information for those with osteoporosis[32], or on a variety of osteoporosis treatments (bisphosphonate, non-bisphosphonate, lifestyle changes).[33] Two of these investigations used only one search engine (google) and all varied in how they measured website quality and content. We focus on bisphosphonate medications, the first-line and most commonly used form of treatment.[34] We utilized four search engines with the first three pages of search results for each, while providing our own subjective interpretation of website balance/slant in addition to the assessment of website content.

Overall, the quality of bisphosphonate information available on the internet was variable, particularly in terms of quantifying benefit and risk information. Quantification is important to help individuals decide whether they are willing to accept some potential risk of side effects, including rare but potentially serious side effects, for a substantial increase in benefit. Future studies can build on this work by assessing older adults’ thoughts and feelings about rare side effects of bisphosphonates, and how they might affect decision-making processes surrounding osteoporosis treatment. Studies can also explore how best to communicate information about bisphosphonates to supplement patient-provider conversations.

Our findings underscore the importance of patient-provider communication about bisphosphonates. Decision-making about osteoporosis treatment may occur both in the setting of a provider’s office, and before or after consultations using online sources. Providers should be aware of high-quality websites for patients to supplement clinical consultations. They should also explore which factors are most important to older adults’ decisions (e.g. daily minor inconvenience versus rare but serious side effects), and follow up with patients to determine level of adherence to bisphosphonates, as well as side effects they may be experiencing. Patients and providers should collaboratively discuss osteoporosis treatment options, with opportunities for patients to voice their concerns about medications and gain understanding from providers about potential side effects that may or may not be supported by empirical evidence. Providers should offer plain language written instructions for day-to-day use of bisphosphonates, which can promote adherence to correct dosing and administration of the medication. Counseling patients about osteoporosis treatment should include discussions about information gathered from online sources so that providers may complement accurate information and dispel potential medication misconceptions.

Abbreviations

AHRQ

Agency for Healthcare Research and Quality

ASPIRE

Advanced Summer Program for Investigation and Research Education

CDC

Centers for Disease Control and Prevention

IBM

International Business Machines

IOF

International Osteoporosis Foundation

IV

Intravenous

PI

Principal investigator

SPSS

Statistical Package for the Social Sciences

Appendix 1

Coding Instrument for Content of Bisphosphonate Information on Websites

Search result name:
Website name:
Website url:
Additional urls linked to:
Date of data collection:
Research team member:
1. Does the website contain basic information about Osteoporosis? (e.g., definition of Osteoporosis, risk factors, consequences of it)? [Data entered in this question’s respective column is not to be used as a coding criteria.]				Yes
				No
2. Does the website contain information about preventing or treating osteoporosis (e.g., calcium and Vitamin D, lifestyle interventions, bisphosphonates)?				Yes
				No
A. Communication about Bisphosphonates
3. Does the site correctly identify the evidence-supported positives/pros of using bisphosphonates? [Y/N]
a.	Keeps bones from getting worse/slows down or prevents bone loss in most people/improves bone density[1]
	Is it quantified? (e.g. X out of 100; X%; 1 in X)  Note quantification here:
b.	Decreases risk of breaking any bone[1]
	Is risk of breaking any bone quantified? (e.g. X out of 100; X%; 1 in X)  Note quantification here:
c.	Is hip bone specified? (Decreases risk of breaking hip)
	If hip specified, is decrease in hip break risk quantified? (e.g. X out of 100; X%; 1 in X)  Note quantification here:
d.	Is wrist specified? (Decreases risk of breaking wrist)
	If wrist specified, is decrease in wrist break risk quantified? (e.g. X out of 100; X%; 1 in X)  Note quantification here:
e.	Is bone in back/spine (vertebrae) specified? (Decreases risk of breaking bone in back/vertebral fracture)
	If back/spine specified, is decrease in vertebral fracture risk quantified? (e.g. X out of 100; X%; 1 in X)  Note quantification here:
f.	Decreases risk of kyphosis/hunched back
	Is risk of kyphosis quantified? (e.g. X out of 100; X%; 1 in X)  Note quantification here:
g.	Other pros/positives? Describe:
	Is it quantified?  Note quantification here:
h.	No pros/positives stated [Mark X here if none of the above apply.]
4. How were the positives/pros numerically quantified? [Mark all that apply.]
a.	Percents (X%)
b.	Frequencies (X out of 100)
c.	Odds ratios or relative risks (cuts risk in half, for example)
d.	1 in X format
e.	Positives not numerically quantified [Mark X here if none of the above apply.]
5a. If positives/pros were quantified, was a time scale used? (e.g. over 10 years) [Y/N]
	List time scale here:
b.	Positives not numerically quantified [X]
6. Does the website use visual diagrams to convey probabilities of positives/pros? (e.g. an icon array, or pictograph, or bar graph, or pie chart) [Y/N]
	List type of visual diagram here:
7. Does the website use visuals to show positives/pros? (e.g. osteoporosis bone images; images of improvements in bone density) [Y/N]
	List type of visual here:
8. For all bisphosphonates: Does the site correctly identify the evidence-supported negatives/cons of using bisphosphonates? [Y/N]
a.	Stomach problems (e.g., nausea, vomiting, diarrhea, general GI upset)
	Is it quantified? (e.g. X out of 100; X%; 1 in X)  Note quantification here:
b.	Osteonecrosis of the jaw
	Is it quantified? (e.g. X out of 100; X%; 1 in X)  Note quantification here:
c.	Atypical femoral fracture
	Is it quantified? (e.g. X out of 100; X%; 1 in X)   Note quantification here:
d.	Other negatives/cons Describe:
	Is it quantified?  Note quantification here:
e.	No negatives/cons stated [Mark X here if none of the above apply.]
8.2 For IV bisphosphonates only: Does the site correctly identify the evidence-supported negatives/cons of using IV bisphosphonates? [Y/N]
a.	Acute-phase reaction or symptoms of acute-phase reaction occurring within 72 hours of administration. {Characterized by low-grade fever, myalgias, and arthralgias. Symptoms defined as characteristic of the acute-phase reaction are fever, diffuse musculoskeletal pain (distinct from severe and persistent), anorexia, eye inflammation, and other symptoms (fatigue, dizziness/vertigo, peripheral edema, flu-like illness, headache, syncope, pain, malaise, nasopharyngitis, thirst, insomnia, tremor)}.[2–4]
	Is it quantified?  Note quantification here:
b.	Other negatives/cons Describe:
	Is it quantified?  Note quantification here:
c.	No negatives/cons stated for IV bisphosphonates [Mark X here if none of the above apply.]
9. How were the negatives/cons numerically quantified? [Mark all that apply.]
a.	Percents (X%)
b.	Frequencies (X out of 100)
c.	Odds ratios or relative risks
d.	1 in X format
e.	Negatives not numerically quantified [Mark X here if none of the above apply.]
10. If negatives/cons quantified, was a time scale used? (e.g. over 10 years) [Y/N]
	List time scale here:
b.	Negatives not numerically quantified [X]
11. Does the website use visual diagrams to convey probabilities of negatives/cons? (e.g. an icon array, or pictograph, or bar graph, or pie chart) [Y/N]
	List type of visual diagram here:
12. Does the website use visuals to show negatives/cons? (e.g. images of osteonecrosis of the jaw, or of a femoral fracture) [Y/N]
	List type of visual here:
13. Were the pros described equally compared with the cons? [Y; N; N/A]
a.	Same quantification (e.g. both use percents; both use odds ratios)
b.	Same denominator if using frequencies (e.g. both use X out of 100 or x/100)
c.	Same time scale (e.g. both state “over 10 years”)
d.	Equal emphasis (same font and size, bolding, underlining)
e.	Visual diagrams used for both pros and cons (e.g. both display some type of diagram for pros and cons (can’t have a diagram for pros, but not for cons))
f.	Other visuals depictions used for both pros and cons
14.1 For oral bisphosphonates only: Does the site correctly list the dosing and administration instructions/requirements for oral bisphosphonates7? [Y/N/NA]
a.	Take on an empty stomach[5]
b.	Take with a glass of water[5]
c.	Do not eat, drink, or take any other medications for at least 30 minutes[5]
d.	Remain upright for at least 30 minutes after taking medications[5]
14.2 For IV bisphosphonates only: Does the site correctly list the dosing and administration instructions/requirements for IV bisphosphonates? [Y/N/NA]
a.	Hydrate prior to infusion
b.	Infusion is performed by a medical professional or infusion is provided at the office of a medical professional
c.	Administration of infusion requires a specified amount of time: 15–30 seconds for ibandronate, at least 30 minutes for zoledronic acid[5]
15.1 Does the site correctly list the contraindications[5] of bisphosphonates? [Y/N]
a.	Past bariatric surgery
b.	Chronic kidney disease
c.	Vitamin D deficiency
d.	Inaccurate contraindications listed Describe:
e.	No contraindications listed [Mark X here if none of the above apply.]
15.2 For IV bisphosphonates only: Does the site correctly list the contradictions of IV bisphosphonates?
a.	Hypocalcemia[5]
b.	Renal impairment[5]
c.	No contraindications listed for IV bisphosphonates [Mark X here if none of the above apply.]
B. Mixed- or Limited-Evidence Information about Bisphosphonates
16. Does the site contain mixed- or limited-evidence information about bisphosphonates? (e.g. a side effect of bisphosphonates that is incorrect) [Y/N]
a.	Esophageal cancer[6]
b.	Atrial fibrillation[6] (Irregular heart beat)
c.	Acute kidney injury[6] (acute renal failure/abrupt loss of kidney function)
d.	Acute phase reaction[6] (ONLY applicable for oral bisphosphonates)
e.	Musculoskeletal pain[6] (ONLY applicable if not in the context of acute phase reaction symptoms for IV bisphosphonates)
f.	Other inaccurate information (not consistent with evidence-based guidelines) Describe:
g.	No inaccurate information [Mark X here if none of the above apply.]
C. Scholarly citations and list of references included in the website
17. Was there a list of references included at the end of particular sections, or at the bottom of the webpage? [Y/N]
18. Were individual citations used in the text of the website? [Y/N]
D. Quality of information included in the website
19. Date of last website update
20. Overall impression of website
User unfriendly interface				User friendly interface
1	2	3	4	5
21. I found the presentation of the website to be: [Adapted from OAQ #4]
a. Slanted toward not taking bisphosphonates		b. Slanted toward taking bisphosphonate medications	c. Balanced
22. Do you think the website included enough information to help older adults decide on therapy for osteoporosis? [Adapted from OAQ #8]
a. Yes		No
23. Number of clicks to get all information
F. Other characteristics of the website
24. Type of organization that hosts the site (indicate all that apply)
a.	Health care-focused organization
b.	Hospital/academic medical center (e.g. Mayo Clinic)
c.	Academic institution
d.	Non-profit organization
e.	Aging-focused organization (e.g. AARP)
f.	General knowledge website (e.g. Wikipedia, About.com)
g.	General medical knowledge website (e.g. WebMD/MedicineNet)
h.	Pharmaceutical company (e.g. Bayer)
i.	Entertainment or media (e.g. Health magazine)
j.	Government (e.g. CDC)
k.	Personal website
l.	Blog
m.	Social media page of organization (e.g. Facebook page of Osteoporosis Foundation)
n.	Other. Describe:
o.	Complementary and integrative health (i.e., lifestyle, herbs, supplements, whether or not evidence-based)
p.	Pharmaceutical or drug product liability litigation (e.g., law firms, pharmaceutical lawsuit news)
25. Authorship [Y/N]
a.	Author is identified
b.	Author qualifications are stated
c.	Author affiliations are stated

Appendix References

1. Qaseem A, Forciea MA, McLean RM, Denberg TD, for the Clinical Guidelines Committee of the American College of Physicians. Treatment of Low Bone Density or Osteoporosis to Prevent Fractures in Men and Women: A Clinical Practice Guideline Update From the American College of Physicians. Annals of Internal Medicine 2017 Jun 6;166(11):818. [doi: 10.7326/M15-1361]

2. Reid I, Gamble G, Mesenbrink P, Lakatos P, Black D. Characterization of and risk factors for the acute-phase response after zoledronic acid. The Journal of Clinical Endocrinology & Metabolism 2010;95(9):4380–4387.

3. Wark JD, Bensen W, Recknor C, Ryabitseva O, Chiodo J, Mesenbrink P, de Villiers TJ. Treatment with acetaminophen/paracetamol or ibuprofen alleviates post-dose symptoms related to intravenous infusion with zoledronic acid 5 mg. Osteoporosis International 2012 Feb 1;23(2):503–512. [doi: 10.1007/s00198-011-1563-8]

4. Black DM, Delmas PD, Eastell R, Reid IR, Boonen S, Cauley JA, Cosman F, Lakatos P, Leung PC, Man Z, Mautalen C, Mesenbrink P, Hu H, Caminis J, Tong K, Rosario-Jansen T, Krasnow J, Hue TF, Sellmeyer D, Eriksen EF, Cummings SR. Once-Yearly Zoledronic Acid for Treatment of Postmenopausal Osteoporosis. N Engl J Med 2007 May 3;356(18):1809–1822. [doi: 10.1056/NEJMoa067312]

5. Rosen HN, Rosen CJ, Schmader KE, Mulder JE. The use of bisphosphonates in postmenopausal women with osteoporosis. 2017.

6. Adler RA, Fuleihan GE-H, Bauer DC, Camacho PM, Clarke BL, Clines GA, Compston JE, Drake MT, Edwards BJ, Favus MJ, Greenspan SL, McKinney R, Pignolo RJ, Sellmeyer DE. Managing Osteoporosis in Patients on Long-Term Bisphosphonate Treatment: Report of a Task Force of the American Society for Bone and Mineral Research. Journal of Bone and Mineral Research 2016 Jan 1;31(1):16–35. [doi: 10.1002/jbmr.2708]