Usability of the evidence-based Japanese integrative medicine (eJIM) information site: analysis with focus group discussion and internet survey on the general population

Abstract

Background

The information site for evidence-based Japanese Integrative Medicine (eJIM) was launched by the Japanese Ministry of Health, Labour and Welfare to enable patients, medical staff, and others to obtain appropriate information, but it has not been objectively evaluated by users. The present study aimed to assess the usability of the eJIM website from the viewpoint of the general population.

Methods

We held a focus group discussion with 12 persons in November 2016, conducted an Internet survey of 1107 people (general population) in February 2017, and assessed their understanding of basic concepts of complementary and alternative medicine and usefulness of the contents after browsing the eJIM website. Levels of functional, communicative, and critical health literacy were also assessed.

Results

Most patients had never visited the eJIM website, because basic terms such as “integrative medicine” and “evidence” were not fully understood. Most overseas information in English was difficult to understand for the general Japanese population, although approximately 50% of the information was translated into Japanese. The degree of appropriate responses depended on health literacy level.

Conclusion

The usability of eJIM website was low, because the currently available information did not meet the needs of the general population. Moreover, simpler expressions and visual tools would help the general population to understand the website contents more easily. A further survey that considers cultural and quantitative perspectives is warranted.

1. Introduction

Since the 1990s, increasing attention has been paid to complementary and alternative medicine (CAM) because of changing disease patterns and an inability of conventional medicine to solve certain health problems.¹ In the United States, at least half of all adults use some types of CAM,² and according to a 2011 survey conducted in Japan, the main focus of CAM was health foods and supplements.³

Integrative medicine, which combines Western medicine and CAM, has also been used in Japan. In 2012, a study group of the Ministry of Health, Labour and Welfare (MHLW) tentatively defined integrative medicine as “medical treatment conducted under a physician's supervision, based primarily on modern Western medicine, and aiming to further improve the quality of life by combining CAM with conventional medicine in various fields.”⁴

A primary problem with CAM is that various therapies are practiced without sufficient evidence. There is no organization in Japan that manages research information and its transmission at a national level, like the National Center for Complementary and Integrative Health (NCCIH) in the United States.⁵ The same MHLW study group suggested the collection of data on safety and efficacy for each category of integrative medicine and the generation of relevant information to help the general population select CAM appropriately.⁴ In response to this suggestion, the eJIM (information site for evidence-based Japanese Integrative Medicine) was launched as a service of the MHLW in March 2014.⁶ The eJIM website has two sections for the general population: 1. “What is ‘integrative medicine’?” and 2. “How to appraise and where to find CAM information for general people?” (Fig. 1).

Fig. 1.

Home page of eJIM and contents of the eJIM website available for general-purpose use. (URL: www.ejim.ncgg.go.jp/en/index.html)

On one hand, the usability of the website for the general population has great significance on the dissemination of information on health and medicine to the public. On the other hand, a survey targeting medical doctors regarding information on CAM revealed needs for information on safety, efficacy, contact address, cost effectiveness, medical staff, and patients.⁷ According to Bolle et al,⁸ elderly patients had usability problems such as difficulties in using navigational elements and tools, understanding the website's layout, and handling many terms and much information.

Health literacy plays an important role in health communication and decision-making. The World Health Organization (WHO) defined health literacy as “the cognitive and social skills which determine the motivation and ability of individuals to gain access to, understand, and use information in ways which promote and maintain good health.”⁸ Nutbeam advocates a model that divides health literacy into three categories: (1) transmission of factual information on health risks and health services utilization (functional), (2) opportunities to develop skills in a supportive environment (communicative), and (3) provision of information on socioeconomic determinants of health, and opportunities to achieve policy and/or organizational change (critical).9, 10, 11

Thus, the objective of this study was to assess the usability of the eJIM website from the viewpoint of the general population and to make recommendations for the provision of information on integrative medicine that meets their needs.

2. Methods

2.1. Subjects and methods

2.1.1. Participants and procedure in focus group discussion

The study method is described by means of a flow diagram (Fig. 2).

Fig. 2.

Flow diagram of the surveys.

A focus group discussion (FGD) involving 12 patients was held in November 2016. An FGD is a “discussion with a small number of participants selected from clearly defined populations in order to have a focused discussion on preselected research themes.”¹² In the FGD, a facilitator, or “moderator” led the discussion based on previously prepared guidelines to encourage voluntary discussion among participants. With respect to participant recruitment, we screened candidates who lived in the suburbs of Tokyo, with diseases and experience of using CAM, from among general monitors at Anterio Inc. (Tokyo, Japan), a marketing research company contracted to conduct the survey, and selected 12 participants. We divided the participants into two groups, that is, six users of health foods and dietary supplements, and six users of acupuncture, moxibustion, or massage therapy.

A two-hour FGD involving both groups was conducted, recorded, and transcribed. We recorded the names of participants who made comments and the order in which the comments were made, and provided a summary of the comments.

2.1.2. Participants and procedure in the Internet survey

An Internet-based survey was undertaken by Anterio Inc. First, 35,176 registered monitors were asked to take part in a screening survey between February 9 and 13, 2017. A total of 10,740 respondents, excluding healthcare professionals, provided valid responses. Based on the responses, active CAM users and discontinued users (those who had used CAM in the past, but do not use now) of product-based CAM (supplements, health foods, etc.) and practice-based CAM (acupuncture, yoga, etc.) were identified. The main survey was conducted between February 17 and 21, 2017, with the participation of 2235 qualified persons selected from the screening survey respondents; 1107 valid responses were obtained.

2.2. Survey item

2.2.1. Characteristics of the study participants

Gender, age, education, marital status, chronic disease, therapies tried within the previous year, and health literacy were asked. Health literacy (HL) refers to the 3-tiered concept (functional, interactive, and critical) proposed by Nutbeam,¹⁰ and the 14-item health literacy scale for Japanese adults (Japanese language version) developed by Ishikawa et al¹³ was used in the present study.

2.2.2. Awareness of and access to the eJIM website

We asked participants if they had previously known about the eJIM website or browsed the web pages. We also asked them if they found it easy to access the web pages.

2.2.3. Challenges of the eJIM website

We asked the participants to express their opinions regarding the overall structure and challenges of the eJIM website freely.

2.2.4. Rate of CAM use and sources of information

To determine rate of CAM use and sources of information, the respondents were asked about 15 sources of information, such as the Internet, over-the-counter pharmacies and drugstores, family, friends, or acquaintances, TV programs/radio programs, and so on, in a multiple-response-allowed fashion.

2.2.5. CAM materials classified by disease and organ

The respondents were asked to choose the most interesting topic of 27 topics on CAM materials classified by disease and organ. The 822 people who consented were asked to read through the content and provide their opinions. The questions included “Do you think a sufficient number of studies have been conducted?”, “Do you think that there are not enough Japanese-translated versions of the studies that you found interesting?”, and “Do you find the Japanese translations reader-friendly?” The respondents rated the topics on a 10-point scale from 1, “Not applicable at all,” to 10, “Very applicable,” and the number of respondents who chose 6–10 (meaning “applicable”) were counted and divided by the total number of respondents to obtain the proportion of respondents who thought that the topics were applicable. We also checked whether Japanese-translated versions were prepared for each of the 27 topics, and obtained the proportion of the 27 topics that had been translated into Japanese.

2.2.6. Rate of information collection of new integrative treatment by different health literacy score groups

We asked the target persons whether they had collected information on efficacy, raw materials, and safety when they tried to use a new integrative treatment. The rate of information collection was calculated for each health literacy score group: high (>2.5), medium (2.0–2.5), and low (<2.0).

2.3. Data analysis

The analysis of the FGD data involved the coding and categorization of the content following repeated examination of the FGD transcript and field notes, to ensure understanding of the overall content, according to the method developed by Lofland et al.¹⁴ The first task in the analysis involved reading the FGD transcript and field notes repeatedly and coding the discussion content and answers to the moderator's questions. In the analysis, at least two analysts performed separate analyses while checking consistency, to avoid subjective bias.

For the data analysis of the Internet survey, the data were collated based on the subject characteristics and usage of each type of therapy.

The statistics package SPSS software version 19.0J for Windows was used for statistical analysis.

2.4. Ethical considerations

The study was approved by the Kanazawa Medical University Ethics Committee (approval number: I-139). The data provided by Anterio Inc. did not include personal information, such as the names and addresses of participants.

3. Results

3.1. Focus group discussion (FGD)

3.1.1. Participants’ characteristics and detailed information

The participants included six women (50.0%) and six men (50.0%), and their mean age was 56.2 years. Detailed information on each participant is provided in Table 1.

Table 1.

Participants’ Characteristics and Detailed information in Focus Group Discussion (n = 12)

ID						Therapies experienced by participantsa
	Discussion group 1: product 2: practice	Sex	Age	Occupation	Disease type	Dietary supplements/health foods	Kampo medicines (OTC)	Acupuncture/moxibustion	Massage
#1	G 1	F	47	Company employee	Digestive diseases, skin diseases, gynecological diseases	✓
#2	G 1	F	55	Housewife	Musculoskeletal diseases, ophthalmic diseases	✓	✓		✓
#3	G 1	F	54	Part-time worker	Skin diseases	✓			✓
#4	G 1	M	66	Company employee	Hypertension, renal and prostate diseases			✓	✓
#5	G 1	M	66	Part-time worker	Hypertension, digestive disease			✓	✓
#6	G 1	M	54	Part-time worker	Hypertension, diabetes, ophthalmic diseases	✓
#7	G 2	F	44	Housewife	Ophthalmic diseases, skin diseases				✓
#8	G 2	F	60	Other	Hypertension, musculoskeletal diseases			✓	✓
#9	G 2	F	55	Company employee	Hypertension, dyslipidemia	✓	✓		✓
#10	G 2	M	55	Freelance	Dyslipidemia, digestive diseases, renal and prostate diseases, musculoskeletal diseases, ophthalmic diseases, skin diseases		✓	✓
#11	G 2	M	52	Company employee	Respiratory diseases, musculoskeletal diseases, duodenal ulcers				✓
#12	G 2	M	66	Civil servant	Hypertension, dyslipidemia	✓			✓

^aTherapies with many users are listed. Participants also used therapies other than these four.

3.1.2. Awareness of the eJIM website

Only one out of the 12 participants had previously used the eJIM website. While some participants expressed the opinion that the eJIM website was reliable because it was created as a Ministry of Health, Labour and Welfare project, others stated that people might be unable to find it, because the Ministry of Health, Labour and Welfare website did not include a link to the eJIM website.

3.1.3. Evaluation of content

Participants provided the rationale for their evaluation of their understanding and the utility of the content of each web page. Opinions of the participants on each item are shown in Table 2.

Table 2.

Opinions on Each Item in Focus Group Discussion

Awareness of the eJIM website

I found eJIM while I was surfing the Internet using “medical information” as a keyword. (#12)

It seems to be reliable, because it is a website made by the Ministry of Health, Labour and Welfare. (#3 positive)

Does the top page of the Ministry of Health, Labour and Welfare's website have a link to eJIM? If it does not have a link to it, you cannot reach eJIM even if you search for it. (#12 negative)

What is integrative medicine?

Because I was doing a job that required English skills, I knew the word “holistic.” It means “integrated” and is used in fields other than medical care. (#5)

I do not know what “evidence” means. (#8 negative)

It would be better to put “This is a website for those looking for information on such and such” on the top page of eJIM. The subtitle “A website that provides information on integrative medicine” gives the impression that a doctor created it for other doctors. People would not know what the eJIM is for. (#12 negative)

It is hard to know what kind of information is provided on this website. (# 3 negative)

How to see through information: beware of the number trick

While other information sites often do not show information on those who did not experience the effects, I like that eJIM posts specific information on it. (#3 positive)

This content reminded me of the importance of “Beware of the number trick” once more. I hope it will be widely advertised so that other people can also benefit from it. (#6 positive)

How to see through information: let us check what is being compared

Since I used to believe everything I saw on the Internet, I am thankful that I had this chance to learn how to check what is being compared. (#4 positive)

I do not think people would understand the expression, “indispensable for a scientific proof.” (#5 negative)

Advanced step in “How to see through information”

As it turns out, it is regrettable to see that we evaluate information as good or bad by ourselves. At first, I thought it was a website about judgment criteria for “good” vs. “bad,” but it was not, so I was disappointed. (#3 negative)

Overseas information: content of the web page regarding different types of CAM

Regarding the web page on calcium, do we need to provide information on American diets even though their physique and diets are different from ours? Calcium intake of 1200 mg a day would be too much for Japanese people. (#10 negative)

Why don’t you provide information on the intake of dietary supplements and nutrients for Japanese people? (#12 negative)

Why does the web page on acupuncture and moxibustion provide overseas information? I think it would be better to post information on Japanese national qualifications for acupuncture and moxibustion. (#4 negative)

Opinions regarding overall structure and content

Too many abstract words (# 5 negative)

I think it would be better to promote Google's custom search a little more actively (#11 suggestion)

I think those who are suffering from health problems would like it if they could go from the table of CAM types on the top page to the detailed explanation page for the type of CAM they like. Do the contents of the “overseas information” page have links to the CAM details pages? I could not find any. (#6 negative)

I found the top page, but it was hard for me to find a link to the next page. (#3 negative)

Opinions regarding layout

There are too many characters on the page, and it is hard to read them. (#7 negative)

I don’t think they considered the age range of the viewers. It is hard for people in their 60s and 70s to read text. It is better to add illustrations, considering the readers. (#11 suggestion)

If it had figures, illustrations, cartoons, etc., it would be easy for us to read and understand. (#3 suggestion)

Comments regarding access

Prescription pharmacies should recommend this website. (#3 suggestion)

The poster should have simple search words. (#6 suggestion)

Since nobody would want to enter the URL of the website, it would be better to make a web page for smart phones and put a QR [quick response] code on it. (#12 suggestion)

Overall issues

Since there is too much information, it is hard for me to find the articles I want to read. I hope to find the target articles more easily by searching with keywords.

Opinions about design were as follows: The letters are too small to read. It would be more understandable to use pictures, movies, or illustrations. Furthermore, we received a suggestion about access. Access to this website might be made easier by linking it to the website of the Ministry of Health, Labour and Welfare and by advertising it through posters and pamphlets supplied to medical facilities and pharmacies.

3.1.4. What is integrative medicine?

When asked about a web page that summarized the concept of integrative medicine, a few participants rated it as “easy to understand” or “useful.” With the exception of those who understood English well, participants did not understand words such as “integrative medicine” or “evidence.” In particular, they did not understand the meaning of “integrative,” and many misunderstood it to refer to comprehensive medical information.

3.1.5. Content regarding “How to see through information”

3.1.5.1. Beware of the number trick

With respect to a web page promoting habits by which to avoid being deceived by the “appearance” of numbers, 83.3% and 58.3% of participants rated it as “easy to understand” and “useful,” respectively.

3.1.5.2. Let us check what is being compared

Regarding a web page advising participants to check what is being compared rather than blindly accepting information provided on the Internet, 50.0% of the participants rated it as “easy to understand” and “useful,” respectively. However, some participants expressed the opinion that the expression was too specialized and difficult to understand.

3.1.5.3. Advanced step in “How to see through information”

With respect to a web page containing advanced steps on how to see through information, types of research involving humans, and “evidence-based medicine,” 41.7% and 41.7% of participants rated it as “easy to understand” and “useful,” respectively. However, most participants were dissatisfied with the recommendation that they should evaluate information on their own.

3.1.5.4. Overseas information: content regarding different types of CAM

With respect to a web page concerning calcium, 50.0% and 66.7% of the participants rated it as “easy to understand” and “useful,” respectively. However, most participants questioned why overseas information regarding various types of CAM was included in the website. In particular, regarding nutritional components, they stated that dietary reference intakes for Japan, rather than other countries, should be provided.

3.2. Internet survey

3.2.1. Attributes of the respondents (Table 3)

Table 3.

Characteristics of the Respondents and Sources of Information in Internet Survey (n = 1107)^a

		n	%
Gender	Female	572	51.7
Age (average)		44.9
Education	Junior high school/high school	364	16.7
	Professional/junior college	242	16.7
	University	408	58.3
	Graduate school	60	8.3
	Other/no response	33	3.0
Marital status	Married	586	52.9
Chronic disease	Yes	312	28.2
Therapies used within the previous one year	Dietary supplements/health foods	426	38.5
	Massage (including various types)	136	12.3
	Acupuncture and moxibustion	37	3.3
	Chiropractic	36	3.3
	Diet therapy	98	8.9
	Kampo medicine (prescribed by a doctor)	59	5.3
	Kampo medicine (excluding those prescribed by a doctor)	48	4.3
	Seitai	88	7.9
	Qigong/yoga/Tai Chi	67	6.1
	Others	132	11.9
Experience of viewing the eJIM site	Yes	31	2.8
Source of information	Internet	348	31.4
	Over-the-counter of pharmacies and drugstores	211	19.1
	Recommended by family	158	14.3
	Recommended by friend or acquaintance	149	13.5
	TV program/radio program	142	12.8
	Recommended by doctor (Western medicine)	135	12.2
	Recommended by practitioner of a clinic, etc.	121	10.9
	A family member is using it	112	10.1
	Articles in books/magazines/newspapers	111	10.0
	Others	322	27.6
	I do not use/buy any treatment/products other than modern Western medicines	238	21.5

^aMultiple responses were allowed.

There were 1107 respondents, and attributes of the study participants are shown in Table 3. Regarding awareness of the eJIM website, only 2.8% (31 out of the 1107 respondents) had visited the eJIM website before this study.

3.2.2. Reason to start using CAM, and source of information (Table 3)

In response to a question asking what prompted their initial use of CAM and the source of the information, 348 (31.4%) respondents said it was the “Internet.”

3.2.3. The most interesting areas of CAM classified by diseases and organs, and percentage of area content translated into Japanese (Table 4)

Table 4.

The Most Interesting Area about CAM Classified by Disease and Organ, and Percentage of Articles Translated into Japanese

	Area	Interested		Translation into Japanese
		n	%	Total number of articles (a)	Presence of Japanese translation (b)	Percentage of articles translated into Japanese (b/a) %
1	Cancer	145	13.1	32	10	31.3
2	Eyes/vision	100	9.0	10	8	80.0
3	Dental and oral health	85	7.7	7	5	71.4
4	Mental health	66	6.0	73	36	49.3
5	Health/welfare	60	5.4	5	2	40.0
6	Tobacco/drug/alcohol	41	3.7	3	2	66.7
7	Gynecologic disorder	36	3.3	26	10	38.5
	Subtotal	822	74.4	857	459	53.6
	None	285	25.7

The mean percentage of Japanese translation in 27 medical areas. The top seven areas are shown.

The areas of interest included cancer (145 [13.1%]), eyes/vision (100 [9.0%]), dental and oral health (85 [7.7%]), mental health (66 [6%]), and health/welfare (60 [5.4%]). On the other hand, only 31.3% of the content and 53.6% of the abstracts on an average (459 out of 857 abstracts) were translated into Japanese.

3.2.4. Information content about CAM classified by organ and disease (Table 5)

Table 5.

Impressions on the Most Interesting Studies about CAM Classified by Organ and Disease (n = 822)^a,b

	N	%
A sufficient number of studies is available.	446	54.3
There are not enough Japanese-translated versions of the studies that I find interesting.	478	58.2
I find the Japanese translations reader-friendly.	452	55.0
The information is easy for the general public to understand.	360	43.8
The information is convincing.	382	46.5

^aWe asked 822 participants after excluding 285 respondents who answered that they were not interested in any topic area covered by Cochrane abstracts.

^bThe number of respondents who chose 6–10 (applicable) on a scale from 1, “Not applicable at all,” to 10, “Very applicable,” and the percentage of the total number of people who responded to the question.

Regarding the information content about CAM classified by organ and disease, slightly more respondents than not thought the number of studies was sufficient and the Japanese translation was easy to read; however, slightly more respondents than not thought that the information content was not easy to understand and not convincing. Furthermore, 478 (58.2%) thought that there were not enough Japanese-translated versions of the studies.

3.2.5. Association of information collection and level of understanding with health literacy (Table 6)

Table 6.

Information on Integrative Medicine that was Collected Before use by Patients with Different Health Literacy Scores (n = 1107)^a

		Health literacy score
		Total	%	≥2.5	%	>2 to <2.5	%	≤2	%
	Total	1107	100.0	557	100.0	272	100.0	278	100.0
1	Safety	581	52.5	386	69.3	115	42.3	80	28.8
2	Adverse reactions	430	38.8	307	55.1	77	28.3	46	16.5
3	Effects	530	47.9	360	64.6	103	37.9	67	24.1
4	Raw materials	141	12.7	117	21.0	18	6.6	6	2.2
5	Not confirmed	315	28.5	72	12.9	98	36.0	145	52.2

^aMultiple answers were allowed.

As for the level of health literacy (HL), 87.1% of the respondents with HL score higher than 2.5, 64% with HL between 2 and 2.5, and 47.8% with HL less than 2 had acquired some information about integrative medicine before trying it. For example, respondents with higher HL score had obtained information on safety more often than those with lower HL before they used integrative medicine as follows: 69.3% of respondents with HL score higher than 2.5, 42.3% with HL between 2.5 and 2, and 28.8% with HL less than 2.

4. Discussion

The present study revealed what the general population thought about the various pages of the eJIM website. An FGD was held, targeting those who had used CAM. Critical opinions and evaluation were obtained from the FGD. On the other hand, the target of the Internet survey was the general population, and the main questions were in regard to the readability and understandability of the overall website.

The Internet survey showed that only 2.8% of respondents had seen or used the eJIM website, revealing that most people were unfamiliar with the site. Respondents also had very low understanding of the fundamental concepts and terms of CAM or integrative medicine.

The contents in the Japanese-translated versions of overseas sites (except some famous sites about vitamin C, calcium, or yoga) did not fit traditional Japanese culture. For this reason, the usefulness of these sites was rated low. Moreover, the Japanese translations of these sites were not very understandable, often causing respondents to read the same sentences repeatedly.

As for the presentation of information, many users thought that there were too many words on the pages. Therefore, providing illustrations and photographs and paying attention to colors and other factors would facilitate users’ understanding.

Users found the overseas information interesting; however, it is doubtful that the information was helpful for the Japanese users, because it was collected in the United States from subjects differing in physique and eating habits from Japanese subjects. In cooperation with academics and research institutions in Japan, the eJIM website should be improved in the area of overseas contents to match the needs of the Japanese people in the future.

Approximately 50% of the Cochrane Review abstracts are translated into Japanese. Increasing the percentage of translations is desirable, and interesting topics for users, as revealed in the present survey, such as cancer, eyes/vision, dental/oral health, and mental health should be given higher priority.

The study had the following two limitations:

First, there were problems with the FGD study. In addition to the small number of participants, participants could have influenced each other and might have found it difficult to express their opinions freely. Moreover, the opinions of shy participants and those whose opinions differed from those of the majority might not have been heard. The expression of some group members’ opinions in group settings was stronger relative to that observed in individual discussion settings. Therefore, it was difficult to generalize the results to the entire participant group.¹¹ Future surveys should include large samples from the general population.

Second, in the Internet survey, the study evaluated individual web pages on the eJIM website, and overall structure such as connections with other web pages and site structure was not examined. In addition, although a user manual has been created,¹⁵ investigation of the usability of the website was not sufficient.

Nevertheless, this study is the first objective assessment of the eJIM website by users from the general population, and its findings have implications for future improvement. The Internet survey was based on opinions expressed by users in preceding interviews, reflecting their actual needs. Meeting the information needs exposed in this study could contribute to the improvement of more user-friendly versions of the eJIM website, promotion of doctor–patient communication, and a safer use of integrative medicine. It is also applicable to the design of web-based health information tools that are useful for provision of medical/health information.

In conclusion, the eJIM website is not well known to the general population, and overseas information is not useful for Japanese users, mainly due to cultural differences. However, respondents to our survey recognized the usefulness of web-based health information tools and indicated their intention to continue to use them in the future. Both usability and accessibility need to be improved.

Conflict of interest

The authors have declare no conflict of interest.