Abstract
Background and Aims:
The first step in each systematic review is selection of the most valid database that can provide the highest number of relevant references. This study was carried out to determine the most suitable database for information retrieval in telemedicine field.
Methods:
Cinhal, PubMed, Web of Science and Scopus databases were searched for telemedicine matched with Education, cost benefit and patient satisfaction. After analysis of the obtained results, the accuracy coefficient, sensitivity, uniqueness and overlap of databases were calculated.
Results:
The studied databases differed in the number of retrieved articles. PubMed was identified as the most suitable database for retrieving information on the selected topics with the accuracy and sensitivity ratios of 50.7% and 61.4% respectively. The uniqueness percent of retrieved articles ranged from 38% for Pubmed to 3.0% for Cinhal. The highest overlap rate (18.6%) was found between PubMed and Web of Science. Less than 1% of articles have been indexed in all searched databases.
Conclusion:
PubMed is suggested as the most suitable database for starting search in telemedicine and after PubMed, Scopus and Web of Science can retrieve about 90% of the relevant articles.
Keywords: Bibliographic databases, Information retrieval, Sensitivity, Accuracy, Telemedicine
1. INTRODUCTION
Systematic literature review is a documented approach to find the answer to a specified question through searching published and unpublished sources. The base of a good systematic review is a comprehensive and efficient search in retrieving previous studies on a topic (1).
Comprehensive search includes search in different electronic databases, internet and also manual search and communication with experts in the field(2). In this process, among thousands present documents, a few ones related to the question of study are selected for description, analysis and conclusion (3).
Today, this type of studies comprises an important part of required evidences for decision making and interventions in health field. In order to collect all relevant information in a systematic literature review in health topics, information retrieving from biomedical databases beside other information sources is essential. Different bibliographic databases are important sources for information retrieval in health field and facilitate this process (4). In addition to Medline, other bibliographic databases such as Cochrane, Psychoinfo, Cinhal, Embase and scientific databases like Science citation index and specific databases in different medical fields such as Pedro for physiotherapy, Herb Med for herbal medicines and other sources provide necessary information. In some systematic reviews, based on the topic, great number of databases is searched to cover more articles (5). The applicant who uses the results of a systematic literature review should be confident that in the performed study all articles or at least all important and efficient articles in that field have been included.
A comprehensive search, in the case of availability of valid information sources, requires financial and human resources and a great amount of time. Therefore, determining the relative value of different information sources causes productivity in all three mentioned aspects. Determining the value of different information sources is a critical activity that increases the efficiency and reliability of a systematic review (6-7). Knowing that which database provides the researcher a greater number of related articles in health topics is beneficial in designing systematic literature reviews (1, 4, 8). For this, several studies have been performed in different scientific fields for evaluation of databases (1, 5, 9).
When the topic of study is related to interdisciplinary fields, selection of information source is more difficult (1, 10-12). Researchers in these fields tend to search several bibliographic databases (13); for example, Telemedicine is a new issue in health sciences field which is highly related to technology. Telemedicine provides this facility for health sciences specialists to benefit from telecommunication technology in evaluation, diagnosis and treatment of patients in a place other than the patient residency (14-15). Studies show the effectiveness of this field in different aspects of health services. Researches in this field which is a combination of new technologies in information sciences and medicine are growing extensively. There are several articles during the last 30 years in support of applying and developing telemedicine and its effects including clinical benefits, financial savings, creating equity in educational opportunities and possibility of consultation (14). Practical use of telemedicine requires providing evidences for decision making and measures based on research documents. In this relation, research documents resulted from reviews are of a great importance, because by evaluating the results of previous studies, they can determine priorities and capabilities of future studies. In this line, recognizing the potentials of different information databases and their comparison in terms of searching different topics increases the confidence of researchers and consequently the productivity of research process.
The number of studies evaluating information databases that are used for search in Telemedicine field is very limited and generally, they have described just the number of retrieved articles from databases and websites and have not evaluated the relevance of retrieved articles with the applicant question and topic (3, 13, 16-17). Therefore, in the present study, in order to determine the best database in providing the most relevant articles in telemedicine field, Telemedicine or its equivalent keywords were searched in four bibliography databases in combination with three topics affecting the quality of health care activities, that is Education, Patients’ satisfaction and cost-effectiveness. Then, based on the retrieved relevant articles, the precision, sensitivity, overlap and uniqueness ratios of databases were determined and compared. These items are among criteria of evaluating the quality of systems in information retrieval (17, 18).
The results of this study provide necessary evidence for researchers who intend to perform systematic reviews in Telemedicine field and also for experts who intends to give service to the researchers by purchasing and providing efficient databases.
2. METHOD
In this cross-sectional study, in order to investigate the precision ratio of electronic databases in information retrieval related to the Telemedicine field, four bibliography databases including Web of science, Pubmed, Scopus and Cinahl were investigated through Internet and from the portal of electronic sources in digital library of Tehran University of Medical Sciences. Selection of these databases was based on two conditions recommended in relevant texts (19, 20) and the availability of them.
Keywords were determined by consultation with a Telemedicine expert and after checking Medical Subject Heading Index (MeSH) in three fields of Education, Patient satisfaction and cost-benefit analysis. For this purpose, telemedicine was selected and its equivalents were searched in Mesh. The suggested terms were Health, Mobile/Mobile health,eHealth,Telehealth. Then, this term and its Mesh equivalents were combined with MeSH equivalents of patient satisfaction (patient satisfaction/patient preference), patient education and cost-benefit analysis.
Databases were searched from 2007-2012 and search was limited to the articles with abstract availability and English language with the following strategy:
(Telemedicine OR telehealth OR Mobile health OR health, mobile) AND (cost-benefit analysis)
(Telemedicine OR telehealth OR Mobile health OR health, mobile) AND (patient education).
(Telemedicine OR telehealth OR Mobile health OR health, mobile) AND [(patient preference) OR (patient satisfaction)].
The results obtained from each database (with abbreviations of the database name and search topic) were sent to the endnote software. In whole, 2251 articles for the three topics were retrieved from the selected databases. After deletion of duplicate articles in each database, articles were separately evaluated by two experts (one telemedicine specialist and one Health information management specialist) based on their titles and abstracts and according to uniform definitions. Accordingly, 941 articles relevant to the search topic were recognized. The articles were analyzed in order to determine the precision ratio and sensitivity ratio of each database in the studied field. The two evaluators participated in a meeting to decide about their differences. In cases that the abstract did not provide required information for decision making, the full text was studied. In this stage, duplicated articles among databases were not omitted for overlap investigation. The retrieved articles were sent to EXCEL software. After the arrangement of articles, each article was labeled for the database from which it had been retrieved (p, s, w, c). A total of 679 articles relevant to the study topic were remained after deleting the duplicate articles. Precision ratio, sensitivity ratio, overlap ratio and uniqueness ratio of each database were determined through analyzing the remained articles. Precision ratio and sensitivity ratio are two qualitative and quantitative criteria in information retrieval (21). Precision ratio shows the ratio of relevant retrieved articles to the total retrieved articles from a database regardless of their relevance. Sensitivity ratio shows the ratio of relevant retrieved articles from a database to the total relevant articles retrieved from all searched databases and is used to compare databases.
For this purpose, by searching valid sources, definitions of questioned ratios in this study were determined. Then, based on these definitions, related formulas were extracted and using collected data in the Excel file, the number of articles in each database and the number of common articles among them were counted. Then, placing numbers in the formulas, ratios were calculated as percents. In order to provide the possibility of comparing databases based on calculated ratios, these ratios have been presented in four separate tables.
Precision, sensitivity, uniqueness and overlap ratios of each database were obtained based on the following definitions.
Precision ratio: the ratio of the number of relevant records retrieved in a database to the total number of records retrieved in the same database (regardless of being relevant or not).
Sensitivity or recall ratio: The ratio of the number of relevant records retrieved in a database to the total number of relevant records retrieved in all studied databases.
Uniqueness ratio: The ratio of the number of retrieved relevant records that have been indexed in a special database to the total number of relevant records in all studied databases.
Overlap ratio: The ratio of the number of common relevant records retrieved in databases to the total number of relevant records in all studied databases.
3. RESULTS
Based on the mentioned strategy, searching Telemedicine topic in PubMed, Scopus, Web of Science and Cinahl databases resulted in a total of 2251 articles. In order to determine the precision ratio, the relevancy of retrieved articles was evaluated and 941 articles were identified as relevant. PubMed had the highest precision ratio (50.7%) followed by Web of Science (41.5%) and Scopus (34.3%).
As it is presented in Table 1, amongst studied databases, PubMed had the highest precision ratio and the lowest precision ratio was found for Cinahl database. Precision ratio of 50.7% for PubMed showed that even in this database, almost half of the retrieved records were not relevant to the search topic; in other words, although this database had the highest precision ratio among other databases, it was not completely satisfactory.
Table 1.
Ratio of precision Articles that were retrieved from the databases in issue of telemedicine
In regard to the sensitivity or recall ratio which is used to compare databases in retrieval of relevant records, the total number of relevant articles retrieved from all four databases, after omitting duplicate articles, was 679 articles. Table 2 shows the sensitivity or recall ratio of four studied databases.
Table 2.
Ratio of sensitivity Articles that were retrieved from the databases in issue of telemedicine
As it is shown, PubMed could provide 61.4% of articles in the field of Telemedicine. Web of Science and Scopus with lower sensitivity ratios showed no considerable difference with each other, but Cinahl showed considerable difference with three other data bases in indexation of Telemedicine articles; that is, just 6% of Telemedicine articles were available in this database. Therefore, it can be seen that PubMed, among four studied databases, has the highest sensitivity ratio; even though, by limiting search to just this database about 39% of articles relevant to the Telemedicine field could not be retrieved.
In order to determine the uniqueness ratio, the number of articles that could be retrieved by searching only a unique database has been counted; in other words, these documents have been indexed in just one of the studied databases.
Based on the uniqueness ratio, from 679 relevant articles, 38.1% could be retrieved by only searching PubMed and about 19% were available only in Scopus. Cinahl was in the last rank in regard to the uniqueness ratio. Therefore, PubMed had the highest uniqueness ratio among the four evaluated databases (Table 3).
Table 3.
Ratio of Uniqueness of articles that were retrieved from the databases in issue of telemedicine
Overlap ratio shows the number of duplicate articles indexed in all evaluated databases. Two/three/four-database overlap ratios were determined through counting the number of duplicate relevant articles in the four databases and the results have been presented in Table 4.
Table 4.
The number of duplicate articles indexed in evaluated databases and Overlap percentage
As it is seen, from 679 relevant retrieved articles, only 5 ones have been indexed in all four databases. In three-database searching, the highest overlap ratio (7.7%) was obtained in searching PubMed, Web of Science and Scopus. In two-database searching, the highest overlap ratio (18.6%) was obtained in searching PubMed and Web of Science. Figure 1 was prepared to select the best database in regard to the availability of relevant articles in the field of Telemedicine.
Figure 1.
Comparison of precision, sensitivity and uniqueness of retrieved articles in four databases
Comparison of four databases in regard to the all precision, sensitivity and uniqueness ratios put Pubmed in the first rank. Scopus had priority over Web of Science in sensitivity and uniqueness ratios, but in terms of precision ratio, it was the opposite. Cinahl database, with precision and uniqueness ratios of less than 10%, stood in the last rank in all three ratios.
4. DISCUSSION
The aim of this study was to compare four bibliography databases in retrieving telemedicine evidences. Based on literature review, these databases have been suggested as important databases for systematic reviews and have been compared in regard to information retrieval in other fields too. For this purpose, the precision and sensitivity (recall) ratios of each database were determined. According to the results, Pubmed, compared to the other three studied databases, can provide the most relevant articles in Telemedicine field (precision ratio=50.7%, sensitivity ratio=61.4%). However, by limiting the search to just this database, about 39% of relevant articles would not be retrieved. Bahaadinbeigy et al have compared Medline, Embase and Cinahl databases for Telemedicine searches and asserted that by limiting search to the Medline database, 9-18% of articles would be missed(3). In the mentioned study, just the rate of retrieval has been investigated and the rate of relevance of articles to the search question has not been investigated. In Dickersin et al study, like our study, 50% of relevant articles in Health field could be retrieved by search in PubMed (2). But according to Brettle et al study, PubMed with precision ratio of 26% in retrieving articles related to the rehabilitation of mental patients has found the third rank among 6 other databases (1). Clapton has searched three topics in the field of social care in 9 bibliography databases and according to the results, PubMed with average retrieving of 5% of the relevant articles has found the 8th rank (11). These results show that the precision and sensitivity ratios of databases differ based on the search topic and also the number of compared databases. Lohonen et al have compared three databases used commonly in searching health information in search of four medical topics and have concluded that the retrieval rate depends on the search topic (4).
In Betran et al study about the precision and sensitivity ratios of 11 databases in identifying studies related to “maternal mortality”, Medline and BIOSIS databases had respectively the highest (62%) and the lowest (13.2%) sensitivity ratios. The researchers have emphasized on searching more databases, particularly regional databases such as HELLIS. ORG, IndMED, IMEMR and AIM; even though, in search of these databases the researcher faces language barriers and they do not have indexation standards (9). In order to compare the performance of different databases for systematic review, Beyer & Wright have evaluated 16 databases in including the studies related to “frozen shoulder”. By comparing precision, sensitivity and uniqueness ratios of these databases, they concluded that for a systematic review in the Health field, at least three databases are necessary. However, the search strategy and topic affect the results. In the mentioned study, Central database had the highest precision ratio, and Science Citation Index had the highest retrieval rate (8). The results of the mentioned study cannot be compared with our results due to studying different databases in the two studies.
In the case of searching combination of two databases, PubMed and Scopus provide approximately 70% of the relevant articles in Telemedicine. Even if this rate is not sufficient for a systematic review, it is valid for knowing previous studies, developing further studies and evidence-based decision taking in this field.
In the present study, the overlap ratio of retrieved articles in the four studied databases varied from 0.7% to 18.6%. This relatively low rate emphasizes on searching more databases in order to have a more comprehensive information retrieval especially in systematic reviews. Overlap ratio of databases varies based on the searched topic, databases’ indexation strategies and the applicants’ skills. For example, in Wong et al study, overlap of Medline and EMBASE varied between 10-85% based on the search topic (18). There are some other studies about overlap of databases for different topics. The results of the present study, in agreement with the results of some of them, emphasize on the necessity of searching more than 3 databases for obtaining 90% of the relevant articles. In Brettle & Long study about the overlap of 6 databases in searching “rehabilitation of patients with mental illnesses”, the highest two-database overlap was 20% (1). Similarly, in the present study, the highest two-database overlap was 18.7% and it was found for PubMed and Web of science databases. Haafkens et al have studied 5 databases, two of them common with our studied databases, with the search topic of “chronic diseases”. They have found low percent of overlap and noticeable percent of uniqueness of the retrieved relevant articles. The percent of uniqueness in the studied databases varies based on the search topics.
In the present study, PubMed and Cinahl had respectively the highest (38.1%) and the lowest (2.9%) uniqueness ratios. In Clapton study with the search topic of “social care” in two databases common with our studied databases (Medline and Cinhal), the uniqueness ratios were respectively 10% and 0% (11) and in Brettle study, the uniqueness ratios for the two mentioned databases were respectively 5% and 2% (1). It should be mentioned that in the present study, the uniqueness ratio has been calculated based on the relevant retrieved records, since retrieved articles that are cited in the final study are of more value for analysis. In Woodman et al study the relevant retrieved articles about “the effect of environmental and social interventions on children obesity, the uniqueness ratio of PubMed and Cinahl databases was 0% 17% (12). The results of the present study, even though it has different search topic, is close to the results of Woodman et al study.
In general, low overlap ratio of studied databases on Telemedicine field shows that searching multiple databases in these fields is necessary and combination of interdisciplinary databases such as Scopus and Web of science with specific databases in Health field like PubMed and Cinahl increases the retrieval rate by 10-20%.
5. CONCLUSION
Determining the efficiency of information databases in search of different research topics saves time and expense of the researcher. The results of the present study and some other studies showed that in retrieval of different topics in one field even one particular database provides different results. Although finding all articles in a systematic review is difficult, search of three or more databases increases the number of retrieved relevant articles (8). According to the results of the present study, for searching records in Telemedicine, PubMed is significantly prior to the other three databases. This database, with providing free access to bibliographic information and full text of some articles, is a valid source for evidence-based decision taking by Telemedicine specialists and other specialists in fields related to medical informatics. The results of the present study suggest attention to Scopus database and its indexed journals in purchasing information sources for Telemedicine.
However, there are some other criteria that are important in selection of an information database for including in a digital library and also for information search. Some of them are the price, the time distance between ordering and access to the database, ease of ordering and accessing, user friendliness, subject coverage and the number and variety of indexed reports. These criteria have not been considered in our study and are suggested for further studies.
Acknowledgment
This study was supported by the School of Health Management and Information Sciences, Iran University of Medical Sciences (IUMS).
Footnotes
CONFLICT OF INTERESTS: NONE DECLARED.
REFERENCES
- 1.Brettle AJ, Long AF. Comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness. Bulletin of the Medical Library Association. [Comparative Study Research Support, Non-U.S. Gov’t] 2001 Oct;89(4):353–362. [PMC free article] [PubMed] [Google Scholar]
- 2.Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ. 1994;309(6964):1286. doi: 10.1136/bmj.309.6964.1286. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Bahaadinbeigy K, Yogesan K, Wootton R. MEDLINE versus EMBASE and CINAHL for telemedicine searches. Telemedicine and e-Health. 2010;16(8):916–919. doi: 10.1089/tmj.2010.0046. [DOI] [PubMed] [Google Scholar]
- 4.Lohonen J, Isohanni M, Nieminen P, Miettunen J. Coverage of the bibliographic databases in mental health research. Nord J Psychiatry. 2010 May 4;64(3):181–188. doi: 10.3109/08039480903337378. [DOI] [PubMed] [Google Scholar]
- 5.Stevinson C, Lawlor D. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary therapies in medicine. 2004;12(4):228. doi: 10.1016/j.ctim.2004.09.003. [DOI] [PubMed] [Google Scholar]
- 6.Crumley ET, Wiebe N, Cramer K, Klassen TP, Hartling L. Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review. BMC Medical Research Methodology. 2005;5(1):24. doi: 10.1186/1471-2288-5-24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bramer WM, Giustini D, Kramer BM, Anderson P. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev. 2013;2:115. doi: 10.1186/2046-4053-2-115. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. 2013;30:49–58. doi: 10.1111/hir.12009. [DOI] [PubMed] [Google Scholar]
- 9.Betrán AP, Say L, Gülmezoglu AM, Allen T, Hampson L. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Medical Research Methodology. 2005;5(1):6. doi: 10.1186/1471-2288-5-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Armstrong R, Field CH, Victoria AEW, Carlton V, Roberts AH, Oliver S. Systematic reviews in public health. Epidemiology and Demography in Public Health. 2010:222. [Google Scholar]
- 11.Clapton J. Social Care Institute for Excellence. London: 2010. Bibliographic databases for social care searching. [Google Scholar]
- 12.Woodman J, Harden A, Thomas J, Brunton J, Kavanagh J, Stansfield C. Searching for systematic reviews of the effects of social and environmental interventions: a case study of children and obesity. Journal of the Medical Library Association: JMLA. 2010;98(2):140. doi: 10.3163/1536-5050.98.2.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Bahaadini K, Yogesan K. Gap analysis in telemedicine. Journal of Telemedicine and Telecare. 2008;14(6):326–328. doi: 10.1258/jtt.2008.080503. [DOI] [PubMed] [Google Scholar]
- 14.Grigsby J, Sanders JH. Telemedicine: where it is and where it’s going. Annals of Internal Medicine. 1998;129(2):123–127. doi: 10.7326/0003-4819-129-2-199807150-00012. [DOI] [PubMed] [Google Scholar]
- 15.Lawrence DW. What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention. 2008;14(6):401–404. doi: 10.1136/ip.2008.019430. [DOI] [PubMed] [Google Scholar]
- 16.Demiris G, Tao D. An analysis of the specialized literature in the field of telemedicine. Journal of Telemedicine and Telecare. 2005;11(6):316–319. doi: 10.1258/1357633054893328. [DOI] [PubMed] [Google Scholar]
- 17.Roine R, Ohinmaa A, Hailey D. Assessing telemedicine: a systematic review of the literature. Canadian Medical Association Journal. 2001 Sep;165(6):765–771. [PMC free article] [PubMed] [Google Scholar]
- 18.Wong SS-L, Wilczynski NL, Haynes RB. Comparison of top-performing search strategies for detecting clinically sound treatment studies and systematic reviews in MEDLINE and EMBASE. Journal of the Medical Library Association. 2006;94(4):451. [PMC free article] [PubMed] [Google Scholar]