Development and validation of an instrument to assess the knowledge and perceptions of predatory journals

Abstract

Objective

The main aim of this study is to develop a valid and reliable instrument to assess levels of knowledge and perceptions of predatory journals.

Methods

The current study employed successive methods framework including (1) item generation through a literature review and theoretical framework development, (2) validity testing in terms of face, content, and construct validity for perceptions construct as well as item analysis for knowledge scale, and (3) reliability testing in terms of Cronbach's alpha, Kuder-Richardson (KR-20), item-to-total correlations, corrected item-to-total correlations, Cronbach's alpha if item deleted, and test-retest reliability. A total of 304 participants were recruited from King Fahad Medical City (KFMC) in Riyadh, Saudi Arabia to evaluate its construct validity and reliability. This was established using exploratory factor analysis (EFA) with principal axis factoring (PFA) and varimax rotation as well as confirmatory factor analysis (CFA) for perception construct.

Results

An instrument was developed from this study called the “Predatory Journals KP Assessment Questionnaire”. The results of EFA and CFA confirmed the construct validity of the perception construct. Item analysis confirmed the construct validity of the knowledge scale. The internal consistency and test-retest reliability were achieved for the knowledge scale items, consisting of 13 items. The results of EFA confirmed the measured constructs of perceptions toward predatory journals. The results of EFA and CFA for perception construct resulted in only one factor with 9 items.

Conclusion

This study has successfully developed a valid and reliable questionnaire to measure knowledge and perceptions of predatory journals among researchers in the clinical and health disciplines. This instrument serves as a valuable guide for future studies that aim to assess researcher's knowledge and perceptions about predatory journals and examine the differences in these measured constructs according to their demographic and professional characteristics.

1. Introduction

Scholarly and scientific journals serve as tools to communicate research results and disseminate evidence to guide and monitor practice and teaching, as well as to share innovations and new ideas across different disciplines [1]. In clinical research, specifically, published scholarly journal articles provide extensive information to address clinical research questions and are considered an important link between research and practice. The knowledge obtained and disseminated through scientific journals forms the basis for building and developing science in a field [1,2].

In recent years, substantial changes in the publication of articles compared to past decades when they were available only in printed form. As a result, many journals now provide open access to scholarly articles in different disciplines, including clinical and health research, which can be freely accessed online. Other journals offer open-access articles for free after charging the author/s an article processing charge (APC), while some provide articles through institutional subscription or online purchasing formats [[1], [2], [3], [4], [5]].

However, with the growing increase in open-access models, the concept of predatory journals or predatory publishing has emerged [1]. According to Grudniewicz et al., predatory publishers and journals are those entities that prioritize their interests over scholarly values. They often report false or deceptive information, depart from established editorial and publication norms, lack transcripancy, and employ aggressive and indiscriminate solicitation procedures [6]. Predatory journals typically encourage submissions solely to charge a publication fee and offer a fast peer-review publication process sometimes without upholding quality standards, resulting in the publication of low-quality published articles [3]. Predatory publishers may also make false claims about the journal's indexing in reputable bibliographic databases such as PubMed, Web of Science, and Scopus [5].

Narmous efforts and attempts have been made to expose and identify predatory publishers. Jeffrey Beall, a librarian at the University of Colorado-Denver, was among the first scholars to dedicate substantial efforts to identifying such journals. He is considered the most prominent guardian against predatory publishers who maintain a catalog of “potential, possible, or probable” predatory open-access publishers and journals [7]. Nevertheless, the list has been forcibly taken down from the internet and could potentially be outdated [8]. There are over 90 checklists available for authors to identify predatory journals based on various characteristics, including poor presentation or the inclusion of terms like ‘international’ in journal titles. This large number of checklists can be overwhelming for authors. Notably, only three of these lists were developed using research evidence. It is evident from the presence of paywalled lists of quality journals and predatory journals that there is a demand for clear and authoritative guidance. However, these lists often exhibit inconsistencies and are occasionally inaccessible [6,9,10]. Additional efforts include publications in Science Magazine [11], which expose compromised peer-review processes in these journals, expository articles in the New York Times [12], in Nature [13], on publisher websites, and various blogs [14]. However, despite these efforts, many prospective authors, including experienced editors and researchers, remain unaware of these predatory journals [15].

Various studies indicated that predatory publishing poses a more significant issue in low- and middle-income countries [3,16,17]. One study indicated that the majority of corresponding authors originate from Asia followed by Africa and Europe [3]. A study by Xie et al. [16] reported that the majority of authors who contribute to predatory journals are early-career and inexperienced scholars hailing from the developing world. However, a study conducted in Sweden indicated that about two-thirds of researchers published in nursing predatory journals hold senior positions, but this prevalence decreased over time [17].

Previous research attempted to address the researcher's knowledge and awareness of predatory journals. It suggested that various authors, especially those engaged in clinical practice rather than academia, are not aware of predatory journals [2,6,[17], [18], [19], [20], [21], [22], [23]]. Swanberg et al. indicated that identifying predatory journals is a major challenge for researchers. On the other hand, most of these studies were based on interviews with potential authors and editors or self-reporting questionnaires, which were constructed without validation and reliability evaluations [22].

Therefore, there is a need for the development of a valid instrument to assess the researcher's knowledge and perceptions of predatory journals. The current study aims to develop and validate the Predatory Journals Knowledge and Perception (KP) Assessment Questionnaire, which is intended to assess clinical and health researchers' knowledge and perception of predatory journals.

2. Materials and methods

2.1. Study design and population

This is a methodological study conducted between March 2021 and August 2021 at King Fahad Medical City (KFMC) in Riyadh, Saudi Arabia. The population of the current study consisted of researchers identified by the IRB at KFMC and the total population size consists of 550 active researchers population including consultants, assistant consultants, residents, physicians, nurses, and pharmacists. The convenience sampling procedure was employed to select participants for this study.

The survey was distributed to participants via Google Forms for data collection. Participants were recruited and reached through invitations sent to their professional emails and WhatsApp. The eligibility criteria were being a registered researcher in the institutional review board (IRB) list at KFMC, having at least one published article, and accepting participation in the study. A total of 304 participants were included in this study for psychometric analysis. The sample size met the recommended minimum sample size (i.e., 200) for conducting factor analysis [[24], [25], [26]].

2.2. Development of the questionnaire

The development process of the Predatory Journals KP Assessment Questionnaire involved three main steps: item construction and concept identification through a literature review and theoretical framework development, validity testing, and reliability testing. The perceptions construct is reflective, and thus items are believed to represent a single underlying construct.

3. Step 1: item generation via theoretical background and extensive literature review

To establish the theoretical background and create items to measure the constructs of knowledge and perceptions regarding predatory journals among researchers, an extensive literature review was initially performed. The search was conducted using keywords in various formats and combinations such as “predatory journal(s)”, “fake journal(s)”, “predatory publishing”, “knowledge of predatory journals”, “knowledge of predatory publishers”, “perceptions and awareness of predatory journals”, and “perceptions and awareness of predatory publishers”, and was searched in the title, keywords, and abstract headings. Conditions for the inclusion criteria were all scholarly published articles were published research over the period from 2012 to 2020 and in peer-reviewed indexed journals in the Scopus (Elsevier) database and PubMed® library as well as to grey literature showing all of the previously published work in this context without any restriction on the documents. However, only articles written in the English language were retrieved. This step resulted in the identification of some papers that had studied the knowledge and perceptions toward predatory journals [3,14,18,19]. Hence, the first draft of the questionnaire was initially developed and evaluated.

3.1. Step 2: validity testing

Validity testing was conducted to ensure the scale measures what is intended to measure. The validation testing process for this study included face validity, content validity, and construct validity [27,28].

3.2. Face validity

The initial draft of the questionnaire was presented to a group of eligible registered researchers to and identify instrument dimensions and items for evaluation. Subsequently, the face validity process was initiated, involving testing the clarity of all items in the instrument to verify and confirm that it is a valid measure of the concept being studied. This process typically evaluates the questionnaire's appearance in terms of its readability, feasibility, style consistency, and clarity of the language used in its construction, as viewed by experts in the field [[29], [30], [31]]. Seven collaborators were engaged in the face validity process. An evaluation form was used to assess the face validity of the questionnaire whereby the items of the questionnaire were evaluated for feasibility, clarity, language accuracy, layout and style, and potential suggestions for the next version of the questionnaire.

3.3. Content validity

Content validity involves extensive analysis of the questionnaire by a panel of experts or raters familiar with the construct in the area of expertise [[29], [30], [31]]. Ten experts in the area of medical and clinical research from different countries and institutions, possessing knowledge and experience in research and publishing ethics, were invited to review the initial version of the questionnaire and provide their comments and feedback on its content validity. The questionnaire was sent to the raters via email and they were asked to review the instrument for clarity, readability, and comprehensiveness. Moreover, the raters were asked to report their level of agreement with each item using a dichotomous rating, with a score of 1 for “favorable” and a score of 0 for “not favorable”. The rating scores were then summed and averaged to calculate the content validity index (CVI). Specifically, the individual CVI (I-CVI) was used for individual items, and the overall CVI (S-CVI) was used for the total scale in each section. The S-CVI is the average I-CVI, calculated by summing I-CVIs and dividing it by the number of items in each scale [27,28,30]. Previous research suggested that an I-CVI of $\ge$ 0.78 obtained from 6 to 10 raters would be acceptable for inclusion in the questionnaire [31,32]. Therefore, the I-CVI was calculated for all items, and items with I-CVI lower than 0.78 were deleted.

3.4. Construct validity

To establish construct validity, a total of 304 participants were recruited in this study. For the knowledge construct, the item response theory (ITR) suggested that the item difficulty index (IDI) and item discrimination index (DI) can be used to assess the effectiveness of each question in a certain knowledge test [[33], [34], [35]]. In particular, IDI measures the percentage of participants who correctly answered each item. The values of IDI ranged from 0 to 1 such that its higher values indicate the question difficulty is low while its lower values indicate the question difficulty is high. The ideal item difficulty index is 85 % for dichotomous questions (correct/incorrect) and 77 % for three multiple-choice questions [34]. Meanwhile, the item discrimination index (DI) for each item distinguishes between participants who performed well and those who performed poorly on the test. It can also be measured using point-biserial correlations (item-to-total correlation). Items with a point-biserial correlation of <0.20 are typically discarded from the questionnaire [35]. As a general rule, an item with an IDI ranging between 20 % and 80 % and a DI value greater than 0.3 is acceptable and indicative of a good item [36,37].

Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were conducted to establish the construct validity for the perception construct. EFA with principal axis factoring (PAF) and varimax rotation was used to assess the construct validity of the perceptions dimension. This was done to demonstrate the uni-dimensionality of the questionnaire items to ensure that the items measure the same trait [31]. Sampling adequacy was assessed using the Kaiser–Meyer–Olkin (KMO) test, with a value greater than 0.6 indicating adequacy, and Barlett's test of sphericity (p-value <0.001) [38,39]. Furthermore, the determination of how many factors to retain was based on the factor having an Eigenvalue of greater than one [24]. Any item with a factor loading greater than 0.30 or 0.40 was considered acceptable and retained in its respective latent constructs [40]. For CFA, any item with a standardized regression weight (factor loading) less than 0.50 was removed from the scale [27,41]. It is worth noting that for knowledge test questions, both EFA and CFA are not applicable [33].

Psychometric researchers also rely on CFA to demonstrate construct validity [27,41]. CFA in particular, employs several statistical tests to determine the adequacy of model fit to the data. That is, model fit is assessed by investigating a combination of several goodness-of-fit indices, including incremental, absolute, and parsimonious fitness measures [41). In the current study, the following fitness indexes were reported: the chi-square/df (CMIN/DF) which is considered acceptable if it is less than 5 [42]; the comparative fit index (CFI); the goodness of fit index (GFI); incremental fit index (IFI); which are considered acceptable if their values exceeded the cutoff point of 0.90 [[42], [43], [44], [45]]. Moreover, root mean square error of approximation (RMSEA), and standardized root mean squared residuals (SRMR) are considered acceptable if it is lower than the cutoff value of 0.08 to 0.06 (45,46). The collected responses were divided into two equal parts for EFA (n = 152) and CFA (n = 152) for the evaluation process [41]. EFA and CFA were performed for perception construct.

3.5. Step 3: reliability testing

Construct reliability of the current instrument was established using two methods including internal consistency and test-retest reliability. Internal consistency measures the consistency of respondents’ responses across items that are supposed to measure the same underlying construct [46]. Test-retest reliability is used to measure the stability of the construct over time and was established by test-retest reliability [29].

The reliability of the knowledge construct was established using Kuder-Richardson (KR-20) with a cutoff value > 0.60 [48). For the perceptions constructs, the coefficient of internal consistency reliability, namely Cronbach's alpha with a cutoff point of greater than 0.70 for acceptable measure [47]. Meanwhile, Wasserman and Bracken suggested that a cutoff point of >0.60 is acceptable to conclude good internal consistency reliability [48].

For the perceptions scale, item-to-total correlations were used to determine whether to retain or discard each item in the construct, based on the threshold point of 0.50. An item with item-to-total correlations higher than 0.50 was retained in the questionnaire [37,49]. Corrected item-total correlation with a threshold point of 0.40 and Cronbach's alpha if the item was deleted was also used to assess the reliability of the perceptions construct.

For test-retest reliability, this study purposively selected a sample of 80 researchers from different departments at KFMC to evaluate the test-retest reliability of the questionnaire. The retest was implemented four weeks after the initial responses to avoid any recall bias, using the same instrument. out of this group, 52 respondents returned and completed the retest. The intra-class correlation coefficient (ICC) was reported to assess test-retest reliability, where ICC values greater than 0.90 indicate excellent reliability, values between 0.75 and 0.90 suggest moderate reliability, values between 0.50 and 0.75 suggest good reliability and values less than 0.50 indicate weak reliability [50].

3.6. Statistical analysis

Descriptive statistics were expressed as mean and standard deviations for continuous variables and as counts and percentages for categorical variables. The I-CVI and S-CVI were calculated by summing and averaging the ratings provided by experts. For the knowledge construct, the difficulty index and discrimination index were computed to evaluate construct validity. Discrimination analysis was performed for the knowledge dimension after transforming responses into correct (yes)/not correct (no, not sure) responses. EFA with PAF method using varimax rotation as well as CFA with maximum likelihood estimation method was executed to evaluate construct validity for a 5-point Likert scale (i.e., perceptions) construct. Questionnaire reliability was evaluated using internal consistency coefficients (e.g., Cronbach's alpha for perceptions construct, and KR-20 for the knowledge scale), and stability using test-retest reliability. All statistical analysis was performed using SPSS version 26.0 (SPSS Inc., SPSS for Windows, Chicago, USA). Moreover, AMOS v.26 was used to execute CFA for practices and perception constructs.

3.7. Ethical consideration

Ethical approval was obtained from the KFMC institutional review board. Participants were recruited into the study voluntarily, and they provided written informed consent after they had read, understood, and signed it. The data were analyzed anonymously.

4. Results

4.1. Step 1: item generation

The authors conducted a comprehensive literature review to generate items and develop constructs for knowledge and perception regarding predatory journals as detailed in the methods section. A total of 722 records were identified in the Scopus database, and 689 records were found in the PubMed® library over the period 2012–2020. Fig. 1 shows the publication trend regarding predatory journals in both the Scopus database and PubMed® library. Most published contents were found in the last four years (2017–2020), constituting approximately 75.0 % of all retrieved documents from both sources. Original articles based on quantitative data analysis accounted for less than one-half of all retrieved records (43.7 %), while the remaining records consisted of opinion-based documents (e.g., letters to the editor, notes, and reviews), conference papers, book chapters, or short surveys.

Fig. 1.

Number of retrieved documents regarding predatory publishing in the Scopus database and PubMed® library over the period 2012–2020.

This step resulted in the identification of some papers that had studied the knowledge and perceptions related to predatory journals [3,14,18,19,23,[51], [52], [53], [54], [55], [56]]. However, most previous studies used self-administered questionnaires and varied widely in terms of questionnaire type. Specifically, in most studies, perception dimensions were assessed using the Likert scale, with different scaling types.

Hence, the questionnaire for the current study was initially drafted by the principal investigator and was subsequently refined through collaboration with all coauthors, all of whom possessed previous knowledge regarding predatory publishers. The questionnaire contains three parts including demographic and professional data of the study participants, knowledge construct, and perception construct. The demographic part contains demographic and professional information about participants.

The knowledge construct included 18 questions related to predatory journals, each offering three response options: “yes”, “no”, and “don't know”. The items underlying this construct were generated utilizing various previous research studies [3,14,18,19,23,[51], [52], [53]].

The perception construct consisted of 10 items rated on a 5-point Likert (i.e., if it was a positive statement, 1 = strongly disagree. 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree; if it was a negative statement, 1 = strongly agree, 2 = agree, 3 = neutral, 4 = disagree, 5 = strongly disagree). The items underlying this latent construct were generated utilizing various previously published related papers [14,18,23,[51], [52], [53],56].

Step 2: Validity testing.

4.2. Face validity

The Predatory Journals KPP Assessment Questionnaire underwent a thorough testing phase, involving a focus group of researchers and experts. This testing aimed to evaluate the questionnaire's readability, comprehensiveness, understandability, and clarity. Several issues were identified during this phase, including problems related to question clarity, response categories, knowledge assessment, the sensitivity of content, instructions, and formatting. Specifically, question 7 in the knowledge scale was identified as needing rewording for improved clarity. Furthermore, most respondents suggested splitting question 8 in the demographic information section into two questions to ensure clarity about the author's publication and whether they have published as a corresponding author or not. Furthermore, the majority of respondents suggested making options for the splintered questions to be as the following: none, 1, 2, 3, or more. These suggestions were discussed with a panel of experts and approved. This enabled reformulating confusing questions and determining the number of problematic items. As a result, the questionnaire was drafted and contained 36 questions comprising 8 questions on demographic characteristics, 18 questions on knowledge, and 10 questions on perceptions about predatory publishers and sent for experts to rate the questionnaire for content validity.

4.3. Content validity

Questionnaire sections on knowledge and perceptions of predatory publishers and journals underwent a thorough review by a panel of 10 experts. The detailed content validity procedure is presented in Table 1 for the knowledge construct and in Table 2 for the perception construct. In the knowledge scale, four questions were discarded because they were irrelevant and repetitive and some of them were not appropriate for the knowledge test. Regarding the perceptions construct, one item was discarded because of low agreement among raters, the CVI is less than 0.78.

Table 1.

The results of the expert validation test for the knowledge scale.

Item	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 6	Expert 7	Expert 8	Expert 9	Expert 10	Number in Agreement	Item CVI
1	X	–	–	X	–	–	X	X	–	X	5	0.5
2	X	X	–	X	X	X	–	X	X	X	8	0.8
3	X	X	X	–	X	X	X	X	X	X	9	0.9
4	–	X	X	X	X	X	X	X	–	X	8	0.8
5	–	–	–	X	X	X	–	X	X	X	6	0.6
6	–	–	X	–	X	X	X	X	–	X	6	0.6
7	–	X	X	–	X	X	–	X	X	–	6	0.6
8	X	X	X	X	–	X	X	X	X	X	9	0.9
9	X	X	X	–	X	X	X	–	X	X	8	0.8
10	X	X	X	X	X	X	X	X	X	X	10	1
11	X	–	X	X	X	–	X	X	X	X	8	0.8
12	X	X	–	X	X	X	X	–	X	X	8	0.9
13	X	X	X	X	X	X	X	X	X	X	10	1
14	X	X	X	X	X	X	–	X	X	X	9	0.9
15	–	X	X	X	X	X	X	X	X	–	8	0.8
16	X	X	X	X	X	X	X	X	X	X	10	1
17	X	X	X	–	X	X	–	X	X	X	8	0.9
18	–	–	X	X	–	–	X	X	–	–	4	0.6
Average I-CVIs (S-CVI) Bolded items were removed												0.800

Table 2.

The results of the expert validation test for perceptions scale.

Item	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 6	Expert 7	Expert 8	Expert 9	Expert 10	Number in Agreement	Item CVI
1	X	X	X	X	X	–	X	X	X	X	9	0.90
2	–	X	–	–	X	–	–	X	X	X	5	0.50
3	X	X	X	X	X	X	–	–	X	X	8	0.80
4	X	X	X	X	X	X	X	X	X	X	10	1.00
5	X	X	X	X	X	X	X	X	X	–	9	0.90
6	X	X	X	X	X	X	X	X	X	X	10	1.00
7	X	X	X	X	X	X	X	X	X	X	10	1.00
8	X	–	X	X	–	X	X	X	X	X	8	0.80
9	X	X	X	X	X	X	X	X	–	X	9	0.90
10	X	X	X	X	X	X	X	X	X	X	10	1.00
Average I-CVIs (S-CVI)												0.855

The global content validity index (CVI) of the questionnaire (S-CVI) was 0.838, which is higher than the value defined as acceptable according to the criteria established in the scientific literature (i.e., 0.78). Therefore, This ratification by the panel of experts established the questionnaire as valid in terms of its structure, content, and language, with only minor revisions. Of the original 36 items, 6 items were deleted 8 items were revised, and one question split into two questions. Thus, the final questionnaire consisted of 31 questions and items at the end of face and content validity. Therefore, the final questionnaire consisted of three sections such that the first section has 9 questions on demographic and professional characteristics, the second section consists of 13 items on knowledge, and the third section consists of 9 items on perceptions towards predatory journals.

4.4. Construct validity

In this stage, a total of 304 participants responded to the Predatory Journals KP Assessment Questionnaire. Before we carried out the construct validity testing, this study reported the descriptive statistics of the participants and represented them in Table 4. The average age was 35.9 ± 9.3 years. The mean years of experience were 10.4 ± 3.8 years. The distribution of the sample by gender was almost similar (50.7 % males and 49.3 % females). As for professional occupation, eighty-one (26.6 %) were senior consultants, 21 (6.9 %) were assistant consultants, 14 (4.6 %) were fellows, 43 (14.1 %) were residents, 23 (7.6 %) were pharmacists, 21 (6.9 %) nurses, and 101 (33.2 %) did not belong to any of the above-mentioned groups. Almost all of them were educated with at least a Bachelor's degree (n = 295, 98.0 %). The majority of participants did not receive prior training regarding predatory publishing (n = 225, 74.0 %). Moreover, 25.4 % of participants had published at least one article (n = 77) and 30.0 % of them served as corresponding authors (n = 70). No issues of nonresponse and missing items were encountered (Table 3).

Table 4.

The results of item analysis for the knowledge construct.

No.	item	IDI	DI	r
K.1	Predatory journals target academicians for financial profit via article processing charges for open-access articles, without checking the academic rigour of the articles nor meeting scholarly publishing standards.	0.68	0.33	0.35
K.2	Predatory journals are journals that follow the standard policies advocated by organizations such as the World Association of Medical Editors (WAME), or International Committee of Medical Journal Editors (ICMJE), the Open Access Scholarly Publishers Association (OASPA), the Committee on Publication Ethics (COPE), and Council of Science Editors (CSE)	0.62	0.31	0.28
K.3	Predatory journals are referring to quality publication processes that have sound peer-review practices or a solid basis in publication ethics	0.63	0.33	0.38
K.4	The scope of interest of predatory journals is vague and not focused	0.59	0.43	0.44
K.5	Predatory journals follow the peer-review publication process.	0.55	0.34	0.34
K.6	Predatory journals have a very short time frame between submission and publication.	0.61	0.45	0.49
K.7	Predatory journals falsely list editorial boards.	0.42	0.57	0.68
K.8	They provide no institutional affiliations and falsely claim to be indexed in any legitimate database	0.47	0.53	0.52
K.9	Predatory journals use fake impact factors	0.52	0.53	0.56
K.10	They lack transparency or honesty regarding the ownership and headquarter location of the publisher of predatory journals.	0.58	0.53	0.53
K.11	They send many spam Emails	0.54	0.41	0.54
K.12	They follow scholarly publishing industry best practices.	0.55	0.47	0.44
K.13	Their article Processing Fees (APF) is unclear	0.42	0.38	0.53
r: point bi-serial correlation or item-to-total correlations

Table 3.

Demographic and professional characteristics of the study participants.

Variable	Measure	Total sample (n = 304)	EFA sample (n = 152)	CFA sample (n = 152)	p-value
Age	Mean $\pm$ SD (years)	35.9 $\pm$ 9.3	36.8 $\pm$ 9.4	35.0 $\pm$ 9.1	0.091
Experience	Mean $\pm$ SD (years)	10.4 $\pm$ 3.8	10.2 $\pm$ 3.7	9.6 $\pm$ 3.4	0.142
Gender, n (%)	Male	154 (50.7)	76 (50.0)	78 (51.3)	0.819
	Female	150 (49.3)	76 (50.0)	74 (49.7)
Educational level, n (%)	Diploma	6 (2.0)	3 (2.0)	3 (2.0)	0.299
	Bachelor	111 (36.5)	53 (34.9)	58 (38.1)
	Masters	61 (20.1)	25 (16.4)	36 (23.7)
	Doctorate	59 (19.4)	31 (20.4)	28 (18.4)
	Subspecialty/Fellowship	67 (22.0)	40 (26.3)	27 (17.8)
Professional occupation, n (%)	Consultant	81 (26.7)	44 (28.9)	37 (24.3)	0.930
	Assistant consultant	21 (6.9)	11 (7.2)	10 (6.6)
	Fellow	14 (4.6)	8 (5.3)	6 (3.9)
	Resident	43 (14.1)	21 (13.8)	22 (14.5)
	Pharmacist	23 (7.6)	10 (6.6)	13 (8.6)
	Nurse	21 (6.9)	11 (7.2)	10 (6.6)
	Other	101 (33.2)	47 (31.0)	54 (35.5)
Received prior training, n (%)	Yes	51 (16.8)	24 (15.8)	27 (17.7)	0.834
	No	225 (74.0)	113 (74.3)	112 (73.7)
	Not sure	28 (9.2)	15 (9.9)	13 (8.6)
No. of publications last 3 years, n (%)	None	115 (37.8)	57 (37.5)	58 (38.2)	0.800
	1	77 (25.4)	39 (25.7)	38 (25.0)
	2	39 (12.8)	17 (11.1)	22 (14.5)
	3 or more	73 (24.0)	39 (25.7)	34 (22.3)
No. of publications as a corresponding author in last 3 years, n (%)	None	155 (51.0)	76 (50.0)	79 (52.0)	0.413
	1	70 (23.0)	38 (25.0)	32 (21.1)
	2	38 (12.5)	15 (9.9)	23 (15.1)
	3 or more	41 (13.5)	23 (15.0)	18 (11.8)

The samples of EFA and CFA were similar in terms of basic demographic characteristics and did not show any statistically significant differences (p-value >0.05).

Of the 13 questions in the knowledge scale, the results of the item analysis using IDI, DI, and point biserial correlation (r) of the retained items from the content validity were reported in Table 5. The results indicate that IDIs ranged from 0.42 to 0.68 and DIs ranged from 0.31 to 0.57, which are higher than their respective threshold. Moreover, all items were greater than the cutoff point of 0.2 for point biserial correlation. Therefore, all items included in the item analysis were retained in the questionnaire, and none of them were discarded. Consequently, the 13-question knowledge scale was determined to be valid as indicated in Table 4.

Table 5.

Exploratory factor analysis results of items in the perception construct.

No.	Item	Factor loadings
		Factor 1
A.1	Predatory journals are exclusively based in the developing world	0.620
A.2	Predatory journals charge no publication fees	0.608
A.3	Predatory journals deceive young and inexperienced authors	0.725
A.4	Predatory journals attract frustrated academics wanting speedy promotions	0.881
A.5	Predatory journals are difficult to distinguish from legitimate open-access journals	0.745
A.6	Predatory journals view open-access medical publishing as a major market. They take authors' money but provide no standard peer review or genuine indexing to disseminate research findings	0.824
A.7	Knowing about predatory journals is not currently a priority for me	0.669
A.8	The academic and clinical community should leave predatory journals to survive or perish by themselves.	0.786
A.9	The academic and clinical community should improve research in the developing world about how to be published in good quality international literature.	0.752
Eigenvalue		4.904
Percentage of Total Variance Explained (%)		49.1 %
KMO		0.775
Bartlett's Test of Sphericity, Chi-square		691.137
Bartlett's Test of Sphericity, p-value		<0.001

As for the perceptions construct, the results of EFA with PAF and varimax rotation are shown in Table 5. The results justified the applicability of performing EFA as the KMO was 0.775 and Bartlett's test of sphericity was statistically significant (p-value <0.001) exhibiting sampling adequacy. The EFA revealed that only one factor with an Eigenvalue greater than 1.0. This factor explained 49.1 % of the total variance in this latent construct. All factor loadings were higher than the cutoff point of 0.30 and therefore all items were retained in the construct.

Following EFA, the current study performed CFA in a different sample for the perception construct. The results of the CFA confirmed the findings of EFA that the perception construct has 9 items. All standardized regression weights exceeded the cutoff value of 0.50. The fitness indexes showed a good fit of the latent construct such that RMSEA (0.056 < 0.08), SRMR (0.058 < 0.08), CFI (0.913 > 0.90), GFI (0.926 > 0.90), IFI (0.942 > 0.90), the ratio of CMIN/DF was 2.942 signifying a good fit of the study data as shown in Fig. 2. As a result, the 9 items perception construct was evaluated as valid.

Fig. 2.

CFA results of Perceptions Construct (standardized regression weights).

4.5. Step 3: reliability testing

The KR-20 reliability coefficient of the knowledge construct was 0.91, indicating higher internal consistency reliability and the items successfully measured in the knowledge construct. This indicates that the items within the knowledge construct effectively measured the intended concepts. As a result, the 9-item knowledge scale was evaluated as reliable.

The coefficient of internal consistency, namely Cronbach's alpha, for the perceptions construct was 0.71, and was higher than the cutoff point of 0.70, which indicates that the items fall under the perceptions construct measured by the respective constructs. All items surpassed the cutoff value of 0.50 for item-total correlation coefficients and the cutoff value of 0.40 for corrected item-total correlation coefficients indicating that the two constructs have good reliability as shown in Table 6.

Table 6.

The reliability results for perception constructs (n = 304).

Construct	Item	Item-Total Correlation	Corrected Item-Total Correlation	Cronbach's Alpha if Item Deleted	Cronbach's alpha
Perception	A.1	0.52	0.47	0.58	0.71
	A.3	0.51	0.46	0.62
	A.4	0.52	0.48	0.58
	A.5	0.62	0.57	0.62
	A.6	0.61	0.56	0.55
	A.7	0.52	0.58	0.58
	A.8	0.63	0.59	0.64
	A.9	0.61	0.58	0.61
	A.10	0.64	0.60	0.67

As for test-rest reliability analysis, the findings showed that the coefficient that the intra-class correlation coefficient for the knowledge scale was 0.54 (p-value <0.001) indicating moderate reliability. The intra-class correlation coefficient and the perceptions construct was 0.76 (p-value <0.001), showing good reliability as indicated in Table 7.

Table 7.

The results of test-rest reliability analysis.

Construct	Intra-class correlation coefficient	95 % confidence interval	p-value	Status
Knowledge	0.54	[0.48, 0.59]	<0.001	Moderate
Perception	0.76	[0.72, 0.81]	<0.001	Good

5. Discussion

During the process of disseminating research and scholarly articles, researchers are exposed to being victims of predatory journals, especially in clinical settings. The development of an effective instrument for assessing researchers’ knowledge and perceptions of predatory publishers and journals can provide significant information to the formulation of effective publication management policies. Beshyah et al. reviewed most journals in the medical research literature on predatory publishing [57].

The current study attempted to develop and validate a measurement tool to raise awareness about predatory journals among researchers, which was based on following successive phases. This included the development of the theoretical framework through an depth literature review that helped in demonstrating the particularities prevailing research publication process and dissemination. Furthermore, this study employed a qualitative study to face and value the questionnaire to ensure its readability, comprehensiveness, and clarity through discussion with potential researchers and some experts in the area of medical and clinical research. The inclusion of ten experts in the content validity contributes to a better version of the questionnaire in terms of content and usability. The items of the questionnaire were continuously revised resulting in a pre-final version to be tested for validity and reliability. Accordingly, the findings of face validity showed an appropriate formulation of the items with small revisions required. The overall content validity index of the questionnaire, S-CVIs for each construct was higher than the acceptable standard criteria reported in the literature [32]. Furthermore, the scaling procedure for raters was carried out by employing a dichotomous rating scale that does not allow for chance agreement among raters. Therefore, it is considered a robust measure of content validity and thus eliminates inconsistency and permits direct explanation [31,32].

Furthermore, the current study included a quantitative study to check the measurement tool for its validity and reliability, which is an important process in tool development to ensure its usability to assess researchers' knowledge and perceptions of predatory journals in the clinical field. This study showed that knowledge items were valid and reliable but there is a gap in participants’ levels of knowledge on predatory publishers as shown by IDI and DI for each item in this scale. This knowledge gap might be attributed to the fact that most respondents had experience in clinical practice rather than academia [[1], [2], [3], [4],14,18,[21], [22], [23]]. This gap may also be considered a barrier to selecting appropriate journals and avoiding such predatory publishers. Therefore, researchers are strongly advised to consistently validate the credentials of journals via reputable sources such as the Committee on Publication Ethics (COPE), Directory of Open Access Journals (DOAJ), and Web of Science. Hence, efforts may also be placed into identifying and validating trusted scholarly journals and publishers [5,6,[14], [15], [16], [17],57].

As for the perceptions construct, the EFA resulted in only one factor. The findings of CFA indicated appropriate fit as evidenced by the fitness indexes and showed that the construct is valid. The reliability of this construct was achieved and showed high internal consistency and good test-retest reliability. Meanwhile, the knowledge construct showed moderate reliability as measured by test-retest reliability. The quantitative results obtained from this study were congruent with several previous findings [14,18,19,22,23,[51], [52], [53], [54], [55], [56]]. For example, Beshyah et al. indicated that less than one-third of physicians were fully aware of predatory publishing while about two-thirds of them did not know Beall's list showing that participants had low awareness of predatory journals as well as they showed that participants had varied attitudes and practices among them [54). Richtig et al. found that predatory journals were known by more than two-thirds (69.7 %) of medical oncologists from Germany and Austria and Beall's list by 27.7 % of them [53]. However, this is the first scale developed in the literature, to the best of the author's knowledge.

As a result, the current questionnaire can be a useful tool to improve the researcher's knowledge and perceptions of predatory publishers, especially clinical and healthcare practitioners. Hence, the current instrument can be a useful tool in the future to ease a better understanding of predatory publishers. Furthermore, the information obtained from data analysis can set the seeds for formulating relevant policies to inform researchers regarding predatory publishers with credible bases for avoiding being a potential victim of such publishers.

The current study followed rigorous methodology during the development of the current questionnaire including the extensive literature review, face validity based on a focus group of some researchers and experts, integration of ten panels of experts to assess the content validity, EFA and CFA to evaluate construct validity and reliability analysis could be considered as a major strength of this study. This scale sheds light on an important international problem and can be performed in different countries and institutions with some modifications.

Nevertheless, this study has a few significant limitations for the current study. First, the data were collected through online platforms, which limited the generalizability. Even though the findings of the current study confirmed the psychometric properties of the instrument, similar studies in healthcare hospital-based and other communities are required to ensure generalizability. Second, the data were collected based on self-reported measures, and thus responses may be exaggerated, participants may be regretful to disclose private details, and biased. Third, the current literature lacks the availability of a similar scale and thus convergent validity was not evaluated since this is the first developed scale, to our knowledge. it was only conducted on and thus the results could not have generalized to other categories of researchers. Finally, this study did not encompass practices for identifying predatory journals and strategies for recognizing reputable publishers. Hence, additional research is required to establish best practices for assessing trustworthy scholarly journals and avoiding predatory ones. For example, a formative practice construct could be developed and validated.

To conclude, the current study has successfully developed a valid and reliable questionnaire to measure knowledge and perceptions of predatory journals among researchers in the clinical and health disciplines. This Predatory Journals KP Assessment Questionnaire serves as a valuable guide for future studies that aim to assess researchers' knowledge and perceptions of predatory journals and examine the differences in these measured constructs according to their demographic and professional characteristics. Finally, this tool can be used to provide valuable information in designing educational programs or training interventions to raise researchers’ knowledge and awareness regarding predatory journals and publishers.

Data availability

Data will be made available upon request.

Funding

This project was funded by King Fahad Medical City, Riyadh Second Health Cluster, Riyadh, Saudi Arabia. The funder did not have any role in the design of the study; the collection, analysis, or interpretation of the data; or the writing of the manuscript.

CRediT authorship contribution statement

Sumayyia Marar: Writing – original draft, Supervision, Methodology, Conceptualization. Muaawia A. Hamza: Writing – review & editing, Writing – original draft, Resources, Project administration, Methodology, Conceptualization. Mohsen Ayyash: Writing – review & editing, Writing – original draft, Methodology, Formal analysis. Amani Abu-Shaheen: Writing – review & editing, Writing – original draft, Methodology, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article.