Risk of Bias Assessment in Non-Randomized Studies of Interventions for Premenstrual Syndrome: A Systematic Review

Abstract

Objective: Premenstrual syndrome (PMS) is a very prevalent condition that affects premenopausal women and can result in monthly debilitating emotional and physical symptoms. The objective of this systematic review was to determine which predictive factors were associated with an increased amount of bias in non-randomized studies (NRSs) of PMS.

Materials and methods: A search of the EMBASE and Medline electronic databases was completed from January 1, 2010 to December 2021. The methodological quality of the included studies was independently evaluated and critically appraised using the Risk of Bias in Non-Randomized Studies - of Interventions (ROBINS-1) tool. Associations of different factors with the risk of bias levels were assessed using a univariate logistic regression. Odds ratio and 95% confidence interval (CI) were reported.

Results: Of the 1668 studies, 38 were determined to be eligible for inclusion. The ROBINS-1 tool identified that 12 studies were of low/moderate risk of bias (31.6%) and 26 were of serious/critical risk (68.4%). Evidence of relationships between the ROBINS-1 score and impact factor (OR=0.20; 95% CI, 0.07 to 0.57; p= 0.003) and number of authors (OR=0.65; 95% CI, 0.43 to 0.99; p= 0.046) were identified, whereas no relationships were found with the number of citations, the sample size, the funding type, or the conflict-of-interest statement.

Conclusion: The systematic review concludes that the methodological rigor of non-randomized studies of PMS can vary, with fewer authors and a lower impact factor showing evidence of association with a decreased quality of evidence.

Introduction

Premenstrual syndrome (PMS), a condition that occurs monthly during the luteal phase of the menstrual cycle, is characterized by a series of recurrent emotional and physical symptoms (1). Typical symptoms of PMS include anger, irritability, depression, abdominal bloating, tenderness in the breasts, fatigue, and anxiety (2). These symptoms can be further exasperated in the severe form of PMS, premenstrual dysphoric disorder (PMDD) (2). PMS first appears in women post-puberty and can substantially impair work, social and family relationships until its resolution with menopause (3). Although the global prevalence of PMS is estimated to be 30-40% (3-8% for PMDD), a 2017 cross-sectional study published by Jarosz et al. revealed that 99% of young multi-ethnic Canadian women suffer from some type of premenstrual symptom (3-4). However, the prevalence of PMS and the impacts of its debilitating symptoms are often under-recognized by many laypeople, workers, governmental organizations and practicing healthcare professionals (3). Even Halbreich et al. noted recently (2004) that new diagnostic criteria for PMS are desperately needed (5). This was attributed to the lack of universal consensus on the nature of PMDD and PMS and the lack of interdisciplinary and universal acceptance of diagnostic criteria at the time (5).

Evidence of the effectiveness of PMS interventions is predominantly from non-randomized studies (NRSs). Many studies on PMS interventions use NRSs as they aim to study the effects between a certain patient group — women who suffer from PMS — and a control. Consequently, the existing evidence does not use randomization to eliminate systematic baseline differences between experimental and control groups (6). Potential biases are thus greater for NRSs as compared to RCTs (7) and NRSs have been found to overestimate the effect of intervention as compared to the results from similar RCTs (8). Fortunately, not all NRSs suffer from a significant amount of bias, and when carefully designed yield valid results (9). Although a wide range of systematic reviews has been conducted on PMS, no studies have explicitly investigated the presence of bias; this is a vital addition where evidence is predominately from non-randomized studies. Furthermore, no systematic reviews of NRSs have yet to be conducted in the context of PMS. Only through contextualizing the evidence and its bias for intervention into PMS can clinicians and policy decision makers confidently act on these data.

The objective of this systematic review is to determine which factors are associated with an increased amount of bias in non-randomized studies of PMS. These predictors include the impact factor of the journal, the number of citations, the source of funding, and the authors’ conflicts of interest statement. The a priori hypothesis for this systematic review is that the authors’ conflicts of interest, a lower journal impact factor, a lower number of citations, and studies that received funding from industry sources will be associated with a greater risk of bias.

Materials and methods

The protocol of this systematic review was conducted according to the guidelines established by the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement (10). The study was further pre-registered in the International Prospective Register of Systematic Reviews (PROSPERO) with the reference number (CRD42020218896).

Search Strategy and Selection criteria : A comprehensive search of the EMBASE and Medline electronic databases was completed from January 1, 2010 to December 2021. The search terms used included “premenstrual syndrome”, “premenstrual tension”, “PMS”, “treat*”, “therap*”, “interven*”, “surg*”, “drug*”, “perscri*”, and “medicat*”. The search strategy was developed with the assistance of a medical librarian and is available in the appendix (Appendix S1).

Publications based on the search strategy were uploaded to Covidence; a systematic review software (www.covidence.org). The abstracts and titles of all papers were screened independently by four reviewers (R.C, S.L, S.S, P.K.), each two in duplicate. Disagreements between reviewers were resolved through a discussion with the primary author (H. T.) to confirm eligibility. To be included in this systematic review, publications were required to meet the following inclusion criteria: (1) non-randomized studies (cohort, case-control, and quasi-randomized studies); (2) trials relevant to PMS (self-identified in the title or abstract); (3) interventions involving a treatment, procedure or drug. Articles were excluded if they met the following exclusion criteria: (1) non-English literature; (2) non-human studies (animal or cadaver); (3) review articles, case reports, and poster abstracts.

When the inclusion criteria were unclear based on the abstract and title alone, the study was included for full-text review. Articles selected for full-text review were once again screened independently by two reviewers, in duplicate. The publications were scrutinized using the same inclusion and exclusion criteria mentioned above. However, papers excluded at this level of the study also had a corresponding reason for exclusion. In the case of disagreements, a discussion between the two reviewers and the primary author (H. T.) took place.

Data Extraction: For all studies included following the full-text review, each paper was extracted independently by two reviewers, in duplicate. Data extracted included the journal name, journal impact factor from Clarivate Analytics, year of publication, total citations from Google Scholar, total number of authors, source of funding (government, institutional, philanthropic, industry), patient number, intervention in the active and control groups, the primary outcome, and conflict of interest statement. For data management purposes, data was extracted using Microsoft Excel (Redmond Wash.).

Quality Assessment: Each paper was assessed for risk of bias independently by two reviewers, in duplicate (4 reviewers in total). The ROBINS-1 tool — a tool developed to assess risk of bias in non-randomized studies of interventions — was used to assess the quality of papers included in this systematic review (11). Bias due to confounding, selection of participants into the study, classification of interventions, deviations from intended interventions, missing data, measurement of outcomes, and selection of the reported result were the domains used by the ROBINS-1 tool to assess study quality. The total score of the scale can be identified as one of four quality ratings: low, moderate, serious and critical risk of bias.

Statistical Analysis: All statistical analyses were performed using the R statistical language (version 4.0.3) (12). Descriptive statistics were calculated using median and interquartile range for continuous and frequency and percentage for categorical variables. The outcome of interest is the overall risk of bias; the tool classifies the studies into low risk, moderate risk, serious risk, or critical risk. For easier interpretation and analysis, the studies were regrouped into two levels; low/moderate and serious/critical risk. We examined the association of these two levels of the risk category with each of the following factors: sample size, number of authors, number of citations, impact factor, conflict of interest (yes/no), and source of funding (non industrial, industrial, or not reported) using univariate logistic regression. We assessed the assumptions of the logistic regression using plots and diagnostic testing. All assumptions were met. Odds ratio and 95% confidence interval (CI) of the regression were reported. Correlation between these different factors was also assessed using Spearman’s rank order correlation.

Results

Search Results: Following the removal of 459 duplicates, 1668 abstracts and titles were preliminarily screened. Of these studies, 66 full-texts were further assessed for eligibility, whereas 38 were determined to be eligible for inclusion (13-51). Search results have been summarized in a PRISMA flow chart (Figure 1).

Figure 1.

PRISMA flow diagram of screened, included or excluded studies

Study Characteristics: The study characteristics have been summarized in Table 1.

Table 1.

The description of PMS study characteristics published between (2010, 2021) which met the eligibility criteria (n=38)

Variable	Value
Sample size of the study; median (IQR)	39.0 (23.2, 77.2)
Number of Authors; median (IQR)	5.0 (4.0, 6.0)
Number of citations; median (IQR)	8.0 (2.2, 20.0)
Impact factor; median (IQR)	1.6 (0.8, 2.4)
Funding source; n (%)
Not industrial	14 (36.8)
Industrial	11 (28.9)
Not Reported	13 (34.2)
Conflict of interest; n (%)
Yes	13 (34.2)
No	25 (65.8)

IQR= Interquartile range

Fourteen studies (36.8%) received government, philanthropic, and institutional funding, while 13 studies (34.2%) did not report a source of funding. Eleven studies (28.9%) received funding from industrial sources. No conflict of interest was reported by the majority of studies (n=25; 65.8%), with the remaining articles including a clear conflict of interest statement (n=13; 34.2%). The primary outcome most frequently evaluated was the reduction of PMS symptoms (n=19; 50%), with the other studies reporting alternative primary outcomes.

Quality Assessment: The methodological quality of the included non-randomized studies was evaluated using the ROBINS-1 tool. Guided by the seven domains of the ROBINS-1 tool, overall risk of bias score of low, moderate, serious or critical was given to each study (appendix S2). Of the 38 studies meeting inclusion, 2 were of low risk (n=2; 5.3%), 10 were of moderate risk (n=10; 26.3%), 22 were of serious risk (n=22; 57.9%), and 4 were of critical risk (n=4; 10.5%). The most common domains which resulted in a downgrade in the risk of bias score were biased due to confounding, missing data, and bias in the measurement of outcomes. To increase the sample size for analysis, the studies were regrouped into two levels; low/moderate and serious/critical risk; 12 studies were of low/moderate risk (n=12; 31.6%) and 26 were of serious/critical risk (n=26; 68.4%).

Factors associated with the Risk of bias : The distribution of characteristics of studies based on methodological quality (ROBINS-1 tool) is summarized in Table 2, with the results of the logistic regression shown in Table 3.

Table 2.

The distribution of characteristics of studies based on methodological quality (ROBINS-1 tool)

Variable	n	low/moderate (n=12)	serious/critical (n=26)
Sample size, median (IQR)	38	58.00 (28.50, 87.00)	37.50 (23.00, 76.00)
Number of authors, median (IQR)	38	6.00 (4.00, 7.00)	4.00 (3.00, 5.00)
Number of citations, median (IQR)	38	12.00 (4.75, 34.25)	7.50 (2.00, 17.00)
Impact factor, median (IQR)	38	2.72 (1.97, 3.28)	1.05 (0.45, 1.70)
Source of funding, n (%)	38
Not industrial		7 (58.3)	7 (26.9)
Industrial		4 (33.3)	7 (26.9)
Not reported		1 (8.3)	12 (46.2)
Conflict of interest, n (%)	38
No		7 (58.3)	18 (69.2)
Yes		5 (41.7)	8 (30.8)
Year of publication-numeric, median (IQR)	38	2013.50 (2011.00, 2014.75)	2015.00 (2013.00, 2017.00)
Year of publication categorical, n (%)	38
2010		2 (16.7)	3 (11.5)
2011		2 (16.7)	1 (3.8)
2012		1 (8.3)	1 (3.8)
2013		1 (8.3)	4 (15.4)
2014		3 (25.0)	2 (7.7)
015		0 (0.0)	4 (15.4)
2016		0 (0.0)	3 (11.5)
2017		1 (8.3)	6 (23.1)
2018		2 (16.7)	2 (7.7)

Table 3.

The logistic regression results for serious/critical risk of bias vs low/moderate risk of bias (n=38)

Variable	Univariate coefficient	Univariate OR (95% CI)	P value
Sample size of the study	0.002	1.00 (1.00, 1.01)	0.520
Number of authors	-0.43	0.65 (0.43, 0.99)	0.046
Number of citations	-0.04	0.96 (0.92, 1.01)	0.089
Impact factor	-1.60	0.20 (0.07, 0.58)	0.003
conflict of interest (Yes vs No)	-0.47	0.62 (0.15, 2.57)	0.512
Source of funding (Reference= Not industrial)			0.105
Industrial	0.56	1.75 (0.35, 8.80)	0.497
Not reported	2.49	12.00 (1.21, 118.89)1	0.034

¹ Very wide confidence interval (imprecise estimate because of the low sample size)

The results of the univariate regression analysis showed that an increase in the number of authors by one would decrease the odds of serious/critical risk of bias by 35% (OR=0.65; 95% CI, 0.43 to 0.99; p=0.046). The impact factor also showed an evident association, where a one unit increase in the impact factor will decrease the odds of serious/critical risk by 80% (OR=0.20; 95% CI, 0.07 to 0.57; p= 0.003). The other factors did not show evidence of association with the risk of bias levels. Although the overall association of the source of funding with the risk of bias was not evident significant (p>0.05), those who did not report any source of funding did show an association with the risk of bias. However, this association showed a wide confidence interval due to the small sample size (n=13) (OR=12.00; 95% CI, 1.21 to 118.89, p=0.03); due to the wide interval generated from a small sample, we are cautious not to interpret this finding.

Discussion

This is the first systematic review to assess the risk of bias in non-randomized studies of interventions for PMS. We hypothesized that the authors’ conflicts of interest statement, a lower journal impact factor, a lower number of citations, and studies that received funding from industry sources will be associated with a greater risk of bias. Our findings revealed that a lower journal impact factor was in fact associated with a greater risk of bias. Reassuringly, the number of citations, conflict of interest statements, and funding from industry sources were not associated with a greater risk of bias. Beyond the a priori hypothesis, an association between the number of authors and the risk of bias was found.

RCTs are recognized as the highest level of evidence in clinical medicine for non-review studies, playing a crucial role in informing policy generation, clinical practice, and patient care (8). However, randomization is not always possible - particularly relevant in the context of PMS - impacting the ability to use evidence to guide evidence-based clinical care. Fortunately, not all NRSs suffer from a significant amount of bias, and when carefully designed, can imitate randomization and yield valid results. The ROB assessment performed in this systematic review found that a third of the studies were of a low/moderate risk of bias, with their conclusions not being limited by confounding, missing data, and biased outcome assessments. However, the remaining two thirds of studies were of high/critical risk of bias. If these biased studies are used to inform clinical practice or policy generation, substantial negative effects on populations and individual patients can then potentially be observed. The analysis performed in the systematic review estimated the associations between ROB and factors including sample size, number of authors, conflicts of interest statement, journal impact factor, number of citations, and funding to inform clinicians and policy makers which studies are more likely to have a higher quality of evidence.

There was no evidence of an association between the number of citations and the ROBINS-1 risk of bias score. The above finding is concerning as citations are used in clinical research to support or refute new research findings. This may indicate that authors of PMS research are not properly evaluating the studies they are citing, acting as a further barrier to evidence-based care for those suffering from PMS. Alternatively, the absence of an association between citation number and ROB score may be attributed to the scope (2010-present) of our systematic review. PMS studies from 2010 are more likely to have a higher number of citations as compared to more recently published studies, regardless of the quality of evidence. However, an evident relationship between the study impact factor and ROBINS-1 risk of bias score was found. An increase in the impact factor was associated with a lower risk of serious and critical bias (p=0.003). This suggests that OB/GYN journals associated with a higher impact factor are more likely to publish PMS non-randomized studies with a higher quality of evidence. The conclusion is further supported by findings in another study by Saha et al. which found that clinicians and researchers consider impact factor as a reasonable indicator of study quality (r² = 0.82, P = 0.001) (51).

Non-randomized studies funded by industrial sources were not methodologically less rigorous when compared with non-randomized studies funded by governmental, philanthropic or institutional sources (p=0.497). This suggests that the source of funding cannot be used as a reliable marker of the quality of evidence for PMS studies. However, an important, but uninterpretable, association was found between the ROBINS-1 risk of bias score and studies which did not report a source of funding (p=0.034). Although cautious not to interpret this finding, the primary author of our systematic review did contact the authors with available contact information to retrieve further data. Of the four responses, three authors stated that their project did not receive any funding. This may indicate that a lack of funding can manifest into a decrease in the quality of evidence of a study, where increased funding should be allocated to PMS research to bolster evidence for patient care.

Similarly, no evidence was found for an association between the ROBINS-1 risk of bias score and the dichotomous conflict of interest statement. This suggests that the conflict-of-interest statement cannot be used as a reliable marker of the quality of evidence for PMS studies. However, the lack of association between the two variables could be attributed to the frequent dichotomization of the variable. A qualitative value of “no” can be attributed to a study either if (1) the author explicitly indicates that there are no conflicts of interest, or (2) no conflict of interest statement is explicitly stated in the study. In order to properly assess this factor, PMS studies should push to be transparent with their conflicts of interest.

Beyond the a priori hypothesis, our systematic review also found a significant association between the ROBINS-1 risk of bias score and the number of authors in the study. The number of authors on low/moderate studies was on average higher than the number of authors on serious/critical studies (p=0.046). This suggests that PMS studies with a higher number of authors are more likely to have a higher quality of evidence. This finding may be linked to the phenomenon of “knowledge diffusion”. By increasing the number of authors, a study can increase its faculty representation and the attention it receives, resulting in increased credibility and theoretical quality of evidence (52). Lastly, no association was found between sample size and ROBINS-1 risk of bias (p=0.52). This suggests that larger sample sizes cannot be used as a reliable marker of the quality of evidence for PMS studies.

It is important to recognize that the following associations are influenced by the number of studies that met the inclusion criteria (n=38) - the most significant limitation of our systematic review. In statistical analysis, a smaller sample size decreases statistical certainty in the results, manifesting as a large confidence interval. With a larger confidence interval, it is more likely for the confidence interval to include a value of “1”, rendering an association statistically insignificant. Greater sample size may thus reveal further associations between the ROBINS-1 risk of bias tool and different study factors.

Conclusion

Our systematic review concludes that the methodological rigor of non-randomized studies of PMS can vary, with the quality of evidence most usually limited by bias due to confounding, missing data, and bias in the measurement of outcomes. Non-randomized studies which had fewer authors and a lower impact factor were associated with a decreased quality of evidence. However, in our current assessment, no significant association was found between the ROBINS-1 risk of bias score and the number of citations, the sample size, the funding type, or the conflict of interest statement.

Conflict of Interests

Authors have no conflict of interests.

Notes:

Citation: Tehfe H, Chow R, Li S, Kim P, Samari S, Hayawi L, et al. Risk of Bias Assessment in Non-Randomized Studies of Interventions for Premenstrual Syndrome: A Systematic Review. J Family Reprod Health 2022; 16(2): 93-101.