Skip to main content
PMC home page
Biosensors logoLink to Biosensors
. 2019 Feb 27;9(1):32. doi: 10.3390/bios9010032

Plasma from Patients with Rheumatoid Arthritis Reduces Nitric Oxide Synthesis and Induces Reactive Oxygen Species in A Cell-Based Biosensor

Herbert Herlitz-Cifuentes 1,2, Camila Vejar 1,3, Alejandra Flores 1, Paola Jara 1, Paulina Bustos 1, Irene Castro 4,5, Evelyn Poblete 1, Katia Saez 6, Marina Opazo 7, Jorge Gajardo 5, Claudio Aguayo 1, Estefania Nova-Lamperti 1,3,*, Liliana Lamperti 1,3
PMCID: PMC6468433  PMID: 30818887

Abstract

Rheumatoid arthritis (RA) has been associated with a higher risk of developing cardiovascular (CV) diseases. It has been proposed that systemic inflammation plays a key role in premature atherosclerosis development, and is therefore crucial to determine whether systemic components from RA patients promotes endothelial cell-oxidative stress by affecting reactive oxygen species (ROS) and nitric-oxide (NO) production. The aim of this study was to evaluate whether plasma from RA patients impair NO synthesis and ROS production by using the cell-line ECV-304 as a biosensor. NO synthesis and ROS production were measured in cells incubated with plasma from 73 RA patients and 52 healthy volunteers by fluorimetry. In addition, traditional CV risk factors, inflammatory molecules and disease activity parameters were measured. Cells incubated with plasma from RA patients exhibited reduced NO synthesis and increased ROS production compared to healthy volunteers. Furthermore, the imbalance between NO synthesis and ROS generation in RA patients was not associated with traditional CV risk factors. Our data suggest that ECV-304 cells can be used as a biosensor of systemic inflammation-induced endothelial cell-oxidative stress. We propose that both NO and ROS production are potential biomarkers aimed at improving the current assessment of CV risk in RA.

Keywords: rheumatoid arthritis, cardiovascular (CV) diseases, endothelial cell-oxidative stress

1. Introduction

Rheumatoid arthritis (RA) is a systemic autoimmune disease of unknown etiology characterized by chronic inflammation of synovial membranes and articular structures of multiple joints [1,2]. RA causes pain, impaired function of the joints, disability and concomitant comorbidities, particularly in the cardiovascular system [2]. Cardiovascular disease (CVD) is the most common comorbidity in RA and the prevalence of CVD in patients with RA is more than doubled compared with the general population [3]. Moreover, CVD is the leading cause of death in patients with RA [4], with a standardized mortality ratio of 1.5 and a 50% risk of death [5]. In fact, the magnitude of risk of CVD in RA has been compared to type-2 diabetes [6], and several studies indicate that RA is an independent risk factor for myocardial infarction or stroke [7,8]. Specific causes of CV events in RA remain unclear [9]. An accelerated or premature atherosclerosis due to systemic inflammation has been proposed as the main cause. Circulating cytokines alter the function of adipose tissue, skeletal muscle, liver cells and vascular endothelium, promoting a spectrum of CV risk factors, including insulin resistance, dyslipidaemia, oxidative stress and endothelial dysfunction [10]. However, these traditional cardiovascular or Framingham risk factors seem to be inadequate in predicting coronary atherosclerosis in patients with RA [11]. In fact, these factors have not been differentially increased in RA patients compared to healthy controls [12,13].

Several studies have reported the presence of early microvascular [14,15,16] and macrovascular [17] endothelial dysfunction in RA patients. In addition, several studies have demonstrated the presence of early subclinical atherosclerosis in RA patients with no previous medical history of CVD [17,18,19]. Furthermore, it has been shown that early diagnosis of endothelial dysfunction correlates with RA disease activity [20]. Therefore, biomarkers of endothelial damage and systemic inflammation may be relevant for the prediction of early cardiovascular events in patients with RA.

Intracellular reactive oxygen species (ROS) production and Nitric Oxide (NO) synthesis are two endothelial cell responses that need to be balanced in order to maintain vascular homeostasis. Whereas NO is a vasoactive factor from the vascular endothelium aim at maintaining an atheroprotective environment [10,20], ROS are associated with injury and endothelial cell activation, thus increased levels of ROS and decreased levels of NO indicate endothelial cell-oxidative stress. This persistent oxidative stress may cause vascular damage, promoting endothelial dysfunction and CVD. Since systemic ROS and NO measurements are not representative of endothelial oxidative stress, it is ideal to use a biosensor that responds like an endothelial cell to mimic the endothelial stress potentially induced by systemic factors from the plasma of RA patients. The aim of this study was to evaluate the capacity of plasma from RA patients to promote ROS production and impair NO synthesis by using ECV-304 cells as an endothelial-like biosensor.

2. Materials and Methods

2.1. Study Protocol

Seventy-three patients with RA from the Regional Hospital in Concepcion, fulfilling the criteria of the American College of Rheumatology for RA were enrolled in this study. As a control group, fifty-two age-matched healthy volunteers were recruited from the University of Concepción. All patients were taking disease modifying anti-rheumatic drugs (DMARDs) at the time of enrolment. Exclusion criteria for RA patients: other autoimmune disease, other inflammatory disease, seronegative arthritis and previous history of coronary heart disease or cerebrovascular accident. Exclusion criteria for healthy volunteers: pregnancy, recent infectious or inflammatory diseases and previous history of coronary heart disease or cerebrovascular accident. All participants gave written informed consent prior enrolment, and samples were processed and analyzed in a blinded fashion. This study was approved by the ethics committee of Concepcion Health Service and University of Concepcion-Chile and our research was conducted in accordance with the Declaration of Helsinki (1964).

Body Mass Index (BMI), waist circumference, blood pressure, hypertension status, diabetes status, smoking status and family history of coronary-artery disease were recorded from all participants. In patients, disease activity was measured by using the Disease Activity Score based on the evaluation of 28 joints (DAS28). This score was calculated using the number of swollen and tender joints (28 joint count), the patient global assessment, the medical global assessment and the erythrocyte sedimentation rate (ESR). The ability to perform activities of daily living was also evaluated by using a health assessment questionnaire (HAQ).

2.2. Sample Collection

Blood samples from patients and healthy controls were obtained after overnight fasting. Plasma was collected after centrifugation and stored at −80 °C. Insulin, glucose, total cholesterol, high-density lipoprotein (HDL) cholesterol, low-density lipoprotein (LDL) cholesterol and triglycerides were measured in all samples. The ESR, anti-cyclic citrullinated peptide antibodies (anti-CCP) and rheumatoid factor (RF) were measured only in RA patient samples.

2.3. ECV-304 Cell Culture

ECV-304 cells were cultured at 37 °C 5% CO2 in 199 medium (Gibco) containing 10% (v/v) foetal calf serum (Thermo Scientific, HyClone Laboratories Inc) and 100 U/mL penicillin-streptomycin. 2 × 104 cells/100 μL/well ECV-304 cells were added in 96-well plates. After a uniform formation of a single layer of endothelial cells was confirmed, the original medium was removed, and the cells were incubated for 12 h with plasma (100 μL/well) from patients with RA or healthy control in triplicate. The ECV-304 cell line was provided by Dr. Juan Carlos Vera.

2.4. Intracellular ROS Production

ROS formation was determined by the method used by Searle et al. [21], using the probe 2,7-dichlorofluorescein diacetate (DCF) (Calbiochem). ECV-304 cells were cultured in 96 well plates before plasma addition. After 12 h of plasma incubation or positive control H2O2 [0.1 mM and 1 mM], ECV-304 cells were washed with PBS and cultured with 2.5 μM DCF for 30 min at 37 °C 5% CO2 in 199 media. Fluorescence intensity was then measured using a microplate reader at emission 540 nm (excitation 488 nm) (Sinergy 2, Biotek). The basal fluorescence intensity was measured prior H2O2 [10 mM] stimulation. Then, the plate was activated with H2O2 [10 mM] and the fluorescence signal of intracellular ROS production was measured at 60, 180, 300 and 600 s. Fluorescence intensity from H2O2-stimulated ECV-304 cells in the absence of DCF was used as a negative control. Fluorescence intensity per sample was normalized by total protein concentration on each well. After obtaining the IF/ug per sample, fluorescence intensity at 0 s (basal level) was subtracted from fluorescence intensity at 0, 60, 180, 300 and 600 s post-activation with H2O2 [10 uM] and tabulated to calculate the kinetics parameters of ROS production. Maximum fluorescence intensity (Fmax) and Km for each sample was obtained according to the Michaelis-Menten equation, using a Graphpad prism (Graphpad prism software, San Diego, CA 92108, USA.). Final values were obtained by dividing the maximum fluorescence intensity (Fmax) by Km (t1/2). (Scheme 1). Cut off values to define patients with high ROS were calculated based on ROC curves.

Scheme 1.

Scheme 1

Open in a new tab

ROS production was measured in ECV-304 cells cultured with plasma from healthy controls and patients with RA. (a) ECV-304 cells were cultured in 96 well plates prior plasma incubation. Then, plasma from healthy controls and patients with RA was incubated for 12 h at 37 °C 5% CO2. Each plate contained a negative control (No DCF) and two positive controls [H2O2 0.1 mM and 1 mM]. ECV-304 cells were then washed with PBS and cultured with 2.5 μM DCF for 30min at 37 °C 5% CO2. Fluorescence intensity was then measured using a microplate reader at emission 540 nm (excitation 488 nm). The basal fluorescence intensity was measured prior H2O2 [10 mM] stimulation. Then, the plate was activated with H2O2 [10 mM] and the fluorescence signal of intracellular ROS production was measured at 60, 180, 300 and 600 s. Finally, protein concentration from each well was quantified after ROS measurement. (b) Fluorescence intensity per sample was normalized by total protein concentration on each well. (c) Fluorescence intensity at 0 s (basal level) was subtracted from fluorescence intensity at 0, 60, 180, 300 and 600 s post-activation with H2O2 [10 uM] and tabulated to calculate the kinetics parameters of ROS production. Maximum fluorescence intensity (Fmax) and Km for each sample was obtained according to Michaelis-Menten equation. Final values were obtained by dividing the maximum fluorescence intensity (Fmax) by Km (t1/2).

2.5. Intracellular Synthesis of NO

NO was determined accordingly by using the method previously described [22]. ECV-304 cells were cultured in 96 well plates before plasma addition. After 12 h of plasma incubation, ECV-304 cells were washed with PBS and incubated with PBS containing 4,5-diaminofluorescein diacetate (DAF-2DA, 5 μM, 30 min, 37 °C, 5% CO2) (Calbiochem). Fluorescence intensity was then measured using a microplate reader at emission 540 nm (excitation 488 nm) (Sinergy 2, Biotek). The basal fluorescence intensity was measured before Histamine [10 mM] addition. Then, the plate was activated with Histamine [10 mM] and the fluorescence signal of NO was measured at 60, 180, 300 and 600 s. Fluorescence intensity from Histamine-stimulated ECV-304 cells in the absence of DAF-2DA was used as a negative control. In addition, the nitric oxide synthase inhibitor N-nitro l-arginine methyl ester (l-NAME) was added 30 min before Histamine activation to confirm NO synthesis. Fluorescence intensity per sample was normalized by total protein concentration on each well. After obtaining the IF/ug per sample, fluorescence intensity at 0 s (basal level) was subtracted from fluorescence intensity at 0, 60, 180, 300 and 600 s post-activation with Histamine [10 uM] and tabulated to calculate the kinetics parameters of NO synthesis. Maximum fluorescence intensity (Fmax) and Km for each sample was obtained according to Michaelis-Menten equation, using Graphpad prism (Graphpad prism software, San Diego, CA 92108, USA). The slope of NO production per sample was obtained by dividing the maximum fluorescence intensity (Fmax) by Km (t1/2) (Scheme 2). Cut off values to define patients with low NO were calculated based on ROC curves.

Scheme 2.

Scheme 2

Open in a new tab

NO synthesis was measured in ECV-304 cells cultured with plasma from healthy controls and patients with RA. (a) ECV-304 cells were cultured in 96 well plates before plasma addition. Then, plasma from healthy controls and patients with RA was incubated for 12 h at 37 °C 5% CO2. Each plate contained a negative control (No DAF-2DA). After 12 h, ECV-304 cells were washed with PBS and cultured with 5 μM DAF-2DA for 30 min at 37 °C 5% CO2. Fluorescence intensity was then measured using a microplate reader at emission 540 nm (excitation 488 nm). The basal fluorescence intensity was measured before Histamine [10 mM] addition. Then, the plate was activated with Histamine [10 mM] and the fluorescence signal of NO was measured at 60, 180, 300 and 600 s. (b) Fluorescence intensity per sample was normalized by total protein concentration on each well. (c) Fluorescence intensity at 0 s (basal level) was subtracted from fluorescence intensity at 0, 60, 180, 300 and 600 s post-activation with Histamine [10 uM] and tabulated to calculate the kinetics parameters of NO synthesis. Maximum fluorescence intensity (Fmax) and Km for each sample was obtained according to Michaelis-Menten equation. The slope of NO production per sample was obtained by dividing the maximum fluorescence intensity (Fmax) by Km (t1/2). (d) Fluorescence intensity from Histamine-stimulated ECV-304 cells in the presence of the nitric oxide synthase inhibitor N-nitro l-arginine methyl ester (L-NAME) was used to confirm NO synthesis.

2.6. IL-6, sVCAM-1 and hs-CRP Levels in Plasma

IL-6 and sVCAM-1 levels in plasma were measured using ELISA according to manufacturer’s instructions (eBioscience ELISA Ready-SET-Go). Concentrations of hs-CRP were quantified using a chemiluminescent immunometric solid phase assay (Siemmens).

2.7. Statistical Analysis

A comparison between RA patients and healthy controls for all parameters was performed. The results are presented as mean ± standard deviation (SD). Normality was verified using the Shapiro-Wilk test. Logarithmic transformation was used in variables with positive bias. Non-parametric Mann-Whitney test and Kruskal-Wallis test were used for those variables that did not fulfil the assumption of normality test. Non-normally distributed data were presented as median and interquartile ranges. All statistical procedures were analyzed with SPSS Statistics and represented with Graphpad prism (Graphpad prism software Inc.,).

3. Results

3.1. RA Patients and Healthy Control Exhibit Similar Traditional CV Risk Factors

The clinical parameters of patients with RA are described in Table 1. Patients exhibited mean disease duration of 14.5 ± 9.7 years, with a moderate disease activity (DAS28 = 3.7 ± 1.2). Their disability was also moderate, and the mean score of the HAQ survey was 1.3 ± 0.9, indicating a moderate self-perceived level of physical disability. In addition, 19 patients (26.8%) were positive for anti-CCP and 45 patients (62.5%) were seropositive for RF. All patients were under DMARD treatment, mainly in combination with corticosteroids (99%). Metabolic parameters of patients with RA and healthy controls are described in Table 2. No significant differences between RA patients and healthy controls were observed when age, weight, height and BMI were compared. However, the waist circumference was found significantly increased in RA patients compared to healthy controls. In fact, the prevalence of central obesity (waist circumference ≥ 88cm in women) was 81% in RA patients compared to 56% in healthy controls (P = 0.0029). Moreover, the systolic and diastolic blood pressure was increased in patients with RA (P < 0.01), and a higher prevalence of hypertension (47% vs. 23%, P < 0.01) was also observed in this group. The prevalence of family history of heart disease was higher in healthy controls than in RA patients (23% vs. 7%, P < 0.01). Although both groups had elevated concentrations of total cholesterol, the control group had significantly higher levels of LDL cholesterol than RA patients. No significant differences in the concentration of triglycerides, HDL cholesterol, insulin and HOMA-IR were found between groups. According to the median HOMA-IR, both groups had borderline values for insulin resistance, using the defined cut-off values for the insulin resistance HOMA1 formula ≥ 2.5 [23].

Table 1.

Disease characteristics in RA patients.

RA Patients (n = 73)
Sex female 66 (90%)
Disease duration (years) 14.5 ± 9.7
VAS (mm) 52.5 ± 24.7
Global assessment (mm) 46.5 ± 23.9
Tender joints 3 ± 3.9
Swollen joints 2.7 ± 3.4
DAS28 3.7 ± 1.2
HAQ 1.3 ± 0.9
ESR (mm/h) 22 ± 15.6
Anti-CCP (UA/mL) † 48.8 ± 90.5
RF (mg/dL) ‡ 241.6 ± 449.3
DMARDs
 Methotrexate
 Leflunomide
 Sulfasalazine
 Hydroxychloroquine		  
64 (88%)
21 (29%)
4 (6%)
1 (1%)
Prednisolone 72 (99%)
Open in a new tab

Results are expressed as number (percentage), mean ± SD, as appropriate. † Anti-CCP measurements were only available for 71 patients (97%). ‡ RF measurements were only available for 72 patients (99%). VAS, visual analogue scale; DAS28, disease activity score in 28 joints; HAQ, Health Assessment Questionnaire; ESR, erythrocyte sedimentation rate; anti-CCP, anti-cyclic citrullinated peptide; RF, rheumatoid factor; DMARDs, disease modifying anti-rheumatic drugs.

Table 2.

Individual CVD risk factors in patients and healthy controls.

RA Patients (n = 73) Control Subjects (n = 52) P †
Sex female
Age(years)		 66/73
58 (50−67)		 42/52
57 (48−62)		  
0.200
Weight (kg) 69 (59−75) 71.7 (61.0−79.7) 0.149
Height (m) ‡ 1.6 (1.5−1.6) 1.6 (1.5−1.6) 0.179
BMI (kg/m2) 27.6 (24.0−30.8) 28.6 (25.0−30.7) 0.499
Waist circumference (cm) 100 (90−106) 95 (82−104) 0.027
SBP (mm Hg) ‡ 125 (120−140) 120 (110−120) <0.001
DBP (mm Hg) 80 (70−90) 77.5 (70−80) 0.007
Total cholesterol (mg/dL) 200 (181−221) 212 (185−230) 0.161
HDL-C 56 (±14) 55 (±14) 0.860
LDL-C 118 (±31) 134 (±40) 0.020
Triglycerides 117 (91–156) 111 (83–133) 0.207
TC/HDL ratio 3.8 (±1.0) 4.1 (±1.3) 0.143
Glucose 91 (83−98) 97 (89−107) -
Insulin 10.1 (7−13.8) 9.1 (7.4−16.2) 0.887
HOMA-IR 2.3 (1.5−3.3) 2.5 (1.6−4.1) 0.482
Hypertension 34 (47%) 12 (23%) 0.007
Diabetes mellitus 11 (15%) 3 (6%) 0.104
Smoking 15 (21%) 12 (23%) 0.735
Family history of coronary-arterial disease 5 (7%) 12 (23%) 0.009
Open in a new tab

Results are expressed as median (interquartile range), mean ± SD or number (percentage). † P value are by Mann-Whitney test, Student’s t test or Fisher exact test, as appropriate, for comparisons among group. ‡ Adjusted for age. BMI, body mass index; SBP, systolic blood pressure; DBP, diastolic blood pressure; HDL-C, high-density lipoprotein cholesterol; LDL-C, low-density lipoprotein cholesterol; TC, total cholesterol; HOMA-IR, homeostasis model assessment insulin resistance.

3.2. ECV-304 Cells Incubated with Plasma from RA Patients Exhibited Reduced NO Synthesis and Increased ROS Production Compared To Healthy Volunteers

In order to evaluate the potential injurious effect of plasma from RA patients in comparison with plasma from healthy volunteers, ECV-304 cells were incubated with plasma from both groups for 12 h. This cell line was used as a biosensor to determine whether plasma from patients with systemic inflammation promotes ROS production and/or impairs NO synthesis. The maximum fluorescence for each probe was obtained incubating ECV-304 cells with 10 mM H2O2 for ROS production and 10 μM histamine for NO synthesis. ECV-304 cells incubated with 10 mM H2O2 or 10 μM histamine in the absence of the fluorescent probe were used as a negative control. Culture medium from non-stimulated ECV-304 cells in the presence of the fluorescent probe was used to obtain the baseline fluorescence for NO synthesis and ROS production (Figure 1a).

Figure 1.

Figure 1

Open in a new tab

Plasma from RA patients induced increased ROS production and reduced NO synthesis compared to healthy volunteers. (a) ECV-304 cells were cultured with 2.5 μM DCF for ROS and 5 μM DAF-2DA for 30 min at 37 °C 5% CO2 in 199 media. The baseline for fluorescence intensity was determined using ECV-304 cells without stimulation. The positive fluorescence signal was obtained using 10 mM H2O2 stimulation for intracellular ROS and 10 μM histamine stimulation for intracellular NO. (b) ROS production and (c) NO synthesis was measured in ECV-304 cells cultured with plasma from RA patients and healthy volunteers after 12 h of incubation. Fluorescence intensity was measured using a microplate reader at emission 540 nm (excitation 485 nm) and confirmed by fluorescent microscope. (d) Protein concentration of hs-CRP, IL-6 and sVCAM-1 in plasma from RA patients and healthy controls was measured using a chemiluminescent immunometric solid phase assay and ELISA. Mann-Whitney test was used to compare RA patients and healthy controls, *** P < 0.0005 and **** P < 0.0001 was considered significant.

Figure 1b shows that ROS production was significantly higher in ECV-304 cell cultures exposed to plasma of patients with RA compared to healthy controls (**** P < 0.0001) (Figure 1b). Concomitantly, Figure 1c shows that NO synthesis was significantly reduced in ECV-304 cell cultures exposed to plasma of patients with RA compared to controls (**** P < 0.0001) (Figure 1c). These results indicate that plasma components from RA patients induce endothelial cell-oxidative stress and impair endothelium-dependent vasodilation compared to healthy subjects. Finally, we evaluated non-traditional cardiovascular risk factors associated with inflammation (hs-CRP and IL-6) and an endothelial activation biomarker (sVCAM-1) in plasma samples from RA patients and healthy controls (Figure 1d). The concentration of hs-CRP, IL-6 and sVCAM-1 were significantly increased in plasma from RA patients compared to healthy controls (Figure 1d), indicating that RA patients had a higher prevalence of presenting non-traditional cardiovascular risk factors. Moreover, RA patients had a higher prevalence of presenting vascular damage considering the results of sVCAM-1.

3.3. Endothelial Cell Oxidative Stress in RA Patients is Not Associated with Traditional CV Risk Factors

Having shown the significant difference between RA patients and healthy controls in ROS and NO synthesis, we defined healthy controls and RA patients with high or normal ROS production, and with low or normal NO production based on the interpretation of ROC curves obtained from all donors. For ROS production, our cut off was 1.66 IF/ug, whereas for NO our cut off was 0.042 IF/ug. When data from individuals with differential ROS production was analyzed, we observed higher concentrations of IL-6 and sVCAM-1 in plasma from RA patients with high ROS in comparison with plasma from healthy controls with normal or high ROS. However, these differences were not statistically significant between RA patients with normal or high ROS (Figure 2). No significant differences were observed when traditional risk factors were compared between groups, except for Glucose levels that were decreased in RA patients with high ROS (Figure 2). Similar results were observed when NO was evaluated (Figure 3). These results suggest that RA patients with high ROS or low NO, exhibit higher systemic inflammation in comparison with individuals without RA, regardless their oxidative stress state. However, inflammatory parameters did not correlate with ROS or NO synthesis in RA patients (Data not shown). Finally, parameters associated with disease activity were not statistically significant between sub-groups of patients with RA, suggesting that biomarkers aimed at detecting early vascular damage, endothelial cell-oxidative stress or systemic inflammation, could improve the current assessment of CV disease in patients with RA (Table 3).

Figure 2.

Figure 2

Open in a new tab

Increased ROS production is associated with inflammatory factors, but not traditional risk factors. Traditional CV risk factors and concentration of hs-CRP, IL-6 and sVCAM-1 were measured in plasma from RA patients with high or normal ROS and plasma from healthy controls (HC) with high or normal ROS. Kruskal-Wallis test was used to compare RA patients and healthy controls, * P < 0.05, ** P < 0.01, *** P < 0.005 and **** P < 0.0001 was considered significant.

Figure 3.

Figure 3

Open in a new tab

Decreased NO production is associated with inflammatory factors, but not with traditional risk factors. Traditional CV risk factors and concentration of hs-CRP, IL-6 and sVCAM-1 were measured in plasma from RA patients with normal or decreased NO and plasma from healthy controls (HC) with normal or decreased NO. Kruskal-Wallis test was used to compare RA patients and healthy controls, * P < 0.05, ** P < 0.01, *** P < 0.005 and **** P < 0.0001 was considered significant.

Table 3.

Disease characteristics in RA patients according to ROS production and NO synthesis.

ROS NO
Normal (n = 15) High (n = 57) P Normal (n = 22) Low (n = 50) P
Disease duration (years) 15.2 (±11.2) 14.0 (±9.3) 0.674 15.3 (±7.9) 14.2 (±10.4) 0.644
Prednisolone dose (mg/day) 8.7 (±2.3) 8.2 (±2.9) 0.553 9.0 (±2.8) 8.0 (±2.7) 0.176
Tender joints 2.3 (±3.8) 3.2 (±4.0) 0.436 3.2 (±3.5) 3.0 (±4.1) 0.748
Swollen joints 2.6 (±3.0) 2.7 (±3.6) 0.938 3.0 (±3.1) 2.5 (±3.6) 0.598
Global assessment (mm) 36.7 (±24.1) 49.1 (±23.7) 0.076 43.2 (±24.2) 47.6 (±24.0) 0.482
VAS (mm) 42.7 (±24.0) 54.8 (±24.6) 0.091 56.0 (±28.1) 50.9 (±23.3) 0.435
DAS28 3.4 (±1.1) 3.8 (±1.2) 0.248 4.0. (±1.0) 3.6 (±1.2) 0.123
HAQ 1.0 (±0.9) 1.2 (±0.9) 0.209 1.4 (±1.0) 1.3 (±0.9) 0.775
ESR (mm/h) 20.6 (±10.2) 22.7 (±16.7) 0.548 24.0 (±12.7) 20.8 (±16.7) 0.420
Anti-CCP (UA/mL) 37.0 (±83.9) 52.6 (±93.1) 0.570 72.6 (±95.4) 38.8 (±87.1) 0.152
RF (mg/dL) 90.1 (±132.6) 282.8 (±493.2) 0.012 * 325.0 (±690.7) 203.8 (±288.6) 0.436
Open in a new tab

Results are expressed as mean ± SD. VAS, visual analogue scale; DAS28, disease activity score in 28 joints; HAQ, Health Assessment Questionnaire; ESR, erythrocyte sedimentation rate; anti-CCP, anti-cyclic citrullinated peptide; RF, rheumatoid factor. * P < 0.05 was considered significant.

4. Discussion

Cardiovascular disease is the leading cause of death in patients with RA [3,4,24]. However, traditional CVD risk factors such as hypertension, hyperlipidaemia, diabetes and smoking, not always correlate with the development of CVD in this autoimmune disorder [12]. As it has been proposed that systemic inflammation plays a role in the development of endothelial cell damage, we evaluated the capacity of plasma from RA patients to promote ROS production and impair NO synthesis by using ECV-304 cells as a biosensor of endothelial cell oxidative stress. Higher oxidative responses in cells cultured with plasma from patients with RA in comparison with plasma from healthy volunteers were observed. Interestingly, similar traditional CVD risk factors were present in both groups. For example, the prevalence of diabetes and smoking was not significantly different between groups, whereas previous history of CVD in first-degree relatives was more prevalent in the control group. Similarly, despite no difference in the lipid profile between RA patients and healthy volunteers, LDL cholesterol was also significantly higher in the control group. Hypertension, on the other hand, was significantly increased in RA patients compared to volunteers, as previously shown [12,13]. This indicates that patients with RA did not exhibit significant differences in terms of traditional CV risk factors in comparison with our control group, suggesting that other factors should be included in order to improve the assessment of CV risk in patients with RA.

ROS production and NO synthesis measurements in the endothelial cell line ECV-304 exposed to plasma from patients with RA have not been reported to date. We have demonstrated for the first time that plasma from patients with RA is injurious to endothelial cells. Hence, plasma from patients with RA induces endothelial oxidative stress, by deregulating ROS production and NO synthesis. Our research group has previously reported a similar system using human umbilical vein endothelial cells incubated with serum from hypercholesteraemic patients, observing reduced NO synthesis and increased ROS formation in patients compared with healthy controls [21]. Both studies support the use of cells as a biosensor of oxidative stress in response to systemic factors, however the use of human umbilical vein endothelial cells present higher donor variability, a time-consuming process to get enough cells to perform experiments and more expensive resources for isolation and cell culture. Thus, ECV-304 cells represent a better candidate as a biosensor to analyze oxidative stress induced by systemic factors. Other studies have analyzed oxidative stress systemically, for example studies from Mateen et al. have compared the antioxidant status between RA patients and healthy controls by monitoring ROS production and oxidative damage markers in periphery. Their results showed higher oxidative stress in RA patients compared to healthy controls, and a positive correlation between ROS production and lipid peroxidation, protein oxidation and DNA damage in RA patients [25]. Interestingly, no significant difference in ROS production between seropositive and seronegative RA patients was observed [25]. In a further study, same authors evaluated whether the presence of inflammatory cytokines correlated with ROS production. Results revealed that levels of cytokines, such as TNF-α, IL-1β, IL-17, IL-10 and IL-6, had a positive and significant correlation with levels of ROS [26]. They also found a positive correlation between ROS production and NO synthesis in the plasma, whereas we found reduced NO synthesis induction by plasma from RA patients in comparison with control plasma. The discrepancy between the studies could be associated with the fact that we measured both parameters in cells, whereas they measured ROS in the haematocrit and NO in the plasma.

The association between systemic inflammation and NO synthesis in RA has also been evaluated as it is well known that a constitutive production of NO is required to maintain an atheroprotective environment within the vascular system. In this context, evaluation of the symmetric (SDMA) and asymmetric (ADMA) dimethylarginine has become an indicator of NO synthesis, as these analogues of L-arginine are endogenous inhibitor of the NO synthase because they compete with L-arginine at the active site of the enzyme. This indicates that high ADMA levels results in low NO synthase activity, and consequently reduced NO synthesis. Recent studies have analyzed the association between inflammatory parameters, SDMA/ADMA levels in plasma and cardiovascular risk factors. Studies from Kwaśny-Krochin et al. showed higher ADMA levels in patients with RA compared to healthy controls [27]. Moreover, ADMA levels in the RA group correlated positively with CRP and disease activity, but not with age, renal function, or the medications used. They also demonstrated that CRP was the only independent predictor of ADMA levels in RA [27]. Another study showed that CRP had a significant positive correlation with ADMA levels after 6 years of follow up, but not at baseline, suggesting that progressive inflammation may be responsible for the reduction of NO synthesis, independently of classical risk factors [28]. A more recent study has also hypothesized that ADMA levels is associated with endothelial function in RA, therefore carotid intima media thickness (cIMT) and arterial stiffness as well as non-invasive assessments were measured and correlated with ADMA levels. The authors concluded that microvascular function, arterial stiffness and cIMT were associated with circulating ADMA levels in RA patients with high inflammatory markers such as erythrocyte sedimentation rate and CRP [29]. Altogether these results suggest that direct or indirect ROS and NO measurements could be potential biomarkers aimed at identifying whether systemic factors promote endothelial cell-oxidative responses.

In term of inflammation, all RA patients in this study have active disease and high levels of inflammation as 58% of RA patients had elevated levels of hs-CRP, 40% exhibited high levels of IL-6 and 55% had high levels of sVCAM-1. It is known that systemic inflammation has detrimental effects in several tissues, particularly in the endothelium where stimulates atherosclerosis [30]. Several biomarkers of systemic inflammation have been associated with endothelial dysfunction and atherosclerosis, for example, IL-6 and CRP have been inversely correlated with the thickness of the intima media of the carotid in RA patients [30,31]. Consistent with our results, CRP levels were higher in patients with high ROS production and low NO synthesis than healthy volunteers with normal ROS or NO. Furthermore, the levels of sVCAM-1 and IL-6 were significantly higher in RA patients than healthy volunteers, either with an altered or normal ROS and NO production. This data indicates that both parameters are valuable candidates to analyze endothelial status or non-specific inflammation systemically. From an autoimmune point of view, 62.5% of patients were positive for the Rheumatoid Factor and only 26.8% were positive for the anti-CCP. One study reported a relationship between the presence of these autoantibodies and endothelial function. They showed that positive patients for anti-CCP and RF have an impaired endothelial function, due to a higher hyperaemic reactive index, independent of other CV risk factors [32].

Since the inflammatory state and the autoimmune components are the main differences between the plasma from RA patients and healthy controls, it is possible that some of these components alone or in combination are causing deregulation in ROS and NO intracellular production, as observed in our model of biosensor cells. Previous analyses have shown phenotypic changes in the endothelium in the onset of RA in an experimental model of arthritis induced by adjuvant. The authors identified mechanisms involved in endothelial dysfunction, as well as a correlation between endothelial dysfunction and biomarkers of systemic inflammation [33]. Despite these efforts, the inflammatory mediators involved in endothelial damage have not been fully defined which are [31,33,34]. It would be interesting to elucidate which of the plasma components is causing endothelial cell damage, however, the combination of these different factors may be more relevant than each component alone in the prognosis of these patients.

Finally, ROS and NO measurements in cell-biosensors, alone or in combination with other vascular parameters could become possible biomarkers for disease activity in RA in terms of endothelial damage. These measurements could help to establish reference values for endothelial cell status or oxidative stress in RA, as ROS production is relatively low in healthy controls. In addition, they could be useful to evaluate therapies aimed at reducing oxidative stress. For example, Khojah et al. have shown that therapy with ascorbic acid (1 g/day) significantly reduced the levels of ROS in RA patients that initially exhibited positive correlations between most of the reactive species and the clinical and biochemical markers of RA. In fact, after treatment with the antioxidant, several parameters were significantly reduced such as DAS28, matrix metalloproteinase 3 and the erythrocyte sedimentation rate [35]. Another study has reviewed the clinical efficacy of dietary antioxidants showing the beneficial role of natural antioxidants in RA and the important role of potential biomarker of oxidative stress for detecting disease pathophysiology in patients with this pathology [36]. These studies suggest that our endothelial biosensor model could help to evaluate potential risk factors individually, as well as potential treatments aimed at reducing endothelial cell-oxidative stress. However, more studies are required to elucidate whether the induction of ROS and NO synthesis by plasma components from patients with RA is associated with endothelial dysfunction and higher incidence of cardiovascular disease in these patients.

5. Conclusions

Our findings demonstrated that ECV-304 cells can be used as a biosensor of oxidative stress induced by plasma components. We demonstrated that plasma from RA patients promote oxidative stress by altering ROS and NO production. Considering the 2015/2016 updated recommendations for the management of CV risk in RA, the inclusion of novel biomarkers of CVD diseases associated with systemic inflammation could improve the assessment of CV risk in patients with RA.

Author Contributions

For research articles with several authors, a short paragraph specifying their individual contributions must be provided. The following statements should be used conceptualization, L.L. and H.H.-C.; methodology, H.H.-C., C.V., A.F., E.P. and P.J.; formal analysis, K.S.; investigation, P.B.; resources, I.C., M.O., J.G. and C.A.; writing—original draft preparation, H.H., L.L. and E.N.-L.; writing—review and editing, E.N.-L.; project administration, L.L.; funding acquisition, L.L. and E.N.-L. We thank Dr. Carmen Hernandez for her scientific advices.

Funding

This research was funded by Grant DIUC 211.072.035-1.0, Universidad de Concepcion-Chile, Grant DIUSS 2012-0007-I, Universidad San Sebastian-Chile and Grant INNOVA 10CHS2674F11-Chile. E.N-L was funded by Fondecyt de Iniciación 11170610 and PAI79170073.

Conflicts of Interest

The authors declare no conflict of interest.

References

  • 1.Scott D.L., Wolfe F., Huizinga T.W.J. Rheumatoid arthritis. Lancet. 2010;376:1094–1108. doi: 10.1016/S0140-6736(10)60826-4. [DOI] [PubMed] [Google Scholar]
  • 2.McInnes I.B., O’Dell J.R. State-of-the-art: Rheumatoid arthritis. Ann. Rheum. Dis. 2010;69:1898–1906. doi: 10.1136/ard.2010.134684. [DOI] [PubMed] [Google Scholar]
  • 3.Solomon D.H., Goodson N.J., Katz J.N., Weinblatt M.E., Avorn J., Setoguchi S., Canning C., Schneeweiss S. Patterns of cardiovascular risk in rheumatoid arthritis. Ann. Rheum. Dis. 2006;65:1608–1612. doi: 10.1136/ard.2005.050377. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Farragher T.M., Goodson N.J., Naseem H., Silman A.J., Thomson W., Symmons D., Barton A. Association of the HLA-DRB1 gene with premature death, particularly from cardiovascular disease, in patients with rheumatoid arthritis and inflammatory polyarthritis. Arthritis Rheum. 2008;58:359–369. doi: 10.1002/art.23149. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Barber C.E.H., Smith A., Esdaile J.M., Barnabe C., Martin L.O., Faris P., Hazlewood G., Noormohamed R., Alvarez N., Mancini G.B.J., et al. Best practices for cardiovascular disease prevention in rheumatoid arthritis: A systematic review of guideline recommendations and quality indicators. Arthritis Care Res. 2015;67:169–179. doi: 10.1002/acr.22419. [DOI] [PubMed] [Google Scholar]
  • 6.Peters M.J.L., van Halm V.P., Voskuyl A.E., Smulders Y.M., Boers M., Lems W.F., Visser M., Stehouwer C.D.A., Dekker J.M., Nijpels G., et al. Does rheumatoid arthritis equal diabetes mellitus as an independent risk factor for cardiovascular disease? A prospective study. Arthritis Rheum. 2009;61:1571–1579. doi: 10.1002/art.24836. [DOI] [PubMed] [Google Scholar]
  • 7.Wolfe F., Freundlich B., Straus W.L. Increase in cardiovascular and cerebrovascular disease prevalence in rheumatoid arthritis. J. Rheumatol. 2003;30:36–40. [PubMed] [Google Scholar]
  • 8.Södergren A., Stegmayr B., Lundberg V., Ohman M.-L., Wållberg-Jonsson S. Increased incidence of and impaired prognosis after acute myocardial infarction among patients with seropositive rheumatoid arthritis. Ann. Rheum. Dis. 2007;66:263–266. doi: 10.1136/ard.2006.052456. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.van den Oever I.A.M., van Sijl A.M., Nurmohamed M.T. Management of cardiovascular risk in patients with rheumatoid arthritis: Evidence and expert opinion. Ther. Adv. Musculoskelet. Dis. 2013;5:166–181. doi: 10.1177/1759720X13491025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Stevens R.J., Douglas K.M.J., Saratzis A.N., Kitas G.D. Inflammation and atherosclerosis in rheumatoid arthritis. Expert Rev. Mol. Med. 2005;7:1–24. doi: 10.1017/S1462399405009154. [DOI] [PubMed] [Google Scholar]
  • 11.Del Rincón I.D., Williams K., Stern M.P., Freeman G.L., Escalante A. High incidence of cardiovascular events in a rheumatoid arthritis cohort not explained by traditional cardiac risk factors. Arthritis Rheum. 2001;44:2737–2745. doi: 10.1002/1529-0131(200112)44:12&#x0003c;2737::AID-ART460&#x0003e;3.0.CO;2-#. [DOI] [PubMed] [Google Scholar]
  • 12.Chung C.P., Giles J.T., Petri M., Szklo M., Post W., Blumenthal R.S., Gelber A.C., Ouyang P., Jenny N.S., Bathon J.M. Prevalence of traditional modifiable cardiovascular risk factors in patients with rheumatoid arthritis: Comparison with control subjects from the multi-ethnic study of atherosclerosis. Semin. Arthritis Rheum. 2012;41:535–544. doi: 10.1016/j.semarthrit.2011.07.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Gonzalez A., Maradit Kremers H., Crowson C.S., Ballman K.V., Roger V.L., Jacobsen S.J., O’Fallon W.M., Gabriel S.E. Do cardiovascular risk factors confer the same risk for cardiovascular outcomes in rheumatoid arthritis patients as in non-rheumatoid arthritis patients? Ann. Rheum. Dis. 2008;67:64–69. doi: 10.1136/ard.2006.059980. [DOI] [PubMed] [Google Scholar]
  • 14.Profumo E., Di Franco M., Buttari B., Masella R., Filesi C., Tosti M.E., Scrivo R., Scarno A., Spadaro A., Saso L., et al. Biomarkers of subclinical atherosclerosis in patients with autoimmune disorders. Mediat. Inflamm. 2012;2012:503942. doi: 10.1155/2012/503942. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Sandoo A., Kitas G.D., Carroll D., Veldhuijzen van Zanten J.J.C.S. The role of inflammation and cardiovascular disease risk on microvascular and macrovascular endothelial function in patients with rheumatoid arthritis: A cross-sectional and longitudinal study. Arthritis Res. Ther. 2012;14:R117. doi: 10.1186/ar3847. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Finckh A., Courvoisier D.S., Pagano S., Bas S., Chevallier-Ruggeri P., Hochstrasser D., Roux-Lombard P., Gabay C., Vuilleumier N. Evaluation of cardiovascular risk in patients with rheumatoid arthritis: Do cardiovascular biomarkers offer added predictive ability over established clinical risk scores? Arthritis Care Res. 2012;64:817–825. doi: 10.1002/acr.21631. [DOI] [PubMed] [Google Scholar]
  • 17.Södergren A., Karp K., Boman K., Eriksson C., Lundström E., Smedby T., Söderlund L., Rantapää-Dahlqvist S., Wållberg-Jonsson S. Atherosclerosis in early rheumatoid arthritis: Very early endothelial activation and rapid progression of intima media thickness. Arthritis Res. Ther. 2010;12:R158. doi: 10.1186/ar3116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Georgiadis A.N., Voulgari P.V., Argyropoulou M.I., Alamanos Y., Elisaf M., Tselepis A.D., Drosos A.A. Early treatment reduces the cardiovascular risk factors in newly diagnosed rheumatoid arthritis patients. Semin. Arthritis Rheum. 2008;38:13–19. doi: 10.1016/j.semarthrit.2007.09.008. [DOI] [PubMed] [Google Scholar]
  • 19.Sandoo A., Dimitroulas T., Veldhuijzen van Zanten J.J.C.S., Smith J.P., Metsios G.S., Nightingale P., Stavropoulos-Kalinoglou A., Kitas G.D. Lack of association between asymmetric dimethylarginine and in vivo microvascular and macrovascular endothelial function in patients with rheumatoid arthritis. Clin. Exp. Rheumatol. 2012;30:388–396. [PubMed] [Google Scholar]
  • 20.Tanasescu C., Jurcut C., Jurcut R., Ginghina C. Vascular disease in rheumatoid arthritis: From subclinical lesions to cardiovascular risk. Eur. J. Intern. Med. 2009;20:348–354. doi: 10.1016/j.ejim.2008.09.005. [DOI] [PubMed] [Google Scholar]
  • 21.Searle A., Gómez-Rosso L., Meroño T., Salomon C., Durán-Sandoval D., Giunta G., Grant C., Calvo C., Lamperti L., Brites F., et al. High LDL levels are associated with increased lipoprotein-associated phospholipase A(2) activity on nitric oxide synthesis and reactive oxygen species formation in human endothelial cells. Clin. Biochem. 2011;44:171–177. doi: 10.1016/j.clinbiochem.2010.10.004. [DOI] [PubMed] [Google Scholar]
  • 22.Knepler J.L., Taher L.N., Gupta M.P., Patterson C., Pavalko F., Ober M.D., Hart C.M. Peroxynitrite causes endothelial cell monolayer barrier dysfunction. Am. J. Physiol. Cell Physiol. 2001;281:C1064–C1075. doi: 10.1152/ajpcell.2001.281.3.C1064. [DOI] [PubMed] [Google Scholar]
  • 23.Mojiminiyi O.A., Abdella N.A. Effect of homeostasis model assessment computational method on the definition and associations of insulin resistance. Clin. Chem. Lab. Med. 2010;48:1629–1634. doi: 10.1515/CCLM.2010.303. [DOI] [PubMed] [Google Scholar]
  • 24.Aviña-Zubieta J.A., Choi H.K., Sadatsafavi M., Etminan M., Esdaile J.M., Lacaille D. Risk of cardiovascular mortality in patients with rheumatoid arthritis: A meta-analysis of observational studies. Arthritis Rheum. 2008;59:1690–1697. doi: 10.1002/art.24092. [DOI] [PubMed] [Google Scholar]
  • 25.Mateen S., Moin S., Khan A.Q., Zafar A., Fatima N. Increased Reactive Oxygen Species Formation and Oxidative Stress in Rheumatoid Arthritis. PLoS ONE. 2016;11:e0152925. doi: 10.1371/journal.pone.0152925. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Mateen S., Moin S., Shahzad S., Khan A.Q. Level of inflammatory cytokines in rheumatoid arthritis patients: Correlation with 25-hydroxy vitamin D and reactive oxygen species. PLoS ONE. 2017;12:e0178879. doi: 10.1371/journal.pone.0178879. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Kwaśny-Krochin B., Głuszko P., Undas A. Plasma asymmetric dimethylarginine in active rheumatoid arthritis: Links with oxidative stress and inflammation. Pol. Arch. Med. Wewn. 2012;122:270–276. doi: 10.20452/pamw.1277. [DOI] [PubMed] [Google Scholar]
  • 28.Sandoo A., Dimitroulas T., Hodson J., Smith J.P., Douglas K.M., Kitas G.D. Cumulative inflammation associates with asymmetric dimethylarginine in rheumatoid arthritis: A 6 year follow-up study. Rheumatology. 2015;54:1145–1152. doi: 10.1093/rheumatology/keu349. [DOI] [PubMed] [Google Scholar]
  • 29.Dimitroulas T., Hodson J., Sandoo A., Smith J., Kitas G.D. Endothelial injury in rheumatoid arthritis: A crosstalk between dimethylarginines and systemic inflammation. Arthritis Res. Ther. 2017;19:32. doi: 10.1186/s13075-017-1232-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Mangoni A.A., Zinellu A., Sotgia S., Carru C., Piga M., Erre G.L. Protective Effects of Methotrexate against Proatherosclerotic Cytokines: A Review of the Evidence. Mediat. Inflamm. 2017;2017:9632846. doi: 10.1155/2017/9632846. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Verma I., Syngle A., Krishan P. Predictors of endothelial dysfunction and atherosclerosis in rheumatoid arthritis in Indian population. Indian Heart J. 2017;69:200–206. doi: 10.1016/j.ihj.2016.10.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Hjeltnes G., Hollan I., Førre Ø., Wiik A., Mikkelsen K., Agewall S. Anti-CCP and RF IgM: Predictors of impaired endothelial function in rheumatoid arthritis patients. Scand. J. Rheumatol. 2011;40:422–427. doi: 10.3109/03009742.2011.585350. [DOI] [PubMed] [Google Scholar]
  • 33.Totoson P., Maguin-Gaté K., Nappey M., Wendling D., Demougeot C. Endothelial Dysfunction in Rheumatoid Arthritis: Mechanistic Insights and Correlation with Circulating Markers of Systemic Inflammation. PLoS ONE. 2016;11:e0146744. doi: 10.1371/journal.pone.0146744. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Moroni L., Selmi C., Angelini C., Meroni P.L. Evaluation of Endothelial Function by Flow-Mediated Dilation: A Comprehensive Review in Rheumatic Disease. Arch. Immunol. Ther. Exp. 2017;65:463–475. doi: 10.1007/s00005-017-0465-7. [DOI] [PubMed] [Google Scholar]
  • 35.Khojah H.M., Ahmed S., Abdel-Rahman M.S., Hamza A.-B. Reactive oxygen and nitrogen species in patients with rheumatoid arthritis as potential biomarkers for disease activity and the role of antioxidants. Free Radic. Biol. Med. 2016;97:285–291. doi: 10.1016/j.freeradbiomed.2016.06.020. [DOI] [PubMed] [Google Scholar]
  • 36.Bala A., Mondal C., Haldar P.K., Khandelwal B. Oxidative stress in inflammatory cells of patient with rheumatoid arthritis: Clinical efficacy of dietary antioxidants. Inflammopharmacology. 2017;25:595–607. doi: 10.1007/s10787-017-0397-1. [DOI] [PubMed] [Google Scholar]

Articles from Biosensors are provided here courtesy of Multidisciplinary Digital Publishing Institute (MDPI)

RESOURCES