Validation of the Russian Version of the Positive and Negative Syndrome Scale (PANSS-Ru) and Normative Data

Evgenia Ivanova; Anzalee Khan; Lora Liharska; Alexander Reznik; Sergey Kuzmin; Olga Kushnir; Alexey Agarkov; Nikolay Bokhan; Tatiana Pogorelova; Olga Khomenko; Ksenia Chernysheva; Margarita Morozova; George Rupchev; Taisia Lepilkina; Alexander Ozornin; Nina Ozornina; Nikolay Govorin; Anna Malakhova; Natalia Hmara; Oxana Shylova; Artsiom Hryhoryeu; Nataljya Ivanchikova; Irina Raevskaya; Pavel Gusak; Marina Skugarevskaya; Lewis A Opler

. 2018 Sep-Oct;15(09-10):32–48.

Validation of the Russian Version of the Positive and Negative Syndrome Scale (PANSS-Ru) and Normative Data

Evgenia Ivanova ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,^15,^✉, Anzalee Khan ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Lora Liharska ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Alexander Reznik ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Sergey Kuzmin ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Olga Kushnir ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Alexey Agarkov ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Nikolay Bokhan ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Tatiana Pogorelova ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Olga Khomenko ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Ksenia Chernysheva ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Margarita Morozova ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, George Rupchev ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Taisia Lepilkina ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Alexander Ozornin ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Nina Ozornina ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Nikolay Govorin ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Anna Malakhova ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Natalia Hmara ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Oxana Shylova ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Artsiom Hryhoryeu ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Nataljya Ivanchikova ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Irina Raevskaya ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Pavel Gusak ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Marina Skugarevskaya ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵, Lewis A Opler ^1,^2,^3,^4,^5,^6,^7,^8,^9,^10,^11,^12,^13,^14,¹⁵

PMCID: PMC6292716 PMID: 30588364

Abstract

Objective: The Positive and Negative Syndrome Scale (PANSS) is widely used to assess psychopathology. The Russian version (PANSSRu) has not been validated, and normative data for the Russian-speaking population currently do not exist. The aims of this study were to 1) complete linguistic validation for the PANSSRu, 2) perform psychometric validation of the Russian translation, and 3) present norms for the Russian and Belarusian population. Design: Validation and norms of the PANSS-Ru occurred in three stages—Stage I: linguistic validation; Stage II: psychometric validation of the translated version for 40 inpatients with schizophrenia and other psychoses; and Stage III: norms for 533 census-matched inpatients, outpatients, and healthy control subjects. Results: The rating criteria (PANSS-Ru), interview guide (SCI-PANSSRu), informant questionnaire (IQ-PANSS-Ru), and scoring form (PANSS QuikScore-Ru) were linguistically and psychometrically validated. Convergent validity between the PANSS subscale scores and total score with the Clinical Global Impressions-Severity Scale (CGI-S) were moderate (r=0.41–0.60) to high (r=0.61–0.80). Cronbach’s α (0.88) verified internal consistency, and intra-class correlation coefficient (ICC) comparisons had a range of 0.83. Percentile normative data collected from 533 subjects are presented. Conclusion: This is the largest population-based study providing linguistic and psychometric validation of the PANSS-Ru. Normative data can provide clinicians with a benchmark of psychopathology and inform the efficacy of treatment interventions.

Keywords: Schizophrenia, Positive and Negative Syndrome Scale (PANSS), psychometric validation, normative data, Russia-Belarus

The field of psychiatry is encountering major challenges in the signal-to-background ratio for the accurate detection of change in antipsychotic treatment response in clinical and research settings. Given the increasing application of psychometric instruments in clinical trials, academic research, and clinical practice, there is a need for validated instruments and normative data for each population that is being measured.

The Positive and Negative Syndrome Scale (PANSS) is used to assess symptom severity and treatment response in schizophrenia and other psychotic disorders.¹ It has become the “gold standard” for measurement of psychopathology in clinical trials. The PANSS evaluates the presence/absence and severity of positive symptoms, negative symptoms, and general psychopathology in individuals with schizophrenia or other psychotic disorders. It can capture and summarize the heterogeneity of psychotic disorders, while being reliable, valid, and sensitive to change over time.²^,³

Literature on cross-cultural translation and adaptation shows that cultural and linguistic disparities can affect the interpretation of quantitative instruments, thus limiting the reliability and validity of a scale’s construct when applied to a specific population.⁴^,⁵ Even when a translated version of a scale is administered in a population’s native language (e.g., Russian), there could be cultural differences in meaning, relevance, and verbal expression of concepts that might affect the validity of results obtained using the translation.⁶

Population-specific normative data for a scale provide valuable information on the performance (results) of any one individual (e.g., in a clinic or outpatient facility), or any group of individuals (e.g., in a clinical trial).⁷ Population norms from a sample of sufficient size and diversity make it possible for individual scores to be reliably compared to demographically similar individuals from a reference sample or from the healthy population. In addition to informing the interpretation of overall scores and levels of pathology, normative data allow for performance in specific domains of psychopathology to be accurately evaluated.⁸ For example, the authors of the PANSS developed norms for the United States population (provided in the PANSS Manual), but these norms are not generalizable to the entire country because they are based on subjects who were recruited primarily from one state and were not census-matched.² Additionally, many scales assessing psychiatric symptoms, such as the Inventory of Depression and Anxiety Symptoms (IDAS-II), the Brief Symptom Inventory (BSI), and the Hopkins Symptom Checklist (HSC), employ norms to provide information on the distribution of specific symptoms and on how these symptoms are related to demographic characteristics.⁹^–¹¹

In Europe, mental health disorders account for the second-highest burden of disease after cardiovascular illnesses.¹² This is especially true in the Russian Federation and surrounding countries currently experiencing economic and social transitions.¹³ However, to date, limited information has been gathered on the psychiatric population of Russian and Belarus. In addition, due to multiple barriers including the political misapplication of psychiatry in the former Soviet Union, psychiatric literature has lacked equitable data regarding the scientific merit and historical judgement of the Russian-Soviet perception of schizophrenia.¹⁴ Jenkins et al¹³ reported that absence of contemporary training materials and evidence-based guidelines hinder effective care of patients with schizophrenia in this region.

The most commonly used contemporary scale for assessing symptoms of schizophrenia is the PANSS. However, there have been no efforts to fully validate the Russian version of the PANSS, or to collect normative data in Russia. As a result, providing a culturally and linguistically validated Russian translation of the PANSS in conjunction with data on available population norms would help to characterize and improve evidence-based care in Russia and Belarus.

The objectives of this three-stage study were to 1) complete linguistic validation of the PANSS-Ru and all associated questionnaires, 2) perform psychometric validation of the Russian translation, and 3) establish normative PANSS-Ru data for the Russian and Belarusian population using scores from a representative sample. Cultural adaptation and linguistic validation of the PANSS-Ru scale, interview guide (SCI-PANSS-Ru), informant questionnaire (IQ-PANSS-Ru), and scoring form (PANSS-Ru QuikScore) for the Russian population examine whether any potential cultural disparities in the meaning of individual items or concepts are present. Normative data for the PANSS-Ru present a benchmark level of psychopathology in Russian patients with schizophrenia and other psychotic disorders as compared to healthy Russian individuals, and provide an indication of a patient’s clinical status as compared to the expected or abnormal range.

METHODS

Sample. Stage I. Linguistic validation of the translated version of the PANSS-Ru included cognitive debriefing on the SCI-PANSS-Ru interviews of 40 Russian participants with schizophrenia or other psychotic disorders from one psychiatric hospital in the Moscow region of Russia and on the IQ-PANSS-Ru interviews of their informants. Subjects and informants participated only in the cognitive debriefing portion of the linguistic validation procedure, which is described in detail further down. Linguistic validation was conducted in 2012.

Stage II. Psychometric validation of the PANSS-Ru included the SCI-PANSS-Ru and IQ-PANSS-Ru interviews of the 40 inpatients and their informants who had participated in the cognitive debriefing step of the linguistic validation procedure. Of the 40 subjects, 37 had a diagnosis of schizophrenia, one had a diagnosis of schizotypal disorder, one had a diagnosis of brief psychotic disorder, and one had a diagnosis of bipolar disorder. The three subjects with diagnoses of other psychotic disorders were included in the sample because assessment of psychotic features with the PANSS is not specific to schizophrenia. Data collection (administration of the SCI-PANSS-Ru interview) was performed by one of two raters present in the room with each subject, and both raters independently scored each SCI-PANSS-Ru interview. Completed IQPANSS-Ru forms were collected for all 40 subjects.

Stage III. Normative data collection utilized a subset of 533 individuals distributed geographically across Russia and Belarus who were representative of the 2010 Russian Federation Census and the 2009 Belarusian Census. Because of the social, historical, and cultural similarities of these populations, they were determined insufficiently different to be treated as two distinct groups.¹⁵^,¹⁶ Accordingly, subjects from Russia and Belarus were collapsed into a single analysis group. Normative data collection in Russia and Belarus was conducted from 2013 to 2015.

Of the 533 subjects, 365 were inpatients or outpatients with schizophrenia or other psychotic disorders (inpatient n=197, outpatient n=168), and 168 were healthy controls. There was a total of 17 site-specific raters from both countries and two independent raters. The role of the two independent raters was to evaluate the quality of the PANSS interviews being conducted by the 17 site-specific raters by listening to recordings of the interviews, independently scoring the interviews, and, if necessary, providing the site raters with feedback for improving the quality of the on-site interviews.

Recruitment. The subject recruitment process was the same for the 40 patients involved in Stage I and Stage II of the study and the 365 patients involved in Stage III of the study. For Stage I and Stage II, data were collected from the Moscow Region of Russia. For Stage III (normative data collection), data were collected from 10 sites across Russia and Belarus, located in Moscow, Saint Petersburg, Smolensk, Ekaterinburg, Tomsk, Chita, Gomel, and Gomel Region. All sites participating in the study were inpatient psychiatric hospitals with outpatient facilities. Substantial variations in working policy and practice between adjacent hospitals/clinics required access to raters and patients to be individually negotiated with each hospital and clinic team.

All raters who participated in the study were also clinicians at the study sites serving as primary treatment providers for the recruited patients. These clinicians knew the study requirements and only approached those of their patients whom they knew would most likely meet the inclusion criteria for the study. As a result, all screened patients were accepted into the study. The 168 healthy control participants were enrolled by word-of-mouth through clinicians’ colleagues, students, interns, friends, acquaintances, and relatives. No advertisements were used to enroll control patients.

Scales. All scales used in the study had existing Russian versions.

Mini-International Neuropsychiatric Interview (MINI). Psychiatric screening for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV)¹⁷ Axis I/II disorders was conducted using the MINI (MINI 60_AU11 0_Russian-Ru),¹⁸ a short diagnostic structured interview (DSI) assessing 17 disorders according to DSM-IV/DSM-IV-TR/ICD–10 diagnostic criteria.¹⁹

Positive and Negative Syndrome Scale (PANSS). Symptoms of schizophrenia were measured according to the subscale scores and total score on the PANSS-Ru,¹^,²⁰^,²¹ which consists of 30 items scored from 1 (Absent) to 7 (Extreme). Scores range from 30 to 210, with higher scores indicating more symptoms.

Clinical Global Impressions-Severity Scale (CGI-S).Global symptom severity was assessed using the CGI-S, a brief, stand-alone assessment of the clinician’s view of the patient’s overall functioning prior to and after initiating a study medication.²² The CGI-S consists of one item scored from 1 to 7, with higher scores indicating poorer functioning or more symptom severity.²²

Inclusion criteria. Each subject participating in one or more stages of the study underwent the same study procedures. After obtaining informed consent from the participant, the site investigator conducted the MINI diagnostic interview and asked about the use of drugs and alcohol (lifetime use and use within the past month). The subject advanced to the next stage of the study only if the MINI confirmed the presence of the subject’s primary diagnosis (or its absence in the case of control participants), and the DSM subtype of the diagnosis according to inclusion criteria. Subjects who did not meet the inclusion criteria for diagnosis continued participating in the study in the control group.

Stages I, II, and III inclusion criteria for all non-control subjects were as follows: 1) primary diagnosis of schizophrenia or other psychotic disorder (schizotypal disorder, schizoaffective disorder, delusional disorder, bipolar disorder with psychotic features, or other non-mood psychotic disorder) as established by the MINI;¹⁸ 2) 18 to 65 years of age (inclusive); 3) Russian as primary language; 4) available informant to complete the informant questionnaire (IQPANSS-Ru).

Stage III inclusion criteria for healthy control subjects comprised no signs of any psychotic disorder, no history of psychiatric disease, and no current formal psychiatric diagnosis prior to enrollment in the study, as confirmed by the MINI.

Exclusion criteria. Stages I, II, and III exclusion criteria were as follows: 1) primary diagnosis of substance-induced psychosis, other organic psychosis, or other psychotic disorders related to medical conditions, dementia, or intellectual disability; 2) medical/psychiatric condition preventing completion of SCI-PANSSRu; 3) more than 65 years of age; 4) inability to speak Russian; 5) no available informant or caregiver; 6) history of traumatic brain injury (TBI).

Data quality. Raters involved in study-related activities completed comprehensive rater training on the study protocol and assessments (PANSS-Ru, MINI, and CGI-S) prior to the start of the study.

To ensure quality of the Stage III PANSS-Ru normative data, the study team designed and implemented comprehensive rater training procedures, and employed an active monitoring process for collection of study data. With permission from study participants, all SCI-PANSS-Ru interviews were audio- or video-recorded for assessment of site rater interview quality. All recorded interviews were rated by one independent off-site rater, and one quarter of the interviews underwent additional co-rating by a second independent off-site rater. Site rater quality was measured using the Rater Applied Performance Scale (RAPS).²³

Statistical analysis. SPSS Version 23.0,²⁴ and SAS Version 9.2 were used for statistical analysis.²⁵

Stage I linguistic validation procedure. PANSS-Ru/SCI-PANSS-Ru/IQ-PANSS-Ru linguistic validation and cultural adaptation. Linguistic validation²⁶ and cultural adaptation²⁷ of the Russian language translation of the PANSS were developed according to standardized, internationally adopted methods (more detailed information is provided elsewhere²⁸). These methods comprise forward translation, back translation, examination of translation quality by bilingual speakers, and pilot testing using inpatients with schizophrenia and other psychotic disorders. The five specific steps employed in the linguistic validation of the PANSS-Ru are as follows:

1.
The PANSS was translated from English into Russian by two professional Russian translator-clinicians who provided two independent forward translations for all PANSS-Ru files (rating criteria, SCIPANSS, IQ-PANSS, and PANSS QuikScore). The wording of seven SCI-PANSS proverbs²⁹ was slightly modified by the translator-clinicians to better adapt the corresponding questions to Russian lifestyle and culture.
2.
The two Russian translator-clinicians reviewed and deliberated their independent Russian translations and produced a single draft of all PANSS-Ru components.
3.
An independent bilingual clinical psychologist, who was not involved in the forward translations of the scale and was blinded to the original PANSS scale and questionnaires, back-translated the drafted PANSS-Ru files into English.
4.
The publisher of the PANSS (Multi-Health Systems Inc. [MHS]) reviewed the back-translated PANSS-Ru version and compared it to the original English version to confirm that all precise meanings had been successfully conveyed.
5.
Cognitive debriefing of the SCI-PANSS-Ru and IQ-PANSS-Ru was performed on 40 Russian participants with schizophrenia and their informants to test their understanding and interpretation of the questionnaire. Patients and their informants were asked to provide feedback on the Russian translations of the SCI-PANSS and IQ-PANSS questions to ensure all translated questions were easily understood and correctly interpreted in Russian.

Stage II psychometric validation procedure. To assess the reliability and validity of the translation, psychometric validation of the final version of the PANSS-Ru was performed using the results from the 40 inpatient interviews. Each of the 40 SCI PANSS-Ru interviews was conducted by one of two certified raters present in the room with each participant, while the other rater observed the interview. Both raters independently scored the PANSS-Ru. Following the debriefing, no modifications to the SCIPANSS-Ru were made.

Descriptive statistics. Descriptive statistics were used to determine the distribution of clinical characteristics and were reported as mean ± standard deviation (SD) or proportion (%).

Reliability. To assess the PANSS-Ru interrater reliability, the study team anticipated that raters would be in agreement at least 80 percent of the time, with a relative error of 20 percent, thus necessitating a sample size of 40 patients.³⁰

Cronbach’s α coefficient was used as an index of internal consistency for each subscale score and total score, and rater concordance (i.e., comparison of scores of independent reviewers and raters) was performed for the healthy control group, inpatient schizophrenia group, and outpatient schizophrenia group. The optimal range of Cronbach’s α was above 0.70. To quantify rater concordance, intraclass correlation coefficients were used.³⁰ Reliability coefficients above 0.80 were considered satisfactory.

Validity. Convergent validity was assessed through Pearson’s correlations between the PANSS and CGI-S. All statistical analyses were conducted at the p=0.05 significance level.

Stage III normative data analysis procedure. Conventional descriptive analyses were used to present the site and sex distributions of the study sample and the raw score distributions of the three subscales, the total score, and the CGI-S score. Student’s t test was used to test for sex differences. Analyses of variance (ANOVA) were used to examine differences among sites. Multiple linear regression analyses were used to assess the effects of sex, age, and site on PANSS scores. All p-values were two-sided and p-values less than 0.05 were deemed statistically significant. Finally, raw PANSS subscale scores and Marder factor scores³² were converted into percentiles separately for demographic variables that turned out to be significant for severity of symptoms in schizophrenia. Rater interviewing skills were evaluated using the Rater Applied Performance Scale (RAPS), and all raters received at least a score of “Good” on all dimensions before receiving approval to begin assessing patients with the PANSS.

RESULTS

Stage I linguistic validation. Following consensus review of PANSS-Ru items, it was noted that seven of the proverbs appearing in the original scale could not be directly translated into Russian. These proverbs were: “plain as the nose on your face,” “carrying a chip on your shoulder,” “too many cooks spoil the broth,” “don’t cross the bridge until you come to it,” “what’s good for the goose is good for the gander,” “a rolling stone gathers no moss,” and “people who live in glass house should not throw stones at others.” Cultural adaptation of these proverbs for a Russian population consisted of substituting each English proverb with a Russian proverb with a culturally equivalent meaning.

Stage II psychometric validation. Forty individuals participated in the PANSS-Ru psychometric validation, of which 21 were male (52.5%) and 19 were female (47.5%). The mean patient age was 39 years (SD 13.99), with an age range of 18 to 65 years. Results substantiated the psychometric qualities of the adapted instrument. A Cronbach’s α coefficient of 0.88 verified the adapted instrument’s internal consistency. Results from intra-class correlation coefficient (ICC) comparisons verified the instrument’s inter-rater reliability (range=0.83). PANSS-Ru subscale scores were normally distributed. Pearson correlation between the subscale scores and the total score ranged from 0.76 to 0.86. Internal consistencies met the minimum criteria (α>0.745). Convergent validity between the PANSS subscale scores and total score with the CGI-S were moderate (r=0.41–0.60) to high (r=0.61–0.80).

Stage III development of normative data. Out of the 533 participants, 260 (49%) were male and 273 (51%) were female. Demographic data per group are presented in Table 1.

TABLE 1.

Demographic characteristics of Stage III sample

SEX	INPATIENT		OUTPATIENT		CONTROL
SEX	n	%	n	%	n	%
Male	100	51	82	49	78	46
Female	97	49	86	51	90	54
AGE	MEAN	SD	MEAN	SD	MEAN	SD
AGE	37.99	12.00	39.00	11.70	37.00	12.00

Open in a new tab

SD: standard deviation

Descriptive statistics for the three subscale scores and the total score for the inpatient, outpatient, and control groups are presented in Table 2 and Figure 1. The score breakdown for each of the groups is as follows:

TABLE 2.

Descriptive statistics for all Stage III groups

ALL GROUPS	GROUP	n	MEAN	95% CONFIDENCE INTERVAL FOR MEAN
ALL GROUPS	GROUP	n	MEAN	LOWER BOUND	UPPER BOUND	MEDIAN	VARIANCE	SD	MIN	MAX	SKEWNESS	KURTOSIS	STD ERROR
PANSS Positive Symptoms	Control	168	12.5	11.96	13.03	12	8.835	2.972	7	22	0.536	-0.028	0.27
	Inpatient	197	21.78	20.67	22.89	21	40.204	6.341	9	36	0.261	-0.549	0.56
	Outpatient	168	17.56	16.53	18.59	17	34.119	5.841	8	34	0.57	-0.245	0.522
PANSS Negative Symptoms	Control	168	10.85	10.31	11.4	10	9.128	3.021	7	22	0.862	0.524	0.275
	Inpatient	197	23.61	22.74	24.48	24	24.807	4.981	7	36	-0.214	0.176	0.44
	Outpatient	168	20.24	19.28	21.2	20	29.539	5.435	8	34	0.026	-0.244	0.486
PANSS General Psychopathology	Control	168	26.92	25.77	28.07	26	40.96	6.4	17	44	0.467	-0.405	0.582
	Inpatient	197	44.27	42.64	45.89	44	86.307	9.29	24	67	0.142	0.039	0.821
	Outpatient	168	37.55	35.92	39.18	37	84.636	9.2	16	58	-0.005	-0.557	0.823
PANSS Total	Control	168	50.26	48.49	52.04	50	97.679	9.883	34	82	0.526	0.158	0.898
	Inpatient	197	89.66	86.61	92.71	89.5	304.133	17.439	51	133	0.175	-0.041	1.541
	Outpatient	168	75.35	72.2	78.5	75	316.972	17.804	36	117	0.152	-0.616	1.592
CGI-S	Control	168	1.33	1.21	1.45	1	0.44	0.663	1	5	2.826	10.389	0.06
	Inpatient	197	4.4	4.22	4.58	4	1.092	1.045	1	7	0.022	0.463	0.092
	Outpatient	168	3.6	3.43	3.77	3	0.968	0.984	1	7	0.517	0.7	0.088

Open in a new tab

PANSS: Positive and Negative Syndrome Scale; Clinical Global Impressions-Severity (CGI-S) SD: standard deviation

FIGURE 1. — Comparison of PANSS-Ru score distributions on Positive Symptom subscale, Negative Symptom subscale, General Psychopathology subscale, and Total Score for control, inpatient, and outpatient groups

• Inpatient: mean Total PANSS-Ru score: 89.82 (SD=17.41); mean Positive Symptoms score: 21.83 (SD=6.35); mean Negative Symptoms score: 23.63 (SD=4.99); mean General Psychopathology score: 44.36 (SD=9.26).

• Outpatient: mean Total PANSS-Ru score: 75.35 (SD=17.88); mean Positive Symptoms score: 17.56 (SD=5.87); mean Negative Symptoms score: 20.25 (SD=5.46); mean General Psychopathology score: 37.55 (SD=9.24).

• Control: mean Total PANSS-Ru score: 50.26 (SD=9.88); mean Positive Symptoms score: 12.5 (SD=2.97); mean Negative Symptoms score: 10.85 (SD=3.02); mean General Psychopathology score: 26.92 (SD=6.40).

The values for skewness and kurtosis suggest positively skewed distributions for the Russian translated version of the CGI-S for the control group, as expected. Therefore, scores representing percentiles are given to constitute normative data (Table 3). Significant differences were observed in age (F=3.552, p<0.001) across groups, with the outpatient group being older. For all other demographic variables, i.e., education (χ²=1.690, p=0.698), site/location (χ²=2.000, p=0.597), and sex (χ²=2.012, p=0.549), no statistical differences were found according to the results of the univariate analysis of variance (ANOVA) and the chi-square tests. Subjects with higher PANSSRu scores were older. Therefore, we did not stratify the sample according to region, sex, or education. All normative data for the control, inpatient, and outpatient groups are stratified by age and presented in Appendix 1.

TABLE 3.

PANSS-Ru subscale scores and total score percentiles for Stage III inpatient, control, and outpatient groups

ALL GROUPS	GROUP	PERCENTILES
		PERCENTILES							5	10	25	50	75	90	95
		PANSS-Ru Positive Symptoms	Control	9	9	10	12	14	17	18
Inpatient	12		13	17.25	21	26	31	33.55
Outpatient	10		11	13	17	22	25	29
PANSS-Ru Negative Symptoms	Control	7	7	9	10	13	15	16
	Inpatient	16	17.9	20	24	27	30	31
	Outpatient	11	12.6	17	20	24	28	30
PANSS-Ru General Psychopathology	Control	18	19	22	26	31	36	39
	Inpatient	28.45	31	38.25	44	49	57	61.55
	Outpatient	22.3	25	31	37	44.5	50	52.7
PANSS-Ru Total	Control	36	37.2	42	50	56	63.6	67
	Inpatient	59.35	69	78.25	89.5	100	113.1	123
	Outpatient	47.3	52.2	62	75	89	99	105.4
CGI-S	Control	1	1	1	1	2	2	2
	Inpatient	3	3	4	4	5	6	6
	Outpatient	2	3	3	3	4	5	5

Open in a new tab

CGI-S: Clinical Global Impressions-Severity; PANSS-Ru: Positive and Negative Syndrome Scale-Russian version

Significant correlations were observed between the Russian CGI-S and the PANSSRu Negative Symptoms subscale, General Psychopathology subscale, and Total Score, for the control, inpatient, and outpatient groups. Scores on the Russian CGI-S in the control group did not significantly correlate with the PANSS-Ru Positive Symptoms subscale, possibly due to the limited variability of scores on the Russian CGI-S. This is shown in Table 4.

TABLE 4.

Correlations of PANSS-Ru and CGI-S

PATIENT TYPE	CORRELATION	POSITIVE SYMPTOMS	NEGATIVE SYMPTOMS	GENERAL PSYCHOPATHOLOGY	PANSS-RU TOTAL
Control	Pearson Correlation	0.149	0.233^*	0.444^**	0.404^**
Control	Sig. (2-tailed)	0.104	0.010	<0.001	<0.001
Inpatient	Pearson Correlation	0.638^**	0.487^**	0.492^**	0.633^**
Inpatient	Sig. (2-tailed)	<0.001	<0.001	<0.001	<0.001
Outpatient	Pearson Correlation	0.552^**	0.532^**	0.568^**	0.637^**
Outpatient	Sig. (2-tailed)	<0.001	<0.001	<0.001	<0.001

Open in a new tab

p≤0.05

^**

p≤0.01; CGI-S: Clinical Global Impressions-Severity; PANSS-Ru: Positive and Negative Syndrome Scale-Russian version

DISCUSSION

Prior to this study, Russian-speaking clinicians did not have access to a fully validated translation of the PANSS and associated SCI-PANSS and IQ-PANSS. Despite the widespread use of the PANSS in clinical and research settings, adequate normative data and reliability and validity data for the Russian language version have not been presented previously. Interpretation of the PANSS has been based on point and percent change over time, and normative data for the PANSS have been derived from a sample predominantly comprising hospitalized English-speaking inpatients in acute care settings in the United States.³ The current study complements the English psychometric properties of the PANSS and provides normative data derived from a sample known to be broadly representative of the general adult population in Russia and Belarus.

Stage I linguistic validation. Almost all the content in the SCI-PANSS interview is generalizable across cultures and therefore not subject to culture-specific modifications of the text. The one common and expected exception to this is the English proverbs used in some SCI-PANSS items. Seven of these proverbs were identified as not being directly translatable into Russian and thus necessitated culture-specific substitutions. For example, “people who live in glass houses should not throw stones at others” was substituted with the Russian proverb “Как аукнется, так и откликнется,” which roughly back-translates to “what goes around comes around.”

Stage II psychometric validation. The reliability of the PANSS-Ru, as measured by Cronbach’s α, was 0.88 for the total score, with a range from 0.62 (Blunted Affect) to 0.92 (Depression). The narrowness of the range associated with the PANSS-Ru indicates that it can be regarded as providing accurate estimates of the internal consistency of the PANSS-Ru in the adult inpatient population. There is no absolute criterion for the reliability of an instrument, but, as a rule of thumb, Anastasi has suggested that Cronbach’s α should be at least 0.85 if the intention is to use the instrument to draw inferences concerning an individual.³⁴ Per this criterion, all the PANSS-Ru total scores can be viewed as possessing adequate reliability. Furthermore, it is not unexpected that Blunted Affect would have the lowest reliability: Khan et al³⁵ observed moderate differential item functioning in a Russian sample when compared to scores of the item in a United States sample. Other studies examining cross-cultural differences in the PANSS also report Blunted Affect as the second most difficult item for interviewers to score for most regions including Southern Europe (Δ=0.30), Eastern Asia (Δ=0.28), Russia and Ukraine (Δ=0.22), and India (Δ=0.10).³⁶ Because Blunted Affect represents decreased emotional expressivity and diminished facial expression,³⁷^–⁴⁰ it could be representative of cultural expectations.

Mean subscale scores and total score were equivalent to the United States general population norms within 13 percent. However, there was a difference of more than five norm-based scoring points for mean General Psychopathology scores. The PANSS-Ru scores were normally distributed, resulting in compatible percentile values reported by the original author.⁴¹

Stage III development of normative data. Normative data from a representative sample of sufficient size and diversity are important to clinicians and researchers because it makes it possible for individual scores to be reliably compared to scores of demographically similar individuals from a reference sample. Normative data also serve as an indication of how a patient’s clinical status compares to the expected conditions (inpatient or outpatient, and/or status on each specific domain). The only previous normative data for the PANSS come from Kay, Opler, and Fiszbein² (N=101) and Perkins et al³ (N=240) for United States samples.

The normative tables presented in Appendix 1 were adopted to permit conversion of raw scores into percentiles for all three PANSS-Ru subscale scores and total score for controls, inpatients, and outpatients. The scores for the present inpatient sample are mean Positive Symptoms score: 21.83 (SD=6.35) and mean Negative Symptoms score: 23.63 (SD=4.99). These means are slightly lower than the norms presented by Kay, Opler, and Fiszbein:² mean Positive Symptoms score=19.86 (SD=6.27) and mean Negative Symptoms score=21.75 (SD=6.21). These minor differences could be due to the fact that the data used by Perkins et al³ were derived from a sample comprising 240 predominantly medicated inpatients.

Limitations. The normative data provided for the PANSS-Ru can be readily applied to clinical cases with inpatients and outpatients. Given that the current sample includes healthy controls, clinicians can easily determine if a patient, whether healthy or presenting symptoms of schizophrenia, falls in the expected or abnormal range. However, some limitations to this data must be considered.

First, given that the current sample was mainly collected from populated urban centers across Russia and Belarus, additional samples from rural settings are necessary for developing a more robust profile of the population. Thus, norms from this sample might not be as appropriate for minorities or individuals from other regions of Russia and Belarus. Additional data representing other parts of Russia (Caucasus region, Kaliningrad exclave, Crimea peninsula, Kamchatka peninsula, and Sakhalin island) and the central and western parts of Belarus (Grodno, Mogilev, Brest, Vitebsk, and Minsk regions) would be beneficial.

Second, although scores on the PANSS and the CGI-S are correlated, these scales are not synonymous,⁴² and data comparing the PANSSRu to other clinical measures (such as the Scale for the Assessment of Negative Symptoms [SANS], the Scale for the Assessment of Positive Symptoms [SAPS], and the Brief Psychiatric Rating Scale [BPRS]) were not available to further assess construct validity.

Third, because the range of psychopathology for this study did not include individuals with severe mental illness, the study has inherent participation bias. Several survey studies suggest that elevated psychopathology contributes to premature dropout.⁴³^–⁴⁵ Additionally, patients experiencing greater psychiatric symptom severity have been reported to be more likely to miss clinic appointments,⁴⁶ and thus are likely to make less cooperative participants in research trials. Therefore, further investigations into the behaviors of participants and nonparticipants in clinical research would help elucidate the degree to which participant selection bias is of concern for research in psychiatric settings. Participation bias also affected the control group; there is extensive evidence that poor recruitment or retention of participants in clinical research is widespread and leads to delays in the start or completion of both academic and commercially funded studies.⁴⁷^–⁵⁰ In Russia, there has been limited investigation on how national-level factors (e.g., communication regarding risks and benefits of research, resources from the community, recruitment processes such as advertising, and regional perception of mental illness) operate as barriers or facilitators to research recruitment.

Fourth, although beyond the scope of the present investigation, it would be valuable to formally examine whether the PANSS-Ru is factorially invariant. In the present study, for example, it was shown that demographic variables (e.g., age) exerted significant effects on PANSS-Ru scores, while sex exerted a negligible effect. Simultaneous multi-group CFA could be employed to test whether this latent structure is invariant across age groups and sex.⁵¹^,⁵² More importantly, this method could be used to examine whether the PANSS is factorially invariant across cultures and across healthy and clinical populations. Examination of this latter issue would provide important information for those using the PANSS in research or practice.

CONCLUSION

The current study is the largest population-based validation of the Russian version of the PANSS and presents normative data for Russia and Belarus. It has shown that within the existing regulations and organizational structures of the Russian Federation, it is possible to establish rigorous training programs and structured clinical trials conducted by multidisciplinary teams to improve linguistic and cultural translations and assess psychometric properties of measurement tools. Although it would be beneficial to compare the results of this validation study with those from neighboring geographic regions, linguistic and cultural variations between countries necessitate the development of country-specific psychometric adaptations of psychiatric rating scales and training programs. These are not currently available.

The present analysis shows that the PANSSRu has robust psychometric properties in a large sample drawn from the general census-matched adult population of healthy controls and participants with schizophrenia. The normative data presented will provide clinicians with an accurate and valid benchmark of level of psychopathology in Russian patients with schizophrenia or other psychotic disorders as compared to healthy Russian individuals. Because the norms presented are based on a sample that was broadly representative of the general adult population in terms of age, sex, and education, these norms will serve as a useful supplement for assessing the schizophrenia profile in Russia and Belarus.

The normative data presented in this paper are the only such currently available data for the PANSS-Ru. Use of these population-specific norms will improve outcome criteria by leaving less opportunity for misinterpretation, as the data will inform the efficacy of behavioral and pharmacologic interventions and the modification of treatment plans and targets. The results of the current study will provide clinicians with the empirical guidance that, until now, has been missing in psychiatry research and practice.

IN MEMORIAM

This work is dedicated to the memory of our wonderful colleague, the brilliant scientist and devoted clinician, Dr. Lewis A. Opler, who recently passed away.

ACKNOWLEDGMENTS

The authors would like to thank the Health and Development Foundation (HDF) (Moscow, Russia) for their assistance in making this study possible. The authors would also like to thank the following individuals: Dr. Mark Opler (MedAvante-ProPhase, New York, New York) for his insight and expertise; Dr. Elena Dmitrieva and Sergey Frolov (HDF, Moscow, Russia) for their support; Dr. Guillermo DiClemente and Dr. Christian Yavorsky (CROnos, CCS; Lambertsville, New Jersey) for their assistance with the data monitoring process used in the study; Dr. David V. Sheehan (University of South Florida, Tampa, Florida) for his recognition of the merit of the study, and for allowing the use of the Russian translation of the MINI in the study; Dr. Artem Aldushin (A.S. Puchkov Department of Health, Moscow, Russia) for his help with the co-rating of study assessments; and James Lefkowitz for developing and providing the compliance and data management system for the study.

Appendix

APPENDIX 1A.

Control group descriptive statistics for PANSS-Ru by age

ALL GROUPS	AGE GROUP	n	MEAN	95% CONFIDENCE INTERVAL FOR MEAN
ALL GROUPS	AGE GROUP	n	MEAN	LOWER BOUND	UPPER BOUND	MEDIAN	VARIANCE	SD	MIN	MAX	SKEWNESS	KURTOSIS	STD.ERROR
PANSS-Ru Positive Symptoms	≤25 years	40	11.62	10.62	12.62	12	4.848	2.202	7	15	-0.332	-0.504	0.48
	26–35 years	43	12.4	11.45	13.34	12	9.435	3.072	7	18	0.279	-1.058	0.468
	36–45 years	37	12.52	11.45	13.58	12	7.259	2.694	9	18	0.328	-0.896	0.519
	46–55 years	28	13.33	11.61	15.06	13	12	3.464	9	22	0.945	1.095	0.816
	≥56 years	20	13.08	10.81	15.36	12.5	12.811	3.579	8	20	0.604	-0.229	1.033
PANSS-Ru Negative Symptoms	≤25 years	40	9.86	8.9	10.82	10	4.429	2.104	7	16	1.026	2.349	0.459
	26–35 years	43	10.7	9.83	11.57	10	7.978	2.825	7	17	0.649	-0.549	0.431
	36–45 years	37	10.7	9.39	12.01	9	10.986	3.314	7	19	0.938	0.025	0.638
	46–55 years	28	11.94	10.06	13.83	12	14.408	3.796	7	22	0.927	1.394	0.895
	≥56 years	20	11.83	10.06	13.61	12.5	7.788	2.791	7	15	-0.337	-1.469	0.806
PANSS-Ru General Psychopathology	≤25 years	40	26.81	23.95	29.67	26	39.362	6.274	18	38	0.253	-1.144	1.369
	26–35 years	43	26.23	24.39	28.07	26	35.802	5.983	17	42	0.639	0.052	0.912
	36–45 years	37	26.04	23.93	28.14	26	28.268	5.317	17	36	-0.06	-0.684	1.023
	46–55 years	28	28.5	24.46	32.54	27.5	66.029	8.126	19	44	0.529	-0.957	1.915
	≥56 years	20	29.17	24.4	33.94	29.5	56.333	7.506	17	41	-0.163	-0.494	2.167
PANSS-Ru Total	≤25 years	40	48.29	44.61	51.96	49	65.314	8.082	34	61	-0.306	-0.981	1.764
	26–35 years	43	49.33	46.35	52.3	49	93.177	9.653	36	77	0.71	0.279	1.472
	36–45 years	37	49.26	46.02	52.5	51	67.046	8.188	34	66	-0.159	-0.419	1.576
	46–55 years	28	53.78	47.4	60.15	52.5	164.418	12.823	37	82	0.516	-0.492	3.022
	≥56 years	20	54.08	46.87	61.3	55.5	128.992	11.357	36	75	-0.002	-0.16	3.279
CGI-S	≤25 years	40	1.33	1.11	1.55	1	0.233	0.483	1	2	0.763	-1.579	0.105
	26–35 years	43	1.37	1.12	1.62	1	0.668	0.817	1	5	3.042	10.506	0.125
	36–45 years	37	1.22	0.97	1.48	1	0.41	0.641	1	4	3.571	14.095	0.123
	46–55 years	28	1.44	1.14	1.75	1	0.379	0.616	1	3	1.085	0.387	0.145
	≥56 years	20	1.25	0.96	1.54	1	0.205	0.452	1	2	1.327	-0.326	0.131
PANSS-Ru Marder Positive	≤25 years	40	13.147	12.318	13.976	13	5.644	2.37579	8	18	0.146	0.097	N/A
	26–35 years	43	12.833	12.106	13.559	12	7.085	2.66175	9	22	1.264	2.686	N/A
	36–45 years	37	13.2	12.284	14.115	13	7.106	2.66569	8	20	0.583	0.384	N/A
	46–55 years	28	14.416	12.76	16.072	14	15.384	3.92225	9	25	1.305	2.198	N/A
	≥56 years	20	13.714	12.123	15.305	13	12.214	3.49489	9	22	0.955	0.184	N/A
PANSS-Ru Marder Negative	≤25 years	40	8.8529	7.9849	9.721	8.5	6.190	2.48794	7	17	1.834	3.719	N/A
	26–35 years	43	9.9630	9.1983	10.727	9	7.848	2.80137	7	20	1.152	1.707	N/A
	36–45 years	37	9.3143	8.3987	10.229	9	7.104	2.66537	7	17	1.025	0.381	N/A
	46–55 years	28	10.208	8.8616	11.555	10	10.172	3.18937	7	19	0.942	0.839	N/A
	≥56 years	20	9.6667	8.3605	10.972	8	8.233	2.86938	7	16	0.904	-0.380	N/A
PANSS-Ru Marder Anxiety	≤25 years	40	8.0000	7.1584	8.8416	8.5	5.818	2.41209	4	13	-0.289	-0.549	N/A
	26–35 years	43	7.4259	6.7774	8.0744	7.5	5.645	2.37600	4	13	0.284	-0.410	N/A
	36–45 years	37	7.4286	6.5616	8.2955	7	6.370	2.52384	4	15	0.904	1.346	N/A
	46–55 years	28	8.5000	7.1589	9.8411	9	10.087	3.17600	4	14	0.120	-1.376	N/A
	≥56 years	20	7.5714	6.4511	8.6917	7	6.057	2.46113	4	13	0.572	0.077	N/A
PANSS-Ru Marder Disorganized	≤25 years	40	10.000	9.0320	10.968	10	7.697	2.77434	7	18	0.796	0.470	N/A
	26–35 years	43	10.388	9.6216	11.156	10	7.903	2.81114	7	17	0.516	-0.685	N/A
	36–45 years	37	10.571	9.6132	11.529	11	7.782	2.78954	7	17	0.385	-0.527	N/A
	46–55 years	28	12.375	10.881	13.868	13	12.505	3.53630	7	19	0.170	-0.795	N/A
	≥56 years	20	11.904	10.245	13.564	12	13.290	3.64561	7	21	0.796	0.507	N/A
PANSS-Ru Marder Hostility	≤25 years	40	10.000	9.0320	10.968	10	7.697	2.77434	7	18	0.796	0.470	N/A
	26–35 years	43	10.388	9.6216	11.156	10	7.903	2.81114	7	17	0.516	-0.685	N/A
	36–45 years	37	10.571	9.6132	11.529	11	7.782	2.78954	7	17	0.385	-0.527	N/A
	46–55 years	28	12.375	10.881	13.868	13	12.505	3.53630	7	19	0.170	-0.795	N/A
	≥56 years	20	11.904	10.245	13.564	12	13.290	3.64561	7	21	0.796	0.507	N/A

Open in a new tab

CGI-S Clinical Global Impressions-Severity; PANSS: Positive and Negative Syndrome Scale; SD: standard deviation; N/A: not applicable

APPENDIX 1B.

Control group average weighted scores with percentiles for the PANSS-Ru by age

ALL GROUPS	AGE GROUP	PERCENTILES
ALL GROUPS	AGE GROUP	5	10	25	50	75	90	95
PANSS-Ru Positive Symptoms	≤25 years	7.1	8.2	10	12	13	14.8	15
	26–35 years	8.2	9	10	12	15	17	17.8
	36–45 years	9	9	10	12	15	16.2	17.6
	46–55 years	9	9	10.75	13	15	19.3	N/A
	≥56 years	8	8.3	11	12.5	15.75	19.4	N/A
PANSS-Ru Negative Symptoms	≤25 years	7	7	8.5	10	11	12	15.6
	26–35 years	7	7	9	10	13	15	16.6
	36–45 years	7	7	8	9	13	15.4	18.2
	46–55 years	7	7	9	12	14.25	16.6	N/A
	≥56 years	7	7.6	9.25	12.5	14	15	N/A
PANSS-Ru General Psychopathology	≤25 years	18	18.2	21.5	26	33.5	35.8	37.8
	26–35 years	18	19	20	26	29	35	38.6
	36–45 years	17	17.8	22	26	30	34	35.2
	46–55 years	19	19.9	20.75	27.5	34.5	42.2	N/A
	≥56 years	17	17.3	23.25	29.5	34.75	40.4	N/A
PANSS-Ru Total	≤25 years	34	35	41	49	56	57.8	60.7
	26–35 years	36.2	37	41	49	54	63.8	66.8
	36–45 years	34.4	35.8	44	51	55	61	64
	46–55 years	37	38.8	42.5	52.5	64.75	70.3	N/A
	≥56 years	36	36.9	43.25	55.5	59	72.6	N/A
CGI-S	≤25 years	1	1	1	1	2	2	2
	26–35 years	1	1	1	1	2	2	3.6
	36–45 years	1	1	1	1	1	2	3.2
	46–55 years	1	1	1	1	2	2.1	N/A
	≥56 years	1	1	1	1	1.75	2	N/A
PANSS-Ru Marder Positive	≤25 years	8.75	10	12	13	14	17	18
	26–35 years	9	10	11	12	14	16	18.75
	36–45 years	8.8	10	11	13	15	17	19.2
	46–55 years	9.25	10	11.5	14	15.75	21	24.75
	≥56 years	9.1	10	11	13	15.5	19	21.7
PANSS-Ru Marder Negative	≤25 years	7	7	7	8.5	10	11.5	16.25
	26–35 years	7	7	8	9	12	13.5	15.25
	36–45 years	7	7	7	9	11	13	14.6
	46–55 years	7	7	7	10	12	14.5	18
	≥56 years	7	7	7	8	12	14.6	15.9
PANSS-Ru Marder Anxiety	≤25 years	4	4	6	8.5	10	10.5	12.25
	26–35 years	4	4	6	7.5	9	10	12.25
	36–45 years	4	4	6	7	9	10.4	13.4
	46–55 years	4	4.5	5.25	9	11	13	13.75
	≥56 years	4	4	6	7	9	11.8	12.9
PANSS-Ru Marder Disorganized	≤25 years	7	7	7	10	12	13.5	15.75
	26–35 years	7	7	8	10	13	14.5	16
	36–45 years	7	7	9	11	12	14.4	16.2
	46–55 years	7	7.5	9	13	15	17.5	19
	≥56 years	7	7.2	9	12	14.5	17.6	20.7
PANSS-Ru Marder Hostility	≤25 years	4	4	4	6.5	8	11	12.25
	26–35 years	4	4	4	6.5	10	12	12
	36–45 years	4	4	5	7	9	10.4	12
	46–55 years	4	4.5	5	8.5	11.75	12.5	13
	≥56 years	4	4	4	6.5	8	11	12.25

Open in a new tab

CGI-S: Clinical Global Impressions-Severity; PANSS-Ru: Positive and Negative Syndrome Scale-Russian version

APPENDIX 1C. — Comparison of PANSS-Ru score distributions on Positive Symptoms subscale, Negative Symptoms subscale, General Psychopathology subscale, and Total Score for control group by age

APPENDIX 1D.

Inpatient group descriptive statistics for PANSS-Ru by age

ALL GROUPS	AGE GROUP	n	MEAN	95% CONFIDENCE INTERVAL FOR MEAN
ALL GROUPS	AGE GROUP	n	MEAN	LOWER BOUND	UPPER BOUND	MEDIAN	VARIANCE	SD	MIN	MAX	SKEWNESS	KURTOSIS	STD ERROR
PANSS-Ru Positive Symptoms	≤25 years	43	21.46	18.88	24.04	21.5	40.738	6.383	9	34	0.06	-0.694	1.252
	26–35 years	43	20.61	18.49	22.72	20	35.559	5.963	11	36	0.393	-0.263	1.038
	36–45 years	42	21.84	19.26	24.42	20	49.34	7.024	11	36	0.529	-0.446	1.262
	46–55 years	41	22.63	20.21	25.05	22	42.033	6.483	10	35	0.102	-0.577	1.184
	≥56 years	26	24.25	20.63	27.87	24	18.786	4.334	19	31	0.279	-1.438	1.532
PANSS-Ru Negative Symptoms	≤25 years	43	22.35	20.23	24.46	21.5	27.515	5.246	13	35	0.572	-0.012	1.029
	26–35 years	43	23	21.35	24.65	24	21.688	4.657	7	29	-1.207	2.85	0.811
	36–45 years	42	23.23	21.6	24.85	23	19.714	4.44	14	31	-0.072	-0.977	0.797
	46–55 years	41	24.6	22.66	26.54	25.5	27.007	5.197	12	33	-0.573	-0.148	0.949
	≥56 years	26	28	24.1	31.9	26.5	21.714	4.66	23	36	0.757	-0.697	1.648
PANSS-Ru General Psychopathology	≤25 years	43	43.88	39.87	47.9	47	98.826	9.941	24	66	-0.306	0.135	1.95
	26–35 years	43	42.88	39.84	45.92	43	73.61	8.58	28	61	0.48	-0.347	1.494
	36–45 years	42	43.71	40.04	47.38	44	100.28	10.014	26	67	0.29	0.239	1.799
	46–55 years	41	45.53	42.15	48.92	44.5	82.051	9.058	24	65	0.199	0.79	1.654
	≥56 years	26	48.62	41.78	55.47	47	67.125	8.193	37	62	0.6	-0.123	2.897
PANSS-Ru Total	≤25 years	43	87.69	81.21	94.17	90	257.582	16.049	56	123	-0.077	-0.085	3.148
	26–35 years	43	86.48	80.59	92.38	84	276.82	16.638	51	126	0.359	-0.024	2.896
	36–45 years	42	88.77	81.89	95.66	89	352.381	18.772	52	132	0.244	0.076	3.372
	46–55 years	41	92.77	85.95	99.58	91.5	332.875	18.245	51	133	0.184	0.484	3.331
	≥56 years	26	100.88	89.27	112.48	99.5	192.696	13.882	80	120	0.141	-0.704	4.908
CGI-S	≤25 years	43	4.54	4.25	4.82	4.5	0.498	0.706	3	6	0.223	-0.073	0.138
	26–35 years	43	4.15	3.8	4.51	4	1.008	1.004	1	6	-0.321	2.433	0.175
	36–45 years	42	4.29	3.8	4.78	4	1.813	1.346	2	7	0.305	-0.586	0.242
	46–55 years	41	4.53	4.16	4.91	5	1.016	1.008	3	7	0.227	-0.057	0.184
	≥56 years	26	4.88	4.18	5.57	5	0.696	0.835	4	6	0.277	-1.392	0.295
PANSS-Ru Marder Positive	≤25 years	43	24.897	22.7155	27.0793	24	45.305	6.73090	13	40	0.325	-0.524	N/A
	26–35 years	43	23.882	22.0018	25.7629	24	44.706	6.68625	10	42	0.156	-0.108	N/A
	36–45 years	42	25.065	22.7116	27.4189	25	62.818	7.92577	11	43	0.443	-0.369	N/A
	46–55 years	41	25.909	23.8504	27.9678	25.5	45.852	6.77141	12	38	-0.050	-0.794	N/A
	≥56 years	26	26.941	23.1960	30.6863	26	53.059	7.28415	10	37	-0.703	0.265	N/A
PANSS-Ru Marder Negative	≤25 years	43	18.615	16.5630	20.6677	20	40.085	6.33127	8	31	0.042	0.811	N/A
	26–35 years	43	20.098	18.5379	21.6582	21	30.770	5.54709	7	31	-0.352	-0.253	N/A
	36–45 years	42	19.891	18.2252	21.5574	20	31.477	5.61042	7	32	-0.214	0.071	N/A
	46–55 years	41	21.34	19.2686	23.4133	22.5	46.462	6.81634	7	36	-0.248	-0.407	N/A
	≥56 years	26	22.00	18.5895	25.4105	22	44.000	6.63325	11	34	-0.194	-0.340	N/A
PANSS-Ru Marder Anxiety	≤25 years	43	10.23	8.9853	11.4762	10	14.761	3.84202	4	19	0.293	-0.511	N/A
	26–35 years	43	9.8039	8.7682	10.8396	9	13.561	3.68250	4	17	0.247	-0.848	N/A
	36–45 years	42	11.195	10.2006	12.1907	11.5	11.228	3.35075	4	21	0.141	0.468	N/A
	46–55 years	41	10.25	9.2950	11.2050	10	9.866	3.14106	4	17	0.249	-0.459	N/A
	≥56 years	26	9.1176	7.3284	10.9069	8	12.110	3.47998	4	16	0.635	-0.721	N/A
PANSS-Ru Marder Disorganized	≤25 years	43	19.461	17.9726	20.9505	20	21.097	4.59317	11	27	-0.193	-1.022	N/A
	26–35 years	43	19.058	17.6679	20.4497	19	24.456	4.94535	10	31	0.250	-0.348	N/A
	36–45 years	42	19.847	18.3985	21.2972	19	23.821	4.88065	9	32	0.216	-0.102	N/A
	46–55 years	41	21.022	19.5706	22.4749	21	22.813	4.77634	11	33	0.665	0.864	N/A
	≥56 years	26	22.47	19.9249	25.0163	23	24.515	4.95123	12	32	-0.281	0.238	N/A
PANSS-Ru Marder Hostility	≤25 years	43	9.1282	8.0073	10.2491	8	11.957	3.45786	4	17	0.566	-0.574	N/A
	26–35 years	43	7.9608	6.8965	9.0250	7	14.318	3.78397	4	18	0.914	-0.006	N/A
	36–45 years	42	9.4783	8.1142	10.8423	8	21.100	4.59342	4	22	0.853	0.281	N/A
	46–55 years	41	9.7273	8.4654	10.9891	9	17.226	4.15045	4	20	0.873	0.074	N/A
	≥56 years	26	8.5294	6.6305	10.4283	8	13.640	3.69320	4	15	0.635	-0.801	N/A

Open in a new tab

CGI-S: Clinical Global Impressions-Severity; PANSS: Positive and Negative Syndrome Scale; SD: standard deviation; N/A: not applicable

APPENDIX 1E.

Inpatient group average weighted scores with percentiles for the PANSS-Ru by age

ALL GROUPS	AGE GROUP	PERCENTILES
ALL GROUPS	AGE GROUP	5	10	25	50	75	90	95
PANSS-Ru Positive Symptoms	≤25 years	10.05	13.4	16	21.5	27	31	32.95
	26–35 years	12.4	13	14.5	20	25.5	28	31.8
	36–45 years	11.6	12.2	18	20	26	33.8	36
	46–55 years	11.1	13.2	18.75	22	28.25	32.9	33.9
	≥56 years	19	19	20.25	24	27.75	N/A	N/A
PANSS-Ru Negative Symptoms	≤25 years	14.05	16	18.75	21.5	25.5	30	33.25
	26–35 years	14	18	19.5	24	26	29	29
	36–45 years	15.8	17.2	19	23	28	28.8	30.4
	46–55 years	13.65	17.1	20.75	25.5	28.25	31	32.45
	≥56 years	23	23	24.25	26.5	32.25	N/A	N/A
PANSS-Ru General Psychopathology	≤25 years	24.7	28.1	38.5	47	49.25	54.2	62.85
	26–35 years	30.1	31.4	37	43	46	57	59.6
	36–45 years	26.6	29.2	38	44	49	58	65.8
	46–55 years	27.3	35.2	40.75	44.5	51	60.3	65
	≥56 years	37	37	44	47	56.5	N/A	N/A
PANSS-Ru Total	≤25 years	56.7	65.7	76.75	90	98.5	109.3	118.45
	26–35 years	59.4	66.8	74.5	84	96	111.4	117.6
	36–45 years	55	62.4	79	89	95	116.6	126.6
	46–55 years	56.5	72.4	81	91.5	103	125.1	129.15
	≥56 years	80	80	90.25	99.5	115.75	N/A	N/A
CGI-S	≤25 years	3.35	4	4	4.5	5	5.3	6
	26–35 years	2.4	3	4	4	4.5	6	6
	36–45 years	2	3	3	4	5	6	7
	46–55 years	3	3	4	5	5	6	6.45
	≥56 years	4	4	4	5	5.75	N/A	N/A
PANSS-Ru Marder Positive	≤25 years	14	16	20	24	30	35	37
	26–35 years	13.2	15	19	24	29	32	34
	36–45 years	12.35	15.7	19	25	30	38	40.95
	46–55 years	13.75	16.5	21	25.5	32.75	35	37.5
	≥56 years	10	14.8	23	26	32.5	36.2	N/A
PANSS-Ru Marder Negative	≤25 years	8	9	14	20	24	28	30
	26–35 years	9.2	12.2	16	21	24	26.8	29.4
	36–45 years	8.05	12.4	17	20	24	27.3	29.3
	46–55 years	8.25	11	16.25	22.5	26.75	28	32.75
	≥56 years	11	11	18.5	22	26	31.6	N/A
PANSS-Ru Marder Anxiety	≤25 years	4	6	7	10	13	16	17
	26–35 years	4	4.4	7	9	12	15	16.4
	36–45 years	6	6	9	11.5	14	15	15.65
	46–55 years	5.25	6	8	10	12	15	16
	≥56 years	4	5.6	6.5	8	12.5	14.4	N/A
PANSS-Ru Marder Disorganized	≤25 years	11	13	15	20	24	26	26
	26–35 years	10.6	12.2	16	19	23	26	27.8
	36–45 years	11.7	13.7	17	19	24	26.3	28.65
	46–55 years	14.25	15	18	21	23.75	27.5	32.75
	≥56 years	12	14.4	19.5	23	27	28	N/A
PANSS-Ru Marder Hostility	≤25 years	4	5	6	8	12	15	16
	26–35 years	4	4	5	7	11	13	16.4
	36–45 years	4	4	6	8	13	16.3	19.6
	46–55 years	4.25	5	7	9	12	17	18.75
	≥56 years	4	4	5.5	8	11.5	15	N/A

Open in a new tab

CGI-S: Clinical Global Impressions-Severity; PANSS-Ru: Positive and Negative Syndrome Scale-Russian version; N/A: not applicable

APPENDIX 1F. — Comparison of PANSS-Ru score distributions on Positive Symptom subscale, Negative Symptom subscale, General Psychopathology subscale, and Total Score for inpatient group by age

APPENDIX 1G.

Outpatient group descriptive statistics for PANSS-Ru by age

ALL GROUPS	AGE GROUP	n	MEAN	95% CONFIDENCE INTERVAL FOR MEAN
ALL GROUPS	AGE GROUP	n	MEAN	LOWER BOUND	UPPER BOUND	MEDIAN	VARIANCE	SD	MIN	MAX	SKEWNESS	KURTOSIS	STD ERROR
PANSS-Ru Positive Symptoms	≤25 years	33	18	15.66	20.34	19	29.182	5.402	11	33	0.754	1.085	1.126
	26–35 years	32	17.55	14.72	20.38	17	40.736	6.382	9	29	0.18	-1.362	1.361
	36–45 years	42	15.42	13.94	16.91	15	20.413	4.518	9	30	1.03	1.632	0.733
	46–55 years	48	19.33	17.07	21.6	20	40.917	6.397	8	34	0.182	-0.619	1.114
	≥56 years	15	19	13.79	24.21	19	46	6.782	11	32	0.745	0.42	2.261
PANSS-Ru Negative Symptoms	≤25 years	33	20.39	17.9	22.88	21	33.067	5.75	9	31	-0.032	-0.087	1.199
	26–35 years	32	19.86	17.66	22.07	20	24.695	4.969	9	28	-0.461	-0.317	1.059
	36–45 years	42	18.84	17.15	20.53	19	26.515	5.149	8	29	0.137	-0.038	0.835
	46–55 years	48	21.55	19.4	23.69	22	36.443	6.037	10	34	0.005	-0.491	1.051
	≥56 years	15	21.89	19.03	24.75	21	13.861	3.723	17	29	0.822	0.186	1.241
PANSS-Ru General Psychopathology	≤25 years	33	38.7	35.31	42.08	38	61.403	7.836	23	53	0.147	-0.543	1.634
	26–35 years	32	37	32.63	41.37	35.5	97.143	9.856	18	56	0.032	-0.688	2.101
	36–45 years	42	34.71	31.79	37.63	34	78.698	8.871	20	58	0.502	0.162	1.439
	46–55 years	48	40.12	36.9	43.34	43	82.547	9.086	21	57	-0.288	-0.486	1.582
	≥56 years	15	38.56	30.05	47.06	38	122.528	11.069	16	52	-0.915	1.083	3.69
PANSS-Ru Total	≤25 years	33	77.09	70.98	83.2	72	199.538	14.126	51	106	0.465	-0.173	2.945
	26–35 years	32	74.41	65.94	82.88	79	364.634	19.095	36	111	-0.104	-0.702	4.071
	36–45 years	42	68.97	63.57	74.38	68	270.567	16.449	44	111	0.604	0.063	2.668
	46–55 years	48	81	74.32	87.68	84	355.312	18.85	42	117	-0.196	-0.573	3.281
	≥56 years	15	79.44	64.46	94.42	76	379.778	19.488	47	109	0.018	-0.366	6.496
CGI-S	≤25 years	33	3.61	3.2	4.02	3	0.885	0.941	2	5	0.194	-0.893	0.196
	26–35 years	32	3.73	3.29	4.16	4	0.97	0.985	2	6	0.28	0.184	0.21
	36–45 years	42	3.34	3.06	3.62	3	0.718	0.847	2	6	0.949	1.688	0.137
	46–55 years	48	3.7	3.35	4.05	4	0.968	0.984	1	5	-0.591	0.406	0.171
	≥56 years	15	4	2.85	5.15	3	2.25	1.5	3	7	1.429	0.825	0.5
PANSS-Ru Marder Positive	≤25 years	33	21.964	20.168	23.7599	21	21.443	4.63067	14	36	0.846	1.782	N/A
	26–35 years	32	20.864	18.365	23.3638	19	56.176	7.49504	12	38	0.613	-.487	N/A
	36–45 years	42	19.6	17.951	21.2482	19	33.633	5.79937	12	35	0.780	.016	N/A
	46–55 years	48	22.878	20.495	25.2602	24	56.960	7.54717	10	37	-0.157	-1.060	N/A
	≥56 years	15	22.5	17.236	27.7639	22	68.636	8.28471	10	40	0.501	.524	N/A
PANSS-Ru Marder Negative	≤25 years	33	18.178	15.929	20.4274	17	33.634	5.79945	7	31	0.511	-0.088	N/A
	26–35 years	32	17.00	15.011	18.9881	17	35.556	5.96285	7	30	0.136	-0.362	N/A
	36–45 years	42	18.22	16.645	19.7948	18	30.706	5.54127	7	29	-0.086	-.0415	N/A
	46–55 years	48	18.585	16.823	20.3470	19	31.149	5.58111	9	31	0.061	-.0478	N/A
	≥56 years	15	18.25	14.93	21.5695	19	27.295	5.22451	9	27	-0.459	0.745	N/A
PANSS-Ru Marder Anxiety	≤25 years	33	9.3571	8.0986	10.6157	9	10.534	3.24567	4	16	0.313	-0.673	N/A
	26–35 years	32	8.8108	7.6779	9.9438	9	11.547	3.39802	4	18	0.468	-0.069	N/A
	36–45 years	42	8.7000	7.6178	9.7822	8	14.500	3.80789	4	18	0.712	-0.480	N/A
	46–55 years	48	10.024	8.9693	11.0795	10	11.174	3.34281	4	17	0.083	-0.762	N/A
	≥56 years	15	9.1667	7.1974	11.1359	9	9.606	3.09936	4	16	0.583	1.285	N/A
PANSS-Ru Marder Disorganized	≤25 years	33	17.321	15.609	19.0331	16	19.485	4.41423	9	26	0.228	-0.578	N/A
	26–35 years	32	16.891	14.981	18.8020	17	32.821	5.72899	7	33	0.510	0.529	N/A
	36–45 years	42	16.20	14.752	17.6480	16	25.959	5.09502	8	29	0.724	0.033	N/A
	46–55 years	48	18.195	16.274	20.1154	19	37.011	6.08366	7	29	0.113	-1.161	N/A
	≥56 years	15	18.083	14.295	21.8710	18.5	35.538	5.96137	8	29	0.074	-0.214	N/A
PANSS-Ru Marder Hostility	≤25 years	33	9.2143	7.7279	10.7006	9	14.693	3.83316	4	18	0.546	-0.291	N/A
	26–35 years	32	7.9189	6.6904	9.1474	7	13.577	3.68464	4	16	0.772	-0.467	N/A
	36–45 years	42	7.5000	6.6212	8.3788	7	9.561	3.09212	4	15	0.578	-0.688	N/A
	46–55 years	48	8.5122	7.4770	9.5474	8	10.756	3.27965	4	16	0.610	-0.279	N/A
	≥56 years	15	8.0833	5.8582	10.3085	7.5	12.265	3.50216	4	14	0.237	-01.404	N/A

Open in a new tab

CGI-S: Clinical Global Impressions-Severity; PPANSS-Ru: Positive and Negative Syndrome Scale-Russian version; SD: standard deviation; N/A: not applicable

APPENDIX 1H.

Outpatient group average weighted scores with percentiles for the PANSS-Ru by age

ALL GROUPS	AGE GROUP	PERCENTILES
ALL GROUPS	AGE GROUP	5	10	25	50	75	90	95
PANSS-Ru Positive Symptoms	≤25 years	11	11	13	19	22	24	31.2
	26–35 years	9	9.3	11.75	17	24	26.4	28.7
	36–45 years	9	10	12	15	17.25	22	24.3
	46–55 years	9.4	11	13.5	20	24	28.2	30.5
	≥56 years	11	11	13	19	23	N/A	N/A
PANSS-Ru Negative Symptoms	≤25 years	9.4	11.8	17	21	23	30	30.8
	26–35 years	9.45	12.3	17.25	20	24	26	27.7
	36–45 years	10.85	11	16	19	21	28	29
	46–55 years	10	14	17	22	25.5	30.6	31.9
	≥56 years	17	17	19.5	21	25	N/A	N/A
PANSS-Ru General Psychopathology	≤25 years	24.2	29.4	33	38	44	50.6	52.6
	26–35 years	18.9	24.6	28.75	35.5	46.25	50.1	55.25
	36–45 years	20	23.8	27	34	41.25	45.2	54.2
	46–55 years	23.1	25.6	32.5	43	46	51.2	56.3
	≥56 years	16	16	32.5	38	48.5	N/A	N/A
PANSS-Ru Total	≤25 years	52.6	60.2	68	72	86	102	105.6
	26–35 years	37.8	49.8	59	79	89.75	96.7	108.9
	36–45 years	44.95	47.8	55.75	68	78.25	94.3	104.35
	46–55 years	46.2	54.8	65.5	84	95	100.8	116.3
	≥56 years	47	47	66.5	76	96.5	N/A	N/A
CGI-S	≤25 years	2	2.4	3	3	4	5	5
	26–35 years	2	2.3	3	4	4	5	5.85
	36–45 years	2	2	3	3	4	4.1	5.05
	46–55 years	1.7	2.4	3	4	4	5	5
	≥56 years	3	3	3	3	5	N/A	N/A
PANSS-Ru Marder Positive	≤25 years	14.9	16	19	21.5	24.75	28	32.4
	26–35 years	12	12	14	19	26	31	38
	36–45 years	12	13	15	19	23	29	31.45
	46–55 years	10.1	12	16	24	29.5	33	33.9
	≥56 years	10	10.6	17.25	22	27.5	37	U+00AD
PANSS-Ru Marder Negative	≤25 years	8.8	11.9	15	17	22.75	26.4	30.55
	26–35 years	7	7	12.5	17	20.5	25.4	28.2
	36–45 years	8.55	10	15.75	18	22	25	29
	46–55 years	9	10.2	15	19	22	26	28.8
	≥56 years	9	9	17.25	19	19.75	26.4	N/A
PANSS-Ru Marder Anxiety	≤25 years	4	5.8	7	9	11.75	14.1	15.55
	26–35 years	4	4	6	9	11.5	13	15.3
	36–45 years	4	4	6	8	11.25	15	16
	46–55 years	4.1	5.2	8	10	13	14.8	15.9
	≥56 years	4	4.6	7	9	10.75	14.8	N/A
PANSS-Ru Marder Disorganized	≤25 years	9.9	11	14.25	16	20	24.1	25.55
	26–35 years	7.9	9	13	17	20.5	24.2	28.5
	36–45 years	9	11	12	16	19	24	26.9
	46–55 years	9.1	11	13	19	22.5	26.8	28
	≥56 years	8	8.9	13.25	18.5	21.75	27.8	N/A
PANSS-Ru Marder Hostility	≤25 years	4	4	6	9	12	15.2	17.55
	26–35 years	4	4	5	7	10.5	14	16
	36–45 years	4	4	4.75	7	10	12	13
	46–55 years	4	4.2	6	8	11	12.8	15.9
	≥56 years	4	4	4.5	7.5	11	13.4	N/A

Open in a new tab

CGI-Severity: Clinical Global Impressions-Severity; PANSS-Ru: Positive and Negative Syndrome Scale-Russian version; N/A: not applicable

APPENDIX 1I. — Comparison of PANSS-Ru score distributions on Positive Symptom subscale, Negative Symptom subscale, General Psychopathology subscale, and Total Score for outpatient group by age

REFERENCES

1.Kay SR, Fiszbein A, Opler LA. The Positive and Negative Syndrome Scale (PANSS) for schizophrenia. Schizophr Bull. 1987;13:261–276. doi: 10.1093/schbul/13.2.261. [DOI] [PubMed] [Google Scholar]
2.Kay SR, Opler LA, Fiszbein A. Toronto, ON: Multi-Health Systems Inc; 2000. The Positive and Negative Syndrome Scale (PANSS) Manual. [Google Scholar]
3.Perkins DO, Stroup TS, Lieberman JA, et al. Rush AJ, Pincus HA, First MB, et al. (eds). Handbook of Psychiatric Measures. Washington, DC: American Psychiatric Association Press Inc.; 2000. Psychotic Disorders Measures. [Google Scholar]
4.Yamada H, Kikumoto K, Masui K, et al. Reliability of the Japanese version of the Positive and Negative Syndrome Scale (PANSS) Jpn J Clin Psych. 1993;22:609–614. [Google Scholar]
5.Yehya A, Ghuloum S, Mahfoud Z, et al. Validity and reliability of the Arabic version of the Positive and Negative Syndrome Scale. Psychopathology. 2016;49:181–187. doi: 10.1159/000447328. [DOI] [PubMed] [Google Scholar]
6.Nelson MA, Palchanes K. A literature review of the critical elements in translation theory. J Nurs Schol. 1994;26(2):113–117. doi: 10.1111/j.1547-5069.1994.tb00928.x. [DOI] [PubMed] [Google Scholar]
7.Strauss E, Sherman EM, Spreen O. New York: Oxford University Press; 2006. A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary. [Google Scholar]
8.Lezak M, Howieson D, Bigler E, Tranel D. (5th ed) New York: Oxford University Press; 2012. Neuropsychological Assessment. [Google Scholar]
9.Nelson GH, O’Hara MW, Watson D. National norms for the expanded version of the inventory of depression and anxiety symptoms (IDAS-II) J Clin Psychol. 2018;74(6):953–968. doi: 10.1002/jclp.22560. [DOI] [PubMed] [Google Scholar]
10.Francis VM, Rajan P, Turner N. British community norms for the brief symptom inventory. Br J Clin Psychol. 1990;29:115–116. doi: 10.1111/j.2044-8260.1990.tb00857.x. [DOI] [PubMed] [Google Scholar]
11.Deane F, Leathern J, Spicer J. Clinical norms, reliability and validity for the Hopkins Symptom Checklist-21. Aust J Psychol. 1992;44(1):21–25. [Google Scholar]
12.WHO Regional Office for Europe. The European health report 2005. Public health action for healthier children and populations. [Accessed 1 Oct 2018.]. http://www.euro.who.int/__data/assets/pdf_file/0004/82435/E87325.pdf.
13.Jenkins R, Lancashire S, McDaid D, et al. Mental health reform in the Russian Federation: an integrated approach to achieve social inclusion and recovery. Bull World Health Org. 2007;85(11):821–900. doi: 10.2471/BLT.06.039156. [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Lavretsky H. The Russian concept of schizophrenia: a review of the literature. Schizophr Bull. 1998;24(4):537–557. doi: 10.1093/oxfordjournals.schbul.a033348. [DOI] [PubMed] [Google Scholar]
15.Russian Federation Federal State Statistics Service. General information about the all-Russia population census of 2010. [Accessed 1 Oct 2018.]. http://www.perepis-2010.ru/perepis-foreigner/
16.National Statistical Committee of the Republic of Belarus. General information about the all-Belarus population census of 2009. [Accessed 1 Oct 2019]. http://www.belstat.gov.by/ofitsialnaya-statistika/bazy-dannyh/baza-dannyh-po-statistike-naseleniya/perepis-naseleniya-2009-goda/.
17.American Psychiatric Association. (4th ed). Arlington, VA: American Psychiatric Association Press, Inc.; 2000. The Diagnostic and Statistical Manual of Mental Disorders; p. 297. [Google Scholar]
18.Sheehan D, Janavs J, Harnett‐Sheehan K, et al. M.I.N.I - Russian Translation - Version 6.0.0. Mapi Research Institute, 2010. [Accessed 1 Oct 2018.]. https://eprovide.mapi-trust.org/instruments/mini-international-neuropsychiatric-interview.
19.Sheehan DV, Lecrubier Y, Sheehan KH, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59(Suppl 20):22–33. quiz 34-57. [PubMed] [Google Scholar]
20.Kay SR, Opler LA, Fiszbein A. North Tonawanda, NY: Multi-Health Systems Inc; 1994. Positive and Negative Syndrome Scale Manual. [Google Scholar]
21.Peralta V, Cuesta MJ. Psychometric properties of the positive and negative syndrome scale (PANSS) in schizophrenia. Psychiatry Res. 1994;53:31–40. doi: 10.1016/0165-1781(94)90093-0. [DOI] [PubMed] [Google Scholar]
22.Guy W. Rockville, MD: U.S. Department of Health, Education, and Welfare; 1976. ECDEU Assessment Manual for Psychopharmacology. (ed.) [Google Scholar]
23.Lipsitz J, Kobak K, Feiger A, et al. The Rater Applied Performance Scale (RAPS): development and reliability. Psychiatric Res. 2004;127:147–155. doi: 10.1016/j.psychres.2004.03.001. [DOI] [PubMed] [Google Scholar]
24.Armonk, NY: IBM Corp; 2015. IBM SPSS Statistics for Windows, Version 23.0. IBM Corp. [Google Scholar]
25.Cary, NC: SAS Institute Inc; 2011. SAS/ACCESS 9.2 Interface to ADABAS: Reference. SAS Institute Inc. [Google Scholar]
26.Acquadro C, Conway K, Giroudet C, et al. Linguistic validation manual for patient-reported outcomes (PRO) instruments. Lyon: Mapi Research Institute. 2004 [Google Scholar]
27.Mear I. Chassany O, Caulin C. Health-Related Quality of Life and Patient-Reported Outcomes: Scientific and Useful Criteria. (eds). Paris: Springer; 2002. Difficulties of international clinical trials: cultural adaptation of quality of life questionnaires. [Google Scholar]
28.Mapi Group. Linguistic Validation. 2017. [Accessed 1 Oct 2018]. http://mapigroup.com/services/language-services/linguistic-validation/.
29.Opler LA, Kay SR, Lindenmayer JP, Fiszbein A. Toronto: Multi-Health Systems Inc.; 1992. Structured Clinical Interview for the Positive and Negative Syndrome Scale (SCI-PANSS) [Google Scholar]
30.Gwet KL. Gaithersburg, MD: Advanced Analytics, LLC; 2010. Handbook of Inter-Rater Reliability (2nd ed) [Google Scholar]
31.30Kramer MS, Feinstein AR. Clinical biostatistics, LIV The biostatistics of concordance. Clin Pharmacol Ther. 1983;29:111–123. doi: 10.1038/clpt.1981.18. [DOI] [PubMed] [Google Scholar]
32.Marder SR, Davis JM, Chouinard G. The effects of risperidone on the five dimensions of schizophrenia derived by factor analysis: combined results of the North American trials. J Clin Psychiatry. 1997;58(12):538–546. doi: 10.4088/jcp.v58n1205. [DOI] [PubMed] [Google Scholar]
33.Kay SR, Opler LA, Lindenmayer JP. Reliability and validity of the Positive and Negative Syndrome Scale for schizophrenics. Psychiatry Res. 1988;23:99–110. doi: 10.1016/0165-1781(88)90038-8. [DOI] [PubMed] [Google Scholar]
34.Anastasi A. New York: Macmillan Publishing Company; 1990. Psychological Testing. [Google Scholar]
35.Khan A, Liharska L, Harvey PD, et al. Negative symptom dimensions of the Positive and Negative Syndrome Scale across geographical regions: implications for social, linguistic, and cultural consistency. Innov Clin Neurosci. 2017;14(11-12):30–40. [PMC free article] [PubMed] [Google Scholar]
36.Khan A, Yavorsky C, Liechti S, et al. A Rasch model to test the cross-cultural validity in the Positive and Negative Syndrome Scale (PANSS) across six geo-cultural groups. BMC Psychol. 2013;1(1):5. doi: 10.1186/2050-7283-1-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Messinger JW, Trémeau F, Antonius D, et al. Avolition and expressive deficits capture negative symptom phenomenology: implications for DSM-5 and schizophrenia research. Clin Psychol Rev. 2011;31:161–168. doi: 10.1016/j.cpr.2010.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Strauss GP, Keller WR, Buchanan RW, et al. Next-generation negative symptom assessment for clinical trials: validation of the Brief Negative Symptom scale. Schizophrenia Research. 2012;142:88–92. doi: 10.1016/j.schres.2012.10.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Kring AM, Gur RE, Blanchard JJ, et al. The Clinical Assessment Interview for Negative Symptoms (CAINS): final development and validation. Am J Psychiatry. 2013;170:165–172. doi: 10.1176/appi.ajp.2012.12010109. [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Galderisi S, Bucci P, Mucci A, et al. Categorical and dimensional approaches to negative symptoms of schizophrenia: focus on long-term stability and functional outcome. Schizophr Res. 2013;147:157–162. doi: 10.1016/j.schres.2013.03.020. [DOI] [PubMed] [Google Scholar]
41.Dikmen SS, Heaton RK, Grant I, et al. Test-retest reliability and practice effects of Expanded Halstead-Reitan Neuropsychological Test Battery. J Int Neuropsychol Soc. 1999;5:346–356. [PubMed] [Google Scholar]
42.Rabinowitz J, Mehnert A, Eerdekens M. To what extent do the PANSS and CGI-S overlap? J Clin Psychopharmacol. 2006;26(3):303–307. doi: 10.1097/01.jcp.0000218407.10362.6e. [DOI] [PubMed] [Google Scholar]
43.Clark MM, Niaura R, King TK, Pera V. Depression, smoking, activity level, and health status: pretreatment predictors of attrition in obesity treatment. Addictive Behaviors. 1996;21:509–513. doi: 10.1016/0306-4603(95)00081-x. [DOI] [PubMed] [Google Scholar]
44.Eaton WW, Anthony JC, Tepper S, Dryman A. Psychopathology and attrition in the epidemiologic catchment area surveys. Am J Epidemiol. 1992;135:1051–1059. doi: 10.1093/oxfordjournals.aje.a116399. [DOI] [PubMed] [Google Scholar]
45.Farmer ME, Locke BZ, Liu IY, Moscicki EK. Depressive symptoms and attrition: the NHANES I epidemiologic follow-up study. Int J Methods Psychiatr Res. 1994;4:19–27. [Google Scholar]
46.Killaspy H, Banerjee S, King M, Lloyd M. Prospective, controlled study of psychiatric out-patient non-attendance: characteristics and outcome. Br J Psychiatry. 2000;176:160–165. doi: 10.1192/bjp.176.2.160. [DOI] [PubMed] [Google Scholar]
47. Watson JM, Torgerson DJ. Increasing recruitment to randomised trials: a review of randomised controlled trials BMC Med Res Methodol. 2006. 6 34 10.1186/1471-2288-6-34. [DOI] [PMC free article] [PubMed] [Google Scholar]
48.Oude Rengerink K, Opmeer BC, Logtenberg SL, et al. Improving participation of patients in clinical trials—rationale and design of IMPACT. BMC Med Res Methodol. 2010;10(1):85. doi: 10.1186/1471-2288-10-85. [DOI] [PMC free article] [PubMed] [Google Scholar]
49.Fletcher B, Gheorghe A, Moore D, et al. Improving the recruitment activity of clinicians in randomised controlled trials: a systematic review. BMJ Open. 2012;2(1):e000496. doi: 10.1136/bmjopen-2011-000496. [DOI] [PMC free article] [PubMed] [Google Scholar]
50.Jenkins VA, Farewell D, Farewell V, et al. Teams Talking Trials: results of an RCT to improve the communication of cancer teams about treatment trials. Contemp Clin Trials. 2013;35(1):43–51. doi: 10.1016/j.cct.2013.02.003. [DOI] [PubMed] [Google Scholar]
51.Byrne BM. New York: Springer-Verlag: 1989. A primer of LISREL: Basic applications and programming for confirmatory factor analytic models. [Google Scholar]
52.Byrne BM. Thousand Oaks, CA: Sage; 1994. Structural equation modeling with EQS and EQS/Windows: Basic concepts, applications, and programming. [Google Scholar]

[B1] 1.Kay SR, Fiszbein A, Opler LA. The Positive and Negative Syndrome Scale (PANSS) for schizophrenia. Schizophr Bull. 1987;13:261–276. doi: 10.1093/schbul/13.2.261. [DOI] [PubMed] [Google Scholar]

[B2] 2.Kay SR, Opler LA, Fiszbein A. Toronto, ON: Multi-Health Systems Inc; 2000. The Positive and Negative Syndrome Scale (PANSS) Manual. [Google Scholar]

[B3] 3.Perkins DO, Stroup TS, Lieberman JA, et al. Rush AJ, Pincus HA, First MB, et al. (eds). Handbook of Psychiatric Measures. Washington, DC: American Psychiatric Association Press Inc.; 2000. Psychotic Disorders Measures. [Google Scholar]

[B4] 4.Yamada H, Kikumoto K, Masui K, et al. Reliability of the Japanese version of the Positive and Negative Syndrome Scale (PANSS) Jpn J Clin Psych. 1993;22:609–614. [Google Scholar]

[B5] 5.Yehya A, Ghuloum S, Mahfoud Z, et al. Validity and reliability of the Arabic version of the Positive and Negative Syndrome Scale. Psychopathology. 2016;49:181–187. doi: 10.1159/000447328. [DOI] [PubMed] [Google Scholar]

[B6] 6.Nelson MA, Palchanes K. A literature review of the critical elements in translation theory. J Nurs Schol. 1994;26(2):113–117. doi: 10.1111/j.1547-5069.1994.tb00928.x. [DOI] [PubMed] [Google Scholar]

[B7] 7.Strauss E, Sherman EM, Spreen O. New York: Oxford University Press; 2006. A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary. [Google Scholar]

[B8] 8.Lezak M, Howieson D, Bigler E, Tranel D. (5th ed) New York: Oxford University Press; 2012. Neuropsychological Assessment. [Google Scholar]

[B9] 9.Nelson GH, O’Hara MW, Watson D. National norms for the expanded version of the inventory of depression and anxiety symptoms (IDAS-II) J Clin Psychol. 2018;74(6):953–968. doi: 10.1002/jclp.22560. [DOI] [PubMed] [Google Scholar]

[B10] 10.Francis VM, Rajan P, Turner N. British community norms for the brief symptom inventory. Br J Clin Psychol. 1990;29:115–116. doi: 10.1111/j.2044-8260.1990.tb00857.x. [DOI] [PubMed] [Google Scholar]

[B11] 11.Deane F, Leathern J, Spicer J. Clinical norms, reliability and validity for the Hopkins Symptom Checklist-21. Aust J Psychol. 1992;44(1):21–25. [Google Scholar]

[B12] 12.WHO Regional Office for Europe. The European health report 2005. Public health action for healthier children and populations. [Accessed 1 Oct 2018.]. http://www.euro.who.int/__data/assets/pdf_file/0004/82435/E87325.pdf.

[B13] 13.Jenkins R, Lancashire S, McDaid D, et al. Mental health reform in the Russian Federation: an integrated approach to achieve social inclusion and recovery. Bull World Health Org. 2007;85(11):821–900. doi: 10.2471/BLT.06.039156. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B14] 14.Lavretsky H. The Russian concept of schizophrenia: a review of the literature. Schizophr Bull. 1998;24(4):537–557. doi: 10.1093/oxfordjournals.schbul.a033348. [DOI] [PubMed] [Google Scholar]

[B15] 15.Russian Federation Federal State Statistics Service. General information about the all-Russia population census of 2010. [Accessed 1 Oct 2018.]. http://www.perepis-2010.ru/perepis-foreigner/

[B16] 16.National Statistical Committee of the Republic of Belarus. General information about the all-Belarus population census of 2009. [Accessed 1 Oct 2019]. http://www.belstat.gov.by/ofitsialnaya-statistika/bazy-dannyh/baza-dannyh-po-statistike-naseleniya/perepis-naseleniya-2009-goda/.

[B17] 17.American Psychiatric Association. (4th ed). Arlington, VA: American Psychiatric Association Press, Inc.; 2000. The Diagnostic and Statistical Manual of Mental Disorders; p. 297. [Google Scholar]

[B18] 18.Sheehan D, Janavs J, Harnett‐Sheehan K, et al. M.I.N.I - Russian Translation - Version 6.0.0. Mapi Research Institute, 2010. [Accessed 1 Oct 2018.]. https://eprovide.mapi-trust.org/instruments/mini-international-neuropsychiatric-interview.

[B19] 19.Sheehan DV, Lecrubier Y, Sheehan KH, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59(Suppl 20):22–33. quiz 34-57. [PubMed] [Google Scholar]

[B20] 20.Kay SR, Opler LA, Fiszbein A. North Tonawanda, NY: Multi-Health Systems Inc; 1994. Positive and Negative Syndrome Scale Manual. [Google Scholar]

[B21] 21.Peralta V, Cuesta MJ. Psychometric properties of the positive and negative syndrome scale (PANSS) in schizophrenia. Psychiatry Res. 1994;53:31–40. doi: 10.1016/0165-1781(94)90093-0. [DOI] [PubMed] [Google Scholar]

[B22] 22.Guy W. Rockville, MD: U.S. Department of Health, Education, and Welfare; 1976. ECDEU Assessment Manual for Psychopharmacology. (ed.) [Google Scholar]

[B23] 23.Lipsitz J, Kobak K, Feiger A, et al. The Rater Applied Performance Scale (RAPS): development and reliability. Psychiatric Res. 2004;127:147–155. doi: 10.1016/j.psychres.2004.03.001. [DOI] [PubMed] [Google Scholar]

[B24] 24.Armonk, NY: IBM Corp; 2015. IBM SPSS Statistics for Windows, Version 23.0. IBM Corp. [Google Scholar]

[B25] 25.Cary, NC: SAS Institute Inc; 2011. SAS/ACCESS 9.2 Interface to ADABAS: Reference. SAS Institute Inc. [Google Scholar]

[B26] 26.Acquadro C, Conway K, Giroudet C, et al. Linguistic validation manual for patient-reported outcomes (PRO) instruments. Lyon: Mapi Research Institute. 2004 [Google Scholar]

[B27] 27.Mear I. Chassany O, Caulin C. Health-Related Quality of Life and Patient-Reported Outcomes: Scientific and Useful Criteria. (eds). Paris: Springer; 2002. Difficulties of international clinical trials: cultural adaptation of quality of life questionnaires. [Google Scholar]

[B28] 28.Mapi Group. Linguistic Validation. 2017. [Accessed 1 Oct 2018]. http://mapigroup.com/services/language-services/linguistic-validation/.

[B29] 29.Opler LA, Kay SR, Lindenmayer JP, Fiszbein A. Toronto: Multi-Health Systems Inc.; 1992. Structured Clinical Interview for the Positive and Negative Syndrome Scale (SCI-PANSS) [Google Scholar]

[B30] 30.Gwet KL. Gaithersburg, MD: Advanced Analytics, LLC; 2010. Handbook of Inter-Rater Reliability (2nd ed) [Google Scholar]

[B31] 31.30Kramer MS, Feinstein AR. Clinical biostatistics, LIV The biostatistics of concordance. Clin Pharmacol Ther. 1983;29:111–123. doi: 10.1038/clpt.1981.18. [DOI] [PubMed] [Google Scholar]

[B32] 32.Marder SR, Davis JM, Chouinard G. The effects of risperidone on the five dimensions of schizophrenia derived by factor analysis: combined results of the North American trials. J Clin Psychiatry. 1997;58(12):538–546. doi: 10.4088/jcp.v58n1205. [DOI] [PubMed] [Google Scholar]

[B33] 33.Kay SR, Opler LA, Lindenmayer JP. Reliability and validity of the Positive and Negative Syndrome Scale for schizophrenics. Psychiatry Res. 1988;23:99–110. doi: 10.1016/0165-1781(88)90038-8. [DOI] [PubMed] [Google Scholar]

[B34] 34.Anastasi A. New York: Macmillan Publishing Company; 1990. Psychological Testing. [Google Scholar]

[B35] 35.Khan A, Liharska L, Harvey PD, et al. Negative symptom dimensions of the Positive and Negative Syndrome Scale across geographical regions: implications for social, linguistic, and cultural consistency. Innov Clin Neurosci. 2017;14(11-12):30–40. [PMC free article] [PubMed] [Google Scholar]

[B36] 36.Khan A, Yavorsky C, Liechti S, et al. A Rasch model to test the cross-cultural validity in the Positive and Negative Syndrome Scale (PANSS) across six geo-cultural groups. BMC Psychol. 2013;1(1):5. doi: 10.1186/2050-7283-1-5. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B37] 37.Messinger JW, Trémeau F, Antonius D, et al. Avolition and expressive deficits capture negative symptom phenomenology: implications for DSM-5 and schizophrenia research. Clin Psychol Rev. 2011;31:161–168. doi: 10.1016/j.cpr.2010.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B38] 38.Strauss GP, Keller WR, Buchanan RW, et al. Next-generation negative symptom assessment for clinical trials: validation of the Brief Negative Symptom scale. Schizophrenia Research. 2012;142:88–92. doi: 10.1016/j.schres.2012.10.012. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B39] 39.Kring AM, Gur RE, Blanchard JJ, et al. The Clinical Assessment Interview for Negative Symptoms (CAINS): final development and validation. Am J Psychiatry. 2013;170:165–172. doi: 10.1176/appi.ajp.2012.12010109. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B40] 40.Galderisi S, Bucci P, Mucci A, et al. Categorical and dimensional approaches to negative symptoms of schizophrenia: focus on long-term stability and functional outcome. Schizophr Res. 2013;147:157–162. doi: 10.1016/j.schres.2013.03.020. [DOI] [PubMed] [Google Scholar]

[B41] 41.Dikmen SS, Heaton RK, Grant I, et al. Test-retest reliability and practice effects of Expanded Halstead-Reitan Neuropsychological Test Battery. J Int Neuropsychol Soc. 1999;5:346–356. [PubMed] [Google Scholar]

[B42] 42.Rabinowitz J, Mehnert A, Eerdekens M. To what extent do the PANSS and CGI-S overlap? J Clin Psychopharmacol. 2006;26(3):303–307. doi: 10.1097/01.jcp.0000218407.10362.6e. [DOI] [PubMed] [Google Scholar]

[B43] 43.Clark MM, Niaura R, King TK, Pera V. Depression, smoking, activity level, and health status: pretreatment predictors of attrition in obesity treatment. Addictive Behaviors. 1996;21:509–513. doi: 10.1016/0306-4603(95)00081-x. [DOI] [PubMed] [Google Scholar]

[B44] 44.Eaton WW, Anthony JC, Tepper S, Dryman A. Psychopathology and attrition in the epidemiologic catchment area surveys. Am J Epidemiol. 1992;135:1051–1059. doi: 10.1093/oxfordjournals.aje.a116399. [DOI] [PubMed] [Google Scholar]

[B45] 45.Farmer ME, Locke BZ, Liu IY, Moscicki EK. Depressive symptoms and attrition: the NHANES I epidemiologic follow-up study. Int J Methods Psychiatr Res. 1994;4:19–27. [Google Scholar]

[B46] 46.Killaspy H, Banerjee S, King M, Lloyd M. Prospective, controlled study of psychiatric out-patient non-attendance: characteristics and outcome. Br J Psychiatry. 2000;176:160–165. doi: 10.1192/bjp.176.2.160. [DOI] [PubMed] [Google Scholar]

[B47] 47. Watson JM, Torgerson DJ. Increasing recruitment to randomised trials: a review of randomised controlled trials BMC Med Res Methodol. 2006. 6 34 10.1186/1471-2288-6-34. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B48] 48.Oude Rengerink K, Opmeer BC, Logtenberg SL, et al. Improving participation of patients in clinical trials—rationale and design of IMPACT. BMC Med Res Methodol. 2010;10(1):85. doi: 10.1186/1471-2288-10-85. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B49] 49.Fletcher B, Gheorghe A, Moore D, et al. Improving the recruitment activity of clinicians in randomised controlled trials: a systematic review. BMJ Open. 2012;2(1):e000496. doi: 10.1136/bmjopen-2011-000496. [DOI] [PMC free article] [PubMed] [Google Scholar]

[B50] 50.Jenkins VA, Farewell D, Farewell V, et al. Teams Talking Trials: results of an RCT to improve the communication of cancer teams about treatment trials. Contemp Clin Trials. 2013;35(1):43–51. doi: 10.1016/j.cct.2013.02.003. [DOI] [PubMed] [Google Scholar]

[B51] 51.Byrne BM. New York: Springer-Verlag: 1989. A primer of LISREL: Basic applications and programming for confirmatory factor analytic models. [Google Scholar]

[B52] 52.Byrne BM. Thousand Oaks, CA: Sage; 1994. Structural equation modeling with EQS and EQS/Windows: Basic concepts, applications, and programming. [Google Scholar]

PERMALINK

Validation of the Russian Version of the Positive and Negative Syndrome Scale (PANSS-Ru) and Normative Data

Evgenia Ivanova, PhD

Anzalee Khan, PhD

Lora Liharska, MS

Alexander Reznik, MD, PhD

Sergey Kuzmin, MD

Olga Kushnir, MD, PhD

Alexey Agarkov, MD, PhD

Nikolay Bokhan, MD, PhD

Tatiana Pogorelova, MD, PhD

Olga Khomenko, MD, PhD

Ksenia Chernysheva, MD, PhD

Margarita Morozova, MD, PhD

George Rupchev, PhD

Taisia Lepilkina, PhD

Alexander Ozornin, MD, PhD

Nina Ozornina, MD, PhD

Nikolay Govorin, MD, PhD

Anna Malakhova, MD, PhD

Natalia Hmara, MD

Oxana Shylova, MD

Artsiom Hryhoryeu, MD

Nataljya Ivanchikova, MD

Irina Raevskaya, MD

Pavel Gusak, MD

Marina Skugarevskaya, MD, PhD

Lewis A Opler, MD, PhD†

Abstract

METHODS

RESULTS

TABLE 1.

TABLE 2.

FIGURE 1.

TABLE 3.

TABLE 4.

DISCUSSION

CONCLUSION

IN MEMORIAM

ACKNOWLEDGMENTS

Appendix

APPENDIX 1A.

APPENDIX 1B.

APPENDIX 1C.

APPENDIX 1D.

APPENDIX 1E.

APPENDIX 1F.

APPENDIX 1G.

APPENDIX 1H.

APPENDIX 1I.

REFERENCES

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases