Validation of English and Spanish-Language Versions of a Screening Questionnaire for Rheumatoid Arthritis in an Underserved Community

Abstract

Objective

Questionnaires to screen for rheumatoid arthritis (RA) have been tested in groups that were primarily well educated and Caucasian. We sought to validate the RA questions of the Connective Tissue Disease Screening Questionnaire (CSQ) in ethnic minorities in an underserved community, and to test a Spanish-language version.

Methods

The Spanish-language version was developed by 2 native speakers. Consecutive English-speaking or Spanish-speaking patients in a community-based rheumatology practice completed the questionnaire. Diagnoses were confirmed by medical record review. Sensitivity and specificity of the questionnaire for a diagnosis of RA were computed for each language version, using 2 groups as controls: patients with noninflammatory conditions, and participants recruited from the community.

Results

The English-language version was tested in 53 patients with RA (79% ethnic minorities; mean education level 11.3 yrs), 85 rheumatology controls with noninflammatory conditions, and 82 community controls. Using 3 positive responses as indicating a positive screening test, the sensitivity of the questionnaire was 0.77, the specificity based on rheumatology controls was 0.45, and the specificity based on community controls was 0.94. The Spanish-language version was tested in 55 patients with RA (mean education level 7.8 yrs), 149 rheumatology controls, and 88 community controls. The sensitivity of the Spanish-language version was 0.87, with specificities of 0.60 and 0.97 using the rheumatology controls and community controls, respectively.

Conclusion

The sensitivity of the English-language version of the RA questions of the CSQ was lower in this study than in other cohorts, reflecting differences in the performance of the questions in different ethnic or socioeconomic groups. The Spanish-language version demonstrated good sensitivity, and both had excellent specificity when tested in community controls.

Case-finding in community-based or population-based epidemiological or clinical research often begins with screening by questionnaire to identify potential participants who might have the condition of interest. Questionnaires with high sensitivities are useful for screening because these identify almost all persons with the condition. Screening questionnaires that are also highly specific would reduce the proportion excluded after screening because they did not have the condition of interest. Although a questionnaire with both high sensitivity and specificity would be the most cost-effective tool, sensitivity is generally valued more than specificity when screening is the purpose¹.

The Connective Tissue Disease Screening Questionnaire (CSQ) was developed to screen populations for connective tissue diseases, including rheumatoid arthritis (RA)². It has been applied in several studies to screen for potential cases of RA or to exclude persons with inflammatory arthritis^3–5. In initial testing, the CSQ questions related to RA were found to have a sensitivity of 0.85 and a specificity of 0.92 for a diagnosis of RA². However, the group tested was primarily well educated and Caucasian, and subgroup analyses suggested that the specificity of the CSQ might be lower among persons of lower socioeconomic status. In this study, we tested the validity of the RA-related questions of the CSQ in a largely ethnic minority cohort living in an under-served area of Washington, DC. We also developed and tested a Spanish-language version which, if valid, would allow monolingual Spanish speakers to be included in community-based or population-based screenings.

MATERIALS AND METHODS

Research participants and study procedures

Subjects were English-speaking or Spanish-speaking patients enrolled in the National Institute of Arthritis and Musculoskeletal and Skin Diseases Community Health Center rheumatology practice in Washington, DC⁶. This practice is located in a neighborhood health center that serves a local area whose residents are primarily African-American or Hispanic/Latino. The health center provides primary medical care to uninsured and underinsured residents, and is the major source of referrals of patients to the rheumatology practice. Other patients are referred from other neighborhood health centers in Washington or from other clinics or practices in the area. Rheumatology care is provided without regard to medical insurance status.

Consecutive patients were asked to complete the RA-related questions of the written CSQ once. Information on age, sex, race/ethnicity, and education level was also collected. The questionnaire was administered by bilingual staff, who instructed patients to answer whether they had ever experienced the symptoms being asked. Rheumatic disease diagnoses were based on review of medical records and all laboratory and radiographic data. Those with RA, meeting the revised American College of Rheumatology (ACR) classification criteria⁷, were the target group. Other rheumatology practice patients were divided into 2 groups: those with inflammatory disorders (systemic lupus erythematosus, other connective tissue diseases, crystal-induced arthritis, seronegative spondyloarthropathy, vasculitis), and those with noninflammatory conditions (osteoarthritis, back pain, soft tissue or regional musculoskeletal problems, fibromyalgia, arthralgia, abnormal serologies alone, other). All rheumatology practice patients provided written informed consent.

In addition, we surveyed English and Spanish-speaking members of the local community to obtain estimates of the specificity of the questionnaire in a setting most like that in which it would be applied in practice. After obtaining informed verbal consent, we asked people at parks, stores, restaurants, workplaces, libraries, and churches to complete a 1-page anonymous questionnaire that included the RA-related screening questions, and questions on age and education level. Survey workers reported respondents’ sex and ethnicity. We excluded people who reported a diagnosis of RA.

The study protocol was approved by the institutional review board. The community survey was exempted from human subjects review by the National Institutes of Health Office of Human Subjects Protection.

RA-related questions on the CSQ and Spanish translation

The CSQ includes 4 questions related to RA symptoms and signs from which 6 RA-related responses are derived: morning stiffness, arthritis in hand joints or wrists, arthritis in 3 or more joint areas, symmetric arthritis, subcutaneous nodules, and rheumatoid factor (RF) test results. The presence of 3 positive responses was considered to represent possible RA, and the presence of 4 positive responses was considered to represent probable RA².

A Spanish-language version was developed by 2 native speakers (from Puerto Rico and Colombia) using translation by committee (Figure 1). In preliminary testing, the format for the question about the affected joints was modified to include written responses for each joint, rather than checkboxes, because some subjects found the checkboxes confusing.

Figure 1.

Spanish-language version of the rheumatoid arthritis-related questions of the Connective Tissue Disease Screening Questionnaire.

Statistical analysis

We computed the sensitivity and specificity of the RA-related questions, categorizing the rheumatology practice patients who had RA by the ACR classification criteria as true positives, and rheumatology practice patients with noninflammatory conditions and the community controls as 2 separate control groups of true negatives. Results were computed alternatively using either 3 positive questionnaire responses or 4 positive questionnaire responses as the level for determining if a respondent had a “positive” screening questionnaire. Specificities were also computed for the rheumatology practice patients with inflammatory conditions, but because these patients often had inflammatory arthritis, these results were not considered in the evaluation of the validity of the questionnaire. We also computed positive predictive values (PPV) and negative predictive values (NPV) for the questionnaire, using population prevalences of RA of 0.5%, 1%, and 2%^8,9. Analyses were performed using SAS programs (version 9.1; Statistical Analysis Systems, Cary, NC, USA).

RESULTS

We enrolled 296 English-speaking participants (53 with RA, 85 rheumatology practice controls with noninflammatory conditions, 82 community controls, and 76 rheumatology practice patients with inflammatory conditions) and 338 Spanish-speaking participants (55 with RA, 149 rheumatology practice controls with noninflammatory conditions, 88 community controls, and 46 rheumatology practice patients with inflammatory conditions). Seventy-nine percent of the English-speaking patients with RA were ethnic minorities, with most being African or African American (Table 1). The mean education level of the English-speaking patients with RA was 11.3 years, while that of the English-speaking control groups was slightly higher. The mean education level of the Spanish-speaking patients with RA was 7.8 years, which was similar to that of the Spanish-speaking rheumatology practice controls but somewhat lower than that of the Spanish-speaking community controls. Among English-speaking rheumatology practice controls with noninflammatory conditions, 53% had osteoarthritis or mechanical back pain, 15% had fibromyalgia, 9% had arthralgias/myalgias, and 7% had soft tissue problems. Among Spanish-speaking rheumatology practice controls with noninflammatory conditions, 44% had osteoarthritis or mechanical back pain, 19% had soft tissue problems, 18% had arthralgias/myalgias, and 9% had fibromyalgia.

Table 1.

Demographic characteristics of participants, by group^*.

	RA	Rheumatology Practice Controls with Non- inflammatory Disorders	Community Controls	Rheumatology Practice Patients with Inflammatory Conditions
English-speaking, n	53	85	82	76
Age, yrs	53.0 ± 12.2	55.3 ± 11.4	56.9 ± 15.7	46.1 ± 13.5
Female, n (%)	36 (68)	63 (74)	60 (73)	56 (74)
Education level, yrs	11.3 ± 3.9	12.3 ± 3.8	12.8 ± 2.9	13.4 ± 3.2
African or African American, n (%)	31 (59)	60 (71)	82 (100)	48 (63)
Hispanic, n (%)	5 (9)	3 (4)	0	9 (12)
Caucasian, n (%)	11 (21)	20 (23)	0	11 (15)
Asian, n (%)	6 (11)	2 (2)	0	8 (10)
Spanish-speaking, n	55	149	88	46
Age, yrs	46.3 ±11.4	48.6 ± 11.9	38.7 ± 12.0	40.0 ±14.1
Female, n (%)	47 (85)	133 (89)	72 (82)	36 (77)
Education level, yrs	7.8 ± 4.9	7.8 ± 5.3	10.9 ± 4.1	8.2 ± 4.7

^*Plus-minus values are mean ± standard deviation.

Among English-speaking participants, the questionnaire had a sensitivity of 0.77 for detecting RA when 3 positive responses were used as the criterion, and a sensitivity of 0.72 when 4 positive responses were used as the criterion (Table 2). The specificities were low using rheumatology practice patients with noninflammatory conditions as controls, but were much higher among community controls. The sensitivity was slightly higher among Spanish-speaking participants than English-speaking participants, as was the specificity using rheumatology practice patients with noninflammatory conditions as controls. The specificity among the Spanish-speaking community controls was 0.97 using either 3 positive responses or 4 positive responses as the criterion.

Table 2.

Sensitivity and specificity of the RA-related questions of the Connective Tissue Disease Screening Questionnaire for the diagnosis of RA^*.

	Responses Needed for Positive Screening	Sensitivity	Rheumatology Practice Controls with Non- inflammatory Disorders	Specificity Community Controls	Rheumatology Practice Patients with Inflammatory Conditions
English-speaking	3	0.77 (0.63, 0.88)	0.45 (0.34, 0.56)	0.94 (0.85, 0.98)	0.45 (0.33, 0.57)
English-speaking	4	0.72 (0.57, 0.83)	0.56 (0.45, 0.68)	0.95 (0.87, 0.99)	0.50 (0.38, 0.62)
Spanish-speaking	3	0.87 (0.74, 0.95)	0.60 (0.51, 0.68)	0.97 (0.91, 0.99)	0.51 (0.36, 0.66)
Spanish-speaking	4	0.78 (0.64, 0.88)	0.69 (0.60, 0.77)	0.97 (0.91, 0.99)	0.63 (0.48, 0.77)

^*Values in parentheses are 95% confidence intervals.

At a population prevalence of 1%, the PPV of the questionnaire ranged from 0.11 to 0.226 (Table 3). The NPV were high, regardless of language and the number of positive responses required.

Table 3.

Positive predictive values (PPV) and negative predictive values (NPV) for rheumatoid arthritis (RA), based on the English-language and Spanish-language questionnaires, for different prevalences of RA in the population, and based on specificities of the community controls.

		0.5%		Prevalence 1%		2%
Group	Responses Needed for Positive Screening	PPV	NPV	PPV	NPV	PPV	NPV
English-speaking	3	0.06	0.998	0.11	0.997	0.207	0.995
English-speaking	4	0.067	0.998	0.127	0.997	0.227	0.994
Spanish-speaking	3	0.127	0.999	0.226	0.998	0.371	0.997
Spanish-speaking	4	0.115	0.999	0.208	0.997	0.346	0.995

DISCUSSION

Screening for cases of RA in community-based studies is problematic because of the relative rarity of the disease, the lack of unique clinical features, and the imprecise use of the term “rheumatoid arthritis” by the general public. Studies that tested the accuracy of self-reported diagnosis of RA (or self-report of a physician diagnosis) have found this to be highly sensitive, but generally not specific, and would miss cases that have not been diagnosed^10–12. Screening by questionnaire is a practical approach that might provide more accurate results than using self-reported diagnoses alone. Although the CSQ has been reported to have a high sensitivity and specificity for RA, it is important to test its performance in all segments of the population.

In our study of primarily ethnic minorities with relatively low levels of education, the sensitivity of the English-language version was somewhat lower than that reported for a largely Caucasian and well educated sample (0.77 vs 0.85), and the specificity using rheumatology practice patients with noninflammatory conditions as controls was markedly lower (0.45 vs 0.87), while the specificity based on community controls was similar (0.94 vs 0.93)². These findings support the suggestion by Karlson and colleagues that the CSQ may be less accurate among persons of lower socioeconomic status². This finding would translate into the need to evaluate a larger number of participants who screen positive for every true case of RA among subjects of lower socioeconomic status. However, the high specificity and negative predictive value based on data from the community controls suggest that this measure would be a useful screening tool to exclude accurately most individuals who did not have RA.

The Spanish-language version performed as well as or better than the English-language version in this less well educated cohort, with a sensitivity of 0.87 (compared to a sensitivity of 0.77 for the English-language version) and a specificity of 0.97 based on data from the community controls. The high specificity and NPV again suggest that the Spanish-language version may be used to exclude accurately most individuals who did not have RA in a screening program. Testing the Spanish-language version in a group with a low education level is important because this is the situation in which a Spanish-language version would be most applied, as those who are more highly educated would more likely be bilingual. However, the Spanish-speaking group demonstrated the same poor specificity as the English-speaking group when rheumatology practice patients with noninflammatory conditions were used as controls.

It is not axiomatic that screening questionnaires perform less well in ethnic minority or less well educated samples. For example, limited evidence suggests that screening questionnaires for depression perform equally well in African American or low-income patients as in Caucasians and those with private medical insurance^13,14. However, other studies have reported ethnic differences in the performance of screening questionnaires for problem alcohol use and asthma^15–17. Reading ability, comprehension, and unfamiliarity with questionnaire formats may affect responses. In clinic-based studies, respondents may believe that failure to endorse a symptom-related item may lead their care providers to take their complaints less seriously or perhaps discharge them from care, which may lead to false-positive responses. This may be particularly important for patients who are uninsured or have experienced past difficulty accessing medical care, as was true for most patients in our practice, or for those who have experienced discrimination. We hypothesize that this perception largely accounts for the difference in specificities between our clinic-based samples, which range from 0.45 to 0.69, and those of an insured sample (0.87)². This motivation would not be present among community controls, and the proportion of false positives would not be expected to be inflated by this concern. In accordance with this interpretation, the specificity of the questionnaire for RA in our community participants was similar to that reported in an insured cohort².

Using 3 positive responses as the criterion for a positive screening test had better sensitivity and comparable specificity to using 4 positive responses as the criterion. RF test results were one of the 6 responses scored, but few community controls reported having had a RF test. In practice, this reduced the number of evaluable responses from 6 to 5 for the community controls. The presence of 3 of 5 clinical features would therefore be considered a positive “screen” for RA when applied in the community. It is important to note that the performance of the questionnaire in the community was based on active surveillance, and not on passive recruitment. Passive recruitment strategies, such as community advertisement, would likely preferentially identify subjects with some rheumatic complaints, and result in a specificity similar to that seen in the rheumatology practice controls.

The strengths of our study include large cohorts enriched for the target group of underserved and less well educated participants, and testing of both English-language and Spanish-language versions. We also tested the specificities of the questionnaires in the community, similar to the setting in which it would be used in epidemiological surveys. However, our study has some limitations. Use of patients with RA from a rheumatology practice might not be representative of the spectrum of RA in the population, which might affect the estimates of sensitivity of the questionnaire. In addition, some patients in this sample may have had experience completing questionnaires and may have answered based on their current symptoms of RA, rather than based on symptoms that had ever been present. To the extent this occurred, the sensitivity would be underestimated. We did not clinically evaluate the community controls for the presence of RA, and may have misclassified some of the controls as not having RA. This misclassification would lead us to underestimate the questionnaire’s specificity. Poor recall of past symptoms might have affected estimates of the sensitivity of the questionnaire. Lastly, studying a larger number of participants would have allowed more precise estimates of the accuracy of the questionnaire.

The English-language and Spanish-language versions of the RA-related questions of the CSQ demonstrated good sensitivity, excellent specificity, and a high NPV in this group of primarily ethnic minority participants in an under-served area. Using a criterion of 3 positive responses would maximize the questionnaire’s sensitivity without compromising specificity.