The Validity of Self-Reported Rheumatoid Arthritis in a Large Cohort: Results from the Black Women's Health Study

Abstract

Objective

To evaluate the positive predictive value (PPV) of three case definitions of rheumatoid arthritis (RA) based on self-reported data on RA diagnosis and use of arthritis medications and to determine if a validated screening survey would increase the PPVs in the three groups.

Methods

Medical records and physician-checklists were reviewed for confirmation of a RA diagnosis among a sample of Black Women's Health Study participants who reported incident RA and were categorized according to reported medications: disease-modifying anti-rheumatic drugs (DMARDs) (N=102), non-steroidal anti-inflammatory drugs (NSAIDs) (N=100), and no arthritis medications (No Meds) (N=101). PPVs for confirmed RA were calculated for each of the medication groups, overall and according to results of the screening survey.

Results

The PPV of confirmed RA was 76%, 61%, and 29% in the DMARDs group, NSAIDs group, and No Meds group, respectively. After exclusion of women who reported other rheumatic conditions or who reported taking only prednisone, the PPV increased in the DMARDs group to 88%, but little improvement was seen in the other groups. The PPVs increased somewhat according to results of the screening survey for the DMARDs group (positive: 92% vs. negative screen: 85%, p=1.00), and increased substantially for the NSAIDs group (89% vs. 38%, p=0.03), but only 43% of participants completed the survey.

Conclusion

We found that self-report of RA, along with DMARDs is a useful case definition for identifying confirmed RA. The validated screening survey could be useful for identifying cases of confirmed RA in some, but not all medication groups.

Rheumatoid arthritis (RA) is a chronic, systemic, inflammatory disease of unknown etiology, which affects approximately 1% of adults (1) and often results in severe pain and disability (2-6). The prevalence of RA increases with age and it is a disease that is two to three times more likely to affect women than men (7).

Since incident RA is relatively uncommon, hospital and clinic based epidemiologic studies often face difficulties in enrolling adequate numbers of cases. Large, population-based studies could advance knowledge of the etiology of RA, but they often rely on self-reported diagnosis information. The positive predictive value of self-reported RA has been studied in various populations including the elderly and disabled, and was found to be low, ranging between 21% and 34% (8-10), possibly due in part to confusion with other forms of arthritis, such as osteoarthritis. In two large population-based studies that investigated risk factors for RA, the Nurse's Health Study and the Iowa Women's Health Study, only 6-7% of self-reported RA diagnoses could be confirmed by medical record review (11,12). To overcome the validity concerns associated with self-reported RA, studies have relied on medical record review to confirm the diagnosis. However, the number of potential cases lost may be substantial because of the difficulty in obtaining consent from study participants to release medical information (11-13). In addition, selective consent to record review could introduce bias in risk factor studies. Therefore, a valid RA case definition based on self-reported information would be very valuable for large questionnaire-based epidemiologic studies.

The primary objective of the present study was to evaluate the predictive value of three case definitions of RA among African-American women, each based on self-reported data on RA diagnosis and use of arthritis medications. A secondary objective was to determine if supplemental self-reported information based on responses to a validated screening survey increased positive predictive values within the three case groups.

Materials and Methods

Population

The Black Women's Health Study (BWHS) is a prospective study of 59,000 Black women throughout the United States that began in 1995. The participants were recruited through mailings to subscribers of Essence magazine, a general readership magazine targeted to African-American women, mailings to members of African-American professional organizations, and mailings to friends and relatives of early participants (14). The 1995 baseline questionnaire obtained information on demographic characteristics, reproductive and medical histories, and lifestyle characteristics such as cigarette smoking and diet. Biennial follow-up questionnaires mailed to all participants elicited information on disease occurrence and selected exposures with 80% or more of participants responding in each follow-up cycle.

Sample

Subjects for the present study include a sample of BWHS participants who reported that they had been diagnosed for the first time with RA in the two years preceding either the 1999 or 2001 follow-up questionnaire. The sample was categorized based on reported medications into three hierarchical groups: disease-modifying anti-rheumatic drugs (DMARDs), non-steroidal anti-inflammatory drugs (NSAIDs), and no reported RA medications (No Meds). The smallest medication group, DMARDs (N=101), set the sample size for the two other medication groups. The three groups were selected as follows:

1. Self-reported RA and use of DMARDs. All women who reported RA and DMARDs in 1999 (N=60) or 2001 (N=41) were included in this group. DMARDs include methotrexate, azathioprine, D-penicillamine, sulfasalazine, etanercept, myochrisine, cyclophosphamide, infliximab, hydroxychloroquine, leflunomide, glucocorticoid steroids (prednisone), and gold.

2. Self-reported RA and use of NSAIDs only. All women who reported RA and NSAIDs in 2001 (N=70) and a random sample of women who reported RA and NSAIDs in 1999 (N=31), but who did not report any DMARDs, were included in this group. NSAIDs include indomethacin, etodolac diclofenac sodium, nabumetone, sulindac, piroxicam, celecoxib, rofecoxib, ibuprofen, naproxen sodium, and ketoprofen.

3. Self-reported RA and no use of DMARDs or NSAIDs (No Meds). A random sample of women who reported RA and no arthritis medications in 2001 (N=101) was included in this group.

Because a sufficient number of women reported RA and No Meds in the 2001 questionnaire, a sample of women who reported RA and No Meds in 1999 was not needed. One participant in the NSAIDs group was subsequently found to have reported using prednisone, and was moved into the DMARDs group, resulting in 102 subjects in the DMARDs group, 100 subjects in the NSAIDs group, and 101 subjects in the No Meds group. The participants ranged from 27 to 73 years of age, with an average age of 48 years.

Screening Questionnaire and Case Confirmation

All women in the sample were mailed a form for consent to release medical information. The consent form requested permission to obtain copies of medical records and to send a checklist to their rheumatologist and/or other physician who could provide information regarding her RA. The checklist consisted of the American College of Rheumatology (ACR) criteria for RA (15). The consent form provided an option to consent only to the physician checklist, but not the release of medical records.

All women were also mailed a RA screening survey, which has been validated in African-American women. The survey consists of the RA portion of questions from the Connective Tissue Disease Screening Questionnaire (16, 17) and elicits information on the year and month of RA diagnosis, the year of symptom onset, medications used to treat RA, RA symptoms, and results of the rheumatoid factor test. A positive screen on the survey was defined as report of at least four of the following six symptoms included in the questionnaire: morning stiffness for at least one hour and present for at least six weeks; swelling of three or more joints for more than six weeks; swelling of wrist, metacarpophalangeal, or proximal interphalangeal joints for more than six weeks; symmetric joint swelling; rheumatoid nodules; and a positive test result for rheumatoid factor (16).

Both the consent form and the screening survey provided the participant the opportunity to refute her initial report of physician-diagnosed RA. Women who did not respond to the initial mailing of the consent form and screening survey were sent up to two additional mailings and were then telephoned. Physicians who did not respond to the initial medical records/checklist request were sent one additional request and were then telephoned. Physicians who had still not responded were then mailed or faxed a modified checklist, which asked for confirmation or refutation of an RA diagnosis and a list of any RA medications prescribed for the subject.

Case Definitions

The ACR criteria for RA (15), documentation of an RA diagnosis, year of diagnosis and year of symptom onset were abstracted from medical records (reviewed by a trained rheumatologist blinded to subjects' reported medications) and physician checklists for all subjects for whom medical information was received. Three case definitions were used to classify subjects, ‘definite’, ‘probable’, and ‘clinical’ RA. Subjects were classified as ‘definite’ RA based on documentation of at least four of seven ACR criteria (15). Limiting cases of RA to only those who meet four criteria may exclude those with less severe disease (7,18). Therefore, ‘probable’ RA was defined as those who met three ACR criteria, along with those classified as having ‘definite’ RA. Finally, because the ACR criteria may not be well documented in the medical records, participants with an RA diagnosis mentioned in their medical records or physician checklist, or who met at least three ACR criteria were defined as having ‘clinical’ RA.

Statistical Analyses

Response rates to the medical records release form and RA screening questionnaire were calculated for each of the three study sample groups. Among those with medical records or physician checklists, the positive predictive values for those classified as confirmed RA (definite, probable, clinical) were calculated for each study sample group. Positive predictive values were also calculated according to the results of the RA screening survey (positive or negative screen) within each study sample group. Many rheumatic diseases have a high degree of symptom overlap and are treated with prednisone. The accuracy of self-reported RA may be improved by excluding subjects who reported other rheumatic conditions or whose only reported medication was prednisone. Therefore, we additionally calculated the positive predictive values of confirmed RA in the three study sample groups excluding those who reported other rheumatic conditions (systemic lupus erythematosus, myositis, dermatomyositis, mixed/undifferentiated connective tissue disease, scleroderma, CREST, Sjogren's syndrome, and sarcoidosis), and in the DMARDs group, excluding those whose only reported DMARD was prednisone. Homogeneity of the positive predictive values of definite, probable, and clinical RA was tested across the three sample groups. In addition, pairwise comparisons of the predictive values of the three sample groups were made for definite, probable, and clinical RA. The utility of the RA screening survey was also evaluated by comparing the positive predictive values of definite, probable, and clinical RA for those who screened positive versus negative within each sample group. Chi-square tests of proportions, or Fisher's exact tests for cells with fewer than five subjects, were used for all comparisons.

Results

Frequencies and percentages of subjects' response to medical records release requests and RA screening surveys, and physicians' response to medical records requests are displayed in Figure 1. Almost half of the DMARDs group consented to release their medical information, while only one-third of the NSAIDs group and one-fifth of the No Meds group were willing to release information. Three (3%) subjects in the DMARDs group refuted their initial report of RA, while 21 (21%) and 15 (15%) subjects refuted the diagnosis in the NSAIDs and No Meds groups, respectively. There was no response to the medical records release or RA screening survey from 37% of the subjects in both the DMARDs and NSAIDs groups, and almost half of the No Meds group did not respond. Among those who consented to release of medical information, medical records or physician checklists were received for 88% of the DMARDs group, 70% of the NSAIDs group and 67% of the No Meds group. The final proportions of those for whom medical information was received in each sample group was 41% for the DMARDs group, 23% for the NSAIDs group, and 14% for the No Meds group.

Figure 1.

Flow chart of subjects' response to medical records release requests and RA screening survey, and physicians' response to medical records requests.

* RA screening surveys were received from all subjects for whom medical records release forms were received.

** A modified physician checklist was sent to non-responding physicians (N=48), which included only information on an RA diagnosis and medications (no ACR criteria for RA). Modified physician checklists were received for 5, 4 and 3 participants in the DMARDs, NSAIDs, and No Meds groups, respectively.

Table 1 describes the frequencies and positive predictive values of confirmed RA by sample group among those for whom medical records or physician checklists were received. Table 2 displays the p-values for tests of homogeneity of proportions across the three sample groups, and pairwise comparisons of positive predictive values of definite, probable, and clinical RA excluding those who reported prednisone only or other rheumatic conditions. Because the modified physician checklist did not include the ACR criteria, the number of subjects for whom medical information was obtained is higher for clinical RA than for definite or probable RA. The positive predictive value of definite RA was highest for the DMARDs group (62%), intermediate for the NSAIDs group (53%), and lowest for the No Meds group (27%). The exclusion of women who had used only prednisone, or who had other rheumatic conditions, increased the positive predictive value in the DMARDs group (75%), but only slightly affected the NSAIDs (50%) and No Meds (30%) groups (p=0.03 for homogeneity). Using the case definition of probable RA did not substantially increase the positive predictive values. After exclusions, positive predictive values for probable RA were 79%, 56%, and 30% in the DMARDs, NSAIDs, and No Meds groups, respectively (p=0.02 for homogeneity). The positive predictive value of clinical RA was 76% in the DMARDs group, 61% in the NSAIDs group, and 29% in the No Meds group. After exclusions, the positive predictive value increased substantially in the DMARDs group to 88%, but little improvement was seen in the NSAIDs or No Meds groups (p=0.0005 for homogeneity).

Table 1.

Frequencies and positive predictive values (PPV) of definite, probable, and clinical RA by sample group among those for whom medical information was received.

Sample Group	Definite RA			Probable RA		Clinical RA
	Medical Information Received		PPV		PPV	Medical Information Received*		PPV
	N	N	%	N	%	N	N	%
DMARDs	37	23	62	25	68	42	32	76
Excluding Prednisone Only	33	22	67	23	70	38	30	79
Excluding Other Rheumatic Conditions	30	22	73	23	77	34	29	85
Excluding Prednisone Only and Other Rheumatic Conditions	28	21	75	22	79	32	28	88
NSAIDs	19	10	53	11	58	23	14	61
Excluding Other Rheumatic Conditions	18	9	50	10	56	22	13	59
No Meds	11	3	27	3	27	14	4	29
Excluding Other Rheumatic Conditions	10	3	30	3	30	13	4	31

^*For Clinical RA, response to the modified physician checklist increased the number of subjects for whom medical information was received.

Table 2.

P-values of Fisher's exact tests or chi-square tests of homogeneity of proportions across the three sample groups, and pairwise comparisons of positive predictive values of definite, probable, and clinical RA excluding those who reported prednisone only or other rheumatic conditions.

Comparison	Definite RA	Probable RA	Clinical RA
Homogeneity	0.03	0.02	0.0005
DMARDs vs. No Meds	0.02	0.02	0.0004
DMARDs vs. NSAIDs	0.08*	0.10*	0.02
NSAIDs vs. No Meds	0.43	0.25	0.16

^*Chi-square test

The positive predictive value of confirmed RA was higher in the DMARDs group than the No Meds group for all case definitions (definite RA: 75% vs. 30%, p=0.02; probable RA: 79% vs. 30%, p=0.02; and clinical RA: 88% vs. 31%, p=0.0004). Higher positive predictive values were found in the DMARDs group compared to the NSAIDs group for all case definitions (definite RA: 75% vs. 50%, p=0.08; probable RA: 79% vs. 56%, p=0.10; clinical RA: 88% vs. 59%, p=0.02). In addition, higher positive predictive values were found in the NSAIDs group compared to the No Meds group for all case definitions.

The performance of the RA screening survey is displayed in Table 3. Within the DMARDs group, the proportion of cases classified to have definite, probable or clinical RA who screened positive was somewhat higher than that of those who screened negative, although no statistically significant differences were found (clinical RA: 92% vs.85%, p=1.00). Within the NSAIDs group, the proportion of cases classified to have definite, probable or clinical RA who screened positive was substantially higher than that of those who screened negative (clinical RA: 89% vs. 38%, p=0.03). Within the No Meds group, the numbers were too small to make any valid comparisons of the RA screening survey results.

Table 3.

Frequencies, positive predictive values (PPV) and Fisher's exact tests of definite, probable, and clinical RA by screening survey results within sample groups excluding those who reported prednisone only and other rheumatic conditions.

Sample Group	Definite RA				Probable RA			Clinical RA
	Medical Information Received		PPV			PPV		Medical Information Received*		PPV
	N	N	%	P-value	N	%	P-value	N	N	%	P-value
DMARDs
+ Screen	9	8	89	0.37	8	89	0.63	12	11	92	1.00
- Screen	19	13	68		14	74		20	17	85
NSAIDs
+ Screen	8	6	75	0.15	6	75	0.19	9	8	89	0.03
- Screen	10	3	30		4	40		13	5	38
No Meds
+ Screen	2	1	50	1.00	1	50	1.00	3	1	33	1.00
- Screen	8	2	25		2	25		10	3	30

^*For Clinical RA, response to the modified physician checklist increased the number of subjects for whom medical information was received.

Discussion

We found that the accuracy of a self-reported diagnosis of RA varies within groups defined by reported medications. While RA could not be confirmed among a high proportion of women who reported taking no arthritis medications or reported taking NSAIDs, most women who reported taking DMARDs were confirmed to have RA. This self-reported case definition improved after excluding women whose only reported DMARD was prednisone and women who reported other rheumatic conditions.

Our finding that the validity of self-reported RA improves substantially with reported DMARDs is similar to that of a validation study of self-reported RA among post-menopausal participants in the Women's Health Initiative (13). Walitt et al. found that the PPV of self-reported RA increased from 14.7% to 62.2% when DMARDs were also reported, but did not evaluate other medication groups, such as NSAIDs (13).

As was expected, the proportion of cases confirmed as clinical RA was greater than the proportions confirmed by ACR criteria. While confirmation of RA by documentation of the ACR criteria has become the “gold standard,” there are several problems with its use in this and other large epidemiologic studies. First, the ACR criteria were developed primarily as a research tool, not as a means of clinically diagnosing patients (15). While physicians use the ACR criteria as a guide, patients are frequently diagnosed and treated for RA without meeting four criteria. It is, therefore, possible for subjects to have a physician-diagnosis of RA, but not meet the criteria for definite RA. In addition, the ACR criteria may not be well documented in the medical record. For example, among the subjects for whom we received both medical records and physician checklists (n=7), an average of four criteria were documented in the checklist, while an average of only one criterion was documented in the medical record. Last, because patients often see more than one physician (i.e. family physician, rheumatologist), the medical records may be in more than one facility, increasing the difficulty of obtaining a complete record.

The RA screening survey has been found to be useful in identifying RA cases in previous research (16), and has been shown to increase the PPV of self-reported RA beyond that of self-reported DMARDs (13). Walitt et al. reported a PPV of 82% among those who reported RA, along with DMARDs and a positive screen on the CSQ, compared to a PPV of 62% among those who reported only RA and DMARDs. We found that among women who reported RA and DMARDs, the screening survey provided little additional information to help identify cases of clinical RA. The CSQ did improve the PPV in our study from 68% to 89% for definite RA, which is likely a more similar outcome to that used by Walitt et al. (13). The survey did appear to identify cases in the BWHS among those who report NSAIDs, but the numbers in this study were too small to be definitive. A drawback of using the survey is that only 43% of women completed it; relying on the survey for validation would, therefore, result in an appreciable loss of cases. Further research should be conducted to determine the usefulness of the RA screening survey in ascertaining cases as a third step after self-reported RA and NSAIDs.

A limitation of our study is the low response to medical records requests, which reduced statistical power and introduced the potential for significant selection bias. Of utmost concern is the large percentage of subjects in the DMARDs group for whom no medical information was received (59%) and the possibility that they, in fact, do not have RA. The subjects were given the opportunity to refute their initial report of RA, however. In addition, when questioned, BWHS participants cited confidentiality concerns as their reason for refusal to allow record review. We found little difference between those for whom medical information was received and those for whom no medical information was received on age in years (48 vs. 46, p=0.48) or smoking (62% vs. 58% ever smoked, p=0.70), and although not statistically significant, a higher percentage of those for whom medical information was received had graduated from college compared to those for whom no medical information was received (48% vs. 38%, p=0.33). In addition, women for whom no medical information was received, but who responded to the RA screening survey (n=16) were compared to the women for whom medical information was received (n=42) and were found to have no differences in mean number of reported RA symptoms (2.6 vs. 2.6, p=0.91). A similar response rate was found for a study of systemic lupus erythematosus in the BWHS (19), and even in a study of female health professionals who reported connective tissue disease, response to requests for medical records was only 28% (20). Difficulties in obtaining medical information in this and other studies illustrate the disadvantage of validating cases through medical record review and underscores the need for a RA case definition based on self-reported information.

Medical information (i.e. medical records and physician checklists) was required to confirm or disconfirm a subjects' report of a RA diagnosis. Therefore, positive predictive values were calculated only among women for whom we received medical information. Women who refuted their initial report of RA were not included in the calculations because it is unclear whether the initial report of RA or the refutation of the diagnosis is accurate and we have no medical information to confirm either. It is possible that these women do not have RA, which would lower the positive predictive values in the sample groups. For clinical RA, the positive predictive value would slightly decrease in the DMARDs group to 80%, but would substantially decrease in the NSAIDs (30%) and No Meds (14%) groups.

We were unable to estimate the number of unreported RA cases in our study population. Ideally, medical records would have been obtained on a sample of women who did not report RA to determine the proportion of cases missed by relying on self-reported diagnoses. However, ascertaining medical records on such a sample was not feasible due to the difficulty with obtaining consent to release medical information among a group of subjects who have not reported the diagnosis under study. Based on previous research, the number of missed cases is likely to be small (10), reducing the likelihood of considerable bias in risk factor analyses using a case definition based on self-reported information.

Self-reported RA together with DMARDs was the best case definition, but its use would introduce some misclassification. Women who self-reported RA and were true cases, but did not report use of DMARDs would be excluded. On the other hand, inclusion of women who self-reported RA and DMARDs, but do not have RA would include some false positives. However, the small number of false positive RA cases in the DMARDs group would result in a very high specificity and, assuming misclassification is non-differential, a highly specific case-definition would result in negligible bias of the effect estimate (21). For example, assuming an incidence rate of RA of 45/100, 000 person-years and 200, 000 person-years of follow-up over a four year period, we'd expect approximately 90 true cases of RA in the BWHS (N≈ 59,000). Based on the best case definition of RA in this study (self-reported RA and DMARDs, excluding those who reported prednisone only and other rheumatic conditions), we would classify 63 subjects as having RA. A predictive value of 88% would result in 55 correctly classified clinical RA cases, providing a sensitivity of 61% (55/90) and a specificity of ≥99% (58,902/58910). With a 50% exposure in the BWHS cohort, disease misclassification would bias a true risk ratio of 2.0 to 1.9, and a true risk difference of 0.001 to 0.0006.

The “gold-standard” for confirmation of an RA diagnosis in research is clinical exam or medical record review for the ACR criteria, which are not feasible in large, geographically dispersed epidemiologic studies as suggested by the low response rate in this and other studies. Previous research has indicated that the positive predictive value of self-reported RA is low (8-10,19), but improves with additional information such as reported medications. We confirmed that self-report of RA, along with DMARDs is a valid case definition for identifying clinical RA, and is sufficient for use after excluding those who report other rheumatic conditions and prednisone as the only reported DMARD. This case definition for RA based on self-reported information may improve and advance research on RA in large, epidemiologic studies.