Interobserver Agreement Among Uveitis Experts on Uveitic Diagnoses: The Standardization of Uveitis Nomenclature Experience

Abstract

• PURPOSE:

To evaluate the interobserver agreement among uveitis experts on the diagnosis of the specific uveitic disease.

• DESIGN:

Interobserver agreement analysis.

• METHODS:

Five committees, each comprised of 9 individuals and working in parallel, reviewed cases from a preliminary database of 25 uveitic diseases, collected by disease, and voted independently online whether the case was the disease in question or not. The agreement statistic, κ, was calculated for the 36 pairwise comparisons for each disease, and a mean κ was calculated for each disease. After the independent online voting, committee consensus conference calls, using nominal group techniques, reviewed all cases not achieving supermajority agreement (> 75%) on the diagnosis in the online voting to attempt to arrive at a supermajority agreement.

• RESULTS:

A total of 5766 cases for the 25 diseases were evaluated. The overall mean κ for the entire project was 0.39, with disease-specific variation ranging from 0.23 to 0.79. After the formalized consensus conference calls to address cases that did not achieve supermajority agreement in the online voting, supermajority agreement overall was reached on approximately 99% of cases, with disease-specific variation ranging from 96% to 100%.

• CONCLUSIONS:

Agreement among uveitis experts on diagnosis is moderate at best but can be improved by discussion among them. These data suggest the need for validated and widely used classification criteria in the field of uveitis.

THE UVEITIDES ARE A COLLECTION OF MORE THAN 30 diseases characterized by intraocular inflammation.^1-3 They can be thought of as a matrix of diseases classified by the primary site of inflammation (anatomic class) and by whether they are infectious in nature, associated with a systemic autoinflammatory or autoimmune disease, or eye-limited and presumed to be immune-mediated.¹ The diagnosis of the specific uveitic disease is made by a combination of the clinical features and selected laboratory testing.^1,2

Classification criteria are a method of diagnosing individual diseases for research purposes.⁴ They differ from clinical diagnostic criteria in that although they seek to maximize sensitivity and specificity (eg, minimize misclassification), when a tradeoff is needed, they emphasize specificity over sensitivity.⁴ The goal of classification criteria is to define a homogeneous group of patients for inclusion in clinical research projects, such as clinical trials, prospective cohort studies, and translational or pathogenesis research. The criteria therefore seek to optimize the likelihood that all patients in the project are generally accepted to have the disease. Among the best recognized classification criteria are those developed for the rheumatic diseases, such as rheumatoid arthritis, systemic lupus erythematosus, spondyloarthritides, and juvenile idiopathic arthritis subgroups.^5-12 These criteria have used datasets of cases submitted for the project(s) to develop and evaluate the criteria. In addition, they have been revised over time as new information has become available.^5-12

In the field of uveitis, diagnostic criteria have been proposed for several diseases, but there has been limited formal evaluation of many of these criteria.^13-22 The Standardization of Uveitis Nomenclature (SUN) Working Group is an international collaboration dedicated to improving clinical research in the field of uveitis.² They are developing classification criteria for the leading 25 uveitides using a formal approach to development and evaluation.^3,23

The SUN Developing Classification Criteria for the Uveitides project is proceeding in 4 phases: 1) informatics, 2) case collection, 3) case selection, and 4) machine learning.^3,23 The informatics phase developed a standardized vocabulary and set of dimensions for describing cases of uveitis. It enabled the development of a standardized, dropdown menu–driven, hierarchical case report form for collecting information on uveitides.^3,23 Case collection was accomplished by individual investigators from each of the 65 clinical centers entering data into a preliminary database using the standardized form. Case selection involved committees of 9 individuals reviewing each case to determine whether they agreed with the diagnosis, thereby developing a final database of cases with supermajority agreement on the diagnosis from the preliminary database. The machine learning phase will evaluate the features of each disease and compare them to other diseases in the differential diagnosis in order to develop a parsimonious set of criteria for each of the 25 diseases under consideration that minimizes misclassification for each disease.

The case selection phase of the project included an independent online voting step by each committee member, in which they decided to accept the case if they agreed with the diagnosis or “reject” it, if they did not. This online voting step permitted the evaluation of the agreement among uveitis experts on the diagnosis of these diseases.

METHODS

• CASE COLLECTION:

Information was entered into the SUN database by the 76 contributing investigators for each of the 25 diseases under consideration. The data were entered into a dropdown menu–driven form using the terminology developed during the informatics phase of the study to maximize discreet data collection and minimize free text. For selected relevant diseases (eg, posterior and pan-uveitides), the investigators also uploaded imaging (eg, fundus photographs, fluorescein angiograms, and optical coherence tomography) sufficient to assist with the diagnosis. Case information was deidentified, and investigators entered cases retrospectively from existing case records. Investigators were instructed to enter the data from the presentation visit, or in the unusual case where there was disease evolution, the visit at which the diagnosis was first known. The target was 250 cases for each disease, and once that number was reached, case entry for that disease was closed. For several diseases, typically less common ones, a final number <250 was achieved. For a few diseases, the number of cases entered exceeded 250 as cases were entered from multiple centers, and extra cases were entered just before closing the disease. In order to have a sufficient number of cases of the several manifestations of sarcoidosis, additional cases were entered and the upper limit was set at approximately 400.

• CASE SELECTION:

The case selection phase of the project was designed to ensure that cases in the final database were widely accepted (ie, accepted by a supermajority of clinicians) as having the disease. Committees of 9 individuals, geographically and “school of thought” dispersed, were constituted to review the cases for each disease. Five committees worked in parallel: anterior uveitides; intermediate and pan-uveitides; posterior uveitides; infectious posterior and pan-uveitides; and other diseases (eg, tubulointerstitial nephritis with uveitis and sarcoidosis). Case selection occurred in 2 steps: online voting and consensus conference calls. During the online voting step, each committee member reviewed the each case’s information independently and then voted to include the case in the final database (“accept”) or exclude it (“reject”). The information was presented on standardized forms, using standardized terminology and descriptive phrases derived during the informatics phase²³ (Supplemental Figure available online at AJO.com). A “forced choice” was required, and committee members were instructed to vote on whether they thought the diagnosis was correct using their clinical judgment. Any case getting a supermajority of online votes to accept (>75%) was included in the final database, and any case getting a supermajority of online votes to reject was excluded. Cases with less than a supermajority were tabled for consensus conference calls. Diseases were addressed sequentially by each committee.

Consensus conference calls were conducted for all tabled cases for each disease on a disease by disease basis using nominal group techniques.²⁴ Nominal group techniques are a formalized discussion approach for achieving consensus, designed to ensure participation from each member and avoid “dominant personality” effects. In short, for each case tabled in the online voting step, each committee member makes a brief, uninterrupted comment about the case; after all committee members had commented, there was anonymous “real time” online voting. Cases getting a supermajority (>75%) of “accept” votes were accepted into the final database without further discussion, and cases getting a supermajority of “reject” votes were excluded. Cases neither accepted nor rejected were subjected to a second round of comments and voting. Cases neither accepted nor rejected after the second round of voting were permanently tabled and were not included in the final database.²⁴

The study adhered to the principles of the Declaration of Helsinki. Institutional review boards at each participating institution reviewed and approved the study; the study typically was considered either minimal risk or exempt by the individual institutional review boards.

The independent online voting step provided the opportunity to evaluate the agreement on diagnosis among uveitis experts. For each disease, pairwise kappas (κ) were calculated between each pair of committee members. Therefore, for each disease there were 36 pairwise κs; for each disease, a mean, standard deviation, and range of the κs were calculated. The κ statistic is an agreement statistic that corrects for agreement by chance alone. It ranges from −1.00 (zero agreement) to 1.00 (perfect agreement), with κ = 0 representing chance agreement. Kappas >0.70 are considered “substantial,” and those >0.85 are considered “almost perfect.”^25,26

RESULTS

A TOTAL OF 5766 CASES WERE COLLECTED FOR THE 25 DISeases under consideration. Because of the important differential diagnosis between serpiginous choroiditis and serpiginous-like tuberculous choroiditis, the latter were collected specifically in addition to other cases of tuberculous choroiditis. Also because of the different manifestations of early and late Vogt-Koyanagi-Harada disease, separate datasets were collected for each.

The mean κ from the online voting for the entire project was 0.39 (moderate agreement) with a range of disease-specific mean κs from 0.23 for toxoplasmic retinitis to 0.79 for cytomegalovirus (CMV) anterior uveitis (Table). For the entire project, only 1 pair of individuals had perfect agreement (κ = 1.00), and this occurred in CMV anterior uveitis. For several diseases, there was at least 1 pair of individuals whose agreement was essentially “chance alone” (κ ~ 0.00); diseases where there were 2 individuals who agreed at or close to “chance alone” (κ < 0.10) included herpes simplex anterior uveitis, juvenile idiopathic arthritis-associated uveitis, pars planitis, multifocal choroiditis with panuveitis, serpiginous choroiditis, serpiginous-like tuberculous choroiditis, CMV retinitis, toxoplasmic retinitis, tuberculous uveitis, and sympathetic ophthalmia.

TABLE.

Interobserver Agreement Among Uveitis Experts on Uveitic Diagnosis

		Online Voting Results			Final Results After Consensus Conference (% Cases)
Disease	No. of Cases Submitted	Mean κ	SD	Range κsa	Accepted	Rejected	Tabled
Cytomegalovirus anterior uveitis	112	0.79	0.15	0.51–1.00	82	17	1
Herpes simplex anterior uveitis	250	0.32	0.14	0.00–0.56	42	56	2
Varicella zoster anterior uveitis	163	0.58	0.10	0.35–0.87	76	23	1
Fuchs uveitis syndrome	249	0.44	0.13	0.16–0.65	59	41	<1
Juvenile idiopathic arthritis chronic uveitis	251	0.29	0.16	−0.02 to 0.64	80	19	1
Spondylitis/human leukocyte antigen-B27–associated uveitis	251	0.47	0.11	0.27–0.71	74	22	4
Tubulointerstitial nephritis with uveitis	125	0.54	0.22	0.16–0.87	76	24	0
Pars planitis	308	0.32	0.15	−0.04 to 0.63	74	25	1
Intermediate uveitis, non–pars planitis type	209	0.49	0.09	0.27–0.67	55	45	0
Multiple sclerosis associated uveitis	183	0.44	0.08	0.31–0.62	62	38	<1
Acute posterior multifocal placoid pigmentepitheliopathy	149	0.44	0.12	0.17–0.84	52	48	0
Birdshot chorioretinitis	257	0.36	0.09	0.20–0.57	81	18	1
Multiple evanescent white dot syndrome	95	0.39	0.12	0.10–0.75	54	44	2
Multifocal choroiditis with panuveitis	251	0.30	0.13	0.02–0.58	57	42	1
Punctate inner choroiditis	250	0.52	0.08	0.32–0.70	58	42	0
Serpiginous choroiditis	157	0.37	0.19	−0.02 to 0.69	78	22	<1
Serpiginous-like tuberculous choroiditis	104	0.28	0.15	−0.02 to 0.55	92	8	0
Acute retinal necrosis	252	0.43	0.18	0.13–0.61	75	25	<1
Cytomegalovirus retinitis	251	0.27	0.16	0.07–0.65	84	16	0
Syphilitic uveitis	250	0.47	0.12	0.15–0.68	86	14	0
Toxoplasmic retinitis	213	0.23	0.14	0.03–0.53	82	17	1
Tuberculous uveitis	254	0.24	0.16	0.01–0.58	71	27	2
Behçet disease	248	0.36	0.13	0.15–0.61	80	18	2
Sarcoid uveitis	383	0.56	0.15	0.23–0.86	72	28	0
Sympathetic ophthalmia	149	0.31	0.12	0.07–0.51	75	25	<1
Vogt-Koyanagi-Harada disease, early	224	0.45	0.14	0.16–0.72	69	30	1
Vogt-Koyanagi-Harada disease, late	177	0.42	0.14	0.13–0.68	58	41	1
Overall	5766	0.39	0.14	−0.04 to 1.00	71	28	1

SD = standard deviation.

^aRange of the κs for the pairwise comparisons within a disease.

For the entire project, after the consensus conference calls, 71% of the submitted cases were accepted into the final database and 28% were rejected. Approximately 1% of submitted cases were permanently tabled. Disease-specific acceptances into the final database ranged from 42% (for herpes simplex anterior uveitis) to 92% for serpiginous-like tuberculous choroiditis. Disease-specific permanent tabling of submitted cases ranged from 0% for several diseases to 4% for spondylitis/human leukocyte antigen-B27-associated uveitis, with the majority of diseases in the 0% to 1% range.

DISCUSSION

OUR DATA SUGGEST THAT WHEN FUNCTIONING INDEPENdently, agreement among uveitis experts on the diagnosis of the specific disease entity is moderate at best, although there is variation by specific uveitic disease. Diseases in which there was good agreement were those for which there is a specific laboratory test, such as CMV anterior uveitis. The diagnosis of CMV anterior uveitis rarely is made without detection of CMV DNA in the anterior chamber on polymerase chain reaction (PCR) testing of anterior chamber paracentesis fluid.

Diseases that had lower κ values were those that are morphologically diagnosed, such as multifocal choroiditis with panuveitis and infectious retinitides. Similar problems arose when there was a laboratory test that is a risk factor (eg, HLA-B27) but that does not automatically diagnose the disease because of the population frequency of a positive test result and the resultant need to determine if the clinical picture is compatible with the disease, and those diseases where a test was necessary but not sufficient to make the diagnosis (eg, testing for tuberculosis), because of the frequency of an unrelated abnormal test result in the population being studied. Disagreements also were encountered where there were 2 possible diagnoses. Herpes simplex virus (HSV) anterior uveitis and varicella zoster virus (VZV) anterior uveitis are illustrative. Although PCR for viruses can be performed to establish the diagnosis, the disease often is diagnosed morphologically. Some related clinical features can be used reliably (eg, the presence of documented HSV keratitis or dermatomal zoster) to aid in diagnosis without laboratory confirmation. Studies using anterior chamber paracentesis and PCR of patients with anterior uveitis and sectoral iris atrophy have shown that in >95% of cases these patients have either HSV or VZV.²⁷ Patients with HSV are younger at onset (<60 years of age with a mean of 34 years), and those with VZV are older (>50 years of age with a mean of 65 years). It can be reasonably inferred that a patient <50 years of age with anterior uveitis with sectoral iris atrophy has HSV anterior uveitis, whereas a patient >60 years of age with anterior uveitis and sectoral iris atrophy has VZV anterior uveitis. However, in the absence of PCR testing or other cardinal features, it may not be possible to be certain of the diagnosis in patients who are 50 to 60 years of age.

These data also suggest that in the absence of validated and widely used classification criteria, case series from different centers may not be reporting the comparable sets of patients. This is of particular concern where the range of pairwise κs included a pair with agreement close to “chance alone” agreement.

However, the ability to reach supermajority agreement on the diagnosis for 99% of cases after the consensus conference calls suggests that uveitis experts are capable of agreement after discussion of the case. In addition, it implies that validated and widely used classification criteria for the uveitides should result in case series, cohorts, and clinical trials from different centers that should be more comparable, because they will contain a more homogeneous group of patients. It was the experience of the individuals involved in the consensus conference calls that during the calls a sense of the essential features for making the diagnosis and excluding the diagnosis began to emerge.

Although the approach was rigorous, and in general the number of cases of each disease reasonably large, there are caveats to the data. For some diseases, the number of cases was not in the 200 to 250 range. The requirement for supermajority agreement for a case to be included in the final database was fundamentally conservative and may exclude cases considered by many investigators to have the disease. However, this requirement is necessary for classification criteria (as opposed to diagnostic criteria) because the goal is high specificity and a homogeneous group of patients generally accepted to have the disease.⁴ In addition, this requirement does not influence the κs derived from the independent online voting—only the results about inclusion in the final database. Group dynamics are susceptible to “dominant personality” effects, and there can be a falsely elevated level of agreement if the group is not representative of the population at large. These effects were addressed by using nominal group techniques, which minimize “dominant personality” effects, and by the international composition of the groups with efforts to make them geographically and school of thought dispersed.²⁴ Moreover, these potential effects would affect the agreement from the consensus conference calls but not the independent online voting being evaluated by the κs reported herein.

The accepted “final” dataset will be used in machine learning approaches to arrive at a parsimonious set of criteria for each disease that minimizes misclassification. The dataset will be split into 2 groups, a learning set and a test set, to develop and test the diagnostic algorithms developed, respectively. Critically, the developed criteria should optimize classification within uveitic class. A variety of statistical techniques may be used, including logistic regression and classification and regression trees.^28,29

In conclusion, our results suggest that currently, in the absence of validated and widely used classification criteria, the agreement among uveitis experts on diagnosis is moderate with disease-specific variation. The results of the consensus conference calls, after which 1% of cases were permanently tabled, suggest that uveitis experts can agree on the diagnosis after discussion and suggest that validated and widely used classification criteria may improve this situation.