The Lessons of QUANTEC: Recommendations for reporting and gathering data on dose-volume dependencies of treatment outcome

Abstract

The 16 clinical articles in this issue review the dose volume dependence of toxicities of external beam radiotherapy. They are limited by the difficulty of synthesizing results from different publications. The major problems stem from incomplete reporting of results and use of incompatible or ambiguous endpoints. Here we specify these problems, give recommendations to authors, editors, and reviewers on standards of reporting, and, provide methods of defining endpoints suitable for the dose-volume analysis of toxicity. Adopting these recommendations will facilitate meta-analysis and increase the utility of individual studies of the dependence of complications on dose distributions.

Requirements for future QUANTEC efforts

Severe normal tissue complications are, by good medical practice, relatively uncommon. Investigations of their causes will inevitably be plagued by the small numbers of events in individual series. It is therefore important to be able to combine complication data from different institutions and protocols. This can be achieved in two ways: by the direct combination of raw data (i.e. pooling of dose distribution and outcome information); or by combination of published results (literature-based meta-analysis). The initial efforts of QUANTEC have concentrated on the second approach (the data-pooling paper by Deasy et al. in this issue discusses the first). In the process of gathering and analyzing the published data for the papers in this issue, the limitations of existing journal articles as sources for meta-analysis have become apparent. They fall into two major categories: first, the low and uneven standards of reporting results; and second, the difficulty of defining common and meaningful clinical endpoints.

1) Reporting standards and statistical requirements for literature based meta-analysis

The Consolidated Standards of Reporting Trials (CONSORT) statement provides a model for good practice in reporting clinical results (http://www.consort-statement.org/). Guidelines based on these standards have been created for radiotherapy by Bentzen (1) however, as noted by Trotti and Bentzen, the CONSORT guidelines are lacking in specifics when dealing with the scope and severity of the adverse effects of radiotherapy (2).

Typically, papers on the dose-volume dependence of complications are not written to maximize their utility for either clinical application or subsequent meta-analysis. The remedies for these defects are often simple. Common failures and specific remedies are given below. Where possible, examples of these failures (sometimes from our own publications) are cited in the heading, and citations to solutions in the following text. To clarify the examples, citations are given to the relevant locations, figures and tables where appropriate.

1.A Lack of basic statistical data on incidence of toxicity (see reference (3): Methods and Materials – Criteria for Radiation Myelopathy)

Wherever relevant, both the number of subjects and the number of events should be reported (see reference (4): figure 1, and reference (5): page 735). Alternatively, when an estimate of incidence is given, the standard error should also be supplied. For late complications, when the Kaplan Meyer method is used, the follow up time at which the estimate is made should be given. If a minimum follow up time is used, this cutoff should be specified.

1.B Lack of numerical labeling of response histograms (see reference (6): figure 9)

Where responses are shown as functions of a dosimetric variable, using quartile or similar plots, the numerical range and median of the variable should be supplied for each group (see reference (7): table 4, and ref (8): table 4).

1.C Complication rates as functions of differences of absolute variables (see reference (9): figure 2)

Where a scientific point may require a response rate as a function of the difference between two variables (e.g. the dose difference between the ipsi- and contra-lateral ears), the corresponding response rate as a function of an absolute variable should also be supplied (e.g. the absolute dose to the ipsi-lateral ear).

1.D Lack of model parameter estimates and their standard errors (see reference (10): pages 698–609)

Where clinical and dose-volume variables of a predictive model are found to be significantly correlated with complications, best fit parameters and their confidence intervals should be given, so the response function can be reconstructed (see reference (11): table 1).

1.E Lack of complication rates associated with constraints (see reference (12): pages 689–690)

When DVH or other constraints are derived, the observed complication rate for treatments above and below the constraint should be stated (see reference (4): figure 1, and reference (13)). These are expected to depend on the degree to which the constraints are violated or respected respectively, and to vary for constraints at different dose levels.

1.F Lack of goodness of fit

While models may be fit, and result in statistically significant correlation with outcome, their quality as descriptions of the data is rarely reported. Goodness of fit statistics should be stated (such as the chi-squared test on binned complication data (see reference (14): pages 887–888, figure 6, and table 4)), since they indicate when models provide adequate fits to data or when alternative models should be considered. Various tests of goodness of fit have been proposed for Cox Proportional Hazards models (15–18).

1.G Lack of ROC curves or other discrimination statistics

It is useful to know if a model efficiently discriminates between responders and non-responders. Plots of predicted versus observed incidence of toxicity are helpful for graphical testing of model performance. If the model is used as a binary classifier the area under the ROC curve should be calculated to summarize the model’s ability to discriminate responders from non-responders throughout the range of cut-points (see reference (19): figure 5b); the false and true positive ratios display the costs and benefits of imposing clinical constraints. Spearman’s rank correlation coefficient (see reference (see ref 6: table 6): table 6), and Kendall’s tau (see reference (14): page 888 and figure 7) may also be used to assess the strength of correlation and the ability of a model to segregate responders from non-responders.

1.H Dose-volume histograms including only partial volumes (see reference (20): page 693, section on conformal plan evaluation)

Segmentation guidelines vary. Some publications relied on planning systems that did not allow overlapping structures, or did not include parts of a structure outside the dose computation grid. Preferably, data used for NTCP analyses should include the full organ volume (see reference (19): page 107, paragraph 2). When this is not possible, either a standard method of normalization, or absolute volumes should be used (see reference (21): page 63 Methods and Materials paragraph 2, and table 1). A clear statement of the definition of the organ volume should be given, paying close attention to such issues as length of linear structures included (e.g. rectum), inclusion or otherwise of the lumen for tubular or cavitary structures (e.g. rectum or bladder), inclusion or otherwise of any overlap with PTV, CTV or GTV.

1.I Complication rates as functions of novel variables (see reference (22): figure 3)

Fragmentation of the literature results when response rates are presented only as a function of a novel variable (e.g. dose-surface histograms, or measures of dose to the circumference of tubular structures). This can be mitigated if the corresponding response rate function for a standard variable is also supplied (see reference (23): table 2, and figures 3 and 4) or if the relationship between the variables is well understood (see reference (24): figure 4).

Without the basic reporting of statistics and treatment variables recommended in items 1.A–.E, a study cannot be included in a meta-analysis; items 1.F and 1.G allow its quality to be assessed, and help decide if it should be included; items 1.H and 1.I help insure that studies are compatible.

Other good research practices (not rising to the level of requirements) include: tests of models reported by others to be statistically significant; tables or graphs of the correlations between dosimetric variables -this can make it obvious whether a dataset really has the ability to discriminate between, say V20 and mean dose as predictors of radiation pneumonitis (see reference (25): figures 5 and 6); the robustness of a fit to statistical variation of the patients included can be assessed using cross validation and bootstraps techniques (see reference (19): page 109 paragraph 1, reference (26): page 16 section 2.2.4. and figure 6, and reference (27): figure 5 and page 987 section on statistical analysis), choice of the most robust fit helps to avoid over-fitting.

Recommendations for referees and editors

The responsibility for accurate and sufficient reporting lies primarily with authors; however reviewers and editors of submitted papers should consider these criteria when judging the suitability of a paper for publication. The requirement of brevity is in conflict with that of complete reporting, and a priority of journals is for concise papers establishing a single scientific point; nevertheless, authors, editors, and reviewers need to keep in mind the specific priorities of this branch of clinical science. As reviewers and editors, we will ask for revisions to articles that do not follow the recommendations in items 1.A.–.I above, since the clinical utility of papers complying with these requests would be greatly increased.

The requirement for comprehensive reporting

The simple expedients given above will allow a reader to critically appraise the validity of a modeling study, and facilitate the statistical synthesis of data from multiple published studies. However, there is a need they do not address: the need for data to be analyzed and presented comprehensively. Individual studies may give dose volume constraints, or perform model fits, but results from different studies cannot be combined unless the same dose volume constraints or model parameters are considered. This can be addressed only partly by diligently exploring the models used by previous authors, as the fundamental difficulty lies in the large number of dose volume variables to be tested. To address this problem through the literature requires adoption and creation of tools for comprehensive reporting such as Dose-Volume and EUD Atlases of Complication Incidence (DV-ACI or EUD-ACI) (4, 13). We also recommend that journals allow reporting comprehensive supplementary material in electronic form. Alternatively, repositories of validated dose distribution, patient characteristics, and outcome data may address this matter (see Deasy et al. in this issue). If such a data base were created, tools such as the EUD-ACI would still be useful for the comprehensive display of the data.

In any kind of retrospective pooling of data from multiple studies, recently discovered clinical or treatment related risk factors may not be available in older data sets. Such factors can be strong confounders of dose-volume response relationships, and pooling the outcome of studies without adjusting for them may not be justifiable. The development of side effects after cancer therapy is inherently multi-factorial and much of the variability in response is currently unexplained by dosimetric factors alone. Another obstacle, described below, is that many studies use different definitions of endpoints.

2) Endpoint definitions

Many endpoints are used to classify normal tissue injury, including patient symptoms (e.g. shortness of breath), formal clinical/functional assessments (e.g. quality of life tools, exercise testing), laboratory tests (e.g. pulmonary function tests, blood counts), and imaging (e.g. CT density). The majority of studies use symptoms to assess toxicity.

Authors should make every effort to analyze standard endpoints. However, the commonly-used clinical grading systems may not be ideal for providing quantitative data for the QUANTEC projects for several reasons.

2.A Most grading systems separate acute and late complication at one time. In specific cases, the timing of this division may not be appropriate. The classic example that does not fit easily into the early-late scheme is radiation pneumonitis which may occur up to ~6 months post-treatment. Unless this is recognized prospectively, pneumonitis cases occurring after the acute cutoff time need to be retrospectively re-assigned by careful examination of individual cases. The use of actuarial statistical methods may be required (28).

2.B Toxicities are grouped into grades, typically: Grade zero: no change; Grade 1: changes of no clinical significance; Grade 2 changes requiring outpatient treatment; Grade 3: changes requiring hospitalization; Grade 4: life threatening changes; Grade 5: lethal changes. Toxicity scales vary in the way these definitions are applied (e.g., prescription of steroids for radiation pneumonitis is considered grade 2 by SWOG, but grade 3 by RTOG). Their clinical application is often subjective, and related to the physician’s clinical practice and perception of the severity of the event. For example, it is often not clear if and when patients should be started on medications [grade 2] and/or admitted to the hospital [grade 3]. The use of a standard toxicity reporting system is needed. For each of the organ specific papers, the authors make recommendations regarding the scoring and grading of toxicity.

2.C Various symptoms referable to a single organ are often grouped together. Thus, dose-volume responses based on clinical grade alone are super-positions of responses from different complications, and in consequence, broader and shallower than that of any one constituent. For example, the response of the bladder may reflect a global organ effect (e.g. reduced capacity and resultant urinary frequency) or a focal effect (e.g. bleeding due to local ulceration). The dose/volume relationships for these two types of responses are likely to be different. Nevertheless, these symptoms are grouped together in most scoring systems used to assess bladder injury. Particular signs and symptoms may not indicate a specific pathogenesis without further work-up.

Similarly, there may be circumstances where different symptoms originate from damage to different portions of an organ, and the superposition of responses seen in any individual clinical series will depend on the probability of irradiation of the different regions. For example, Heemsbergen (29) and Peeters (30) examined rectal complications of prostate radiotherapy using patient reported symptoms. They identified several distinct endpoints in patients treated on a prostate dose escalation trial, and analyzed their dose-volume and regional dependence separately. Bleeding and soiling were associated with irradiation of different regions of the rectal wall. Thus, analysis of portmanteaux endpoints, such as ≥ RTOG/EORTC grade 2 rectal complications, will produce response functions that are unpredictable super-positions of the responses for bleeding, incontinence, increased stool frequency, and increased mucous discharge. Where possible, it is preferred to report the data in a more granular fashion (i.e. with the different endpoints reported separately), as well as in the aggregate form.

2.D Symptoms are not always easily attributable to specific organs. For example, shortness of breath can result from injury to the lung, heart and trachea, as well as anemia due to bone marrow effects. Thus, a modest degree of lung injury, along with anemia, can result in a relatively high degree of shortness of breath. Should that be scored as a low grade lung and bone marrow effect, or a high grade lung effect? Further, the presence and grade of toxicity may be related to concurrent clinical factors unrelated to treatment (e.g. pre-existing anemia or pre-RT lung disease may make patients more prone to developing dyspnea). Studies considering such endpoints should at least acknowledge this issue, and, where possible, report the information concerning the confounding factors (e.g. considering and reporting anemia concurrent with dyspnea). Since toxicity may occur, at least in part, due to pre-RT conditions, scoring systems that acknowledge the contribution to such baseline factors may also be useful in dose/volume/outcome studies.

2.E The number of different endpoints for a single complication is large. Aside from the differences in the grades of complication analyzed, published results use different grading schema, and it is often not possible to combine these cleanly.

In contrast with the statistical issues, those related to endpoint definition are challenging and complex.

The requirement that endpoints be objective and quantitative naturally leads to those based on functional imaging and testing (see Jeraj et al. in this issue). Nevertheless, for such tests to be useful, they must be related to clinically relevant outcomes.

It is important to separate symptoms into coherent disease entities when attempting to associate functional imaging studies with clinical outcome. Failure to eliminate complications arising from confounding disease entities will obscure the correlations between measures of functional change and their clinical consequences. A proper understanding of the causes and anatomical origins of distinct clinical complications is also important in this regard.

As an example, pre- and post- treatment studies of saliva collected from head and neck patients have had sporadic success in establishing that this endpoint is related to xerostomia. Studies vary in the way this saliva is collected: stimulated vs un-stimulated flow, saliva collected from the parotids alone (31, 32) vs whole mouth (33, 34). They also vary in the degree to which the whole mouth is involved in the treatment. Xerostomia is not totally determined by the parotid gland function; other glands in the oral cavity (particularly the sub-mandibular and sub-lingual glands) produce watery or sticky saliva (35). Therefore, the degree to which parotid dose or parotid function correlates with the incidence of xerostomia may depend on the dose to the rest of the oral cavity (36–38). A number of pre- and post- treatment imaging studies of the function of the parotid and other glands of the oral cavity are currently underway and may resolve this issue.

We therefore recommend that grading schemes based on symptoms of coherent clinical syndromes (rather than organ specific collections of disparate symptoms) be utilized for dose-volume toxicity studies. Where necessary, these may need to be developed, with the aid of our biological and anatomical understanding of the causes and usual timing of the onset of these complications. Standard coalescing scoring systems (e.g. RTOG) may provide a useful way to summarize the data, however, the underlying discrete symptom-specific information should be retained. This may facilitate studies to determine the pathophysiological cause of complications.

The SOMA-LENT system (39–43) was devised to specifically address four different ways to quantify toxicity; based on Subjective findings (e.g. symptoms of shortness of breath), Objective findings (e.g. changes in respiratory rate), Management interventions (e.g. the institution of steroids or oxygen), and Analytic data (e.g. pulmonary function tests or blood gas results). Findings in each of these four domains are independently quantified. These granular data, in each of the four domains, can be combined, in either a pre-determined or situation-specific manner, to generate a global toxicity score. Thus the SOMA-LENT system has the flexibility to be widely applicable in a wide variety of clinical scenarios. As demonstrated in the NKI group’s studies of rectal complications, patient reported symptom checklists may be helpful (see also Brunner et al. in this issue). In this regard, the NCI is developing patient reported versions of the CTCAE symptom items (https://wiki.nci.nih.gov/display/CTMS/Task+2+-+Develop+PRO-CTCAE+items). Currently, some 27 items are listed at the website (e.g. erectile dysfunction, dyspnea). In addition, the NIH is developing PROMIS, a databank of validated patient reported instruments (http://www.nihpromis.org/default.aspx). In some cases, determining the appropriate endpoints to analyze for separate dose-volume outcome correlates is a key part of the research project itself.

To address the complex problems outlined here, there is a need for a peer-reviewed central repository of dose-volume constraint standards. This might include atlases, contouring standards, endpoint definitions, grading schemes, and toxicity data/rates for a variety of common situations. Expert working groups would need to be formed and maintained with the charge of responsible for overseeing the repository and keeping it up-to-date.. These needs will require specific infrastructure solutions, such as the data banks outlined in Deasy et al. in this issue.

Advanced imaging and other quantitative metrics afford a unique opportunity to systematically relate dose volume variables with outcome. Carefully defined objective endpoints are required for the success of such projects. Studies of this kind, together with the recent explosion in radiotherapy planning and delivery tools, leading to more diversity in dose distributions, may provide an exciting time when the complex responses of normal tissues to radiation can be better understood.