Reproducibility in the Assessment of the Components of a Clinical Complexity Index

INTRODUCTION

Clinical complexity (CC) represents one of the most relevant challenges of modern medicine, and its quantification is crucial both to stratify the clinical risk and for a fair hospital reimbursement policy.^1–3 CC is a multifaceted condition that encompasses biological (e.g., age, multimorbidity, frailty) and non-biological (e.g., socioeconomic, cultural, environmental, and behavioral) components.^{4, 5} On this basis, a vectorial model of CC has been set up in which each vector expresses the dynamic changes over time of each component.⁵ Although this model cannot but be an approximation of CC, it is able to take into account its determining factors all at once. To allow its use, each vector has been graded following a consensus meeting during which the five most representative variables of each CC domain were selected.⁶ However, a score made of a series of variables could be a source of variability. For this reason, we have performed an interobserver agreement study.

METHODS

In June–August 2018, three healthcare professionals conducted the present study: a senior physician (observer #1, gastroenterology consultant), a research nurse (observer #2), and a young physician (observer #3, internal medicine resident). The first two observers were preliminarily trained by testing the CC index with roughly 100 patients, whereas only its theoretical bases were taught to the third observer. The CC index (Fig. 1) was administered by each observer to 30 consecutive adult patients (mean age 68 years, range 31–88; 16 females) admitted to an academic internal medicine ward. The three observers followed a randomization list, so that each of them administered the CC index to the same patient for the first, second, and third times in three consecutive days. Patients with a poor prognosis (< 48 h) were excluded. Each observer timed her/himself with an electronic chronometer during the CC index administration.

Figure 1.

Clinical complexity index assessed in the present study. The clinical complexity index was built after a consensus meeting held in 2017, involving 25 panelists Figure 1 contains poor-quality text inside the artwork. Please do not re-use the file that we have rejected or attempt to increase its resolution and re-save. It is originally poor; therefore, increasing the resolution will not solve the quality problem. We suggest that you provide us the original format. We prefer replacement figures containing vector/editable objects rather than embedded images. Preferred file formats are eps, ai, tiff, and pdf.I apologize for the inconvenience. I have now uploaded the original pdf and pptx versions of the Figure, so you can use the best one.⁶. CIRS, Cumulative Illness Rating Scale.

We rescaled each variable (Fig. 1) to have a score of 0 if the answer was “no” and a score of 2 if the answer was “yes”; these were summed up within each domain (range 0–10) and over all domains (range 0–50). Agreement of the scores computed for each domain and overall between each of the three pairs of observers was assessed with the intraclass correlation coefficient (ICC) with 95% confidence intervals (CI). The strength of agreement was interpreted as follows: poor, < 0.50; moderate, 0.50–0.75; good, 0.75–0.90; excellent, > 0.90.

This study represents a sub-analysis of the San MAtteo Complexity study (NCT03439410) that was approved by the local Ethics Committee.

RESULTS

Table 1 reports the results of the interobserver agreement. Overall agreement was excellent (> 0.90) between observers #1 and #2, and good (≥ 0.80) between observers #1/#2 and #3. Domain agreement showed some weaknesses regarding the cultural domain (> 0.50, moderate agreement) between any two observers, and generally showed poorer results for all domains when comparing observers #1/#2 vs #3. The agreement between observers #1 and #2 was good (≥ 0.80), particularly for the biological and socioeconomic domains.

Table 1.

Partial and Total Intraclass Correlation Coefficient Among the Three Observers for the Evaluation of an Index of Clinical Complexity. The Total Index Is Computed over 25 Items and Each Sub-domain over 5 Items

	Obs #1 vs Obs #2 ICC (95% CI)	Obs #1 vs Obs #3 ICC (95% CI)	Obs #2 vs Obs #3 ICC (95% CI)
Total index	0.92 (0.84–0.96)	0.82 (0.64–0.91)	0.80 (0.63–0.90)
Sub-domains
Biological	0.84 (0.71–0.92)	0.72 (0.50–0.86)	0.78 (0.59–0.89)
Socioeconomic	0.86 (0.72–0.93)	0.53 (0.07–0.78)	0.45 (0.0–0.73)
Behavioral	0.73 (0.51–0.86)	0.41 (0.08–0.66)	0.39 (0.06–0.65)
Environmental	0.73 (0.47–0.86)	0.39 (0.04–0.66)	0.34 (0.0–0.62)
Cultural	0.53 (0.23–0.75)	0.56 (0.22–0.77)	0.57 (0.25–0.77)

CI, confidence interval; ICC, intraclass correlation coefficient; Obs, observer

The mean times needed to complete the CC index were 21 ± 7 (observer #1), 24 ± 6 (observer #2), and 22 ± 8 min (observer #3).

DISCUSSION

The CC index studied has a high interobserver agreement among different healthcare professionals who were trained to administer this tool (observers #1 and #2). Observer #3 had a high agreement rate when considering the total CC index. However, the assessment of environmental and cultural domains showed lower agreement, possibly because some of their variables could be open to diverging interpretations. Hence, a training period consisting of becoming familiar with the meaning of the variables and practicing with this tool is warranted. A larger study involving more observers is needed to confirm our findings.

The completion of the CC index takes approximately 20 min, including the time needed to collect relevant patient medical history. We believe that this amount of time is acceptable, particularly in a research setting, and could be potentially reduced by computer-assisted data collection.

To conclude, the results of the present study support the feasibility of the use of the CC index in an internal medicine setting, and could provide background for its use in future studies. A simplification of the CC index, i.e., with fewer, but more specific, variables, will be considered at the end of its validation phase (NCT03439410).

Abbreviations

CC

Clinical complexity

CI

Confidence interval

ICC

Intraclass correlation coefficient

Author Contributions

All authors participated in the drafting of the manuscript or critical revision of the manuscript for important intellectual content, and provided approval of the final submitted version. Individual contributions are as follow: GRC, MVL designed the study, organized data collection, and drafted the manuscript; MVL, ASB, IB conducted the study and enrolled patients; MC, GB, CK contributed to data collection and analysis; GRC made the final critical revision for important intellectual content. All authors approved the final version of the paper.

Funding

This research is part of a project for the study of clinical complexity funded by San Matteo Hospital Foundation - Italian Ministry of Health (Progetto di Ricerca Corrente 2017 - PI Prof. Gino Roberto Corazza).

Compliance with Ethical Standards

This study represents a sub-analysis of the San MAtteo Complexity study (NCT03439410) that was approved by the local Ethics Committee.

Conflict of Interest

The authors declare that they do not have a conflict of interest.