Qualitative Assessment and Reporting Quality of Intracranial Vessel Wall MR Imaging Studies: A Systematic Review

Abstract

BACKGROUND:

Over the last quarter-century, the number of publications using vessel wall MR imaging has increased. Although many narrative reviews offer insight into technique and diagnostic applications, a systematic review of publication trends and reporting quality has not been conducted to identify unmet needs and future directions.

PURPOSE:

We aimed to identify which intracranial vasculopathies need more data and to highlight areas of strengths and weaknesses in reporting.

DATA SOURCES:

PubMed, EMBASE, and MEDLINE databases were searched up to September 2018 in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

DATA ANALYSIS:

Two independent reviewers screened and extracted data from 128 articles. The Strengthening the Reporting of Observational Studies in Epidemiology guidelines were used to assess the reporting quality of analytic observational studies.

DATA SYNTHESIS:

There has been an exponentially increasing trend in the number of vessel wall MR imaging publications during the past 24 years (P < .0001). Intracranial atherosclerosis is the most commonly studied intracranial vasculopathy (49%), followed by dissections (13%), aneurysms (8%), and vasculitis (5%). Analytic observational study designs composed 48% of the studies. Transcontinental collaborations showed nonsignificantly higher reporting quality compared with work originating from single continents (P = .20).

LIMITATIONS:

A limitation is the heterogeneity in study designs.

CONCLUSIONS:

Investigations on the diagnostic utility of vessel wall MR imaging in less commonly studied intracranial vasculopathies such as dissections, aneurysms, and vasculitis are warranted. More consistent adherence to the Strengthening the Reporting of Observational Studies in Epidemiology guidelines should improve transparency and maximize effective synthesis for clinical translation. Diverse collaborative teams are encouraged to advance the understanding of intracranial vasculopathies using vessel wall MR imaging.

Vessel wall MR imaging (VWI) is being increasingly used worldwide to evaluate intracranial vasculopathies.¹ This increased utilization has paralleled a rise in the number of publications using VWI. Many narrative reviews report the application and utility of VWI for different types of vasculopathies.^2,3 However, no study has systematically assessed the frequency or trends of VWI publications for all intracranial vasculopathies or the reporting quality.

Interpretation of data and secondary analyses from observational studies is often limited by the methodology and completeness of reporting. Reporting quality is important to critically assess the strengths of studies, weaknesses, and generalizability as well as for investigators who want to assess the reproducibility of a study. The clinical and scientific utility of research data may be lost in poorly reported studies. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines were developed to improve the quality of reporting of observational studies in medical research.^4,5 We set out to systematically survey VWI publications by examining the types of vasculopathies studied as well as assessing the quality of reporting of analytic observational studies using the STROBE checklist. The aims of this study were to identify which intracranial vasculopathies may need more data and to highlight areas of reporting that could be improved.

MATERIALS AND METHODS

Search Strategy

The systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. PubMed, EMBASE, and MEDLINE were searched on September 12, 2018. To identify eligible studies, we searched keywords using the Boolean operators “OR” and “AND.” Keywords covered imaging, vessel wall imaging, intracranial circulation, vasculopathy, and vascular disease terms (On-line Table 1). A manual review of the citations of each included article was also performed. All foreign language articles were translated.

Study Selection

Two researchers independently reviewed all publications for inclusion. Inclusion criteria were the following: 1) case series or observational studies, 2) imaging humans, 3) intracranial arteries, 4) intracranial vasculopathies, 5) imaging of the arterial wall, and 6) MR imaging. Single case reports, conference abstracts, animal studies, and studies of the pediatric population were excluded.

Data Extraction

Two reviewers independently screened and extracted data from each study that fulfilled the inclusion and exclusion criteria. Disagreements were resolved by consensus. We collected the following data: publication characteristics (year of publication, countries of publication, funding sources), type of vasculopathy studied, study design (case series or analytic observational study), and subject enrollment design (prospective or retrospective). Analytic observational studies were identified using a published study design classification algorithm⁶ and included studies with comparator groups or that were designed as prevalence/cross-sectional or diagnostic accuracy studies.^7–9 Studies that reported obtaining informed consent or explicitly reported prospective enrollment by the authors were categorized as prospective. Transcontinental collaborative publications were identified by author affiliations. Reporting quality was assessed using the STROBE checklist.⁴ Each STROBE item was assessed as 1 = reported, 0.5 = partly reported, or 0 = not reported. A complete reporting score (CRS) per publication was calculated by summing the total number of items divided by 22 (the total number of STROBE criteria). Scores by article section (introduction, methods, results, and conclusions) were also calculated.

Statistical and Sensitivity Analysis

Categorical variables are expressed in counts and percentages. Distributions of continuous variables are summarized with means and SDs or medians and interquartile ranges. Agreement was calculated with an unweighted Cohen κ. Based on the goodness of fit, exponential and linear regression analyses were fit to test trends across time in the total number of VWI studies and analytic observational studies by year, respectively. The Shapiro-Wilks test was used to test normal distributions of CRS measures. CRS measures were calculated by a summary statistic (mean or median) based on the test for a normal distribution. The Kruskal-Wallis test assessed differences among CRS scores by continent. Two sensitivity analyses assessed the robustness of the results. The first sensitivity analysis considered fulfillment of STROBE items reported in footnotes, bylines, and different sections of the article. A second analysis was conducted by considering fulfillment of STROBE items that were “partly reported” in any part of the article, including footnotes and bylines. Publication bias was assessed using the likelihood ratio χ² test to compare the distributions of the intracranial vasculopathies of the included studies versus the excluded conference abstracts. SPSS Version 19 (IBM, Armonk, New York) was used for statistical analysis.

RESULTS

Search

The search strategy identified 2431 publications, among which 1635 were screened by title/abstract (κ = 0.77; 95% CI, 0.71–0.83; P < .01). Of those, 234 articles were selected for full-text review (κ = 0.89; 95% CI 0.82–0.96; P < .01). Manual review of the citations of the included articles yielded 807 citations, which were further screened by title/abstract (κ = 0.73; 95% CI, 0.52–0.93; P < .01). Full data extraction for qualitative synthesis was performed on 128 articles identified from the initial and manual citation review (Fig 1; references in the On-line Appendix).

Fig 1.

Systematic review of the data bases. From the initial data base search, 165 articles were identified for full-text review. Manual review of the citations of those 165 articles identified 807 citations that were further screened by title and abstract. This resulted in 69 articles for full-text review from the manual citation search. A total of 234 articles underwent full-text review, from which 128 articles met the predetermined inclusion and exclusion criteria.

Publication Trends

The first article evaluating intracranial vessel wall characteristics from this systematic review was reported in 1994 evaluating intramural hematomas in dissections.¹⁰ The second publication was in 1995 and evaluated vessel walls for cerebral atherosclerosis.¹¹ An exponentially increasing trend in the number of VWI publications during the past 24 years was seen (Fig 2) (β = 0.14; 95% CI, 0.11–0.17; P < .0001). Asia published the highest number of publications (61%) during the 24 years, followed by North America (n = 19) and Europe (n = 18). In 2014, transcontinental collaborative publications emerged, composing 10% of the included publications. Asia was part of 92% of the transcontinental collaborations (Fig 3A), and 46% of the transcontinental collaborations were between Asia and North America.

Fig 2.

Annual number of publications using vessel wall MR imaging.

Fig 3.

Distribution of publications by funding source, continent and intracranial vasculopathy. A, Distribution of publications by continent is shown. Distribution of transcontinental collaborations is further elaborated to show 4 different groups of collaborating partners. B, Distribution of different funding sources and the combination of funding sources are shown. C, Distribution of studied intracranial vasculopathies are shown. RCVS indicates reversible cerebral vasoconstriction syndrome.

Most investigations were solely federally funded (39%). Publications with mixed types of funding sources accounted for 21%, with the most common combination being federal and medical society sources (54%). No funding source was reported for 30% of all publications (Fig 3B).

Survey of studies focused on 1 vasculopathy revealed intracranial atherosclerotic disease (ICAD) to be the most commonly studied vasculopathy (49%, n = 62, references in the On-line Appendix), followed by arterial dissection (13%, n = 16),^10,12–26 aneurysm (8%, n = 10),^27–36 vasculitis (5%, n = 6),^37–42 Moyamoya disease (3%, n = 4),^43–46 postendovascular changes (2%, n = 3),^47–49 and reversible cerebral vasoconstriction syndrome (1%, n = 1)⁵⁰ (Fig 3C). Among the 16 publications investigating arterial dissections, 11 used VWI^{12,14–18,22–26} and the other 5 publications^{10,13,19–21} assessed intramural hematoma signal characteristics on conventional MR imaging. Note that 21% of the studies examined ≥2 types of vasculopathies; a breakdown of the types of vasculopathies studied among these publications is further illustrated in the pie-in-pie chart.

Publication bias was assessed by comparing the number of intracranial vasculopathy types that were included in this study with excluded conference abstracts; the comparison showed no significant difference (On-line Table 2, P = .95).

Study Designs

Case series composed 52% of the publications, and 48% were analytic observational study designs. There was a significant yearly increase in analytic observational studies since 2000 (β = 0.39; 95% CI, 0.26–0.51; P < .0001). Most studies were conducted with prospective subject enrollment (50%) compared with retrospective subject identification (44%). Examples of prospective case series include studies that obtained written informed consent to study circle of Willis cadaveric specimens and characterize intracranial atherosclerotic plaque components,⁵¹ recruited 3 subjects to describe atherosclerosis enhancement characteristics by VWI,⁵² and methodologic articles reporting interrater/intrarater reliability⁵³ and scan-rescan reproducibility.⁵⁴

STROBE Reporting Assessment

The 62 analytic observational studies were evaluated for reporting quality using the STROBE checklist (Table) (κ = 0.76; 95% CI, 0.72–0.79; P < .001). The mean CRS for all studies was 0.64 ± 0.10. The introduction section had the highest and the methods had the lowest scores. Transcontinental collaborative publications showed higher scores (CRS_all = 0.67 ± 0.05) compared with single-continent studies (Table) and multisite/single-continent collaborations (CRS_all = 0.63 ± 0.12), though these results did not reach statistical significance (P = .20).

STROBE complete reporting scores^a

	CRSAll	CRSIntroduction	CRSMethods	CRSResults	CRSDiscussion
All (n = 62)	0.64 (0.10)	1.00 (0)	0.56 (0.22)	0.60 (0.20)	0.63 (0.25)
North America (n = 11)	0.62 (0.10)	1.00 (0)	0.52 (0.11)	0.56 (0.22)	0.69 (0.28)
Asia (n = 40)	0.65 (0.11)	1.00 (0)	0.61 (0.17)	0.65 (0.20)	0.63 (0.25)
Europe (n = 6)	0.58 (0.14)	1.00 (0)	0.42 (0.15)	0.60 (0.15)	0.63 (0.19)
Trans-continental collaborations (n = 5)	0.67 (0.05)	1.00 (0)	0.60 (0.07)	0.64 (0.11)	0.73 (0.10)

^aCRS measures reported in means (SDs) or medians (interquartile range). Summary statistics were chosen on the basis of the test for normality.

Two sensitivity analyses were performed that showed the same direction of the results but notably did not reach statistical significance. First, a sensitivity analysis evaluating each study for items reported in any part of the article, including footnotes and bylines, showed higher scores from transcontinental collaborations compared with single-continent studies (P = .30). A second sensitivity analysis considered all partially reported criteria as fulfilling reporting and also showed higher CRS measures by transcontinental collaborations than North American and European studies. The studies from Asia had only marginally higher CRS measures (P = .27) (On-line Table 3).

On-line Table 4 reports the mean score per STROBE item for the 62 articles and checklist descriptions. Items 1 and 22 were scored separately because they reflect title/abstract and funding reporting, respectively. Most studies provided an informative abstract but did not indicate the study design in the title (item 1) to fully meet the STROBE criterion, resulting in a score of 0.52 ± 0.13. Also, 70% of the studies did not disclose whether there was a funding source (item 22), resulting in a score of 0.71 ± 0.46. The 2 introduction section criteria, evaluating the reporting of the scientific background and rationale (item 2) and specific objectives or hypothesis (item 3), scored the highest among all sections.

The methods section included 9 criteria. No publication reported a sample size determination (item 10). Explicitly presenting key elements of the study (item 4) also scored low (0.08 ± 0.24). Although 73% of the studies reported a prospective or retrospective subject enrollment method, few studies named the study design type.⁵ Clearly defining outcomes, exposures, predictors, confounders, and diagnostic criteria (item 7) also had a low score (0.34 ± 0.32); for example, only 14% of the 43 publications studying ICAD reported diagnostic criteria for cardiovascular risk factors such as hypertension. The importance of reporting diagnostic criteria for hypertension is emphasized in light of the new 2017 American Heart Association classifications of hypertension and evolving definitions.⁵⁵

Five criteria were included in the results section. Reporting of the study participants and information on exposures and potential confounders (item 14) was suboptimal (0.45 + 0.19). Publications lacking information on confounders such as race/ethnicity were scored as partial reporting. Only 8 studies^18,56–62 reported race/ethnicity, among which 5 studies^18,56–59 reported the information in the discussion section as a limitation of generalizability. Twenty-four publications originated from Asia, were single-center studies, and did not report a description of ethnicity; in these studies, one could assume that all enrolled subjects were Asian but clarity in reporting could be strengthened.

In the discussion section, generalizability (item 21) scored the lowest (0.26 ± 0.44). Most studies were single-center studies (74.2%), but this was not commonly addressed as a limitation for external validity. A discussion on limitations and reporting direction/magnitude of potential bias (item 19) also scored poorly (0.55 ± 0.24), with most studies partially fulfilling this item due to an absence of a discussion on the direction and magnitude of the bias.

DISCUSSION

The increasing trend in the number of VWI publications and more transcontinental collaborations during the past 24 years suggest the widespread interest in the diagnostic utility of VWI. Evaluation of the reporting quality of analytic observational studies using the STROBE checklist highlighted strengths and weaknesses in the current literature. Subgroup analyses showed that transcontinental collaborations yielded higher CRS measures compared with single-center/single-continent and multicenter/single-continent publications, suggesting an advantage for multicultural, diverse collaborative teams. Notably, the results did not reach significance, potentially due to the relatively small number of transcontinental publications.

A survey of the literature to date shows Asia as the leading contributor of VWI publications and ICAD to be the most commonly studied vasculopathy. Fewer VWI studies were identified evaluating arterial dissections, aneurysms, vasculitis, and Moyamoya disease. More effort is warranted to further understand the diagnostic utility of VWI in these less commonly studied intracranial vasculopathies.

Given that stroke is one of the leading causes of morbidity and mortality worldwide and a widespread public health problem, it is not surprising to see federal funding supporting many of the VWI investigations across all continents. Medical societies such as the Dutch Heart Foundation, American Society of Neuroradiology, Radiological Society of North America, and American Heart Association also emerged as common funding sources, showing societal missions to improve stroke outcomes. Nearly one-third of the studies did not disclose a funding source, reflecting either no funding source or funding that was not disclosed.

The availability of the STROBE guidelines has encouraged many research domains and medical specialties to improve reporting quality^63,64 and ultimately improve research reproducibility. Assessment of reporting quality of VWI publications has not yet been evaluated. Our results show a CRS of 0.64 for the 62 analytic observational VWI studies. The results of this study highlight areas where reporting of analytic observational studies are good, along with other areas where improvements are needed. In particular, there is a need for studies to improve the clear reporting of definitions of exposures, predictors, and potential confounders (item 7). The lack of clearly defined diagnostic criteria becomes important when, for example, guidelines are updated; in 2017, the American Heart Association changed the blood pressure guidelines, with some secondary analyses suggesting that the new classification correlates with different prevalence rates and outcomes.^65,66

Study size calculations (item 10) were not reported in any study. Sample size estimates are informative because they indicate the magnitude of the aimed effect and address whether there were challenges with recruitment due to drop-out or attrition bias.⁵

Additional areas of weakness in reporting included the description of study design (item 4) and settings and locations (item 5). Reporting a prospective or retrospective enrollment design only is insufficient. For example, a cross sectional study could assess imaging findings on a subject at 1 time point using a prospective enrollment design or the subject could be identified retrospectively from a registry.

In the discussion, reporting the generalizability or external validity (item 21) of the results was also noted to be a weakness. Generalizability is evaluating the extent to which the results of the publication can be applied to other settings and populations and is important for the reader to see whether the results are applicable to his or her own practice setting.

Analysis of reporting quality by continent showed that transcontinental collaborations had higher CRS measures (Table). Collaborations are encouraged for the cross-pollination of ideas, to increase recruitment, or to enhance generalizability by coordinating a multisite study, among other advantages.⁶⁷ Our findings suggest an additional advantage of higher reporting quality. Multilingual authors and increased diversity within teams from different continents may be possible explanations for this finding.

This systematic review has some limitations. First, neuroradiology research often reports innovative techniques and drives cutting-edge methods that are not tested as part of larger epidemiologic studies. These technologic-advancement publications are typically of smaller scale and are “proof of concept” studies that are designed differently but may have larger impact. The STROBE guidelines do not always fit this radiology research framework. However, as an assessment for completeness of reporting with consistency in evaluation by 2 independent raters, the results provide at least a basis of reporting quality among VWI publications. Second, classifying studies as case series or analytic observational studies was often challenging due to the heterogeneity in study designs. Many of the included studies did not fit traditional epidemiologic study designs. This challenge has been addressed in other systematic review methodologies, and an algorithm has been tested to appropriately classify study designs for systematic reviews. We based our classification as descriptive (case series) or analytic observational studies following a simplified version of this algorithm.⁶ Third, methodologic quality was not assessed because the aim was to identify trends in VWI publications rather than conduct a quantitative meta-analysis. Instead, an assessment of reporting quality was conducted to highlight areas for improvement.

Fourth, use of the STROBE checklist is inconsistently recommended in author guidelines among journals, which could be a confounder in our analysis. Notably, some journals have their own checklists requiring a statement on data-sharing and open-source availability, which also improve transparency of the conducted research. As a future direction, an assessment of the journals can be conducted to assess CRS measures by journal Impact Factor as well as explicit author instructions to follow the STROBE checklist. Finally, the CRS was calculated using the STROBE guidelines, and items were scored by section as determined by the STROBE checklist. Studies that did not report items in the correct section per the STROBE guidelines were not considered as fulfilling the criteria. This method may underestimate CRS measures. To account for this issue, we performed a sensitivity analysis with items scored as reported if present in any part of the article, including footnotes and author bylines. This sensitivity analysis yielded similar conclusions. A second sensitivity analysis considering partially reported items as completely reported showed a similar direction of the results.

CONCLUSIONS

We systematically assess VWI publications to identify trends and assess reporting quality. Our results show that ICAD is the most common intracranial vasculopathy studied. The utility of VWI for ICAD, and for other types of intracranial vasculopathies will likely benefit from additional rigorous studies. We also highlight deficiencies in the reporting of analytic observational VWI studies. Transcontinental collaborative effort yielded a higher reporting quality, though this result did not reach statistical significance. Nonetheless, there may be advantages to diverse and multilingual cross-cultural teams. More consistent adherence to STROBE guidelines should improve transparency and maximize effective synthesis and clinical translation of findings for future studies.

ABBREVIATIONS:

CRS

complete reporting score

ICAD

intracranial atherosclerotic disease

STROBE

Strengthening the Reporting of Observational Studies in Epidemiology

VWI

vessel wall MR imaging