Qualitative Assessment and Reporting Quality of Intracranial Vessel Wall MR Imaging Studies: A Systematic Review

Abstract

Background:

Over the last quarter century, the number of publications using vessel wall MR imaging (VWI) has increased. Though many narrative reviews offer insight into technique and diagnostic applications, a systematic review of publication trends and reporting quality has not been conducted to identify unmet needs and future directions.

Purpose:

We aimed to identify which intracranial vasculopathies need more data and highlight areas of strengths and weaknesses in reporting.

Data Sources:

PubMed, EMBASE and MEDLINE databases were searched up to September 2018 in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

Data Analysis:

Two independent reviewers screened and extracted data from 128 articles. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline was used to assess reporting quality of analytic observational studies.

Data Synthesis:

There is an exponentially increasing trend in the number of VWI publications over the last 24 years (p<0.0001). Intracranial atherosclerosis is the most commonly studied intracranial vasculopathy (49%), followed by dissections (13%), aneurysms (8%), and vasculitis (5%). Analytic observational study designs comprised 48% of the studies. Trans-continental collaborations showed non-significantly higher reporting quality compared to work originating from single continents (p=0.20).

Limitations:

Heterogeneity in study designs.

Conclusions:

More efforts are warranted to further understand the utility of VWI in less commonly studied intracranial vasculopathies such as dissections, aneurysms, and vasculitis. More consistent adherence to STROBE guidelines should improve transparency and maximize effective synthesis for clinical translation. Diverse, collaborative teams are encouraged to advance the understanding of intracranial vasculopathies using VWI.

Introduction

Vessel wall MR Imaging (VWI) is increasingly being used world-wide to evaluate intracranial vasculopathies.¹ This adoption has paralleled a rise in the number of publications using VWI. Many narrative reviews report on the applications and utility of VWI for different types of vasculopathies.^{2, 3} However, no study has systematically assessed the frequency or trends of VWI publications for all intracranial vasculopathies or the reporting quality.

Interpretation of data and secondary analyses from observational studies is often limited by the methodology and completeness of reporting. Reporting quality is important to critically assess a study’s strengths, weaknesses and generalizability as well as for investigators who desire to assess a study’s reproducibility. The clinical and scientific utility of research data may be lost in poorly reported studies. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines were developed to improve the quality of reporting of observational studies in medical research.^{4, 5} We set out to systematically survey VWI publications by examining the types of vasculopathies studied as well as assess the quality of reporting of analytic observational studies using the STROBE checklist. The aims of this study were to identify which intracranial vasculopathies may need more data as well as highlight areas of reporting that could be improved.

Materials and Methods

Search Strategy

The systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. PubMed, EMBASE, and Medline were searched on September 12, 2018. To identify eligible studies, keywords were searched using the Boolean operators “OR” and “AND.” Keywords covered imaging, vessel wall imaging, intracranial circulation, vasculopathy, and vascular disease terms (Supplemental Table 1). A manual review of the citations of each included article was also performed. All foreign language articles were translated.

Study selection

Two researchers independently reviewed all publications for inclusion. Inclusion criteria were: (1) case series or observational studies; (2) imaging humans; (3) intracranial arteries; (4) intracranial vasculopathies; (5) imaging arterial wall; and (6) magnetic resonance imaging. Single case reports, conference abstracts, animal studies and studies of the pediatric population were excluded.

Data extraction

Two reviewers independently screened and extracted data from each study that fulfilled the inclusion and exclusion criteria. Disagreements were resolved by consensus. The following data were collected: publication characteristics (year of publication, country(ies) of publication, funding sources), type of vasculopathy studied, study design (case series or analytic observational study), and subject enrollment design (prospective or retrospective). Analytic observational studies were identified using a published study design classification algorithm⁶ and included studies with comparator groups or designed as prevalence/cross-sectional or diagnostic accuracy studies.^7–9 Studies that reported obtaining informed consent or explicitly reported prospective enrollment by the authors were categorized as being prospective. Trans-continental collaborative publications were identified by author affiliations. Reporting quality was assessed using the STROBE checklist.⁴ Each STROBE item was assessed as 1=reported, 0.5=partly reported, or 0=not reported. A complete reporting score (CRS) per publication was calculated by summing the total number of items divided by 22 (the total number of STROBE criteria). Scores by manuscript section (introduction, methods, results, and conclusions) were also calculated.

Statistical and Sensitivity Analysis

Categorical variables are expressed in counts and percentages. Distributions of continuous variables are summarized with means and standard deviations or medians and inter-quartile ranges. Agreement was calculated with an unweighted Cohen’s kappa. Based on goodness of fit, exponential and linear regression analyses were fit to test trends over time in the total number of VWI studies and analytic observational studies by year, respectively. Shapiro-Wilks test was used to test normal distributions of CRS measures. CRS measures were calculated by a summary statistic (mean or median) based on the test for a normal distribution. Kruskal-Wallis test assessed differences among CRS scores by continent. Two sensitivity analyses assessed the robustness of the results. The first sensitivity analysis considered fulfilment of STROBE items reported in footnotes, bylines, and in different sections of the manuscript. A second analysis was conducted by considering fulfilment of STROBE items that were “partly reported” in any part of the manuscript, including footnotes and bylines. Publication bias was assessed using the likelihood ratio Chi-square test to compare the distributions of the intracranial vasculopathies of the included studies versus the excluded conference abstracts. SPSS v19 (Chicago, IL) was used for statistical analysis.

Results

Search

The search strategy identified 2,431 publications among which 1,635 were screened by title/abstract (κ=0.77, 95% CI 0.71–0.83, p<0.01). Of those, 234 articles were selected for full-text review (κ=0.89, 95% CI 0.82–0.96, p<0.01). Manual review of the citations of the included articles yielded 807 citations, which were further screened by title/abstract (κ=0.73, 95% CI 0.52–0.93, p<0.01). Full data extraction for qualitative synthesis was performed on 128 articles identified from the initial and manual citation review (Figure 1; Supplemental References).

Figure 1:

Systematic Review of the Databases

From the initial database search, 165 articles were identified for full-text review. Manual review of the citations of those 165 articles identified 807 citations, which were further screened by title and abstract. This resulted in 69 articles for full-text review from the manual citation search. A total of 234 articles underwent full-text review, from which 128 articles met the pre-determined inclusion and exclusion criteria.

Publication Trends

The first article evaluating intracranial vessel wall characteristics from this systematic review was reported in 1994 evaluating intramural hematomas in dissections.¹⁰ The second publication was in 1995 and evaluated vessel walls for cerebral atherosclerosis.¹¹ An exponentially increasing trend in the number of VWI publications over the last 24 years was seen (Figure 2) (β=0.14, 95% CI 0.11–0.17, p<0.0001). Asia published the highest number of publications (61%) over 24 years, followed by North America (n=19) and Europe (n=18). In 2014, trans-continental collaborative publications emerged, comprising 10% of the included publications. Asia was part of 92% of the trans-continental collaborations (Figure 3A) and 46% of the trans-continental collaborations were between Asia and North America.

Figure 2:

Annual Number of Publications using Vessel Wall Imaging

Figure 3:

Distribution of Publications by Continent, Funding source, and Intracranial vasculopathy

1. Distribution of publications by continent is shown. Distribution of trans-continental collaborations is further elaborated to show four different groups of collaborating partners.
2. Distribution of different funding sources as well as the combination of funding sources are shown.
3. Distribution of studied intracranial vasculopathies are shown.

Abbreviations: ICAD, intracranial atherosclerotic disease, RCVS, reversible cerebral vasoconstriction syndrome

Most investigations were solely federally funded (39%). Publications with mixed types of funding sources accounted for 21% with the most common combination being federal and medical society sources (54%). No funding source was reported for 30% of all publications (Figure 3B).

Survey of studies focused on 1 vasculopathy revealed intracranial atherosclerotic disease (ICAD) to be the most commonly studied vasculopathy (49%, n=62, Supplemental References), followed by arterial dissection (13%, n=16)^{10, 12–26}, aneurysm (8%, n= 10)^27–36, vasculitis (5%, n=6)^37–42, moyamoya disease (3%, n=4)^43–46, post-endovascular changes (2%, n=3)^47–49, and reversible cerebral vasoconstriction syndrome (1%, n=1)⁵⁰ (Figure 3C). Among the 16 publications investigating arterial dissections, 11 used VWI^{12, 14–18, 22–26} and the other 5 publications^{10, 13, 19–21} assessed intramural hematoma signal characteristics on conventional MRI. Note 21% of the studies examined 2 or more types of vasculopathies; a breakdown of the types of vasculopathies studied among these publications are further illustrated by the pie-chart inset.

Publication bias was assessed by comparing the number of intracranial vasculopathy types that were included in this study versus excluded conference abstracts, which showed no significant difference (Supplemental Table 2, p=0.95).

Study Designs

Case series comprised 52% of the publications and 48% were analytic observational study designs. A significant yearly increase in analytic observational studies emerged since the year 2000 (β= 0.39, 95% CI 0.26–0.51, p<0.0001). Most studies were conducted with prospective subject enrollment (50%) as compared to retrospective subject identification (44%). Examples of prospective case series include studies that obtained written informed consent to study circle of Willis cadaveric specimens and characterize intracranial atherosclerotic plaque components,⁵¹ recruited 3 subjects to describe atherosclerosis enhancement characteristics by VWI,⁵² and methodological papers reporting inter-rater/intra-rater reliability⁵³ and scan-rescan reproducibility of the imaging technique.⁵⁴

STROBE Reporting Assessment

The 62 analytic observational studies were also evaluated for reporting quality using the STROBE checklist (Table 1) (κ=0.76, 95% CI 0.72–0.79, p<0.001). The average CRS for all studies was 0.64±0.10. The introduction section had the highest and the methods had the lowest scores. Trans-continental collaborative publications showed higher scores (CRS_all=0.67±0.05) compared to single-continent studies (Table 1) and multi-site/single-continent collaborations (CRS_all=0.63±0.12), although these results did not reach statistical significance (p=0.20).

Table 1.

STROBE Complete Reporting Scores

	CRSAll	CRSIntroduction	CRSMethods	CRSResults	CRSDiscussion
All (n=62)	0.64 (0.10)	1.00 (0)	0.56 (0.22)	0.60 (0.20)	0.63 (0.25)
North America (n=11)	0.62 (0.10)	1.00 (0)	0.52 (0.11)	0.56 (0.22)	0.69 (0.28)
Asia (n=40)	0.65 (0.11)	1.00 (0)	0.61 (0.17)	0.65 (0.20)	0.63 (0.25)
Europe (n=6)	0.58 (0.14)	1.00 (0)	0.42 (0.15)	0.60 (0.15)	0.63 (0.19)
Trans-continental collaborations (n=5)	0.67 (0.05)	1.00 (0)	0.60 (0.07)	0.64 (0.11)	0.73 (0.10)

CRS measures reported in means (standard deviations) or medians (interquartile range). Summary statistic chosen based on test for normality.

Two sensitivity analyses were performed that showed the same direction of the results but notably did not reach statistical significance. First, a sensitivity analysis evaluating each study for items reported in any part of the manuscript, including footnotes and bylines, showed higher scores from trans-continental collaborations compared to single-continent studies (p=0.30). A second sensitivity analysis considered all partially reported criteria as fulfilling reporting and also showed higher CRS measures by trans-continental collaborations than North American and European studies. The studies from Asia had only marginally higher CRS measures (p=0.27) (Supplemental Table 3).

Supplemental Table 4 reports the mean score per STROBE item for the 62 articles and checklist descriptions. Items 1 and 22 were scored separately, as they reflect title/abstract and funding reporting, respectively. Most studies provided an informative abstract but did not indicate the study design in the title (item 1) to fully meet the STROBE criterion resulting in a score of 0.52±0.13. Also, 70% of the studies did not disclose whether or not there was a funding source (item 22) resulting in a score of 0.71±0.46. The two introduction section criteria, evaluating the reporting of the scientific background and rationale (item 2) and specific objectives or hypothesis (item 3), scored the highest among all sections.

The methods section included nine criteria. No publication reported a sample size determination (item 10). Explicitly presenting key elements of the study (item 4) also scored low (0.08±0.24). Although 73% of the studies reported a prospective or retrospective subject enrollment method, few studies named the study design type.⁵ Clearly defining outcomes, exposures, predictors, confounders, and diagnostic criteria (item 7) also had a low score (0.34±0.32); for example, only 14% of the 43 publications studying ICAD reported diagnostic criteria for cardiovascular risk factors such as hypertension. The importance of reporting diagnostic criteria for hypertension is emphasized in light of the new 2017 American Heart Association classifications of hypertension and evolving definitions.⁵⁵

Five criteria were included in the results section. Reporting of the study participants and information on exposures and potential confounders (item 14) was suboptimal (0.45+0.19). Publications lacking information on confounders such as race/ethnicity were scored as partial reporting. Only 8 studies^{18, 56–62} reported race/ethnicity, among which 5 studies^{18, 56–59} reported the information in the discussion section as a limitation of generalizability. 24 publications originated from Asia, were single-center studies, and did not report a description of ethnicity; in these studies, one could assume that all enrolled subjects were Asian but clarity in reporting could be strengthened.

In the Discussion section, generalizability (item 21) scored the lowest (0.26±0.44). Most studies were single center studies (74.2%), but this was not commonly addressed as a limitation for external validity. A discussion on limitations and reporting direction/magnitude of potential bias (item 19) also scored poorly (0.55±0.24), with most studies partially fulfilling this item due to an absence of a discussion on direction and magnitude of the bias.

Discussion

The increasing trend in the number of VWI publications and more trans-continental collaborations over the past 24 years suggests the widespread interest in the diagnostic utility of VWI. Evaluation of the reporting quality of analytic observational studies using the STROBE checklist highlighted strengths as well as weaknesses in the current literature. Subgroup analyses showed that trans-continental collaborations yielded higher CRS measures compared to single-center/single-continent and multi-center/single-continent publications, suggesting an advantage for multi-cultural, diverse collaborative work. Notably, the results did not reach significance, potentially due to the relatively small number of trans-continental publications.

Survey of the literature to date shows Asia as the leading contributor of VWI publications and ICAD to be the most commonly studied vasculopathy. Fewer VWI studies were identified evaluating arterial dissections, aneurysms, vasculitis, and moyamoya disease. More efforts are warranted to further understand the utility of VWI in these less commonly studied intracranial vasculopathies.

Given stroke is one of the leading causes of morbidity and mortality world-wide and a wide-spread public health problem, it is not surprising to see federal funding supporting many of the VWI investigations across all continents. Medical societies such as the Dutch Heart Foundation, American Society of Neuroradiology, Radiologic Society of North America, and American Heart Association (AHA) also emerged as common funding sources, showing societal missions to improve stroke outcomes. Nearly one-third of the studies did not disclose a funding source, reflecting either no funding source or funding that was not disclosed.

The availability of the STROBE guidelines has encouraged many research domains and medical specialties to improve reporting quality^{63, 64} and ultimately improve research reproducibility. Assessment of reporting quality of VWI publications has not yet been evaluated. Our results show a CRS of 0.64 for the 62 analytic observational VWI studies. The results of this study highlight areas where reporting of analytic observational studies are good along with other areas where improvements are needed. In particular, there is a need for studies to improve the clear reporting of definitions of exposures, predictors and potential confounders (item 7). The lack of clearly defined diagnostic criteria becomes important when, for example, guidelines are updated; in 2017, the AHA changed the blood pressure guidelines with some secondary analyses suggesting the new classification correlates to different prevalence rates and outcomes.^{65, 66}

Study size calculations (item 10) were not reported in any study. Sample size estimates are informative as they indicate the magnitude of the aimed effect and address whether there were challenges with recruitment due to drop out or attrition bias.⁵

Additional areas of weakness in reporting included the description of study design (item 4) and settings and locations (item 5). Reporting a prospective or retrospective enrollment is insufficient. For example, a cross-sectional study assessing imaging findings on a subject at one time-point may be prospectively enrolled or retrospectively collected from a registry.

In the Discussion, reporting the generalizability or external validity (item 21) of the results was also noted to be a weakness. Generalizability is evaluating the extent to which the results of the publication can be applied to other settings and populations and is important for the reader to see if the results are applicable to his/her own practice setting.

Analysis of reporting quality by continent showed trans-continental collaborations had higher CRS measures (Table 1). Collaborations are encouraged for the cross-pollination of ideas, to increase recruitment or enhance generalizability by coordinating a multi-site study, among other advantages.⁶⁷ Our findings suggest an additional advantage of higher reporting quality. Multilingual authors and increased diversity within teams from different continents may be possible explanations for this finding.

This systematic review has some limitations. First, neuroradiology research often reports innovative techniques and drive cutting-edge methods that are not tested as part of larger epidemiologic studies. These technological advancement publications are typically smaller scale and “proof of concept” studies that are designed differently but may have larger impact. The STROBE guidelines do not always meet the radiology research framework. However, as an assessment for completeness of reporting with consistency in evaluation by 2 independent raters, the results provide at least a basis of reporting quality among VWI publications. Second, classifying studies as case series or analytic observational studies were often challenging due to the heterogeneity in study designs. Many of the included studies did not fit traditional epidemiologic study designs. This challenge has been addressed in other systematic review methodologies and an algorithm has been tested to appropriately classify study designs for systematic reviews. We based our classification as descriptive (case series) or analytic observational studies following a simplified version of this algorithm.⁶ Third, methodologic quality was not assessed, as the aim was to identify trends in VWI publications rather than conduct a quantitative meta-analysis. Instead, an assessment of reporting quality was conducted to highlight areas for improvement. Fourth, use of the STROBE checklist is inconsistently recommended in author guidelines among journals, which could be a confounder in our analysis. Notably, some journals have their own required checklists requiring a statement on data sharing and open source availability, which also improve transparency of the conducted research. As a future direction, an assessment of the journals can be conducted to assess CRS measures by journal impact factor as well as explicit author instructions to follow the STROBE checklist. Finally, the CRS was calculated using the STROBE guidelines and items were scored by section as determined by the STROBE checklist. Studies that did not report items in the correct section per the STROBE guidelines were not considered fulfilling the criteria. This method may underestimate CRS measures. To account for this, a sensitivity analysis was performed with items scored as reported if present in any part of the manuscript, including footnotes and author bylines. This sensitivity analysis yielded similar conclusions. A second sensitivity analysis considering partially reported items as completely reported showed a similar direction of the results with higher reporting quality CRS by trans-continental collaborative work compared to all, North American, and European studies, while Asia had a marginally higher CRS measure.

Conclusions

We systematically assessed VWI publications to identify trends and assess reporting quality. Our results show that ICAD is by far the most common intracranial vasculopathy studied. The utility of VWI for ICAD and importantly, for other types of intracranial vasculopathies, will likely benefit from additional rigorous studies. We also highlight deficiencies in the reporting of analytic observational VWI studies. Trans-continental collaborative efforts yielded a higher reporting quality, although this result did not reach statistical significance. Nonetheless, there may be advantages to diverse and multi-lingual cross-cultural teams. More consistent adherence to STROBE guidelines should improve transparency and maximize effective synthesis and clinical translation of findings for future studies.