Variability of functional outcome measures used in animal models of stroke and vascular cognitive impairment – a review of contemporary studies

Abstract

Despite promising preclinical data, few novel stroke therapies have shown efficacy in man. Efforts to improve standards in conduct and reporting of preclinical research are ongoing. In clinical trials, inconsistency in outcome measures led to regulatory agencies and funders mandating use of a core set of functional outcomes. Our aim was to describe functional outcome measures in preclinical stroke and vascular cognitive impairment (VCI) studies. From 14 high impact journals (January 2005–December 2015 inclusive), 91,956 papers were screened with 1302 full texts analyzed for stroke (ischemic and hemorrhagic) and 56 for VCI studies. In total, 636 (49%) stroke and 37 (66%) VCI papers reported functional outcome measures. There were 74 different functional assessments reported in stroke and 20 in VCI studies. Neurological deficit scores (74%) and Morris water maze (60%) were most commonly used in stroke and VCI, respectively. However, inconsistencies in methods used to assess and score recovery were noted. Neurological and behavioural functional outcome measures are increasingly used in preclinical stroke or VCI studies; however, there is substantial variation in methods. A strict standardized outcome set may not be suitable for translational work, but greater consistency in choice, application and reporting of outcomes may improve the science.

Introduction

The traditional translational pathway, where compounds are first trialed in animals and then humans, has provided limited success in stroke and vascular cognitive impairment (VCI). Many compounds show promise in early phase trials but fail to deliver benefit when tested in patients with cerebrovascular disease.¹ Following a number of neutral studies of putative neuroprotectants,² the stroke research community, pharmaceutical industry and regulatory agencies suggested methods to improve the translational pathway. Resulting guidance documents, from STAIR (Stroke Therapy Academic Industry Roundtable)^3,4 and others,^5,6 were designed to raise standards in both preclinical and clinical stroke research. To prove the efficacy of any new treatment, a robust measure of effect is required. In a disabling condition such as stroke, measures of function are important metrics of assessment. Functional measures are now recommended as the primary outcome in acute stroke interventional trials and certain regulatory agencies and funders mandate their collection in stroke studies. Similar efforts towards a consensus approach for clinical VCI research have recently been proposed.⁷

Systematic reviews of assessments applied in contemporary clinical trials have described substantial heterogeneity in choice and application of functional outcome measures in both stroke and VCI studies.^8,9 Even in more niche areas, such as assessment of post-stroke cognitive and mood disorders, there are almost as many outcome measures employed as there are studies.¹⁰ This variation in assessment is inefficient as it precludes meaningful comparative analyses of studies and complicates any attempt to pool data across studies. In recognition of this, consensus statements on the preferred functional outcomes for use in stroke and dementia clinical studies have been created.^11,12

There are numerous functional outcome assessments available for preclinical ischemic stroke and VCI studies, especially rodent models,¹³ and with choice comes the potential for inconsistency in assessment. Classically in preclinical stroke research, one measures sensorimotor impairment. Motor problems are common following stroke in man but additional impairments are also observed and an exclusive motor focus may be overly reductionist. In VCI models, assessments often rely on memory impairment, although novel assessments that capture other cognitive domains more aligned with the clinical VCI phenotype have been described.¹⁴ Arguments for the importance of using functional outcome measures in humans may also be true for animal studies. As in human cerebrovascular research, there are many potential functional assessment paradigms for different animal models.

The aim of this study was to characterize preclinical stroke (ischemic and hemorrhagic) and VCI trials from the last 11 years, looking at 14 highly cited, exemplar journals from the fields of cerebrovascular science. Our objectives were to describe the frequency of use, methodology applied for functional (sensorimotor, behavioural and neurocognitive) outcome measures as well as temporal trends. To put these results in context, we also described preference of species, sex and disease models employed.

Methods

Study design

Best practice guidance in systematic review (preferred reporting items for systematic review and meta-analysis (PRISMA)) was followed, where appropriate. As the aim was to describe the use of outcomes, no assessments of the quality of the included trials’ design, methods or conclusions were made. A pre-specified protocol (researchregistry1509) was followed. Our approach was based on a previous review of functional assessments in clinical stroke trials.^8,10

Paper screening

Following initial scoping of the literature, the search was limited to 14 journals, representing a broad field of translational cerebrovascular research. Choice of titles was made by the author team. Based on impact factor, reputation within the field and frequency of publication of relevant stroke and/or VCI studies, the senior authors suggested a list of potential titles that could represent the following themes: clinical stroke (three titles chosen in this area); pre-clinical stroke; experimental neurology; neuroscience; neurorehabilitation and vascular science. Through discussion, a consensus was reached as to the top two titles for each category. The primary criterion for selection was visibility within the international research community (Supplementary Table I). They were: Brain (Oxford University Press); Circulation (American Heart Association, AHA); Experimental Neurology (Elsevier); Hypertension (AHA), International Journal of Stroke (World Stroke Organization, Sage Publishers); Journal of Cerebral Blood Flow and Metabolism (International Society of Cerebral Blood Flow and Metabolism, Sage Publishers); Journal of Neuropathology and Experimental Neurology (American Association of Neuropathologists, Oxford University Press); Journal of Neuroscience (Society for Neuroscience); Journal of Stroke and Cerebrovascular Disease (National Stroke Association, Elsevier); Nature Neuroscience (Nature Publishing Group); Neurobiology of Disease (Elsevier); Neurorehabilitation and Neural Repair (Sage Publishers); Stroke (AHA); Translational Stroke Research (Springer).

Inclusion and exclusion criteria

The search was limited to original papers published January 2005–December 2015 inclusive to capture papers before and after quality guidelines such as STAIR.^3,4 Four independent researchers (T.M.H., C.O., Z.P., L.F.) hand searched the chosen publications and screened titles/abstracts. Inclusion criteria were: original research, published in one of the chosen journals, use of animal stroke or VCI model. To ensure no potential manuscripts were missed, all journal content was reviewed, including letters, editorials and short reports. Additional methodology described in online or paper supplements was assessed, where available. As the focus was around the content of published papers, where aspects of methodology were unclear, authors of the papers were not contacted. When looking for the original citation describing a model that was not described in the index publication, if more than three further citations were cross-checked and lacked original scaling, these methods were considered as not found. Where more than one paper described the same dataset, only the primary publication was included, unless other publications reported differing outcome measures.

The aim of the study was to collate stroke and VCI animal trials with functional outcome measures, where ‘functional outcome’ was defined as a quantified measure across any of the WHO-ICF domains of impairment, activity (disability) or participation (handicap). A functional measure was accepted, where it was used as either the primary or secondary end-point of the study. For the purposes of this review, ‘trial’ was defined as any research describing the effects of an active intervention. WHO criteria for stroke was used to define the scope and within the stroke rubric, models of both ischemic and hemorrhagic stroke were included. VCI studies were defined as studies, where vascular interventions were used to emulate one (or more) of the neuropathological changes associated with VCI and where the purpose of the model was to create dementia or a cognitive impairment phenotype.

Data extraction

From eligible papers, relevant data were extracted onto a standardized, piloted, proforma spreadsheet. Items of interest were: journal title, year of publication, animal model (sex, species ± genus), stroke model, VCI model, functional outcome measure(s) (primary and secondary). The name of the functional outcome assessment used was taken directly from the text of the paper. If the assessment was not named or described in the text, accompanying citations were referred to. Where an assessment was described as “modified”, content was compared to the primary scale. If fundamental aspects of measurement differed, the measure was included as a distinct outcome. A test was counted as a single score if it was a battery or composite or various assessments that were combined as a single result (e.g. neuroscore).

Functional assessments were categorized as either neurological scales (closest to impairment measures using WHO-ICF terminology) and functional tasks (closest to activity measures using WHO-ICF). Temporal trends in the use of functional outcomes were described.

Statistical analysis

Workflow was described with a PRISMA style flow diagram. Frequency was described as functional outcomes with the use of basic statistics, non-parametric or proportional, as required. Each category was described as number of papers as well as proportion. Results were reported as the top 10 measures. Interquartile range (IQR) was calculated to show data dispersion. Numbers of papers were compared with an unpaired t-test comparing articles grouped into two groups: published 2005–2010 and papers published 2011–2015, inclusive with 2010 chosen as a time-point when core outcomes and guidance in reporting of outcomes become the standard in clinical stroke research. All analyses were performed using Prism for Windows (4.00, Graphpad) software and p < 0.05 was considered significant.

Results

Frequency of functional measures and temporal trends

Of 91,956 papers screened, 23,802 had a stroke focus and 317 VCI. Primary screening revealed 2165 and 229, respectively, as potentially relevant and of these, 863 (40%) and 88 (38%), respectively, did not meet the inclusion criteria. For stroke, full text review was conducted on 1302 papers with 636 (49%) used for final analysis. The majority of exclusions (666, 51%) were due to no reporting of a functional measure. For VCI, 56 papers were assessed in full, 19 (34%) exclusions reported no functional outcome measures and 37 (66%) papers were included in the final analysis (Figure 1).

Figure 1.

Strategy implemented in the focused literature search. Papers were selected on the basis of set inclusion/exclusion criteria and after further comprehensive review, papers that had no reported functional outcome measures as their primary or secondary endpoint were excluded from the final analysis. This yielded 636 papers for stroke and 37 for vascular cognitive impairment (VCI).

The number of stroke (Figure 2(a)) and VCI- (Figure 2(b)) related papers reporting functional outcome measures increased over the 11 years studied. A significant increase in stroke papers describing functional outcome assessments was observed after 2010, from a median of 32.5 (range: 30–38; IQR: 7) papers annually before 2010 to a median of 82 (range:75–101; IQR:19.5) papers annually since 2011 (t-test; p < 0.0001).

Figure 2.

Number of papers published each year with functional assessments. The number of preclinical stroke (a) or VCI (b) papers reporting functional assessments over the 11 years of the review period. The number of papers per year in stroke and VCI, respectively were: 2005: 29,1; 2006: 30, 1; 2007: 32, 2; 2008: 37, 1; 2009: 38, 5; 2010: 33, 2; 2011: 97, 3; 2012: 75, 3; 2013: 101, 5; 2014: 82, 2; 2015: 82, 12. A significant increase in stroke papers describing functional outcome assessments was observed after 2010 (unpaired t-test; ***p < 0.0001 comparing articles from 2000 to 2010 vs. articles 2011–2015, inclusive). Data generated from 14 specified peer-reviewed journals (Supplementary Table I).

Animal species, sex and model preferences

Rodent models were most commonly used in both stroke and VCI studies (Figure 3(a)). In stroke papers, there was variation in the animal models employed, with 622 (98%) of the papers using rodent models (339 rat, 282 mice, 1 gerbil) and 18 (2%) using non-rodent animal models (6 rabbit, 3 monkey, 2 pig, 2 baboon, 3 macaque, 1 marmoset, 1 dog). All papers within the VCI scope used rodents, with 28 (76%) of papers carried out in mice and 9 (24%) in rats.

Figure 3.

The prevalence of species and sex of animals used in stroke and VCI papers. (a) The percentage of papers using each species in stroke and VCI studies. (b) The percentage of papers using each sex across stroke and VCI studies. Percentage was calculated as portion of total amount of papers included in the final analysis (n = 636 and n = 37, respectively).

There was a sex bias in animal models employed, with the majority using males, irrespective of species (Figure 3(b)). For stroke papers, 535 (84%) used males, 22 (3%) females, 28 (4%) both male and female and 51 (8%) sex unspecified. Males were used in 23 (62%) VCI studies, females in 1 (3%), both male and female in 6 (16%) and 7 papers (19%) did not specify sex. In both stroke and VCI studies, when females were used, it was most commonly in conjunction with male animals. Only 22 stroke and 1 VCI paper(s) used solely female animals.

There were 10 distinct disease models used in stroke and 11 in VCI studies (Supplementary Table II). Ischemic stroke was more commonly studied with 559 (88%) papers, while hemorrhagic stroke was investigated in 84 (13%) of papers. Transient middle cerebral artery occlusion (tMCAO) was the most common stroke model, used in 342 (54%) papers with permanent MCAO in 109 (17%) papers. Other models of ischemic stroke included thrombotic stroke, endothelin-1 induced stroke, hypoxia–ischemia and global ischemia. Hemorrhagic stroke papers used procedures directed to create an intracerebral, subarachnoid or intraventricular bleed (Supplementary Table II).

In VCI research, permanent global hypoperfusion was the most common model, utilized in 11 (30%) papers. Alzheimer’s disease (AD) models, such as amyloid precursor protein (APP) mutations, were the second most common model, with AD pathology being used on its own or in conjunction with other co-morbidities such as hypertension, dyslipidemia and stroke. Other models included post-stroke (tMCAO) dementia, microinfarcts, hypertension or dyslipidemia-induced vascular dementia and transforming growth factor-induced cerebral fibrosis (Supplementary Table II).

Variation in functional outcome measures used in stroke studies

There were 74 different functional outcome measures recorded across stroke papers (Supplementary Table III). Of these, 42 described functional tasks, where a specific behavior was monitored and recorded, and 32 described various neurological deficit scores (NDS). Examining only the top 10 outcome measures used in stroke studies, NDS were the most frequently named functional measure (Figure 4(a)), used in 471 papers with several variations used (Supplementary Table IV). The Bederson et al.¹⁵ NDS scale was the most commonly cited in 175 papers. Modified versions of original scales were considered as separate measures where scale content, scoring or application differed from the primary description. Ten different variations of scoring were found amongst those citing Bederson¹⁵, ranging from 0–3 to 0–20 (Supplementary Table V). In 29 (6%) papers, NDS scales were not referenced or the method was left unnamed. Of the specific functional tests, Rotarod was the most commonly used in 94 (10%) papers (Figure 4(a)). Rotarod, grid walk/foot fault, adhesive label removal, cylinder, and string/wire/swing tests were the five most frequently used sensorimotor assessments and together were present in 327 (34%) of studies.

Figure 4.

Top 10 functional assessments used in preclinical stroke or VCI papers. Top 10 functional assessments in (a) stroke outcome measures (n = 967) and (b) VCI outcome measures (n = 65) included in the final analysis. Percentage was calculated as a portion of the number of functional assessments made.

Considering functional outcome measures used according to the most commonly used stroke models, in papers using intraluminal filament to induce tMCAO or pMCAO, the NDS (81% or 46% respectively) and rotarod (17% or 21% respectively) were most commonly used (Table 1). For the endothelin-1 tMCAO model, the most commonly reported outcome measure was the skilled-reaching/staircase test and cylinder tests (both 19%) (Figure 5(a)). In thrombotic stroke models, the preferred outcome measure reported was the NDS (32%) and adhesive removal test (15%) (Figure 5(b)).

Table 1.

Prevalence of outcome measures used in papers using the intraluminal filament tMCAO and pMCAO stroke models.

Functional assessment	tMCAO, n(%)	pMCAO, n(%)
Neurological deficit score	246 (81)	48 (46)
Rotarod	53 (17)	22 (21)
String/wire/grip/swing test	31 (10)	8 (8)
Adhesive removal test	26 (9)	16 (15)
Beam/rope test	22 (7)	1 (1)
Grid walk/foot fault	22 (7)	13 (13)
Cylinder test	19 (6)	9 (9)
Locomotor activity	16 (5)	9 (9)
Corner turn test	14 (5)	8 (8)
Limb placing test	13 (4)	12 (12)
Morris water maze	12 (4)	11 (11)
Tail suspension/body swing	8 (3)	6 (6)
Side-walking/circling/rotation	5 (2)	2 (2)
Postural reflex	4 (1)	3 (3)
Gait analysis	4 (1)	2 (2)
T or Y or radial arm maze	3 (1)	1 (1)
Barnes maze	3 (1)	1 (1)
Elevated plus or O maze	2 (1)	NA
Skilled-reaching/staircase task	2 (1)	8 (8)
Ladder test	2 (1)	6 (6)
Novel object recognition	2 (1)	NA
Whisker/tactile	2 (1)	1 (1)
Passive avoidance	1 (0.3)	2 (2)
Fear conditioning	1 (0.3)	1 (1)
Parallel bar crossing	1 (0.3)	1 (1)
Inclined plane test	1 (0.3)	NA
Right forelimb resting motor threshold	1 (0.3)	NA
Drinking efficiency	1 (0.3)	NA
Social novel odour recognition task	1 (0.3)	NA
Hindlimb retraction	1 (0.3)	NA
Rotameter task	1 (0.3)	NA
Forced swim test	NA	1 (1)
Stress test (max speed)	NA	1 (1)
Step test	NA	1 (1)

Note: The preferred functional outcome measure used in papers using the intraluminal filament to induce tMCAO or pMCAO was determined. Percentage was calculated as a portion of all papers included (n = 305 and n = 104, tMCAO and pMCAO, respectively).

Figure 5.

Prevalence of outcome measures used in papers according to stroke model. The preferred functional outcome measure used in papers using (a) endothelin-1 or (b) the thrombotic model to induce experimental stroke was determined. Percentage was calculated as a portion of all outcome measures used (a: n = 48; b: n = 101).

The median number of outcome measures used per trial was 1 (range 1–10; IQR: 1). Only 57 (9%) stroke papers reported functional outcome measures as their primary endpoint. The prevalence of reporting more than one outcome measure demonstrated the majority (344 papers, 54%) used one (Figure 6(a)); this dropped to 159 papers, 25% for 2 measures and continued downwards with increasing number of measures (Figure 6(a)). The top two most commonly employed measures, NDS and Rotarod, were shared across rodent models. In the 20 non-rodent studies, NDS were used as the only functional measure in all but one study. Within the nine primate studies that applied NDS, there were five different original scales cited. The six rabbit studies all applied a different NDS. The two piglet stroke studies each employed a different version of the same NDS.

Figure 6.

Prevalence of the number of different outcome measures reported in preclinical stroke or VCI papers. The number of preclinical stroke papers using one or more functional outcome measures was determined for (a) stroke and (b) VCI papers. These represented for increasing number of outcome measures employed: (a) 1 = 344 papers (54%), 2 = 159 (25%), 3 = 81 (13%), 4 = 34 (5%), 5 = 7 (1%), 6 = 6 (1%), 7 = 1 (0.2%), 8 = 0 (0%), 9 = 1 (0.2%), 10 = 4 (1%) and (b) 1 = 20 papers (54%), 2 = 10 (27%), 3 = 4 (13 = 11%), 4 = 1 (3%), 5 = 0 (0%), 6 = 0 (0%), 7 = 0 (0%), 8 = 1 (3%), 9 = 0 (0%), 10 = 0 (0) 11 = 1 (3%). Percentage was calculated as a portion of all papers included.

Variation in functional outcome measures used in VCI studies

There were 20 different types of assessments used (Supplementary Table VI), which were predominantly tasks involving memory (8 assessments) or motor coordination (8 assessments). Only 7 (19%) VCI studies reported functional outcome measures as their primary endpoint. Examining, the top 10 most commonly used outcome measures in VCI papers (Figure 4(b)), demonstrated the Morris water maze (MWM)¹⁶ was most commonly used, irrespective of species and was used in 22 papers (with one completing 2 trials, hence 23 trials). Other commonly applied tasks were novel object recognition, T or Y radial maze, locomotor activity and NDS, found in 27 studies (Figure 4(b)). The median number of functional assessments used in VCI papers was 1 (range 1–11, IQR: 1). The prevalence of reporting more than one outcome measure demonstrated the majority (20 papers, 54%) used one (Figure 6(b)); this dropped to 10 papers, 27% for 2 measures and continued downwards with increasing number of measures (Figure 6(b)).

Within papers using the MWM, there was notable heterogeneity within the methodology of the test (Table 2). Some studies used a submerged platform (14 trials; 61%), whereas others a visible platform (4 trials; 17%) or a combination (5 trials; 22%). There were also differences in external cues used to assist in finding the platform (56% used cues versus 44% no cues) and whether these cues were visual or olfactory/auditory. There was a preference towards including a probe trial, where the platform is removed and the swimming latency is recorded to test retention memory, but this was inconsistent across all studies (78% and 22%, respectively). There was further variability in the number of acquisition days (between 0 and 10 days, IQR: 2.5) allowed before endpoint assessment; the number of trials performed per day (1–8 trials per day, IQR: 2); the timing of the probe trial (range 3–10 days, IQR: 1.25), and whether there were further trials after the probe trial (39% studies applying probe trials had post-probe trials, 61% did not).

Table 2.

Variation in the characteristics of Morris water maze (MWM).

Variation used	Papers using MWM, n (%)
Platform type (n = 23)
Submerged platform	14 (61)
Both types of platform	5 (22)
Visible platform	4 (17)
Cue type (n = 23)
Visual cues	12 (52)
No cues	10 (44)
Olfactory/auditory/spatial	1 (4)
Presence of probe trial (n = 23)
Probe trial	18 (78)
No probe trial	5 (22)

Note: Differences in characteristics of MWM used in the papers shown as differences in set-up, presence and type of cues as well as with the presence of a probe trial, where platform was removed from the pool. Percentages were calculated as total amount of papers using MWM (n = 23).

Discussion

The traditional paradigm for assessing efficacy in preclinical stroke and VCI trials has been a combination of neurological assessments with measurements of infarct volume. Use of functional assessments as primary outcome or to complement other outcomes is increasingly recommended in both clinical and preclinical research. In the present review, papers from 14 journals from the past 11 years were investigated to describe the functional outcome measures used in preclinical trials of stroke and VCI. Although functional assessments are mandated in clinical stroke trials, functional assessments were the primary outcome in only a minority of outcomes used in preclinical models. The landscape may be changing, as there was an increase in papers reporting functional outcome measures over the time horizon of the study. Where functional outcomes were used, there was substantial heterogeneity in the assessment and application of the assessment. There was greater consistency in the sex and species of animal model, although a predominant focus on male animals is not in keeping with best practice.

The observed increase in stroke papers reporting functional measures from 2010 onwards could be partly attributed to the publication of the revised STAIR guidelines in 2009.⁴ These guidelines emphasize not only the importance of using functional outcome measures, but also problems inherent in using multiple outcome measures. Indeed, a recent editorial by Zerna et al.¹⁷ highlighted the choice of endpoint remains challenging in the preclinical stroke field. While not as many papers were identified for VCI preclinical studies, those reporting outcome measures increased temporally across the 11-year period, which may reflect a shift in research focus towards dementia.¹⁸

The predominant (98%) or exclusive use of rodent models in stroke and VCI studies, respectively, is consistent with other animal disease models. Their use is favored for several reasons – maintenance costs are low, use is preferred ethically, availability of transgenic strains and the vascular anatomy is similar to human (reviewed in Macrae¹⁹ and Howells et al.²⁰). Considerations for stroke/VCI studies include that rodents have less white matter than humans since they have lissencephalic brains. STAIR⁴ recommend that once efficacy is established in a rodent model that larger animal models are studied prior to clinical translation. Another key consideration in the translation pipeline is the inclusion of disease-relevant comorbidities and risk factors, such as hypertension and aging for stroke/VCI studies, highlighted in STAIR and other recent guidelines.^4,21 Presence of existing comorbidities across the studies described herein was not included in the inclusion/exclusion criteria as based on previous studies, consideration of such co-morbidities occurred in a minority (3%) of reports of therapeutic interventions in stroke.²²

Even though women are more likely affected by post stroke disability,²³ males are more often used in preclinical research and within those studies included in this review, 84% of stroke and 62% of VCI studies used male animals. The focus on males most likely limits cohort sizes and variability through removal of the effect of the estrous cycle in females. However, the importance of considering sex as a biological variable has been discussed extensively.^4,24,25 This followed the publication of the National Institute of Health guidance highlighting that sex differences should be factored into experimental design.²⁶ Sex has significant effects on many biological processes (reviewed in Chauhan et al.²⁷) and certain interventions, such an inhibition or knockout of neuronal nitric oxide synthase (nNOS) or poly-ADP ribose polymerase-1 (PARP-1), have shown protection in male animals subjected to experimental stroke and exacerbation of injury in females.²⁸ Within the studies included here, when females were used, this was most commonly in conjunction with male animals. The recent publication of the ‘Sex and Gender Equity in Research’ guidelines²⁹ combined with grant review considerations may lead to a shift in the inclusion of both sexes in experimental stroke and VCI studies in coming years.

Eleven models of both stroke and VCI were described. Most stroke studies used an ischemic model with tMCAO being the most widely employed. For VCI, the permanent global hypoperfusion model was the most commonly used. The choice of experimental stroke model used will be tailored to the therapeutic intervention being trialed and considerations such as whether reperfusion should occur or the size and location of the lesion required. Indeed, differences in the choice of preferred outcome measure were evident comparing across stroke models. The relative merits and limitations of each model are reviewed elsewhere.^19,20,30,31 Similarly, for VCI studies, the biological question being addressed will determine the most appropriate model to use. Models that produce lesions in distinct locations require outcome assessments that are relevant to the affected brain region. For example, the use of the skilled reach/staircase test or cylinder test when the endothelin-1 stroke model was used reflects the cortical lesion produced. However, an exclusive focus on impairment measures specific to the lesion is a reductionist approach and potentially fails to capture the biological variability in response; the remote consequences of targeted lesions and the limited correlation between neuroanatomy and function. Global assessments of functional outcome should be relevant to any stroke lesion and could form part of a core set of outcomes complemented by other specific tests.

There were 74 different functional measures used in stroke preclinical studies. The most frequently used was NDS, although within this group there was marked heterogeneity in NDS used and the application of the test. These scales (or scores) generally reflect overall condition, assessing reflexes, simple motor function and balance (reviewed in Balkaya et al.³²). They are often preferred as they can be performed soon after experimental stroke and generally do not require specialized equipment. However, NDS are not a distinct behavioral measure per se, rather NDS are normally composed of various components that quantify the global stroke-related impairment over time with inherent limitations due to subjectivity of scores. The most commonly used NDS, Bederson,¹⁵ gives a total score of 0–3 in the original scale. This limited range of possible scores lacks sensitivity for assessing change with subsequent effects on sample size requirements and has led to a range of modified Bederson scales with a greater number of components.

A further challenge with the NDS was distinguishing between the methodologies used. From the NDS or battery tests described, most were named by referencing the original authors who first developed the tools, or by referencing authors who had previously used the tools. The referencing was further complicated by publication of subsequent and different modified versions of the originals, each with their own authors further modifying the scale. Comprehensive descriptions of the methodology of each assessment were infrequent and the original method was often left unclear. There were also multiple papers where methods were not clarified with appropriate citations or descriptions. Without clear methodology, interpreting between NDS becomes complex. Poor reporting makes it difficult to replicate experiments in independent cohorts and ultimately limits the validity of the research and potential to use the data for comparative or pooled analyses such as systematic reviews and meta-analyses.³³ Within the clinical arena, there have been numerous efforts leading to a consensus in preferred outcome measures in clinical trials³⁴ and our findings suggest that this could be applied in preclinical research.

In VCI studies, 20 different functional assessments were recorded. While common standards exist for identifying and describing cognitive impairment clinically, these recommendations do not extend to preclinical VCI study reporting.³⁵ Nevertheless, assessing cognitive impairment is crucial to validate the disease model and to further assess the effects of interventions. The MWM was the most commonly used outcome measure across all studies in mice or rats. However, there was clearly marked inconsistencies across the methodology used in these studies which would, again, make pooled analyses difficult. Indeed, this is consistent with a recent systematic review of AD mouse models where substantial variation in methodology in the use of the MWM was found with 57 studies using the probe phase using 59 different approaches.³⁶ The substantial variations in the characteristics of the MWM set-up alter which aspects of memory retention and learning are being assessed.³⁷ In stroke studies, use of a pre-determined performance criterion within assessment of the MWM has been recommended in order to avoid potential misinterpretation of data.³⁸ Furthermore, it has been shown that rats outperform mice when using the MWM as an assessment measure.³⁹ Many of the assessments used in VCI models were first proposed for animal models of other dementia pathologies, e.g. AD. Assessments that predominantly describe memory may miss important impairments in other domains commonly seen in VCI, for example the complex constructs of executive function. Recently, models used in VCI preclinical trials have been under review,⁴⁰ but for more effective clinical translation, the appropriateness of functional outcome measures for each model should also be evaluated.

There were strengths and limitations to the search conducted. The large and increasing stroke literature precluded a comprehensive review of outcomes across all published preclinical stroke and VCI trials. The analysis was limited to journals with a large readership across the disciplines of clinical and translational neurovascular research over an 11-year period. Consequently, the sampling frame is potentially biased. Important studies describing VCI could be published in a variety of journals. Our focus on cerebrovascular disease and non-inclusion of dementia specific titles may have affected the yield of VCI papers. However, the intention was to describe the outcome measures used in journals with the greatest scientific impact, rather than across the complete stroke and VCI literature. Our focus was predominantly on measures of sensorimotor or cognitive outcome. These may not be the only “functional” outcomes of relevance. Recent priority setting exercises have suggested that emotional and mood symptoms are the issues of greatest importance to stroke survivors.⁴¹ Although not comprehensive, the search strategy was systematic. A systematic approach was necessary for describing the assessment methods employed. As numerous papers only referred to the assessment method used, the stated functional outcome tool was cross-referenced, sometimes across several papers, to obtain the original description of the assessment method used. The intention was purely to describe current outcome assessment methods and no attempt was made to compare the strengths and weaknesses of different instruments or to assess the methodological approaches described in the trials. There have been recent efforts to evaluate the validity of various assessments in certain models of stroke,^42,43 and there would be value in expanding this evaluation to include the many stroke and VCI preclinical models described in the literature.

The need to determine neurological deficit and function as an outcome measure within stroke and VCI studies is imperative for potential clinical translation. The marked heterogeneity described is perhaps unsurprising as there are no agreed pre-set guidelines for functional assessment measures preclinically. Consistency across studies will facilitate easier between study comparisons and pooling of preclinical data. In clinical trials, work by groups such as EQUATOR (Enhancing the Quality and Transparency of Health Research) has raised standards in reporting of research methods. The work of collectives such as the CAMARADES group is trying to replicate this success in preclinical models. It is encouraging to see emerging guidance around preferred assessments, albeit this is limited to a specific aspect of cerebrovascular research. For example, the Stroke Recovery and Rehabilitation Roundtable have recently produced guidance for behavioural outcome measures.^44,45 Considerable efforts and advances have been made in improving design, conduct and analysis in stroke research,^21,45,46 with improvements in terms of rigor and reduced bias reflecting the changes implemented.⁴⁷ Indeed, in comparison to other areas, preclinical stroke research in particular, is performing well against targets to enhance reproducibility and promote translation.⁴⁸

In this context of increasing standardization of methods in clinical research, one possible interpretation of our results is that the preclinical stroke research community should move towards a core set of preferred functional outcomes measures. While few would argue against improving rigor and transparent reporting, restricting the choice of outcome assessments available may not suit translational and discovery science. The nature of preclinical research is often more exploratory compared to the confirmatory nature of clinical trials and indeed, over-standardization of laboratory studies may have the opposite intended effect and result in poor reproducibility.⁴⁹ For hypothesis generating pre-clinical studies, it is arguable whether functional assessment adds value to other surrogate outcomes such as neuroimaging. Perhaps a more suitable suggestion would be that for larger scale preclinical studies, researchers use at least one functional outcome measure with standardized approaches to assessment and scoring in addition to any assessments specific to the scientific question of interest.

In conclusion, we have demonstrated that functional outcome measures are increasingly being used in preclinical cerebrovascular research but when they are employed, substantial heterogeneity in the measures chosen and their application exists. This inconsistency in conduct and reporting limits the potential for comparative or meta-analyses. The clinical research community have developed preferred outcomes but strict control of the tests available to researchers may not always be suitable in preclinical work. There are other avenues available, including standardized operating procedures, standardized scoring criteria, guidance on reporting outcome assessments and many others. The preclinical research community must now work together to improve consistency and transparency and we would welcome any initiatives that look to develop these resources.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: TMH is supported by a Cunningham Trust PhD studentship (CU15007.0001). TJQ is supported by a Chief Scientist Office/Stroke Association Clinical Senior Lectureship (TSA LECT 2015/05).

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Authors’ contributions

TMH, CO, ZP, LF completed the data collation and primary analysis; TMH, CO, CM, LMW & TQ wrote the manuscript.

Supplementary material

Supplementary material for this paper can be found at the journal website: http://journals.sagepub.com/home/jcb