The Neurological Impairment Scale: reliability and validity as a predictor of functional outcome in neurorehabilitation

Abstract

Purpose

To examine the construct validity and inter-rater reliability of the Neurological Impairment Scale (NIS) and compare ratings by medical and multidisciplinary teams in a mixed neurorehabilitation sample. To assess its concurrent and predictive validity as a predictor of outcome and functional gains during inpatient rehabilitation.

Methods

The NIS was rated in a consecutive cohort of patients (n = 428) recruited from nine specialist neurorehabilitation units in London. Dimensionality and internal consistency were explored through principal components analysis with Varimax rotation. Inter-rater reliability and the relationship between NIS and functional outcome (UK Functional Assessment Measure (FIM + FAM)) were analysed in a sub-sample (n = 94) from one centre.

Results

Factor analysis identified two principal domains (“Physical” and “Cognitive”) together accounting for 35% of the variance: their Cronbach’s alpha values were 0.76 and 0.67, respectively. Inter-rater reliability was excellent for overall scores between doctors (ICC = 0.95 (95% CI = 0.91–0.97)) and acceptable between the medical and multidisciplinary team (ICC = 0.92 (95% CI = 0.88–0.95)). Change in NIS-physical score predicted 29% of the variance in functional gain (FIM + FAM change).

Conclusion

These findings provide the first formal evidence for the validity and reliability of the NIS as a measure of neurological impairment for use in general neuro-rehabilitation settings. Its further application and exploration are now warranted.

Implications for Rehabilitation

• The extent of neurological recovery occurring during rehabilitation can make an important contribution to functional gains. In order to interpret measurement of functional outcome, we need to be able to identify changes at the level of impairment.

• Many of the available tools to measure severity of impairment are condition specific. The Neurological Impairment Scale (NIS) was developed for use across a broad range of disabling conditions alongside the UK FIM+FAM.

• This first formal examination of its psychometric properties provides evidence for its scalability, reliability and validity.

• The NIS has potential to provide useful information for case-mix adjustment and as a predictor of functional gain in general neurorehabilitation settings.

Introduction

The World Health Organisation’s (WHO) International Classification of Functioning, Disability and Health (ICF), allows description of the impact of any health condition at a number of levels, including impairment of body structures and function, limitation in activities and restriction of participation [1]. Standardised measures of functional independence in daily living activities (such as the Barthel Index) [2], the Functional Independence Measure [3–5] and Functional Assessment Measure (FIM + FAM) [6–8] are widely used across the Western world to evaluate global disability and the outcome of rehabilitation programmes. However, in neurological rehabilitation, patients present with diverse conditions and different combinations of physical, cognitive, communicative and behavioural impairments. They also present at varying stages of evolution of their condition, so that intervention may take place against a background trajectory of natural recovery (e.g. following acute stroke or brain injury) or deterioration (e.g. in progressive or neuro-degenerative disease). Functional gains are expected to be smaller in these latter conditions, but even the prevention of deterioration can still improve the experience of patients and their families [9,10].

In order to make meaningful comparisons across different practices, programmes and populations, it is therefore helpful to have a standardised assessment of neurological impairment, against which any change in functional independence can be evaluated. A number of impairment sets exist for specific conditions, such as the NIH scale for stroke [11] and the ASIA scale for spinal cord injury [12]. However, these are not necessarily applicable across the broader range of neurological conditions.

Within the UK, a national clinical dataset is now collated for all episodes of specialist in-patient rehabilitation through the UK Rehabilitation Outcomes Collaborative (UKROC) database [13]. The dataset gathers a common dataset of patient-level information on needs, inputs and outcomes across all conditions presenting for neurological rehabilitation. From April 2013, it is mandated for collection by all NHS specialist neurological rehabilitation services in England. The UK FIM + FAM is widely used in the UK as a global measure of disability [14] and has been adopted as the principal outcome measure for the UKROC dataset, and the Neurological Impairment Scale (NIS) as the measure of impairment.

The NIS is not new. It evolved from the impairment set that was developed alongside the UK FIM + FAM in the 1990s, and has been included within the original “minimum dataset” of the UK FIM + FAM software programme since 1999 (see below for more details). Until now, however, there has been neither formal evaluation of its internal psychometric properties, nor of its usefulness as a predictor of disability or of potential to make functional gains in rehabilitation. Neither is it clear when and by whom the NIS should be recorded in the rehabilitation process.

The aim of the present study was to perform a first examination of the reliability and validity of the NIS as a measure of an individual’s specific and overall neurological impairment in a sample of patients with a diverse range of neurological disabilities.

The specific objectives were:

1. to investigate its construct validity through an exploratory factor analysis and examination of internal consistency;

2. to determine inter-rater reliability and to compare ratings recorded by the medical staff and the multidisciplinary team;

3. to examine concurrent validity in terms of the relationship between changes in impairment and disability;

4. to assess predictive validity of the NIS as a predictor of outcome and functional gains made during inpatient rehabilitation.

Methods

The Neurological Impairment Scale

The UK version of the FIM + FAM was developed in the mid-1990s, by the UK FIM + FAM Development Group, which consisted of experienced FIM + FAM users from 25 rehabilitation centres across the UK [8]. The group set out to improve consistency of scoring of the FIM + FAM between different centres, and to establish a central database to facilitate sharing of outcome data for neurological rehabilitation. As part of this development, the Group also identified a “minimum dataset” be collated alongside FIM + FAM data in a central database [15]. In keeping the WHO model of illness at the time – the International Classification of Impairment Disability and Handicap (ICIDH) [16] – the group developed a simple checklist of neurological impairments that were likely to be confounders of functional gain.

This checklist provided a crude identification of different types of impairment, but it gave no indication of severity and was therefore insensitive to change even where neurological recovery (or indeed deterioration) was taking place. Consequently, later versions of the database not only expanded the range of impairments but also introduced a simple grading of severity (“None”, “Mild”, “Moderate”, “Severe”). After the International Classification of Functioning (ICF) was published in 2002 [1], the NIS items were mapped onto ICF codes to support data collection using the common language of the ICF.

Over the course of some 15 years and several iterations, the tool has evolved into an ordinal measure of impairment that has potential applicability across a wide range of neurological conditions. The NIS (the NIS-version 8, as tested here), comprises 17 items (each rated 0–2 or 0–3 giving a total score range 0–50). An example of the NIS score sheet and main scoring principles is given in Appendix 1. The full NIS together with detailed instructions for rating is available through the UKROC website (http://www.csi.kcl.ac.uk/ukroc).

Design

A cohort analysis of patients admitted for in-patient neurorehabilitation.

In part 1, we used a principal components analysis with Varimax rotation to examine the dimensionality and internal consistency of the NIS in a multicentre sample (n = 428) from nine specialist neurorehabilitation units.

In part 2, we performed a more detailed analysis of inter-rater reliability and the relationship between NIS and functional outcome (UK FIM + FAM scores) in the cohort sample (n = 94) from just one of the contributing centres.

Ethics approval

Approval for part 1 was granted by the Bromley Research Ethics Committee (Ref no: 09/H0805/25) and subsequently R&D centres of the seven participating NHS trusts. The remaining two recruiting centres were in the independent sector - approval for recruitment was obtained through their internal clinical and research governance processes.

The centre in part 2 of the study gathers NIS and UK FIM + FAM data routinely in the course of clinical practice. Approval has been granted by the Harrow Research Ethics Committee (Ref no: 04/Q0405/81) for reporting these data retrospectively for research and audit purposes.

Part 1: Construct validity (dimensionality and internal consistency)

Participants

For this part of the study, NIS ratings were examined at discharge from in-patient rehabilitation to maximise representation across the score range. Participants were recruited as part of a larger study of community rehabilitation services delivery post-discharge between September 2009 and March 2011 [17]. A consecutive cohort of 467 patients discharged over 1 year from each of nine tertiary neurorehabilitation units in London during this period were assessed for eligibility. Of these, 428 consented to participate and 403 had complete NIS scores, giving 25 with missing data. The demographic characteristics of the recruited sample (n = 428) are shown in Table 1.

Table 1.

Demographics of the study populations for Parts 1 and 2.

	Part 1 – n = 428		Part 2 – n = 94
	Mean	SD	Mean	SD
Age (years)	49.5	15.3	42.9	14.5
Length of stay (days)	–	–	93	62
Mean time since onset (days)	–	–	104	71
	n	%	n	%
Males:Females	270:158	63:37	59:35	63:37
Diagnosis
Acquired brain injury	315	74%	79	84%
Vascular (stroke, SAH)	212	50%	41	44%
Traumatic	63	15%	28	30%
Other (e.g. Hypoxic/inflammatory)	40	9%	10	11%
Spinal cord injury	38	9%	7	7%
Guillain-Barré and other peripheral neuropathies	26	6%	5	5%
Multiple sclerosis	21	5%	7	–
Others	27	6%	3	3%

SAH = Sub-arachnoid Haemorrhage.

Data collection and analysis:

NIS scores were recorded by the Multi-disciplinary (MD) treating team within 7 days before discharge to the community. Training in NIS rating was provided for teams who were not already familiar with the NIS as part of the UK FIM + FAM.

Exploratory factor analysis was used to examine dimensionality. The Kayser–Meyer–Olkin measure of sampling adequacy and Bartlett’s test of sphericity both indicated that the correlation matrix was acceptable for factor analysis [18]. We used principal components analysis with Varimax rotation to identify the major sources of variance and to demonstrate whether a set of items can be meaningfully summarised as a single total score. Analysis was completed with IBM SPSS Statistics Package Version 19 (Chicago, IL). After initial inspection of item total correlations and a scree plot, we extracted components with eigenvalues over 2, which were then rotated using Varimax rotation. The internal consistency (i.e. Cronbach’s α) was also calculated for the full scale and the first two principal components we identified.

Part 2: Inter-rater reliability and the relationship between changes in impairment and disability

Participants and data collection

In this more detailed single-centre analysis we examined NIS and UK FIM + FAM scores on both on admission to, and discharge from, in-patient rehabilitation. This unit is the lead centre of training and dissemination of the UK FIM + FAM and NIS, so use of both scales is well established in this setting and all staff are fully trained in their use. The UK FIM + FAM is routinely recorded by the multidisciplinary team within 10 days of admission and repeated within 7 days of discharge. The NIS is recorded by the MD Team for all patients during the admission FIM + FAM rating. During the study period it was also repeated at the discharge rating in n = 73 patients.

In addition, the medical team independently undertook NIS ratings, based on presentation of findings collected at the admission medical clerking within 48 h of admission for n = 77 patients. To test inter-rater reliability of this method, two doctors independently recorded NIS scores for a sub-sample of n = 47 patients.

Patients were included in the analysis sample if they had (a) paired NIS ratings on admission and discharge (n = 73) or (b) paired ratings by the medical and MD team (n = 77) (some patients were included on both counts). The demographic characteristics of this single-centre sample (total n = 94) are compared with that of the multicentre cohort in Table 1.

Analysis

Where item scores were recorded as untestable (e.g. due to low awareness state, etc) scores of three were allocated to generate the total and subscale NIS scores.

Reliability

Inter-rater agreement between doctors, and between the medical and MD teams, was evaluated using linear-weighted kappa coefficients computed in Stata version 10 (StataCorp, College Station, TX) and interpreted according to Landis and Koch [19]. Intra-class correlation (ICC) coefficients were used to compare total scores.

All other statistical analysis was undertaken using IBM-SPSS v.19. Because the NIS and UK FIM + FAM generate ordinal data, we report the median and interquartile range (25th–75th percentile) as well as means and standard deviations. However, as the samples were reasonably large and the data fell within acceptable limits of a normal distribution (one-sample Kolmogorov–Smirnov tests, p > 0.05), parametric statistics were applied throughout. Within-subjects effect sizes were calculated as Cohen’s d with adjustment to allow for correlation between the means [20] using an online calculation package [21].

Concurrent validity

To examine the relationship between impairment and disability, we examined Pearson correlations between NIS and UK FIM + FAM scores on admission, and also between the change of scores from admission to discharge.

It was anticipated that, in this general neuro-rehabilitation population, a significant proportion of patients would not demonstrate change in impairment. To determine whether functional change was greater in patients who changed at the level of impairment during rehabilitation, we divided the dataset retrospectively into “impairment responders” (defined as those patients whose NIS scores reduced by 2 or more points between admission and discharge) and “impairment non-responders” (whose NIS score remained static or increased). Change in UK FIM + FAM scores was compared both within- (paired t-tests) and between (unrelated samples t-tests) these groups.

Predictive validity

Stepwise multiple linear regression modelling was used to evaluate MD team-rated NIS physical and cognitive subscale scores as predictors of functional status at discharge (UK FIM + FAM) and change in function during rehabilitation. Criteria for earning entry to the model were set at an alpha level of 0.05, and for removal at 0.1.

Results

Part 1: Exploratory factor analysis – dimensionality and internal consistency

Table 2 presents the results of the principal components analysis after Varimax rotation. In fact the pattern of loadings on the first two unrotated principal components and on the two rotated factors were almost identical. Two components had eigenvalues >2, which together accounted for 35% of the variance in total scores. The most significant source of variance is explained by a Physical component and the second major source of variance is a Cognitive component. Only two of the 15 items (“Hearing” and “Other”) achieved loadings of <0.4 on both components mainly due to a preponderance of zero scores (90% and 85%, respectively). In subsequent analyses, “Hearing” was included in the cognitive subscale. The “Other” score (which most commonly include impairments such as seizures or pressure sores) was included in the physical subscale. Cronbach’s alpha for the full 17-item NIS was 0.75. For the 10 Physical and seven Cognitive items, Cronbach’s alphas were 0.76 and 0.67, respectively.

Table 2.

NIS score ranges, item-total correlations and loadings on first two principal components using Varimax rotation.

	Descriptives		Item total correlations	Rotated component matrix
	Median (IQR)	Range		Factor 1 Physical	Factor 2 Cognitive
Eigen value				3.6	2.3
% variance				20.4	14.6
NIS Items
Left upper limb	0 (0–1)	0–3	0.43**	0.493
Right upper limb	1 (0–2)	0–3	0.44**	0.502
Left lower limb	1 (0–2)	0–3	0.47**	0.677
Right lower limb	1 (0–2)	0–3	0.50**	0.693
Trunk	1 (0–1)	0–2	0.50**	0.670
Tone/contractures	1 (0–2)	0–3	0.63**	0.697
Sensation	1 (0–2)	0–3	0.55**	0.629
Perception	0 (0–1)	0–3	0.38**		0.498
Speech and language	1 (0–2)	0–3	0.45**		0.580
Cognitive	1 (1–2)	0–3	0.33**		0.808
Behaviour	0 (0–0)	0–3	0.23**		0.635
Mood	1 (0–1)	0–3	0.40**		0.526
Vision	1 (0–1)	0–3	0.31**		0.562
Hearing	0 (0–0)	0–2	0.09
Pain	1 (0–1)	0–3	0.50**	0.505
Fatigue	1 (1–2)	0–3	0.52**	0.543
Other	0 (0–0)	0–2	0.12*

^**significant at p < 0.001

^*significant at p < 0.05.

Part 2: Reliability and relationship with functional outcome

(a) Inter-rater reliability

Table 3 reports the level of inter-rater agreement found between the different ratings. Within the medical team there was high overall agreement between the two doctors reflected by a kappa coefficient of 0.81 for total NIS score and ICC 0.95 (95% confidence interval 0.91–0.97). Item-by-item agreement ranged from “substantial” to “almost perfect” with the exception of the fatigue item.

Table 3.

Inter-rater agreement: item-by-item linear-weighted kappa coefficients interpreted according to Landis and Koch [19].

Item	Agreement between two doctors (n = 47)			Agreement between Medical and MD Team scores (n = 94)
	Kappa	95% CI*	Interpretation	Kappa	95% CI	Interpretation
Left upper limb	0.94	0.72–1.0	Almost perfect	0.76	0.61–0.92	Substantial
Right upper limb	0.90	0.69–1.0	Almost perfect	0.74	0.59–0.90	Substantial
Left lower limb	0.93	0.72–1.0	Almost perfect	0.82	0.66–0.97	Almost perfect
Right lower limb	0.85	0.65–1.0	Almost perfect	0.76	0.92–0.59	Substantial
Trunk	0.87	0.64–1.0	Almost perfect	0.43	0.27–0.59	Moderate
Tone	0.63	0.44–0.82	Substantial	0.45	0.31–0.60	Moderate
Sensation	0.82	0.59–1.0	Almost perfect	0.65	0.51–0.79	Substantial
Perception	0.85	0.60–1.0	Almost perfect	0.63	0.47–0.79	Substantial
Speech	0.90	0.68–1.0	Almost perfect	0.81	0.66–0.96	Almost perfect
Cognitive	0.84	0.63–1.0	Almost perfect	0.66	0.52–0.79	Substantial
Behaviour	0.94	0.69–1.0	Almost perfect	0.13	0–0.27	Slight
Mood	0.75	0.54–0.96	Substantial	0.57	0.43–0.72	Moderate
Vision	0.80	0.56–1.0	Substantial	0.68	0.52–0.84	Substantial
Hearing	0.79	0.52–1.0	Substantial	0.77	0.59–0.95	Substantial
Pain	0.76	0.58–0.94	Substantial	0.43	0.29–0.56	Moderate
Fatigue	0.58	0.40–0.77	Moderate	0.39	0.27–0.52	Fair
Other	0.84	0.63–1.0	Almost perfect	0.44	0.29–0.59	Moderate
Total NIS Kappa	0.81	0.63–0.99	Almost perfect	0.69	0.56–0.95	Substantial
ICC	0.95	0.91–0.97		0.92	0.88–0.95

^*95% Confidence intervals (CI) were calculated as +/−1.96 s standard error and the upper limit truncated at a maximum of 1.0.

Less strong agreement was expected between the ratings by the medical and MD team as up to 10 days elapsed between the assessments. Nevertheless, agreement in total scores was still acceptable (Kappa 0.69 for total NIS score, ICC 0.92 (95% CI 0.88–0.95). Agreement for individual items was moderate to strong for 10/13 items. Only slight or fair agreement was observed, however, for the items for behavior and fatigue (see “Discussion” section).

Figure 1 shows a box and whiskers plot of the total NIS scores rated on admission and discharge. Medical teams tended to record slightly lower ratings than the MD team, although this did not reach significance, either for the total scores or at item level. The MD team ratings were considered to be more reliable (see “Discussion” section) and were therefore used in the further evaluation of NIS as a predictor of functional outcome.

Figure 1.

Box and whisker plots for total NIS scores at admission and discharge. Figure 1 shows a box and whiskers plot of the total NIS scores as rated by the MD and medical teams on admission and the MD team at discharge. There was no significant difference between the two admission ratings (mean difference 0.03, 95% CI −1.16–1.11, t = 0.045, p = 0.96). However, there was a significant reduction in team-rated total scores between admission and discharge (see Table 4).

(b) Concurrent validity – the relationship between impairment and disability

Table 4 summarises the admission, discharge and change scores for the NIS and UK FIM + FAM scores, and Table 5 shows the correlation between them. Strong negative correlations were seen between NIS Physical and FIM + FAM Motor subscales on admission (Pearson r = −0.86), and similarly between the respective cognitive subscales (r = −0.90). Weaker, but still significant correlations were seen between physical and cognitive domains of the respective scales (r = 0.62, p < 0.001 in each case).

Table 4.

Descriptive statistics for NIS and UK FIM + FAM scores as rated by the MD Team on admission and discharge (n = 73).

Admission	Mean	SD	Median	IQR	Range	Paired sample t tests
						t	p	ES*
Neurological Impairment Scale (NIS)
Physical Subscale	14.0	6.7	13	10–19	1–28
Cognitive Subscale	8.2	6.2	7	4–11	0–21
Total Score	22.2	11.6	19	15–28	4–48
Discharge
Physical Subscale	11.8	6.8	11	7–17	0–27
Cognitive Subscale	6.7	5.6	5	2–8	0–21
Total Score	18.3	11.5	15	10–22	3–46
Change
Physical Subscale	−2.2	2.9	−2	−4–0	−11–5	6.4	<0.001	0.73
Cognitive Subscale	−1.5	2.4	−1	−3–0	−11–3	5.3	<0.001	0.64
Total Score	−3.7	4.1	−3	−7–−1	−18–5	7.8	<0.001	0.90
UK Functional Assessment Measure (FIM + FAM)
Motor Subscale	53.9	30.0	51	21–86	16–108
Cognitive Subscale	56.3	27.0	61	32–79	14–95
Total Score	110.1	52.0	123	63–156	30–189
Discharge
Motor Subscale	74.3	33.3	87	47–102	16–111
Cognitive Subscale	69.5	24.6	77	59–88	14–98
Total Score	143.8	55.6	166	109–188	30–206
Change
Motor Subscale	20.4	16.5	17	6–33	−2–62	−10.5	<0.001	−1.26
Cognitive Subscale	13.3	11.9	11	5–18	−5–55	−9.5	<0.001	−1.14
Total Score	33.7	23.4	31	14–52	−3–93	−12.3	<0.001	−1.48

^*Effect size (ES) calculated as Cohen’s d allowing for the correlation between the mean.

Table 5.

Pearson correlations for admission and change scores (n = 73).

	Admission scores						Change scores
	NIS Physical	NIS Cognitive	NIS Total	FIM + FAM Motor	FIM + FAM Cognitive	FIM + FAM Total	NIS Physical	NIS Cognitive	NIS Total	FIM + FAM Motor	FIM + FAM Cognitive
Admission
NIS Physical
NIS Cognitive	0.62***
NIS Total	0.91***	0.89***
FIM + FAM Motor	−0.86***	−0.62***	−0.85***
FIM + FAM Cognitive	−0.62***	−0.90***	−0.83***	0.67***
FIM + FAM Total	−0.81***	−0.82***	−0.90***	0.92***	0.90***
Change score
NIS Physical	−0.18	0.24*	0.02	0.09	−0.17	−0.20
NIS Cognitive	−0.28*	−0.42***	−0.39**	0.27*	0.35*	0.28*	0.19
NIS Total	−0.29*	−0.07	−0.21	0.22	0.09	0.17	0.82***	0.72***
FIM + FAM Motor	−0.16	−0.45***	−0.34*	−0.06	0.36**	0.12	−0.56***	−0.02	−0.42***
FIM + FAM Cognitive	0.07	0.17	0.13	−0.22	−0.42***	−0.35**	−0.28*	−0.51***	−0.49***	0.34**
FIM + FAM Total	−0.08	−0.23*	−0.17	−0.15	0.05	−0.13	−0.54***	−0.27*	−0.54***	0.88***	0.75***

^*p < 0.05

^**p < 0.01,

^***p < 0.001.

NIS = Neurological Impairment Scale, FIM + FAM = UK Functional Assessment Measure.

Both the NIS and the FIM + FAM showed significant changes between admission and discharge for total scores and subscales (see Table 4). Changes in NIS scores were significantly correlated with change in their respective components of the UK FIM + FAM (r = −0.51 to −0.56). However, although there was a strong negative correlation between change in the FIM + FAM Motor score and the NIS cognitive score on admission (r = −0.45, p < 0.001), no such relationship was seen with the admission NIS physical score.

Within this dataset, approximately two-thirds (n = 49) were “impairment responders” (i.e. their NIS scores reduced by 2 or more points between admission and discharge. The remaining 24 impairment “non-responders” showed no such reduction - indeed 10 of them showed an increase in total NIS score ranging from 1 to 5. Both groups made significant functional gains during in-patient rehabilitation as illustrated in Figure 2. However, the “impairment “responders’ showed a significantly greater change in FIM + FAM motor score (mean change 24.8, sd 17.0) compared with “non-responders” (mean change 11.4, sd 11.0) giving a mean difference of 13.3 (95% CI 6.6, 20.0) t −4.4 p < 0.001. Similarly, impairment responders showed a greater change in FIM + FAM cognitive score (mean change 16.6, sd 12.6) than the non-responders (mean change 6.5, sd 6.6) giving a mean difference of 10.0 (95% CI 5.5, 14.5) t −4.5 p < 0.001.

Figure 2.

Box and whisker plots for change in total FIM + FAM domain scores between admission and discharge in the impairment “responder” and “non-responder” groups. Figure 2 shows box and whiskers plots of the FIM + FAM change scores, in patients who did and did not demonstrate change in the NIS score during their rehabilitation programme. Both groups improved overall, but impairment “responders” made significantly greater gains in both motor and cognitive function than the “non-responders”.

(c) Predictive validity – the NIS as a predictor of outcome and functional gains made during inpatient rehabilitation

To determine whether the NIS adds to the prediction of functional outcome, two linear multiple regression models were tested using total discharge FIM + FAM score as the independent variable.

In model 1, the FIM + FAM total admission score, and NIS physical and cognitive scores on admission all met the criteria for entry and were entered stepwise into the prediction model. The admission FIM + FAM score was entered first and accounted for 82% of the variance. The NIS cognitive score contributed a further 4% and the NIS physical score contributed 2%, so that together they predicted 88% of the total FIM + FAM discharge score.

In model 2, in addition to the FIM + FAM total admission score, NIS physical and cognitive change scores from admission to discharge were entered stepwise into the model. Once again the admission FIM + FAM score accounted for 82% of the variance. The NIS physical change score accounted for a further 5% but the NIS cognitive change score was excluded as a predictor variable.

To examine the factors predictive of change in function during rehabilitation, the following variables were entered stepwise into the model using the FIM + FAM total change score as the independent variable: (a) Admission FIM + FAM total score, (b) Admission NIS Physical and Motor scores, (c) NIS Physical and Motor change scores. Within this model, admission FIM + FAM score and admission NIS physical and cognitive scores and change in NIS cognitive score were all excluded as predictor variables. Only the change in Physical NIS score was entered into the model where it accounted for 29% of the variance.

Discussion

The NIS was developed as a measure of severity of neurological impairment across a broad range of disabling conditions. This first examination of its psychometric properties has provided evidence for its scaling properties, reliability and concurrent and predictive validity. Exploratory factor analysis in a large multi-centre sample demonstrated two distinct principal components, which led to the identification a 10-item sub-scale of physical impairment, and a 7-item sub-scale of cognitive impairment each with acceptable internal consistency. The “Hearing” and “Other” items did not load on either factor, reflecting the preponderance of zero scores in this sample. Whilst this might suggest a degree of item redundancy, when they do occur the impairments captured by these items (deafness, seizures, pressure sores) can potentially have substantial impact on rehabilitation. Therefore, they have been retained in the scale for their clinical importance.

Inter-rater agreement between two doctors was very high for both the total score and in item-by-item analysis, confirming that the NIS is highly reliable when applied to the same information. It was expected that agreement between the medical and MD team ratings would be less strong, due to the time lag between the two assessments. Although overall agreement was still good and ratings were highly comparable for most items, agreement for some was marginal – in particular for behaviour and fatigue – and medical ratings tended to be lower. The MD team ratings were considered to be more reliable as they were based on a longer period of examination by a wider spectrum of clinicians. The findings suggest that, for clinical purposes, the NIS can be applied either by the medical team on admission or subsequently by the MD Team. However, the latter may provide a more comprehensive assessment and is perhaps more likely to identify subtle impairments, such as mood, pain, behaviour and fatigue as they emerge over time, with familiarity.

The expected concurrent and divergent relationships were seen between the physical and cognitive domains of the NIS and UKFIM + FAM. However, the moderate correlations between these measures confirms that the underlying constructs of impairment and disability are distinct, each requiring measurement in their own right.

As in other series that have used the FIM alone, the admission FIM + FAM score was the strongest predictor of outcome [22,23], and other authors have also noted the influence of baseline cognitive function on outcome [22,24]. In this series, functional gains during rehabilitation (as measured by change in total FIM + FAM) were significantly predicted by both the level of cognitive impairment on admission and by change in physical impairment (as measured by the NIS). To our knowledge, this is the first study to examine the influence of changing neurological impairment on functional gains recorded by UK FIM + FAM. Our findings suggest that the NIS can make a useful contribution to the prediction of functional outcome in a mixed-diagnosis group of patients with severe/complex neurological disability.

The authors recognise a number of limitations to this study:

1. The findings from our exploratory factor analysis need to be confirmed in a wider population, preferably from centres outside London.

2. Further examination of the scaling properties of the NIS using modern psychometric techniques (such as the Rasch analysis) is required and indeed is underway.

3. Part 2 of this study was conducted in a single centre and needs to be expanded in a multi-centre analysis, across a range of clinical settings and with other samples of neuro-rehabilitation patients and clinicians.

4. The fact that a tool has been in use for some time does not necessarily mean that it is as good as it can be. As a result of this study, clinical teams have highlighted a number of areas of shortfall in the NIS including the evaluation of musculoskeletal impairment and bladder and bowel dysfunction. Further development is currently being explored to extend its scope and range.

Nevertheless, the findings presented in this first psychometric analysis of the NIS demonstrate it to be a promising measure of neurological impairment, suitable for use across a broad range of neurological conditions. They demonstrate that, even in its current form, the NIS can provide useful information for case-mix adjustment, over and above the admission FIM + FAM score, as a predictor of functional gain. Such information will assist the interpretation of functional outcomes from inpatient rehabilitation of people with complex neurological disabilities.

Appendix 1. Neurological Impairment Set for LTNC and Functional Categories. Version 8

Declarations of interest

This article presents independent research commissioned by the National Institute for Health Research in England (NIHR) under its Health Research Health Services and Delivery Research (NIHR HS&DR) programme (project number 08/1809/235) and its Programme Grants for Applied Research funding scheme (RP-PG-0407-10185). Please visit the NIHR website for more information. The views expressed in this article are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. Financial support for the preparation of this manuscript was also provided by the Dunhill Medical Trust, the Luff Foundation.

Outcome measurement is a specific research interest of our centre. The NIS and the UK FIM + FAM were both developed through this department, but are disseminated free of charge. None of the authors has any personal financial interests in the work undertaken or the findings reported.