An Evaluation of the Euroncap Crash Test Safety Ratings in the Real World

Abstract

We investigated whether the rating obtained in the EuroNCAP test procedures correlates with injury protection to vehicle occupants in real crashes using data in the UK Cooperative Crash Injury Study (CCIS) database from 1996 to 2005. Multivariate Poisson regression models were developed, using the Abbreviated Injury Scale (AIS) score by body region as the dependent variable and the EuroNCAP score for that particular body region, seat belt use, mass ratio and Equivalent Test Speed (ETS) as independent variables. Our models identified statistically significant relationships between injury severity and safety belt use, mass ratio and ETS. We could not identify any statistically significant relationships between the EuroNCAP body region scores and real injury outcome except for the protection to pelvis-femur-knee in frontal impacts where scoring “green” is significantly better than scoring “yellow” or “red”.

Since EuroNCAP was established in 1996, many vehicles have been tested under the crash protocols proposed by this International Association [EuroNCAP, 2005]. The aim of the program is to provide a fair, meaningful and objective assessment of the impact performance of cars [EuroNCAP, 2004 (a)]. The details for the crash protocols and safety rating system are described elsewhere [EuroNCAP, 2004 (b) and EuroNCAP, 2004 (c)]. Briefly, crash test procedures proposed within the EuroNCAP program are based on previous work of the European Enhanced Vehicle Safety Committee [EuroNCAP, 2004 (a)]. The dummy values obtained in different body regions during each crash test determine the safety scores the vehicle receives for each of those body regions, and these scores combined with other characteristics of the performance of the vehicle during the impact, result in a summary overall safety score.

Currently, there are three different crash tests under the EuroNCAP program: frontal, side and pedestrian, together with an optional side pole test in case the vehicle has scored the maximum in the side test. Because of the relative novelty of the pedestrian test we focused this paper on frontal and side impacts only.

FRONTAL IMPACT

The front of the vehicle impacts a deformable barrier at 64 km/h with a 40% overlap to the driver’s side (Figure 1). Two 50^th percentile Hybrid–III dummies sitting in the front seats (driver and passenger) are used in this test. For each dummy, forces impacting four main body areas are measured: 1) head and neck, 2) thorax, 3) pelvis, femur and knee, 4) lower leg, ankle and foot. Using established injury criteria to assess the likelihood of sustaining injuries of a specific severity, different safety points are given. Except in the lower leg region (where the risk assessed is that of AIS 2 tibia fracture), and in the chest (where AIS4 injuries are considered in one of the two injury criteria) the level of severity most commonly assessed is AIS3+. For example, the dummy driver in Figure 1 shows the results of a test indicating that the probability of sustaining a MAIS (maximum AIS) 3 or higher in neck or head was less than 5%. If this probability is not exceeded, the body region receives the maximum safety score and this is presented visually using colored segments within body outlines (in this case green). If this performance is not reached then the body region receives a lower score and a different color. So injury likelihood is represented in a color code where green, yellow, orange, brown, and red indicate increasing probabilities of sustaining more severe injuries. The exact probabilities of MAIS 3+ at which the color code changes from green to yellow, orange, brown or red varies for each specific body region can be found in [EuroNCAP, 2004 (b)].

Figure 1.

Frontal crash test procedure and color scale for driver and passenger. Source: www.euroncap.com.

The score a vehicle receives for frontal impact protection is based on the driver scores, unless any body region for the passenger receives a lower score, in which case that worse score becomes the score for that vehicle’s performance.

SIDE IMPACT

A deformable mobile barrier hits the side of the vehicle between A and B pillars at 50 km/h. Only one Eurosid II dummy (driver position) is sitting at the near side of the impact (Figure 2). Four body regions are considered: head, chest, abdomen and pelvis. Like the frontal test, the safety color for each body region is determined by the probability of sustaining injuries of a specific level of severity (mostly, AIS3+ injuries). Vehicles that have scored the maximum in this test can be optionally tested under the pole impact procedure. This procedure considers a fifth body region (head–neck) in a side impact against a rigid pole at 30 km/h. However, we have not included this optional test in our evaluation. The color code has the same interpretation as in the frontal test. The green star over a driver’s head means that the vehicle has passed the pole test.

Figure 2.

Side crash test procedure and color scale for the driver. Source: www.euroncap.com.

With the information of frontal (driver and front seat passenger) and side (driver) tests, an overall score is produced to generate the commonly known star ratings. A rating system that ranks the overall vehicle safety from 1 star (worst) to 5 stars (best).

As the EuroNCAP documents state, this program has been designed to provide a “fair, meaningful and objective assessment of the impact performance of cars…” and it is stated that “cars that perform well in these tests should provide better protection in accidents than cars which protect less well” [EuroNCAP, 2004 (a)].

The program has not been extensively evaluated in real crashes. To the knowledge of the authors, only four studies from 2000 to 2005 have addressed the evaluation of EuroNCAP. Lie and Tingvall (2000) concluded that cars awarded with three or four stars produced approximately 30% less fatal and serious injuries compared to vehicles receiving less stars, although statistical significance of this estimate was not reported. In addition, the source of the real world data used in this study was Swedish Police data where no information is given about the specific injury sustained by the victims. Fails and Minton (2001) found statistical significance only in some comparisons between different numbers of stars, using CCIS data in the study. The lack of sufficient number of cases was suggested as the most likely explanation for not identifying any more statistically significant findings. Frampton et al. (2004) performed a descriptive analysis to identify the relationship between EuroNCAP ratings for body region protection and real world injury risk. They found that the EuroNCAP scoring seemed to reflect trends in real crash injuries except for the chest, where they did not find correlation. Further improvements to EuroNCAP test procedures are suggested in this paper. Last, Newstead et al. (2005) compared a new crashworthiness scale developed by the Safety Rating Advisory Committee (SARAC, a group of experts which is directly aligned to the European Commission DG TREN) to EuroNCAP scores. They used police reported crashes. They mentioned a trend towards reduced severe injury risk in police reported crashes with increased EuroNCAP star rating. However, they also found that other vehicle factors, apart from those summarized in the overall EuroNCAP score, were determining real crash outcomes.

Thus, we set as our objective to evaluate whether real world data confirmed the expectation that vehicles with better ratings correlate with occupants with less severe injuries.

DATA AND METHODS

We combined real world crash data from the British Cooperative Crash Injury Study (CCIS) with information from the EuroNCAP program.

The CCIS is one of the most comprehensive on-going real world crash investigation programs in Europe. It collects data on approximately 1,500 crashes per year and has been in existence since 1983. The program selects cases for investigation using a stratified random sampling procedure based on injury severity. The accident sampling selects crashes involving towed cars less than 7 years old at the time of the accident in geographical regions selected to represent urban and rural roads in Great Britain. CCIS examines about 80% of serious and all fatal injury crashes meeting the selection criteria. For a comprehensive description of this procedure, the reader is referred to Mackay et al. (1985).

The EuroNCAP official website (www.euroncap.com) provided information on the per body region safety ratings of the vehicles eligible for our study, which was appended to the CCIS data. The body-region specific information is an ordinal scale where colors are used instead of numbers. The best safety rating is represented with the green color, whereas the worst safety rating is represented in red. Values in between are represented in yellow, orange and brown (from best to worst). Thus, the body region safety rating is a 5-point ordinal scale. To allow the statistical models to identify effects beyond linear ones, we created four dummy variables per test to characterize whether a vehicle had obtained a score of yellow, orange, brown or red. Green was the reference category for our analyses. Not all the test results for all makes and models and years are publicly available at the EuroNCAP website. Thus, we set an additional exclusion criteria related to whether the occupants’ vehicle safety information was not readily available.

For this study, data from the CCIS were gathered for the years 1996 –2005. In order to be eligible for this analysis, cases had to be in passenger vehicles of known make, model and year, be of model year 1996 or newer, be drivers or frontal seat passengers, have known safety belt status, known crash severity information (as measured by the Equivalent Test Speed, ETS), be in frontal or near-side impacts as defined by principal direction of force, and have known car-to-car mass ratios.

Since EuroNCAP safety ratings are so closely linked to the probability of sustaining MAIS3+ in different body regions, we created one dummy variable per region and per test to summarize whether the occupant sustained a MAIS3+.

We investigated whether CCIS crashes met conditions similar to (or less severe than) those represented in the crash test. For example, we identified whether occupants in frontal crashes were belted, in crashes of ETS <= 64 Km/h, and against vehicles for which their mass ratio ranged from 0.28 to 3.5. Amongst drivers in side crashes, we investigated whether they were belted, in crashes of ETS <=50 km/h and against other vehicles with mass ratios ranging from 0.28 to 2.2. We excluded from all analysis occupants who were in crashes with undetermined or invalid mass ratio, ETS or safety belt use.

Our regression models were developed using Poisson-based statistical methods because our dependent variables followed Poisson distributions as opposed to Normal ones. We built 12 Poisson multivariate regression models (1 per each of 4 body regions per each of 3 tests –driver frontal, front seat passenger frontal, driver side). The models were applied only to those occupants meeting the inclusion criteria. We ran these models twice. The first time we controlled for confounding factors by restricting the analysis to those cases that met crash conditions less severe or as severe as the ones in the crash test (herein referred to as “restrictive models”).

We then repeated the 12 logistic multivariate regression models, but instead of restricting the analysis to those occupants with crash conditions most similar to the experimental ones, we opened the models to all occupants that met inclusion criteria but whose crash conditions may differ slightly from the experimental ones. That is, in these second set of 12 models (“non-restrictive models”), we controlled for confounding factors by including information on safety belt, mass ratio and crash severity as covariates into the model.

Last, in an effort to increase sample size, and to investigate the robustness of our findings, we grouped together drivers and passengers involved in frontal impacts. We repeated the analyses building 8 Poisson regression models (1 per each of 4 body regions per each test: frontal and side) using the non-restrictive approach described in previous paragraph.

Results are reported as Odds Ratio (OR) and its corresponding 95% confidence intervals (CIs). Statistical significance was established at the p<0.05 value. Analysis were performed using Stata 9.0.

RESULTS

There were 3,862 drivers and 1,219 front seat passengers in frontal crashes, and 1,753 drivers in side crashes in the CCIS data for years 1996–2005. EuroNCAP data on the safety ratings of their vehicles was publicly available in aproximately 50% of cases, reducing the 3 groups of occupants to 1,934, 600 and 909 respectively.

Filtering to obtain those cases where belt use was known and mass ratio and ETS were within valid limits, yielded a total of 688 drivers, 218 front seat passengers and 161 drivers involved in side impacts that were considered for the analyses. Table 1 summarizes the car safety ratings of the vehicles involved. No vehicle was rated as “green” across all four body regions in frontal impacts. In contrast, 4.4% of the vehicles tested under the side impact protocol were rated as “green” across all four body regions considered.

Table 1.

Occupant and crash characteristics

	Frontal		Side
	Driver	Front seat passenger	Driver
Meeting inclusion criteria+	3,862	1,219	1,753
Car information available	1,934	600	909
Belt use, mass ratio, ETS known and reasonable++	688	218	161
Belted (N)	620	202	141
Unbelted (N)	68	16	20
Car safety ratings:
Head/neck or head	688	218	161
Green	373	204	154
Yellow	196	4	4
Orange	102	2	2
Brown	0	0	0
Red	17	8	1
Thorax	688	218	161
Green	22	12	38
Yellow	234	81	41
Orange	226	113	7
Brown	185	12	34
Red	21	0	41
Abdomen	N/A	N/A	161
Green			53
Yellow			48
Orange			45
Brown			11
Red			4
Pelvis/femur/knee or pelvis	688	218	161
Green	46	152	78
Yellow	113	30	47
Orange	218	27	16
Brown	113	9	0
Red	198	0	0
Lower leg/ankle/feet	688	218	N/A
Green	15	93
Yellow	115	120
Orange	140	4
Brown	165	0
Red	253	1
MAIS	688	218	161
0	85	32	14
1	381	119	76
2	103	36	25
3	60	9	14
4	8	3	10
5	14	2	9
6	3	1	5
Unknown	34	16	8
MAIS3++++(yes) N
head/neck or head	20	5	20
Thorax	45	8	29
abdomen	N/A	N/A	9
pelvis/femur/knee or pelvis	49	1	5
lower leg/ankle/feet	8	1	N/A
Fatally injured	26	5	22

⁺Data from 1996 – 2005, passenger vehicles of known make, model year >=1996, drivers or frontal seat passengers, known safety belt status, known crash severity information (ETS), frontal or near-side impacts (principal direction of force), and known mass ratios.

⁺⁺We excluded occupants whose safety belt status was unknown, their mass ratio was <=0 or missing, and ETS was missing.

⁺⁺⁺Variable distributed according a Poisson distribution (μ=σ²).

Table 1 also summarizes the main crash and injury severity characteristics of these occupants. As for the distribution of injuries, 2.8% of occupants in frontal collisions sustained MAIS3+ injuries to the head, 5.8% to the chest and 5.5% to the pelvis-femur-knee. However, only 0.9% of the casualties sustained injuries MAIS3+ to the lower leg-ankle-foot region in frontal crashes. In case of side crashes, 12% of the occupants sustained MAIS3+ injuries to the head, 18% to the chest, 5.6% to the abdomen and 3.1% of the cases sustained severe injuries to the pelvis. 53 occupants died in these crashes, although all of them had MAIS 3+ in at least one body region.

FRONTAL IMPACT

After appending the driver and front seat passenger datasets, 906 occupants were involved in frontal crashes comparable to the EuroNCAP frontal test procedure. The bivariate distribution of MAIS3+ injuries by body region and NCAP safety score for that body region is presented in Table 2. The total number of injuries by body region and NCAP score without considering the AIS level is also presented within this table in brackets. The Chi-Square Test for this contingency table (which uses an underlying normallity distribution assumption) shows significant differences between casualties sustaining MAIS3+ injuries or not depending on the EuroNCAP color of the body region for the pelvis-femur-knee and lower leg-ankle-foot regions (p-value = 0.027 and p-value=0.021, respectively). The other two body regions do not show any significant result.

Table 2.

MAIS3+ injuries according to body region and NCAP score in frontal crashes. Known belt use, ETS and mass ratio⁺.

FRONTAL(driver+passenger)	Head and neck N=856	Chest N=857	Knee, Pelvis, Femur N=858	Leg, Foot N=858*
Green	16 (542)	1 (30)	3 (183)	0 (98)
Yellow	2 (191)	21 (297)	12 (135)	1 (222)
Orange	6 (100)	14 (322)	14 (234)	1 (141)
Brown	0 (0)	15 (187)	5 (116)	0 (155)
Red	1 (23)	2 (21)	16 (190)	7 (242)

⁺The total number of injuries by body region and NCAP score without considering the AIS level is also presented within this table in brackets.

^*N varies slightly because of missing AIS value per body region.

As for the restrictive Poisson multivariate regression models (when only belted occupants in crashes with mass ratios and ETS values similar to EuroNCAP are used), statistically significant results were obtained only for one of the two regions: in the femur-knee-pelvis region, “yellow” and “red” are significatively worse than green (p-values= 0.028 and 0.025, respectively). However, the pseudo R² value of this regression was very low (Pseudo R²=0.03). Similar low values for Pseudo R² (a measure of the variability in dependent variable explained by independent variables) were obtained in all the other models.

In the non-restrictive Poisson multivariate regression models (i.e., those including ETS, mass ratios and seat belt use as covariables, instead of using these variables as restrictive criteria), the pseudo R² value was higher than in the previous analysis ranging from 0.14 (lowest) to 0.29 (highest). Significant results were obtained in all the models with regards to effect of the severity of the crash (ETS), with increased likelihood of sustaining MAIS3+ in all models with unit increases in ETS. Seat belt use was found to be statistically significant in thorax and femur-pelvis-knee protection, with protective effects in decreasing the likelihood of sustaining MAIS3+. Adjusted odds ratios for this reduction were OR=0.44 (95%CI 0.22, 0.85) and OR=0.47 (95%CI 0.23, 0.97), respectively. Mass ratios were also statistically significant when assessing the protection for the pelvis-femur-knee region: increasing mass ratio resulted in an increase of the likelihood of being severely injured (OR=2.42, 95%CI 1.22, 4.78). The only significant EuroNCAP value was the increasing of the injury severity comparing “red” to “green” cars in the femur-pelvis-knee region (p-value=0.049), as it is shown in Figure 3c. However, if the model was further expanded to control by seating position (driver or passenger) this coefficient was found to be also non-significant. Results obtained in these models for every body region are shown in Figure 3 using the point estimates and their 95% CIs. It should be pointed out that, except for the femur-pelvis-knee, all the CI (when convergence is achieved) include the value OR=1, thus no statistical significance is achieved. Moreover, no convergence was achieved in the model related to the protection of lower leg-ankle-foot-region.

Figure 3.

Adjusted OR of front occupants (driver and passenger) in frontal crashes. Non restrictive models (*)

(*) Except for “green” which is the reference value for comparisons of OR, OR=1 with no CI (shown as white square) means that the variable was dropped from the model.

SIDE IMPACT

161 occupants were included in the analyses for side impact. Distribution of MAIS3+ injuries for those occupants where the information about the injury level, the use of seatbelt, the crash severity (ETS value) and the mass ratio of the vehicles involved were known is presented in Table 3. In the bivariate analysis (Chi-Square Test), the only signigicant result was the protection observed in cars with the best safety scores in EuroNCAP for the pelvis region (p-value=0.017).

Table 3.

MAIS3⁺ injuries according to body region and NCAP score in lateral crashes. Known belt use, ETS and mass ratio. ⁺

SIDE(driver)	Head N=153	Thorax N=153	Abdomen N=153	Pelvis N=134*
Green	18 (146)	5 (34)	3 (50)	0 (74)
Yellow	1 (4)	4 (41)	3 (45)	2 (45)
Orange	1 (2)	1 (6)	3 (43)	2 (15)
Brown	0 (0)	7 (32)	0 (11)	0 (0)
Red	0 (1)	12 (40)	0 (4)	0 (0)

⁺The total number of injuries by body region and NCAP score without considering the AIS level is also presented within this table in brackets.

^*N varies slightly because of missing EuroNCAP scores.

However, when the analyses were done using the restrictive Poisson regression model, no significant results were found. The non-restrictive regression model, the one including severity of the crash, use of safety belt and mass ratio showed that ETS values were always significant (i.e., the higher the crash severity, the higher the likelihood of sustaining MAIS3+ injuries in the crash).

Mass ratio was significant (OR=4.18, 95%CI 1.005, 17.41) only in the case of head protection (the higher the mass ratio, the higher the probability of being severely injured in the crash). Safety belt use did not reach statistical significance in any model. No significant results were found for any of the EuroNCAP ratings. Results obtained from these models are presented in Figure 4 with point estimates and 95% CIs in the same way used for Figure 3.

Figure 4.

Adjusted OR of drivers in side crashes. Non restrictive models. (*)

(*) Except for “green” which is the reference value for comparisons of OR, OR=1 with no CI (shown as white square) means that the variable was dropped from the model and OR=0 (shown as grey diamond) means that no convergence was achieved.

DISCUSSION

One of the interesting findings of this paper is to show the feasibility of evaluating the EuroNCAP rating system using an in-depth crash database such as CCIS. The CCIS provides a rich dataset with body region-level information on severity of the occupants and information on the severity and characteristics of the crash. Only one of the previous evaluations of EuroNCAP included non fatal outcomes; and more specifically, no previous evaluation has been able to statistically assess injuries amongst occupants at the body region level and AIS severity level. However, as with many other in-depth crash investigations, a substantial number of cases is lost for analyses because of missing or invalid values in some variables.

In addition to the loses due to the difficulties of gathering these data, we lost another 50% of the cases because information concerning the EuroNCAP scores was not available on the program website. Yet, despite these sample size loses, we have found statistically significant results for some variables in all the analysis carried out.

Interestingly, for frontal impacts, 64% of the evaluated vehicles presented “green” scores for the head region. When analyzing side impacts, these results are even more overwhelming as 96% of the vehicles are rated as “green” in head protection and 44% received the same score for the pelvis region. It has been reported in Alzueta (2005) that in recent years more and more vehicles are reaching the best ratings, which may make an evaluation of the effectiveness of the system even more difficult in the future.

We have built several models to check on the relationship between EuroNCAP scores and the likelihood of being injured (MAIS3+) in a real crash. Our first approach was a bivariate contingency table showing a few significant results. However, this type of analysis (common in other papers) does not incorporate the non-normal distribution of the data under evaluation. Thus, our next analyses included creating Poisson regresion models. These models ranged from more restrictive (stratifying by type of user and type of crash and eliminating subjects in crashes more severe than those represented in the experimental tests) to less restrictive (allowing crash configurations outside the experimental ones). We also investigated merging together drivers and front seat passengers in frontal crashes. Yet, out of all evaluations of the EuroNCAP safety scores, only the comparison of red to green rated cars for the pelvis-femur-knee region in frontal crashes yielded statistical significance. No other significant coefficient was identified for any other body regions in either frontal or in side crashes.

Although statistical power could be argued as one possible reason for the lack of statistical significance, we would like to point out here our multiple significant findings for the covariates in the models, particularly in our evaluation of the frontal tests.

For example, crash severity comes out as a statistically significant risk factor for MAIS3+ in all body regions and crash settings. Interestingly, the use of the seat belt was a significant protective factor in frontal crashes preventing MAIS3+ injuries to the thorax (OR=0.44; 95% CI: 0.22, 0.85) and to the pelvis-femur-knee region (OR=0.47; 95% CI: 0.23, 0.97). Thus, if EuroNCAP scores are not showing significance it may be concluded that the magnitude of their effect is, at best, smaller then that observed for these covariates.

In frontal crashes, we repeated the Poisson regression models including also the position of the passenger (driver or front seat passenger) as covariate. The presence of this variable did not influence the models for the head, the thorax and the lower leg-ankle-foot region (in fact, it was not statistically significant). Nevertheless, in case of the pelvis-femur-knee region, the seating position reached statistical significance changing the result we had obtained when comparing red to green vehicles for this body region. When controling for seating position, none of the EuroNCAP ratings were significant.

It is worth pointing out that the variability explained by the regression models increases considerably when we include the severity of the crash, the use of seat belt and the mass ratios as covariables. We have found high values of pseudo R² in several models (ranging from 14% to 29% in frontal crashes and from 21% to 41% in side impacts). In these models, EuroNCAP scores were not significant (with the exception of the comparison between red and green cars for the pelvis-femur-knee region in frontal collisions) Yet, the same analysis methodology shows a statistically significant relationship between injury outcome and crash severity, belt use and mass ratios. This finding supports the conclusion that EuroNCAP scores, as currently defined, do not explain much of the variability in injury severity found in real crashes.

CONCLUSIONS

We were unable to detect a statistically significant relationship between many of the EuroNCAP body region safety ratings and less frequent and severe injuries amongst drivers and front passengers in frontal or lateral crashes.

Because of our choice of outcome (MAIS3+ per body region) and choice of evaluation of the EuroNCAP scores (at the body region level) it is very difficult to compare our findings with those other evaluations in the literature. From the four evaluations previously mentioned, two of them use Police data [Lie and Tingvall, 2005; Newstead et al., 2005] whilst the other two, use in-depth crash data [Fails and Minton, 2001; Frampton et al., 2004]. Two deal with overall evaluation of the behavior of the car [Lie and Tingvall, 2005; Newstead et al., 2005], but the others consider the specific scores per each body region [Fails and Minton, 2001; Frampton et al., 2004]. One of them [Fails and Minton, 2001] is just a collection of descriptives of cases. Two of them just show crude analysis [Lie and Tingvall, 2005; Frampton et al., 2004] and the last one is the only one to report statistical significance of its findings [Newstead et al., 2005]. None observes the non normal distribution of severity scores.

Given the opportunity, it would be helpful to re-run the analysis adding the 50% of cases where EuroNCAP information was not publicly available. Furthermore, we would like to review the safety scores assigned to some vehicles whose model year was in between EuroNCAP tested model years. For example, if a particular make and model was 1996 and EuroNCAP data for that make and model was available for 1995 and 1997 (but not 1996), we assigned to this car the 1995 ratings. Acting conservatively also, if the vehicle of the example was a 1998 model, we left it with missing EuroNCAP data since we did not have the ability to learn whether significant vehicle redesign had taken place since 1997.

Despite its limitations, we believe this is one of the most comprehensive evaluations to date of the EuroNCAP program. Even though it is possible that power limitations prevent us from finding association between the safety test and real world performance in the case of the lateral test, we were unable to find many associations for the frontal test, using a much larger sample of crashes. Although the concept of a performance experimental test is very appealing and intuitive, it is also likely that the state-of-the-art regarding force limits and actual injury likelihood is still in need of further development. Others have suggested that instead of experimental data, maybe EuroNCAP should be fed with real world crash data evidence [Langwieder et al., 2003]. Hopefully, evaluations such as this allow us to stimulate further research that will cover those gaps.