Re-Searching for Hidden Costs: Evidence from the Adoption of Fuel-Saving Technologies in Light-Duty Vehicles

Hsing-Hsiang Huang; Gloria Helfand; Kevin Bolon; Robert Beach; Mandy Sha; Amanda Smith

doi:10.1016/j.trd.2018.08.009

. Author manuscript; available in PMC: 2019 Dec 1.

Published in final edited form as: Transp Res D Transp Environ. 2018 Dec;65:194–212. doi: 10.1016/j.trd.2018.08.009

Re-Searching for Hidden Costs: Evidence from the Adoption of Fuel-Saving Technologies in Light-Duty Vehicles

Hsing-Hsiang Huang ^a,^*, Gloria Helfand ^b, Kevin Bolon ^c, Robert Beach ^d, Mandy Sha ^e, Amanda Smith ^f

PMCID: PMC6260967 NIHMSID: NIHMS991725 PMID: 30505207

Abstract

A variety of fuel-saving technologies have been implemented in light-duty vehicles since 2012 under the U.S. Environmental Protection Agency’s (EPA) and Department of Transportation (DOT)’s light-duty vehicle greenhouse gas emissions and fuel economy standards. Questions have arisen whether there are hidden costs that have not been included in the net benefit calculations as a result of adoption of the new technologies. In this paper, we replicate and expand results from Helfand et al. (2016). We define hidden costs of the new technologies as problems with operational characteristics such as acceleration, handling, ride comfort, noise, braking feel, and vibration, not all of which are easily measured by objective criteria. We overcome the empirical challenge by using data coded from online professional auto reviews that qualitatively evaluate fuel-saving technologies and operational characteristics for model years 2014 and 2015 vehicles. We estimate relationships of fuel-saving technologies and operational characteristics, including an overall vehicle assessment, and find little correlation of hidden costs with the technologies themselves. Variable quality of implementation of technologies across automakers may better explain negatively evaluated operational characteristics. The results imply that automakers have typically been able to implement fuel-saving technologies without harm to vehicle operational characteristics.

Keywords: light-duty vehicles, fuel economy standards, vehicle greenhouse gas standards, content analysis, vehicle fuel-saving technology, vehicle operational characteristics

1. Introduction

Fossil fuel combustion in transportation has contributed approximately one-fourth of greenhouse gas (GHG) emissions in the United States in recent years (U.S. Environmental Protection Agency (EPA), 2016a). In an effort to reduce GHG emissions and improve energy security, the U.S. Environmental Protection Agency (EPA) and the Department of Transportation (DOT) established vehicle GHG and fuel economy standards for light-duty vehicles for model years (MYs) 2012 through 2025. In the presence of the standards, vehicle manufacturers have implemented a wide range of fuel-saving technologies (EPA 2016b; EPA, DOT and California Air Resources Board (CARB) 2016, Chapter 5).¹ Assessments of the standards have found enormous net benefits to society, including significant net benefits from fuel savings for new vehicle buyers (e.g., EPA and DOT, 2010, 2012; EPA, DOT, and CARB 2016). However, questions have been raised about whether there are hidden costs that have not been included in the net benefit calculations as a result of adoption of the new technologies (Allcott and Greenstone, 2012; Gillingham and Palmer, 2014; Helfand et al., 2016). In particular, hidden costs that exceed the net positive financial benefits from fuel reduction for new vehicle buyers might explain why markets had been slow to adopt fuel-saving technologies on light-duty vehicles in the absence of the standards. Given the wide range of fuel-saving technologies developed and adopted in recent years, it is important to understand whether any of the new technologies impose hidden costs.

We consider hidden costs to be negative impacts of the technologies on performance, drivability, ride comfort, and other characteristics that would cause losses to consumer welfare and are difficult to measure. For instance, if six-speed automatic transmissions were clunky or otherwise worse to drive than traditional four-speed automatic transmissions, buyers of vehicles with six-speed transmissions would suffer welfare losses from the hidden costs and thus would be less interested in buying them. As the effects of the GHG standards depend critically on consumers buying vehicles with fuel-saving technologies, an evaluation of the new technologies should consider potential hidden costs.

One set of literature relevant to hidden costs as a result of adoption of fuel-saving technology has focused on estimating the tradeoffs between fuel economy and horsepower and weight (e.g., Knittel, 2011; Klier and Linn, 2012, 2016; MacKenzie and Heywood, 2015). This literature has focused on estimating this relationship as technological, not involving consumer response; see EPA, DOT, and CARB (2016), Chapter 4.1.3, for further discussion.

This paper is closely related to Helfand et al. (2016), which investigated whether there are hidden costs in a range of operational characteristics. Though operational characteristics that consumers may care about are not well measured by quantified vehicle attributes, they are usually evaluated qualitatively by professional auto reviewers. Helfand et al. (2016) gathered data on both operational characteristics and fuel-saving technologies for MY 2014 by conducting a content analysis of online auto reviews of MY 2014 vehicles. Content analysis involves systematic coding of text; it can be used to convert qualitative information to quantitative (Krippendorff, 2013). They did not find systematic evidence of negative operational characteristics associated with adoption of a variety of fuel-saving technologies, suggesting that it is possible to use the technologies on light-duty vehicles without imposing hidden costs on consumers.

This paper builds on Helfand et al. (2016) in several ways. First, this paper adds evaluations from professional auto reviews for MY 2015 vehicles to the dataset. These additional data provide an opportunity for validation of the results of the original study.

Second, instead of only using cross-sectional variation in technology adoption, the use of year fixed effects allows for control of more unobserved factors that may be correlated with changes in technology adoption and evaluation results, such as changes in consumer preferences. The larger dataset also helps avoid small sample size for some technologies. The estimation results of this paper, using the pooled data and adjusted standard errors dealing with potential small sample bias, are consistent with Helfand et al. (2016)’s conclusion that fuel-saving technologies generally appear not to be associated with negative operational impacts.

Third, to further explore the role of variable implementation quality for fuel-saving technologies, proposed by Helfand et al. (2016), we estimate whether negative evaluations of operational characteristics are correlated with negatively reviewed technologies, conditional on the presence of the technologies. We find evidence of positive relationships between negatively evaluated technologies and negatively evaluated operational characteristics, suggesting that poorly implemented technologies, instead of the presence of the technologies themselves, may be correlated with hidden costs.

Lastly, we examine whether fuel-saving technologies are associated with the overall assessment of the reviewed vehicles concluded by each reviewer. An overall rating, advising whether to purchase the vehicle, may be explicit, or it may be inferred from the evaluation of vehicle characteristics and comparison with vehicles in the same segment. The overall rating is expected to include any factors that the reviewer may consider, even if they are not specifically evaluated. We do not find evidence of associations between negative overall ratings of vehicles and the presence of fuel-saving technologies. Instead, we find negatively rated technologies and negatively rated operational characteristics are highly associated with the overall rating, further suggesting that lower quality technologies, with their adverse effects on operational characteristics, play key roles in getting a negative overall assessment.

These results suggest that, to date, automakers generally have been able to implement fuel-saving technologies without imposing hidden costs on consumers. This finding implies that net benefits from fuel savings suggested in the literature are higher than potential hidden costs of adoption of fuel-saving technology.

The remainder of this paper is structured as follows. The next section describes our data. Section 3 covers our estimation approach. Section 4 presents our results for the relationship of fuel-saving technologies and operational impacts. Section 5 presents our results for the relationship of the technologies with the overall assessment. Section 6 concludes.

2. Data and Content analysis

The data for this study come from online professional auto reviews of MY 2014 and MY 2015 new vehicles. Professional auto reviews provide qualitative evaluation of both technologies and operational characteristics. For both of these categories, quality may be difficult to quantify but is very important to consumers. Hidden costs emerge if negative impacts on these operational characteristics exist as a result of adoption of fuel-saving technology.

Content analysis provides a systematic approach to evaluate reviewers’ evaluations of the quality of fuel-saving technologies and operational characteristics of the vehicles they review. This method involves breaking text into words and phrases that can be categorized and analyzed using specified definitional codes (Krippendorff 2013). Content analysis has been widely used in the humanities and social sciences to classify, measure, and evaluate themes and symbols in various communications media. See Sha and Beach (2015) and Sha et al. (2016) for further background and detail, and Helfand et al. (2016) for further examples of vehicle-related content analyses.

2.1. Identification of relevant websites of professional auto reviews

As detailed in Sha and Beach (2015) and Sha et al. (2016), we followed a set of specific procedures to identify the websites used in this study. In particular, we aimed for websites that contained reviews from professional auto reviewers and that consumers are most likely to consult when making vehicle buying decisions in the United States. First, using Google and Yahoo internet search engines, we sought websites on the first page of search returns for keywords “new cars,” “buying a new car,” and “auto reviews.” Second, we excluded websites that did not have national and professional auto reviews. Third, we used monthly unique views from Quantcast.com and Compete.com to gauge website popularity, and excluded websites that had less than one million unique views in both Quantcast.com and Compete.com. Finally, we screened websites to include only professional reviews that evaluated vehicles and technologies. Each review must have gone beyond a basic specification list, have an independent assessment of vehicle quality, and show evidence of the reviewer having test-driven the reviewed vehicle. For MY 2014 vehicles, six websites were selected by following the sampling procedures above: Automobile Magazine, Auto Trader, Car and Driver, Consumer Reports, Edmunds, and Motor Trend. For MY 2015 vehicles, we started with the six websites for MY 2014 vehicles, and followed the same procedures to identify other potential websites. One new website, Cars.com, was added in MY 2015, because its web viewership met our criteria for inclusion. As in Helfand et al. (2016), this study included all reviews of new MY 2014 and 2015 vehicles subject to the light-duty GHG standards. We dropped the reviews of Volkswagen and Audi diesel vehicles due to concerns over compliance with emissions standards, as well as medium-duty vehicles not subject to the light-duty vehicle standards. Table 1 reports the number of reviews by website in our analysis. Our dataset includes 2,238 separate reviews over the two model years, including 1,003 for MY 2014 and 1,235 reviews for MY 2015.

Table 1.

Auto Reviews by Website

Website	MY 2014		MY 2015		Pooled

	Review Count	%	Review Count	%	Review count	%

automobilemag.com	144	14	138	11	282	13
autotrader.com	224	22	336	27	560	25
caranddriver.com	216	22	202	16	418	19
cars.com	0	0	90	7	90	4
consumerreports.org	86	9	79	6	165	7
edmunds.com	112	11	105	9	217	10
motortrend.com	221	22	285	23	506	23

Total	1,003	100	1,235	100	2,238	100

Open in a new tab

The vehicles reviewed in our sample appear to be roughly representative of vehicles offered, based on data from fueleconomy.gov (U.S. Department of Energy and EPA, 2014; 2015), although the reviews do not reflect sales (see Table 2). Vehicles offered may be a better comparison group than sales, because potential buyers examine auto reviews based on the choice set, rather than what other people buy. Grouped at the vehicle class level as presented in Table 3, the percentage of auto reviews by class is roughly similar to the national fleet-wide breakdown (again based on fueleconomy.gov data) of MY 2014 and MY 2015 vehicles. While the number of reviews of mid-sized cars are over-represented for MY 2014 vehicles, it becomes slightly under-represented for MY 2015 vehicles.

Table 2.

Auto Reviews by Make, Compared with fueleconomy.gov Counts

Make	MY 2014				MY 2015

	Auto Review		fueleconomy.gov		Auto Review		fueleconomy.gov

	Count	%	Count	%	Count	%	Count	%

Acura	24	2.4	16	1.3	22	1.8	10	0.8
Audi	37	3.7	48	3.9	60	4.9	55	4.3
BMW	69	6.9	98	8.0	77	6.2	121	9.4
Bentley	11	1.1	7	0.6	16	1.3	8	0.6
Buick	27	2.7	16	1.3	11	0.9	16	1.3
Cadillac	36	3.6	35	2.8	21	1.7	29	2.3
Chevrolet	85	8.5	77	6.3	101	8.2	92	7.1
Chrysler	4	0.4	14	1.1	28	2.3	13	1.0
Dodge	24	2.4	35	2.8	41	3.3	35	2.7
Ferrari	7	0.7	13	1.1	0	0.0	14	1.1
Fiat	8	0.8	7	0.6	4	0.3	10	0.8
Ford	47	4.7	88	7.2	79	6.4	78	6.1
GMC	17	1.7	36	2.9	21	1.7	51	4.0
Honda	34	3.4	30	2.4	30	2.4	27	2.1
Hyundai	19	1.9	38	3.1	64	5.2	44	3.4
Infiniti	25	2.5	29	2.4	23	1.9	31	2.4
Jaguar	28	2.8	20	1.6	22	1.8	23	1.8
Jeep	42	4.2	35	2.8	15	1.2	39	3.0
Kia	44	4.4	35	2.8	44	3.6	38	2.9
Lamborghini	0	0.0	7	0.6	5	0.4	4	0.3
Land Rover	15	1.5	13	1.1	17	1.4	11	0.9
Lexus	23	2.3	25	2.0	54	4.4	32	2.5
Lincoln	6	0.6	16	1.3	22	1.8	21	1.6
Maserati	0	0.0	6	0.5	1	0.1	6	0.5
Mazda	49	4.9	25	2.0	15	1.2	24	1.9
Mercedes-Benz	74	7.4	85	6.9	84	6.8	83	6.5
Mini Cooper	11	1.1	46	3.7	9	0.7	44	3.4
Mitsubishi	17	1.7	19	1.5	10	0.8	18	1.4
Nissan	40	4.0	51	4.1	54	4.4	50	3.9
Porsche	34	3.4	52	4.2	47	3.8	61	4.8
Ram	7	0.7	13	1.1	8	0.6	15	1.2
Rolls Royce	9	0.9	7	0.6	4	0.3	7	0.6
Scion	4	0.4	9	0.7	8	0.6	7	0.6
Smart	1	0.1	4	0.3	0	0.0	0	0.0
Subaru	25	2.5	23	1.9	59	4.8	21	1.6
Tesla	0	0.0	3	0.2	4	0.3	8	0.6
Toyota	63	6.3	58	4.7	75	6.1	53	4.1
Volkswagen	32	3.2	50	4.1	44	3.6	46	3.6
Volvo	5	0.5	13	1.1	36	2.9	21	1.6
Other^*	0	0.0	27	2.2	0	0.0	18	1.4

Total	1,003		1,229		1,235		1,284

Open in a new tab

Other includes Alfa Romeo, Aston Martin, Bugatti, BYD, Lotus, McLaren, Mobility Ventures LLC, Pagani, Roush, and SRT.

Table 3.

Auto Reviews by Vehicle Class, Compared with fueleconomy.gov Counts

Vehicle Class	MY 2014				MY 2015

	Auto Review		fueleconomy.gov		Auto Review		fueleconomy.gov

	Count	%	Count	%	Count	%	Count	%

Subcompact Cars	84	8.4	101	8.2	111	9.0	107	8.3
Minicompact Cars	11	1.1	52	4.2	13	1.1	56	4.4
Compact Cars	181	18.1	201	16.4	271	21.9	207	16.1
Two Seaters	88	8.8	93	7.6	75	6.1	100	7.8
Midsize Cars	227	22.6	215	17.5	149	12.1	203	15.8
Large Cars	93	9.3	104	8.5	119	9.6	113	8.8
Small Station Wagons	26	2.6	36	2.9	56	4.5	35	2.7
Midsize Station Wagons	6	0.6	4	0.3	21	1.7	8	0.6
Passenger Vans	1	0.1	16	1.3	1	0.1	14	1.1
Minivans	15	1.5	14	1.1	28	2.3	14	1.1
Small SUVs	130	13.0	179	14.6	212	17.2	196	15.3
Standard SUVs	94	9.4	116	9.4	121	9.8	134	10.4
Small Pickup Trucks	1	0.1	14	1.1	27	2.2	24	1.9
Standard Pickup Trucks	42	4.2	54	4.4	28	2.3	45	3.5
Other^*	4	0.4	30	2.4	3	0.2	28	2.2

Total	1,003	100	1,229	100	1,235	100	1,284	100

Open in a new tab

Other includes special purpose vehicle and cargo vans.

2.2. Coding qualitative assessments of vehicle characteristics

This study codes both fuel-saving technologies and operational characteristics discussed in auto reviews. The set of technologies coded included most of the technologies proposed for compliance purposes in EPA and DOT (2010 and 2012). The set of operational characteristics was developed from judgment of factors likely to be relevant to drivers, with refinements based on experience with the reviews. To ensure consistency of coding between MY 2014 and MY 2015 vehicles, and thus allow for better assessment of replication of results, the same coders and coding definitions of fuel-saving technologies and operational characteristics were used for both samples (Sha et al. 2016). One new fuel-saving technology, fuel cell, was added for MY 2015. Table 4 and Table 5 list the coded fuel-saving technologies and operational characteristics, respectively, in the study. A hybrid vehicle is a special case, because all hybrids have stop-start and CVT. For this study, stop-start and CVT are only possible for non-hybrid vehicles; “hybrid” is considered a package including stop-start and CVT.

Table 4.

Total Number of Positive, Negative, and Neutral Evaluations of Fuel-Saving Technologies by Auto Review

Fuel-Saving Technology	MY 2014							MY 2015							Pooled
Fuel-Saving Technology	Negative		Neutral		Positive		Total	Negative		Neutral		Positive		Total	Total

Active Air Dam	0	0%	0	0%	6	100%	6	0	-	0	-	0	-	0	6
Active Grill Shutters	0	0%	0	0%	1	100%	1	1	14%	0	0%	6	86%	7	8
Active Ride Height	0	0%	1	33%	2	67%	3	0	-	0	-	0	-	0	3
Low Resistance Tires	4	24%	5	29%	8	47%	17	4	31%	1	8%	8	62%	13	30
Electronic Power Steering	45	22%	42	20%	121	58%	208	22	14%	19	12%	116	74%	157	365
Turbocharged	20	9%	23	10%	180	81%	223	43	13%	35	10%	264	77%	342	565
GDI	6	9%	6	9%	54	82%	66	4	6%	6	9%	55	85%	65	131
Cylinder Deactivation	1	3%	4	11%	30	86%	35	4	16%	3	12%	18	72%	25	60
Diesel	7	12%	9	15%	44	73%	60	5	28%	2	11%	11	61%	18	78
Hybrid	16	23%	10	14%	45	63%	71	10	21%	5	11%	32	68%	47	118
Plug-In Hybrid Electric	4	14%	6	21%	18	64%	28	4	22%	3	17%	11	61%	18	46
Full Electric	2	9%	6	27%	14	64%	22	0	0%	3	15%	17	85%	20	42
Fuel Cell	0	-	0	-	0	-	0	0	0%	0	0%	1	100%	1	1
Stop-Start	14	27%	7	14%	30	59%	51	15	31%	9	19%	24	50%	48	99
High Speed Automatic	60	14%	81	20%	273	66%	414	96	20%	76	16%	310	64%	482	896
CVT	35	31%	20	18%	57	51%	112	38	30%	14	11%	75	59%	127	239
DCT	16	24%	10	15%	42	62%	68	18	17%	10	10%	77	73%	105	173
Elec Assist / Low Drag Brakes	1	14%	3	43%	3	43%	7	0	0%	0	0%	2	100%	2	9
Lighting-LED	1	5%	2	10%	17	85%	20	0	0%	1	4%	25	96%	26	46
Mass Reduction	0	0%	9	12%	65	88%	74	3	6%	2	4%	43	90%	48	122
Passive Aerodynamics	4	10%	7	18%	29	73%	40	2	11%	0	0%	17	89%	19	59
Efficiency Totals	378	16%	391	16%	1,668	68%	2,437	463	18%	310	12%	1,783	70%	2,556	4,993

Open in a new tab

Table 5.

Total Number of Positive, Negative, and Neutral Evaluations of Operational Characteristics by Auto Review

Operational Characteristics	MY 2014							MY 2015							Pooled
Operational Characteristics	Negative		Neutral		Positive		Total	Negative		Neutral		Positive		Total	Total

Handling
Steering Feel	147	20%	163	22%	442	59%	752	173	21%	119	15%	517	64%	809	1,561
Cornering Ability	92	14%	116	17%	471	69%	679	135	18%	131	17%	500	65%	766	1,445
General Drivability	116	15%	146	18%	531	67%	793	173	19%	130	14%	621	67%	924	1,717
General Handling	82	12%	130	20%	450	68%	662	139	17%	116	14%	571	69%	826	1,488
Acceleration
Acceleration Feel	76	15%	73	15%	343	70%	492	158	26%	47	8%	405	66%	610	1,102
Acceleration Capability	164	16%	231	23%	630	61%	1,025	254	21%	232	19%	744	60%	1,230	2,255
General Acceleration	24	17%	27	19%	89	64%	140	47	18%	39	15%	170	66%	256	396
Braking
Brake Feel	46	13%	58	17%	246	70%	350	99	27%	46	12%	226	61%	371	721
Stopping Ability	31	9%	78	22%	249	70%	358	49	14%	73	21%	228	65%	350	708
General Braking Noise	21	17%	18	15%	83	68%	122	17	17%	14	14%	72	70%	103	225
Tire-Road Noise	72	24%	74	25%	153	51%	299	113	34%	48	14%	172	52%	333	632
Wind Noise	29	14%	46	21%	139	65%	214	50	21%	36	15%	147	63%	233	447
Interior Noise	16	32%	6	12%	28	56%	50	20	40%	4	8%	26	52%	50	100
Powertrain Noise	145	25%	104	18%	330	57%	579	149	24%	68	11%	400	65%	617	1,196
General Noise	58	14%	33	8%	332	78%	423	61	15%	24	6%	332	80%	417	840
Vibration
Chassis Vibration	7	70%	3	30%	0	0%	10	2	67%	0	0%	1	33%	3	13
Powertrain Vibration	10	40%	8	32%	7	28%	25	10	56%	2	11%	6	33%	18	43
General Vibration	19	42%	13	29%	13	29%	45	14	42%	2	6%	17	52%	33	78
Ride Comfort	149	19%	171	22%	462	59%	782	201	22%	135	15%	585	64%	921	1,703
Fuel Economy	161	22%	178	24%	396	54%	735	239	27%	163	19%	474	54%	876	1,611
Range	7	16%	11	24%	27	60%	45	7	18%	8	21%	23	61%	38	83
Charging	3	30%	0	0%	7	70%	10	2	14%	2	14%	10	71%	14	24

Operational Totals	1,475	17%	1,687	20%	5,428	63%	8,590	2,112	22%	1,439	15%	6,247	64%	9,798	18,388

Open in a new tab

Coding processes were also consistent over both datasets. For each auto review, every mentioned fuel-saving technology and operational characteristic was coded as Positive, Negative, or Neutral. For instance, a passage of text containing a negative evaluation of stop-start technology would be coded as Negative – Stop-Start, while another passage of text containing a positive evaluation of steering feel was coded as Positive – Steering Feel. It was coded as Neutral when the reviewer did not demonstrate an intensity of opinion that could be clearly discerned to be positive or negative.

In addition to categorizing each mention of a technology and operational characteristic as positive, negative or neutral, the overall assessment, or recommendation, of the review was coded as well. Review summaries and conclusions often provide a general idea of the reviewer’s attitude about the reviewed vehicle (e.g. a reviewer might opine whether consumers should/shouldn’t purchase a particular car.) Reading the review as a whole led to assigning a positive, negative, or mixed evaluation to each review to capture the overall assessment.

The analysis here uses what Helfand et al. (2016) call review-level data: that is, the codes are aggregated for each review. If a specific technology or characteristic is mentioned multiple times in an auto review and all the codes for the mentions of the technology are the same throughout the auto review, then it is listed once in the relevant column. For instance, if all of the mentions of turbocharging are negative in an auto review, then that review is coded once for Turbocharged – Negative. On the other hand, if a technology or characteristic receives more than one kind of evaluation – for instance, both positive and negative evaluations in the same auto review – the review is coded once for each evaluation of the technology – in this case, once positive and once negative. Helfand et al. (2016) found little difference for results when using individual codes compared to the review-level analysis.

2.3. Summary statistics

Table 4 reports the number of auto reviews that have positive, negative, or neutral evaluations of fuel-saving technologies. The fuel technologies examined are not mentioned very frequently in the reviews. There are about 1.52 and 1.27 codes of fuel-saving technologies per review for the MY 2014 and MY 2015 data, respectively. The most mentioned technologies are high speed automatic, turbocharged, electronic power steering, and continuously variable transmissions (CVT). Among the four most evaluated technologies, turbocharged and high speed automatic have substantially more mentions for MY 2015 vehicles than MY 2014 vehicles.

In the data, positive evaluations exceed negative evaluations for all the technologies examined for both years. As reported in Table 4, in the aggregate, positive evaluations are about 70% of the totals, while negative evaluations are less than 20%. CVT, stop-start, and low rolling resistance tires are the most frequently negatively reviewed fuel-saving technologies for the data. However, even these most frequently negatively reviewed technologies have majority positive evaluations. For example, CVT has 51% and 59% positive evaluations for the MY 2014 and MY 2015 data, respectively, while it has about 30% negative evaluations for vehicles of both model years. These results suggest that it is possible to implement these technologies without significant hidden costs.

Some technologies have limited reviews in a single model year data, such as plug-in hybrid electric, passive aerodynamics, and low resistance tires. As shown in the last column of Table 4, all but five technologies have more than 30 reviews in the pooled data; the exceptions are active air dam, active grill shutters, active ride height, fuel cell, and electric assist or low drag brakes. The technologies with greater than 30 reviews most frequently mentioned positively in percentage terms include LED lighting, mass reduction, gasoline direct injection (GDI), cylinder deactivation, turbocharged, and passive aerodynamics. The technologies with greater than 30 reviews least frequently mentioned negatively in percentage terms are the same ones, except that full electric replaces turbocharged. The most frequently negatively reviewed technologies over the two years by percentage are CVT, stop-start, low-rolling-resistance tires, hybrid, and dual-clutch transmissions (DCT). As noted, though, these all are rated positively for more than 50 percent of the reviews where they are mentioned.

As reported in Table 5, mentions of the operational characteristics in the aggregate have more than 60% positive evaluations, about 20% neutral evaluations, and about 20% negative evaluations. The reviews of operational characteristics are slightly more negative (17% to 22% negative) than the reviews of fuel-saving technologies (16% to 18% negative). Among the operational characteristics, chassis, powertrain, and general vibration have the highest percentage of negative reviews across both model years, followed by interior and tire-road noise. Mentions of vibration are relatively infrequent; only charging for plug-in electric vehicles is mentioned as infrequently. It may be that vibration is mentioned only when there is a problem.

Figure 1 shows a summary of the overall assessments of the vehicles reviewed. Similar to the aggregated operational characteristics, about 65% of vehicles are positively reviewed on the overall assessment. While MY 2015 vehicles have slightly more mixed evaluations than MY 2014 vehicles, only about 8% of the reviews have an overall negative evaluation.

Overall Assessment of the Quality of the Vehicle Reviewed

In Panel (A) of Figure 2, we divide our pooled data into two groups: one (red) includes the auto reviews that mention the technology listed on the vertical axis, and the other (blue) includes the auto reviews that do not mention the technology. Then we compare the shares of auto reviews with an overall negative assessment between the two groups. For instance, while 8 percent of reviews that do not mention GDI have a negative overall assessment, only 2 percent of reviews that do mention GDI have a negative overall assessment. As Panel A indicates, vehicles with most fuel-saving technologies are less likely to have negative overall assessments than vehicles without the technologies, with the exception of vehicles with CVT, hybrid, low-rolling-resistance tires, LED lights, and high speed automatic.

Comparison of Evaluation of Overall Assessment and Operational Characteristics by Technology Using Pooled Data

In Panel (B) of Figure 2, we compare the total number of negative evaluations of operational characteristics between the two groups. It suggests that vehicles with most fuel-saving technologies get fewer negative evaluations of operational characteristics than vehicles without the technologies, with the exception of CVT, hybrid, low-rolling-resistance tires, and plug-in hybrid-electric vehicles.

It is important to note that the difference between the two groups does not imply causality: that is, that the presence of the technology would reduce the likelihood of having negative evaluations of operational characteristics or the overall assessments. Instead, they present the difference in means conditional on mention of a technology between the two subgroups. Many other factors, as we describe later in this paper, are expected to contribute to the difference.

3. Empirical Approach

3.1. Specifications and Estimation

The content analysis data were used to build a linear probability model (LPM) exploring the relationship of fuel-saving technologies with the various operational characteristics. Following Helfand et al. (2016), we run the following LPM as our baseline model predicting I(NegativeOper)_i,j,t, an indicator variable equal to 1 if operational characteristic j was negatively reviewed on model-year t vehicle in auto review i:

I {(N e g a t i v e O p e r)}_{i, j, t} = \sum_{k} β_{k} I {(T e c h)}_{i, k, t} + F i x e d E f f e c t s + ϵ_{i, j, t}

(1)

in which I(Tech)_i,k,t is a vector of k indicator variables representing all fuel-saving technologies examined in this study. The indicator variable is equal to 1 if a technology was mentioned in an auto review. FixedEffects include, at a minimum, website, class, and make fixed effects, to address potential unobserved heterogeneity in factors that might be correlated with both the technology mentioned and the operational characteristic.

When we pool the MY2014 and 2015 data, we also control for the following fixed effects: year (e.g., market conditions common to all manufacturers), year-by-website (e.g., a website’s year-specific review standards and preferences), year-by-class (e.g., year-specific market conditions for a vehicle class common to all manufacturers), and year-by-make (e.g., a company’s year-specific innovation and/or production strategy). The interactions of fixed effects play important roles in identifying our variables of interest β_k, as they control for factors that do not vary over the make-year, class-year, and website-year. Model-by-class fixed effects interacting with model year will also be included in our robustness check, although within model-by-class-by-year variation in the variables of fuel-saving technologies could be reduced.

We estimate this specification using a standard fixed effects regression. A positive coefficient of β_k indicates that the mentioned technology is associated with an increased likelihood of a hidden cost; a negative coefficient, on the other hand, indicates that the technology is associated with a reduction in the likelihood of a hidden cost.

One hypothesis suggested in Helfand et al. (2016) is that, while it appears possible for all the technologies to be used without imposing hidden costs, problems may arise due to variation in quality of implementation of some technologies in some vehicle models. In that paper, the authors compare effects on operational characteristics when the technology is mentioned, to effects on characteristics when the technology receives a negative evaluation; they find more correlations with negatively reviewed characteristics for negatively reviewed technologies than for the presence of the technologies. In this paper, we directly estimate whether negative operational impacts are responsive to a negatively reviewed technology conditional on the presence of the technology:²

I (N e g a t i v e O p e r)_{i, j, t} = \sum_{k} α_{k} I {(N e g a t i v e T e c h)}_{i, k, t} + \sum_{k} β_{k} I {(T e c h)}_{i, k, t} + F i x e d E f f e c t s + ϵ_{i, j, t}

(2)

I(NegativeTech) equals 1 if fuel-saving technology k was negatively reviewed in auto review i of model year t, and equals 0 if technology k was positively or neutrally reviewed, or not mentioned. The underlying idea is that, if a technology was not implemented well (e.g., poor quality in production of the technology, poor installation, and /or poor other adjustments to the technology), auto reviewers would give a negative review for the technology.³

Conditional on the mention of fuel-saving technologies and the fixed effects, this specification directly tests whether a negatively reviewed technology is correlated with negative rating of an operational characteristic. Also, specification (2) seeks to address the selection bias arising from the possibility that a negatively reviewed technology is more likely to be mentioned than a technology that is working well. The coefficients of interest, β_k and α_k, are estimates of the increase in the probability of a negative review due to either the presence of technology k or a negative evaluation of technology k, respectively. A positive value for α_k, especially if combined with a reduction in the value for β_k, supports the hypothesis that poor implementation of technologies is associated with negative ratings of operational characteristics, perhaps more than the mere presence of the technologies.

Lastly, we replace operational characteristics on the left hand side of specification (2) by using the overall vehicle assessment to estimate its relationship with fuel-saving technologies.

3.2. Potential Estimation Concerns

Including the fixed effects is useful for identifying the relationship between fuel-saving technologies and reviewed operational characteristics by addressing a variety of potential confounders. Yet, we recognize that there remains a possible selection bias. For instance, if some technologies were put into low-quality vehicles, and the low quality is not fully controlled for by the fixed effects, the estimated coefficient would be biased upward. As a result, we describe the results as correlations between technologies and operational characteristics rather than claiming causality. We thus focus on whether statistically significant positive coefficients are consistently estimated from our regression models.

It is important to note that absence of a mention in a review does not mean that the technology is absent; it means that the reviewer did not mention it. It is plausible that auto reviewers would notice and comment on undesirable features more than on positive features. If so, the estimated relationship between a technology and a hidden cost may be biased upward.

In addition, one concern with LPM is that estimated probability may be not bounded between −1 and 1. However, with binary dependent variables, LPM has some advantages over logit models in that causal analysis is valid and does not require functional form assumptions about the error term (Angrist and Pischke, 2009). In fact, almost all of our estimated coefficients in this study are between −1 and 1.

Last, while robust standard errors are generally used for non-constant error variance with LPM, they are subject to small sample bias and high sampling variance (Angrist and Pischke, 2009, p. 307). As a result, robust standard errors may be too small by accident and thus increase rejection of the null hypothesis. In this study, in additional to using conventional robust standard errors, we follow the suggestion of Angrist and Pischke (2009) of using the maximum of the conventional standard error and a robust standard error as our best measure of precision.

4. Estimated Relationship of Technologies and Operational Characteristics

This paper focuses on examining whether the estimated relationships using multiple datasets (MY 2014, MY 2015, and pooled) with a rich set of fixed effects are consistent and robust. For each dataset, we run a separate regression of each of 22 operational characteristics on all fuel-saving technologies examined and a set of fixed effects. There are 20 technologies for MY 2014 data and 21 for MY 2015 data. Thus, we have 440 estimated coefficients for the fuel-saving technologies for MY 2014 data (20 coefficients for each operational characteristic), and 462 estimated coefficients for MY 2015 and for the pooled data (21 coefficients for each operational characteristic).

In this section, first, we present the estimated results of the initial specification, separately for each model year. Second, we report detailed results using the pooled data, with its advantage of more observations as well as additional fixed effects (i.e., website-year, class-year, and make-year). This section also includes robustness checks.

4.1. Overview of the Results across Datasets

Figure 3 provides the number of significant coefficients in the 22 regressions across alternative datasets based on significance at the 10 percent level, and use of the measure for standard errors suggested by Angrist and Pischke (2009); the detailed estimated results are reported in Appendix Table A.1 and Table A.2 for MY 2014 and MY 2015 vehicles, respectively. Recall that a positive coefficient indicates that fuel-saving technology is associated with a negative review of operational characteristic. As Figure 3 shows, only 2.7% (12 out of 440) of coefficients for the MY 2014 data, and 4.5% (21 out of 462) of coefficients for both the MY 2015 and the pooled data are positive and statistically significant. The results for positive relationships, our focus, do not seem to be sensitive to the standard errors we use. The general pattern of MY 2014 results, of few positive and statistically significant coefficients, continues for MY 2015.

Overview of Estimation Results using Alternative Datasets and Standard Errors (10% significance level)

Only four of the 12 coefficients, or about 1 percent of the 440 coefficients for MY 2014, that are positive and statistically significant using MY 2014 data remain positive and statistically significant using MY 2015 data: hybrid is associated with a negative rating for brake feel, plugin hybrid electric is associated with a negative rating for powertrain noise, and CVT is associated with negative ratings for general drivability and powertrain noise. The small number of consistently significant associations between technologies and negatively reviewed characteristics raises the question whether they are significant by chance. On the other hand, in a single model-year, the small sample sizes for some technologies may make statistically significant relationships difficult to detect.

Except with the maximum of the conventional standard error and a robust standard error for the MY 2015 data, there appear to be more statistically significant negative coefficients than positive ones: that is, there may be more cases of hidden benefits than hidden costs. It is noteworthy that the number of negative relationships is substantially reduced (from over 60 to under 20 in either MY 2014 and MY 2015 reviews) using the approach suggested by Angrist and Pischke compared to the estimated results using robust standard errors. This observation demonstrates how this approach can affect interpretation of results by creating a more stringent standard for significance.

4.2. Results of Pooled Data

The results from estimating specification (1) with the pooled data from MY 2014 and MY 2015 include year, year-by-website, year-by-class, and year-by-make fixed effects, in addition to the fixed effects from the individual-year analyses. The estimation results for all operational characteristics are summarized in Figure 3 and detailed in Table 6.

Table 6.

Relationships between the Presence of Fuel-Saving Technologies and Negatively Reviewed Operational Characteristics, Pooled MY 2014 and 2015 Data

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)	(11)

	Negative Steering Feel	Negative Cornering Ability	Negative General Drivability	Negative General Handling	Negative Acceleration Feel	Negative Acceleration Capability	Negative General Acceleration	Negative Brake Feel	Negative Stopping Ability	Negative General Braking	Negative Tire-Road Noise

Active Air Dam	−0.00	−0.11	−0.02	0.23	0.01	−0.13	−0.06	0.03	−0.01	0.01	0.04
Active Grille Shutters	−0.10*	−0.12***	−0.09**	−0.05	0.10	−0.12	−0.02	−0.04	−0.05**	−0.02*	−0.08*
Active Ride Height	−0.24**	−0.04	−0.01	−0.03	−0.16***	0.17	0.00	−0.02	0.01	0.02	0.01
Low Resistance Tires	0.01	0.14	0.02	0.01	−0.02	−0.08	−0.00	−0.09**	0.05	0.02	0.06
Electronic Power Steering	0.11***	0.03	0.01	−0.02	−0.00	0.01	−0.01	0.01	0.02*	−0.01	0.04*
Turbocharged	0.03	0.00	0.00	−0.01	−0.00	0.04*	−0.01	0.01	0.00	−0.01	0.03
GDI	−0.03	−0.06***	−0.02	−0.02	−0.00	0.03	−0.01	−0.03	0.01	−0.00	−0.00
Cylinder Deactivation	0.07	0.02	−0.02	−0.05	0.03	0.01	0.04	0.09*	0.02	−0.02*	−0.02
Diesel	0.03	−0.01	−0.04	−0.01	−0.01	0.12*	0.06*	−0.02	0.01	−0.02*	−0.02
Hybrid	0.02	−0.06*	0.05	−0.02	0.05	0.01	0.02	0.12***	−0.01	0.01	−0.05*
Plug-In Hybrid Electric	−0.00	0.05	−0.01	0.05	−0.03	−0.01	0.01	0.21**	0.00	0.02	0.02
Full Electric	0.04	−0.04	0.08	−0.04	0.04	0.10	0.03	0.08	0.01	0.01	−0.05
Stop-Start	−0.01	−0.06**	−0.06*	−0.06*	−0.03	−0.09**	−0.00	−0.05	−0.00	−0.01	−0.02
High Speed Automatic	−0.02	0.00	−0.03**	−0.00	−0.01	−0.04**	−0.01	0.01	0.00	−0.01	0.00
CVT	0.02	0.07**	0.11**	0.04	0.02	0.14***	−0.00	0.01	0.03	0.02	0.02
DCT	0.01	0.07**	−0.00	0.02	0.01	−0.02	0.00	0.02	0.00	−0.01	0.06*
Elec Assist / Low Drag Brakes	−0.03	−0.08*	−0.07*	−0.05	0.02	−0.04	−0.05**	−0.07**	0.05	−0.01	−0.04
Lighting-LED	−0.06	−0.04	0.04	0.06	−0.07*	−0.06	0.00	−0.04	−0.03	−0.02	0.09
Mass Reduction	0.02	−0.00	−0.06***	−0.01	−0.05***	−0.03	−0.02	0.04	−0.02	0.00	−0.01
Passive Aerodynamics	−0.05	−0.03	−0.05	−0.01	−0.00	−0.01	0.02	0.05	−0.02	0.03	0.01
Fuel Cell	−0.18***	−0.17***	−0.29***	−0.04	−0.11*	−0.32***	−0.03	−0.02	−0.05	0.02	−0.06

	(12)	(13)	(14)	(15)	(16)	(17)	(18)	(19)	(20)	(21)	(22)

	Negative Wind Noise	Negative Interior Noise	Negative Powertrain Noise	Negative General Noise	Negative Chassis Vibration	Negative Powertrain Vibration	Negative General Vibration	Negative Ride Comfort	Negative Fuel Economy	Negative Range	Negative Charging

Active Air Dam	−0.08	0.01	−0.06	−0.09	−0.01	−0.01	0.01	0.02	0.03	0.00	0.00
Active Grille Shutters	−0.01	0.00	−0.13**	−0.02	0.00	−0.01	−0.00	−0.11**	−0.03	−0.01	−0.00
Active Ride Height	0.02	−0.01	−0.23***	0.02	0.00	−0.01	−0.00	0.01	−0.20**	0.01	0.01
Low Resistance Tires	−0.04**	−0.01	−0.00	0.05	−0.00	0.03	0.03	−0.03	−0.11**	0.04	−0.01
Electronic Power Steering	0.00	−0.00	0.00	−0.02*	0.01	0.00	−0.00	−0.00	0.01	−0.00	−0.00
Turbocharged	0.01	0.01*	0.03	−0.02	0.00	0.01**	0.00	−0.01	0.02	−0.01**	−0.00
GDI	0.01	−0.01	0.01	0.03	−0.01	−0.00	0.01	0.01	−0.05	0.00	0.00
Cylinder Deactivation	−0.04***	−0.01	0.01	−0.03	−0.02*	0.00	0.01	0.09	−0.15***	0.00	−0.00
Diesel	−0.01	0.01	−0.05	0.01	−0.00	0.01	−0.01	−0.11**	−0.09*	−0.00	0.00
Hybrid	−0.00	−0.01	0.02	−0.00	−0.00	−0.01*	−0.01*	−0.01	−0.00	−0.01**	−0.00
Plug-In Hybrid Electric	−0.01	0.01	0.40***	−0.02	−0.00	0.00	0.05	−0.04	0.07	0.22***	0.05
Full Electric	0.00	−0.02*	−0.13**	−0.05	−0.00	−0.02	−0.02*	−0.04	−0.13***	0.02	0.02
Stop-Start	−0.06***	−0.02*	−0.02	0.00	−0.01	0.01	0.02	−0.04	−0.05	0.01	−0.00
High Speed Automatic	−0.02*	−0.01	0.01	−0.04***	0.00	−0.00	−0.00	0.01	−0.02	−0.01***	−0.00*
CVT	0.05**	0.01	0.14***	0.02	0.01	0.00	0.01	−0.09**	−0.07	−0.01	−0.00
DCT	−0.02	0.02	0.01	−0.00	0.01	0.00	0.01	0.09**	−0.01	−0.01	−0.00
Elec Assist / Low Drag Brakes	−0.03	−0.01	0.03	−0.07*	−0.01	−0.01	−0.01	−0.01	−0.11*	0.12	−0.01
Lighting-LED	−0.05***	0.05	−0.05	−0.04	0.01**	−0.01	0.00	0.00	−0.06	−0.00	0.00
Mass Reduction	−0.01	−0.01	−0.01	−0.02	−0.01	0.00	0.00	0.10***	−0.06**	0.01	0.01
Passive Aerodynamics	0.02	0.01	−0.08**	−0.03	−0.01	−0.02**	−0.03***	−0.06	−0.01	−0.00	−0.00
Fuel Cell	0.00	0.01	0.02	0.05*	−0.00	−0.00	−0.04	−0.19***	−0.42***	−0.01	−0.00

Open in a new tab

Notes: Number in a cell indicates the estimated probability of negative rating tor the column variable conditional on technology named in the row. Number in bold means the significant result holds when we use the maximum of robust standard errors and conventional standard errors as the best measure of efficiency, instead of using robust standard errors. For all columns, the sample size is 2,238 from the pooled data of model years 2014 and 2015. For each column, the linear probability model regresses the column variable on all 21 technologies and a set of fixed effects, including year, website, vehicle class, vehicle make, website-by-year, class-by-year, and make-by-year. Asterisks indicate the level of statistical significance: 10% (*), 5% (**), and 1% (***) levels.

Similar to the results with single-year datasets, there continue to be relatively few cases of fuelsaving technologies correlated with a negative rating for an operational characteristic, i.e., positive coefficients. Out of 462 coefficients, 21 coefficients are positive and statistically significant using the measure of precision suggested by Angrist and Pischke; with robust standard errors as our measure of precision, 24 coefficients are statistically significant and positive. In addition, the magnitudes are small; only 5 of the 462 coefficients are associated with an increased probability of a negative impact of more than 0.15.

Among the 21 positive and significant coefficients, five coefficients associate CVT with negative evaluations, for cornering ability, general drivability, acceleration capability, wind noise, and powertrain noise. These coefficients are fairly small: the maximum estimated increased probabilities are 0.14, for acceleration capability and powertrain noise. Plug-in hybrid vehicles have three of the largest positive significant coefficients, for brake feel (0.21), range (0.22), and powertrain noise (0.4).

We note again that the correlations may be affected by unobserved variables. For instance, it is possible that, instead of CVT itself, lower quality implementation of CVT contributes to the negative associations with negative rating of operational characteristics. Also, it is possible that CVTs were put into vehicles with a relatively loud powertrain noise or other problems that are not captured by the fixed effects. We consider these concerns in the next two subsections.

4.2.1. Variation in Implementation Quality for Fuel-Saving Technologies?

The analyses above are based on the presence of the technology; negative effects associated with the presence of the technology may be due to an inherent property of the technology. In contrast, specification (2) seeks to distinguish between problems that are inherent to a technology, and problems associated with particular use, installation of the technology, or other adjustments made to the vehicle with the technology. We do so by including variables for both the presence of the technology (β_k) and for a negatively reviewed technology (α_k).

Complete estimated results of α_k, the coefficient for negatively reviewed technology k, are reported in Table 7, while complete estimated results of β_k, the coefficient for any mention of technology k, are in Appendix Table A.3. Summarized in Figure 4, the results using the pooled data and the Angrist and Pischke measure of precision find 57 out of 462 α_ks are positive and statistically significant, while five coefficients are negative and statistically significant. In addition, for β_k, eight out 462 are positive and statistically significant, substantially less than the 21 positive coefficients from estimating specification (1). Because a positive coefficient indicates a fuel-saving technology (either mentioned (β_k) or negatively reviewed (α_k)) is associated with a negative review of an operational characteristic, the results suggest that negatively reviewed technologies, instead of the presence of the technologies themselves, are more likely to be associated with negative ratings of operational characteristics. The results suggest that vehicles that did not get negative evaluations on fuel-saving technologies may have been able to implement the technologies without harm to operational characteristics. We repeat, though, that these data are not sufficient to demonstrate causality. For instance, it is possible that technologies are negatively reviewed most often in the context of a negatively reviewed operational characteristic.⁴

Table 7.

Estimated Relationships (a_k) between Negatively Reviewed Fuel-Saving Technologies and Negatively Reviewed Operational Characteristics, Pooled MY 2014 and 2015 Data

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)	(11)

	Negative Steering Feel	Negative Cornering Ability	Negative General Drivability	Negative General Handling	Negative Acceleration Feel	Negative Acceleration Capability	Negative General Acceleration	Negative Brake Feel	Negative Stopping Ability	Negative General Braking	Negative Tire-Road Noise

Active Air Dam	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Active Grille Shutters	−0.06	0.05	0.14	−0.04	−0.12	−0.01	−0.05	0.16***	−0.02	0.05**	0.11*
Active Ride Height	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Low Resistance Tires	0.12	0.33	0.41**	0.21	0.17	0.05	0.14	−0.05	−0.04	−0.08	−0.06
Electronic Power Steering	0.80***	0.14**	0.10**	0.04	0.02	0.03	−0.03*	0.03	0.05	0.01	0.04
Turbocharged	0.12**	0.13**	0.15**	−0.01	0.27***	0.34***	0.03	0.01	0.03	−0.00	0.03
GDI	0.14	−0.08**	−0.14**	−0.16***	0.16	0.14	−0.03	−0.02	−0.08**	−0.01	0.10
Cylinder Deactivation	0.18	0.11	0.15	0.38*	−0.02	−0.15*	−0.09**	−0.19***	−0.06	−0.00	0.18
Diesel	−0.09	−0.01	0.04	0.08	0.09	0.45***	0.27**	−0.03	0.07	−0.01	−0.07
Hybrid	0.02	−0.00	0.17	0.02	0.07	0.22**	0.05	0.11	−0.01	0.05	−0.02
Plug-In Hybrid Electric	0.06	0.03	−0.13	0.01	0.05	−0.04	0.03	−0.22	0.15	−0.03	−0.00
Full Electric	0.47	−0.06	−0.10	0.02	0.51	0.37	0.52*	0.48	−0.12	−0.05	0.01
Stop-Start	0.05	−0.11**	0.10*	0.06	0.12	−0.04	−0.01	−0.02	0.08	0.03	0.14*
High Speed Automatic	0.10***	0.08**	0.07**	0.09***	0.19***	0.15***	0.03*	0.06**	0.02	0.00	0.03
CVT	0.05	0.05	0.17**	0.17***	0.14**	0.33***	0.03	−0.00	0.00	−0.04*	0.05
DCT	−0.03	0.12	0.24***	0.07	0.15**	0.02	0.04	0.08	0.08	0.00	0.09
Elec Assist / Low Drag Brakes	−0.16	0.01	−0.15*	−0.02	0.99***	0.90***	0.02	0.02	−0.05	−0.01	0.03
Lighting-LED	−0.13	−0.17**	−0.25***	−0.28***	−0.06	−0.10	0.01	−0.08	−0.13	0.01	−0.27***
Mass Reduction	0.76***	0.19	−0.13	0.65**	0.03	−0.37***	0.01	−0.11**	−0.07	0.01	−0.04
Passive Aerodynamics	0.01	−0.06	−0.08	0.31	0.39**	0.31*	0.34*	−0.18***	−0.04	−0.05	0.08
Fuel Cell	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

	(12)	(13)	(14)	(15)	(16)	(17)	(18)	(19)	(20)	(21)	(22)

	Negative Wind Noise	Negative Interior Noise	Negative Powertrain Noise	Negative General Noise	Negative Chassis Vibration	Negative Powertrain Vibration	Negative General Vibration	Negative Ride Comfort	Negative Fuel Economy	Negative Range	Negative Charging

Active Air Dam	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Active Grille Shutters	0.01	−0.00	−0.02	0.03	−0.01	0.02	−0.04	−0.06	−0.11	0.00	0.00
Active Ride Height	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Low Resistance Tires	0.04	−0.02	0.14	0.18	−0.02	−0.08	0.09	0.35*	−0.03	−0.01	−0.01
Electronic Power Steering	0.01	−0.01	−0.03	0.07*	0.02	−0.01	−0.01	−0.00	0.02	0.01	0.00
Turbocharged	−0.02	−0.02	0.15***	0.01	0.02	0.03	−0.00	−0.01	0.16**	0.00	0.00*
GDI	−0.00	−0.04**	0.19	0.00	0.00	−0.01	0.21*	−0.04	0.02	−0.01	−0.00
Cylinder Deactivation	0.02	0.01	0.30	0.04	0.02	0.00	−0.05	−0.03	0.26	0.01	−0.00
Diesel	−0.06	−0.03	0.29**	−0.04	−0.00	−0.03*	0.02	−0.03	0.18	−0.00	−0.01
Hybrid	0.01	−0.04	0.13	0.03	0.00	0.00	−0.00	0.06	0.15	0.01	−0.00
Plug-In Hybrid Electric	−0.02	0.01	0.25	−0.06	0.01	0.01	0.14	0.01	0.16	−0.27***	0.08
Full Electric	0.05	0.04	0.26	0.65**	−0.02	−0.01	−0.06	0.01	−0.05	1.00***	−0.06
Stop-Start	0.06	−0.01	−0.02	0.05	−0.00	−0.01	0.04	0.20**	0.05	−0.02	0.00
High Speed Automatic	0.02	0.03*	0.07**	−0.00	−0.01*	0.02*	−0.00	0.13***	0.12***	0.00	−0.00
CVT	0.10**	0.01	0.26***	0.06	0.01	0.01	0.03	0.15**	0.14**	0.00	−0.00
DCT	−0.00	0.08	−0.02	0.04	−0.01	0.03	0.07	0.20**	−0.01	0.00	−0.00
Elec Assist / Low Drag Brakes	−0.06	−0.01	−0.07	0.04	0.05*	0.02	0.02	0.03	0.21**	−0.12	−0.07
Lighting-LED	−0.06	−0.07	−0.06	0.81***	0.03**	0.01	0.01	−0.46***	0.07	0.02	0.00
Mass Reduction	−0.00	−0.00	0.50	0.31	−0.01	−0.02	−0.01	0.45	0.11	−0.01	−0.00
Passive Aerodynamics	0.24	0.13	0.01	0.01	0.01	−0.02	0.01	−0.10	0.38*	−0.00	−0.01
Fuel Cell	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Open in a new tab

Notes: Number in a cell indicates the estimated probability of negative rating for the column variable conditional on technology named in the row. Number in bold means the significant result holds when we use the maximum of robust standard errors and conventional standard errors as the measure of efficiency, instead of using robust standard errors. For all columns, the sample size is 2,238. For each column, the linear probability model regresses the column variable on all technologies that were negatively reviewed and a set of fixed effects, including year, website, vehicle class, vehicle make, website-by-year, class-by-year, and make-by-year. Asterisks indicate the level of statistical significance: 10% (*), 5% (**), and 1% (***) levels.

Overview of Estimation Results for Variation in Implementation Quality (10% significance level)

Using CVT as an example, Table 7 shows that eight of the 57 positive coefficients (and one of the 21 negative coefficients) of α_k are with CVT, and there are no positive coefficients (but two negative coefficients) of β_k with CVT. One explanation for this finding is that poor implementation of CVT (from the negative reviews of CVT in certain models, α_k), rather than CVT itself (from mention of the technology, β_k), is related to negative rating of operational characteristics. Recall that specification (1) found that the presence of CVT was associated with negative ratings for five operational characteristics; specification (2) suggests that negatively evaluated CVTs may contribute to negative ratings related to drivability, acceleration, noise, ride comfort, and fuel economy, as shown in Table 7.

Similarly, negatively reviewed high speed automatic and turbocharging technologies are associated with negative ratings of 13 and 7 operational characteristics, respectively, while the presence of the two technologies shows little relationship to negatively reviewed operational characteristics based on both specifications (1) and (2). This observation suggests that specification (2) may provide a clearer signal of the difference between the effects of the presence of a technology, and the effects of a poorly reviewed technology, on operational characteristics.

These results are consistent with Helfand et al. (2016)’s proposal that quality of implementation, rather than technologies themselves, is associated with vehicle qualities.

4.2.2. Potential Model-by-Class-by-Year Specific Unobserved Vehicle Attributes

Another potential concern that could affect our findings is that fuel-saving technology put into vehicles might be systematically related to other attributes or issues that are unobserved by researchers. To assess the concern, we adjust specification (2) by using model-by-class fixed effects interacted with model year, instead of using make-by-year and class-by-year fixed effects.⁵ Coefficients of α_k and β_k are identified within model-class-year variation in whether the fuel-saving technologies we examine are present or not. These fixed effects control for all vehicle characteristics that stay constant for the model-class in a model year. For example, vehicle characteristics in the error term in the previous specifications that are common to the Toyota Highlander standard SUVs in MY 2014 and would affect the operational characteristics will be controlled by these fixed effects separately from vehicle characteristics common to Toyota Highlander small SUVs.

The inclusion of these fixed effects still does not completely address the potential for selection bias. For instance, if a CVT were put into a trim that is already noisy, our estimates of β_k would be biased upward for the relationship between powertrain noise and the technology. In addition, these fixed effects will reduce the variation of fuel-saving technologies for some models. For instance, there is no variation of hybrid technology within the model Toyota Prius C; the estimated results would be based on other models adopting hybrid technology. We present the analysis for the purpose of a robustness check. If the results show a different pattern of the relationships between fuel-saving technologies and operational characteristics, our findings above may be affected by selection bias.

Using model-by-class fixed effects interacting with model year, estimated results of adjusted specification (2) are similar to the results in subsection 4.2.1. Here, 41 out of 462 coefficients of the negatively reviewed technologies (α_k ) are positive and statistically significant, as reported in Table A.4, compared to 57 with the more limited fixed effects (in Table 7); as in the previous specification, 5 out of 462 coefficients of the presence of the technologies (β_k ) are positive and statistically significant, as reported in Table A.5. Negatively rated CVT, turbocharging, and high speed automatic show a pattern of relationships with negative ratings of operational characteristics consistent with the results in Table 7.

In sum, controlling for the model-class-year-specific effects provides a similar pattern of relationships between fuel-saving technologies and operational characteristics. That is, positive and significant relationships between the presence of fuel-saving technologies and negative rating of operational characteristics are a small proportion of the possible relationships, and are outnumbered by the 18 negative and significant relationships. Also, we continue to find that negatively reviewed technologies, instead of the presence of the technologies themselves, are more likely to be associated with negative ratings of operational characteristics. The results suggest that there might be no serious selection bias, though we still cannot rule out a potential bias arising from the possibility that the presence of a new technology is endogenous for a trim within a vehicle model.

5. Relationship of Technologies and Overall Assessment of the Vehicle

We have shown in the previous section that the presence of fuel-saving technologies is infrequently associated with negative evaluations of operational characteristics; when there is a relationship, it is slightly more likely negative, implying hidden benefits, than positive. Although some technologies show a positive relationship with several negative operational characteristics, problems with implementation of the technologies may be a better explanation for the relationship than the existence of the technology.

It may also be useful to know how the technologies are associated with the overall summary evaluation of a vehicle – the recommendation whether a vehicle is worth buying. Auto reviewers’ overall rating is usually highlighted after consideration of vehicle characteristics and comparison with similar vehicles in the same vehicle segment. It is expected to summarize all positive and negative impacts on vehicle quality, including any impacts not separately addressed in the review. The overall rating is expected to matter for vehicle buyers’ decisions, as it provides a recommendation about whether it is reasonable or not to purchase the vehicle among the models sharing similar features in the market.

In this section, we investigate the relationship between the overall rating and fuel-saving technologies, using a coded variable denoting that the overall assessment provided by a reviewer is positive, negative, or mixed. We begin by substituting the left hand side of specification (2) with (NegativeOverall), an indicator variable for whether the vehicle reviewed got a negative overall assessment, to obtain the following specification:

I {(N e g a t i v e O v e r a l l)}_{i, t} = \sum_{k} α_{k} I {(N e g a t i v e T e c h)}_{i, k, t} + \sum_{k} β_{k} I {(T e c h)}_{i, k, t} + F i x e d E f f e c t s + ϵ_{i, t}

(3)

Columns (1a) and (1b) of Table 8 report the results of estimating specification (3) using the maximum of conventional and robust standard errors as our measure of precision. We do not find evidence that the presence of fuel-saving technologies is positively related to negative overall assessments, as shown in (1a). Instead, we find that the presence of three technologies – turbocharged, cylinder deactivation, and high speed automatic – is associated with a reduced likelihood of a negative overall assessment. In addition, all estimated coefficients are less than 0.1 in absolute value no matter whether they are statistically significant. The fairly small coefficients indicate that the overall rating of a vehicle appears not to be responsive to the presence of the fuel-saving technologies.

Table 8.

Relationship between the Presence of a Fuel-Saving Technology, a Negatively Reviewed Technology and Operational Characteristic, and the Overall Assessment of the Vehicle

	(1a)	(1b)	(2a)		(2b)

Technology	Presence of Technology	Negative Review of Technology	Presence of Technology	Operational Characteristics	Negative Review of Operational Characteristics

Active Air Dam	−0.02	0.00	0.00	Steering Feel	0.06***
Active Grill Shutters	−0.08	−0.02	0.00	Cornering Ability	−0.01
Active Ride Height	−0.02	0.00	−0.02	General Drivability	0.15***
Low Resistance Tires	−0.03	0.29*	0.08	General Handling	0.09***
Electronic Power Steering	−0.02	0.08**	−0.01	Acceleration Feel	0.07***
Turbocharged	−0 04***	0.12**	−0.03*	Acceleration Capability	0.08***
GDI	−0.03	0.02	−0.03	General Acceleration	0.06
Cylinder Deactivation	−0.07*	−0.06	−0.07*	Brake Feel	0.09***
Diesel	−0.05	−0.08	−0.06*	Stopping Ability	0.02
Hybrid	−0.04	0.21**	0.00	General Braking	−0.02
Plug-In Hybrid Electric	0.03	−0.05	−0.04	Tire-Road Noise	0.03
Full Electric	−0.04	0.03	−0.05	Wind Noise	0.02
Stop-Start	−0.03	−0.00	−0.01	Interior Noise	0.07
High Speed Automatic	−0.04***	0.19***	−0.00	Powertrain Noise	0.06***
CVT	−0.05	0.23***	0.01	General Noise	0.02
DCT	−0.00	0.07	−0.00	Chassis Vibration	−0.02
Elec Assist Or Low Drag Brakes	−0.06	0.08	−0.05	Powertrain Vibration	0.11*
Lighting-LED	−0.00	−0.09	−0.01	General Vibration	−0.05
Mass Reduction	−0.04	−0.16	−0.04	Ride Comfort	0.04**
Passive Aerodynamics	−0.02	0.16	0.00	Fuel Economy	0 07***
Fuel Cell	−0.06	0.00	0.05	Range	−0.01
				Charging	0.00

Open in a new tab

Notes: There are 2,238 observations for the two specifications. Dependent variable for the two specifications is an indicator variable for negative overall assessment. Independent variables of the first specification include the presence of the technologies (column 1a) plus negatively reviewed technologies (column 1b). Independent variables of the second specification include the presence of the technologies (column 2a) plus negatively reviewed operational characteristics (column 2b). Number in a cell indicates the estimated coefficient. All specifications include a set of fixed effects, including year, website, vehicle class, vehicle make, year-by-website, year-by-class, and year-by-make. Asterisks indicate the level of statistical significance using the maximum of robust standard errors and conventional standard errors as the measure of efficiency: 10% (*), 5% (**), and 1% (***) levels.

Column (1b) of Table 8 shows six negatively reviewed technologies -- low resistance tires, CVT, electronic power steering, hybrid, high speed automatic, and turbocharged -- are significantly correlated with negative overall assessments. The results again raise the possibility that poorly implemented technologies (1b) in some vehicle models, instead of the technologies themselves (1a), are associated with an increased likelihood of a negative overall assessment.

Next, instead of using negatively reviewed technologies as controls in specification (3), we include the 22 negatively reviewed operational characteristics as controls. A negative operational characteristic (NegativeOper)_,, equals 1 if the operational characteristic j was negatively reviewed, and equals 0 otherwise. It is sensible that negative operational characteristics would contribute to negative overall qualitative assessment. If operational characteristics are correlated with the presence of the technologies, our estimates of the technologies may be biased. We examine whether our technology estimates are robust by controlling for the potential confounders.

Columns (2a) and (2b) of Table 8 report the estimated results of the revised specification. Conditional on the negatively reviewed operational characteristics, the estimates of fuel-saving technologies shown in column (2a) are consistent with the results of column (1a); the estimates are fairly small, and all of the statistically significant coefficients are negative. That is, the presence of turbocharged, cylinder deactivation, and diesel, are associated with a reduction in the likelihood of a hidden cost. The results continue not to find the technologies themselves associated with negative overall evaluations.

Column (2b) of Table 8 reports that a negative overall review, conditional on the presence of the technologies, has a positive association with ten negatively reviewed operational characteristics: steering feel, general drivability, general handling, acceleration feel, acceleration capability, brake feel, powertrain noise, powertrain vibration, ride comfort, and fuel economy. The results not only suggest that operational characteristics in the study are the major factors in a reviewer’s overall assessment, but suggest a potential explanation for the association between negatively reviewed technologies and the negative overall rating shown in column (1b). In particular, these results are consistent with a scenario that a negatively reviewed technology leads to a negatively reviewed operational characteristic, which in turn leads to a negative overall assessment. Subsection 4.2.1 and Table 7 indicate that negatively reviewed CVT and high speed automatic are positively correlated with several negatively evaluated operational characteristics related to handling, acceleration, braking, noise, ride comfort, and fuel economy. Negatively reviewed technologies may be associated with negative overall assessment through their associated negative operational impacts.

In sum, we do not find evidence that the presence of technologies is associated with negative overall assessments of a vehicle’s quality. Rather, it may be that the inclusion of the technologies in some vehicle models affects overall quality via their effects on operational characteristics. With little evidence that the technologies by themselves are associated with negatively evaluated characteristics, and somewhat stronger suggestions that quality of implementation, rather than technologies themselves, affect vehicle characteristics, it appears that hidden costs are not an inevitable effect of fuel-saving technologies.

6. Limitations

The limitations of this analysis are the same as those in Helfand et al. (2016). First, this study relies on opinions of professional auto reviewers rather than vehicle buyers. We suspect that auto reviewers are more likely to notice negative vehicle characteristics and operational impacts and better able to make comparisons across vehicles than the general vehicle buyers. If so, this study may overestimate negative impacts. Second, our analysis is short-run in that we do not capture longer-term issues, such as reliability or maintenance, which are not experienced by auto reviewers. Third, vehicle models that have undergone a significant redesign may be more likely to be selected to be reviewed. If redesigned vehicles are more likely to adopt new fuel-saving technologies, our data may over-represent the presence of new technologies in MY 2014 and MY 2015. Fourth, as discussed elsewhere in this paper, our data are not sufficient to consider our results causal. Finally, and perhaps most importantly, this study relies on the assumption that auto reviews contain useful information and are not systematically biased. The fact that results are very similar between MY 2014 and MY 2015 suggests that reviewers are, at a minimum, consistent rather than random in their evaluations.

Also, it is important to note that, for some rarely mentioned technologies with small sample sizes in our data, this paper cannot answer questions about their relationships with operational characteristics. Nevertheless, we demonstrate that the use of the maximum of robust and conventional standard errors can affect conclusions drawn from the analysis, and may be especially influential for interpretation when sample sizes are small.

7. Conclusion

Energy and transportation policies have been enacted to improve vehicle fuel economy and reduce vehicle GHG emissions in many countries, including the U.S. As a variety of fuel-saving technologies have been implemented under the standards, understanding the potential hidden costs and benefits due to adoption of fuel-saving technologies contributes to understanding the full impacts of these policies.

In this paper, using professional auto reviewers’ evaluations of MY 2014 and 2015 vehicles, we find that fewer than 20 percent of evaluations of fuel-saving technologies in individual vehicle models were negative. We then estimate the relationships of a variety of fuel-saving technologies to operational characteristics. Our results, which serve to check and validate the findings of Helfand et al. (2016), suggest that it is possible to implement these technologies without imposing hidden costs. In addition, they suggest that problems with implementation in some vehicle models, rather than something inherent in the technologies, may contribute to occasional negative operational impacts.

This paper also examines the association of the technologies with auto reviewers’ overall assessments of vehicle quality. The results similarly do not provide evidence that the presence of the technologies leads inherently to negative overall ratings, but rather that negatively reviewed technologies in some vehicle models are associated with negative overall ratings. Further, the overall assessment of vehicle quality is more strongly associated with reviewers’ evaluation of vehicle characteristics than the presence of the technologies. Our results suggest the importance of operational characteristics for vehicles’ overall assessment.

Thus, based on MY 2014 and MY 2015 vehicles, fuel-saving technologies appear to have been adopted without significant tradeoffs for other operational characteristics. Rather than technologies themselves, the quality of implementation of the technologies in some vehicles is more likely to be associated with the quality of operational characteristics and the overall assessment. If problems arise due to implementations, rather than the inherent natures of the technologies, then it appears that automakers have the ability to mitigate any problems arising with these fuel-saving technologies.

This paper explores whether negative operational characteristics were associated with fuelsaving technology from the perspective of vehicle consumers who operate the vehicles. On the production side, successful deployment of fuel-saving technology involves costs, strategies, and the diligence of the manufacturer. This analysis suggests that manufacturers have, for the most part, risen to this challenge.

Acknowledgements

RTI International conducted the content analysis under EPA contract EP-C-11–045, WA 3–01 and 4–08. We thank Michael McWilliams, participants at the 2017 Society for Benefit-Cost Analysis annual conference, and commenters at the EPA’s Office of Transportation and Air Quality for helpful comments.