Tracking Biases: An Update to the Validity and Reliability of Alcohol Retail Sales Data for Estimating Population Consumption in Scotland

Abstract

Aims

To highlight the importance of monitoring biases when using retail sales data to estimate population alcohol consumption.

Methods

Previously, we identified and where possible quantified sources of bias that may lead to under- or overestimation of alcohol consumption in Scotland. Here, we update findings by using more recent data and by quantifying emergent biases.

Results

Underestimation resulting from the net effect of biases on population consumption in Scotland increased from −4% in 2010 to −7% in 2013.

Conclusion

Biases that might impact on the validity and reliability of sales data when estimating population consumption should be routinely monitored and updated.

INTRODUCTION

Retail sales data are thought to provide the most accurate means of estimating population level alcohol consumption (World Health Organization, 2000) and are preferred over their less objective alternative, self-reported survey data. However, retail sales data have not been subject to the same level of scrutiny as survey data. Acknowledging this, Robinson et al. (2013) identified and, where possible, quantified biases in retail sales data when estimating annual population alcohol consumption levels.

Quantification of biases was illustrated around alcohol sales per adult in Scotland in 2010. This provided a snapshot of the extent to which identified biases impacted on the estimate of population alcohol consumption in this location at this time. It was found that, on balance, retail sales data were more likely to underestimate than to overestimate alcohol consumption per adult and that this was driven largely by unrecorded alcohol. Nonetheless, it was concluded that retail sales provide a robust source of data for estimating alcohol consumption per adult. Importantly, however, it was noted that the robustness is subject to changes in biases over time, some of which may be in response to alcohol-related or other policy changes.

To illustrate the potential effect of changing biases or new data upon estimates of per adult consumption, here we provide an update to the biases estimated by Robinson et al. (2013).

METHODS

Alcohol retail sales data for 2010 and 2013 were obtained from market research specialists Nielsen and CGA Strategy (Nielsen/CGA). Detailed information on the methods used by Nielsen/CGA to estimate retail sales is available elsewhere (Thorpe et al., 2012). Best estimates of the size of different biases were quantified using the same data source as in the original study unless a new data source was identified. The data sources used to estimate the size of newly identified or quantified biases are described, where relevant, in the Results section. To enable any change in magnitude of over- or underestimation of population consumption to be illustrated, we assessed the impact of biases on per adult consumption estimates in 2010 (11.41 l per adult) and 2013 (10.53 l per adult) (Beeston et al., 2014). Biases are quantified in absolute terms and also relative to the population level estimate to enable direct comparison between years.

RESULTS

Figure 1 summarizes the estimated size of each bias relative per adult consumption in Scotland in 2010 and 2013.

Fig. 1.

Bias size relative to per adult consumption estimated from retail sales data in Scotland, 2010 and 2013. Note: Estimates are based on available data and assumptions deemed to be most robust.

New data

Since the original paper (Robinson et al., 2013) data have become available for two previously unquantifiable sources of bias. Additionally, a new more robust estimate of alcohol waste has become available.

Online sales

Sales of alcohol from online specialists are not captured in Nielsen/CGA data. These include wine clubs and direct orders by the case, but not alcohol bought from online grocery suppliers. The Wilson Drinks Report (WDR, 2015) provides annual estimates of sales through online specialists in the UK. Analysis of these data suggests that sales by online specialists account for around 1% of the total natural volume of alcohol sold in the UK. When applied to pure alcohol volume sales in Scotland in 2013, this translates to an additional 0.11 l of pure alcohol consumed per adult (1% of population estimate).

Census population estimates

A comparison of the original and revised population estimates for Scotland and England/Wales following the 2011 census demonstrated that the adult population was underestimated by an increasing degree each year between 2002 and 2011. This had a knock on effect of overestimating alcohol consumption per adult. In Scotland, the maximum impact was an overestimation of alcohol consumption by 0.11 l per adult in 2011 (1% of population estimate). In England/Wales, our main comparator, the bias did not exceed 0.06 l per adult in the same time period.

Waste

In the absence of a substantiated estimate of alcohol waste at the time of the original report, an industry estimate of <10% was applied (Rehm et al., 2010), resulting in an underestimation of 1.14 l per adult using updated 2010 sales estimates. A new report by the Waste Resource Action Programme (WRAP) uses a more systematic approach to quantify waste based on the triangulation of data from three main sources: measurement of food and drink waste from 1800 households; a week-long food and drink diary from 950 households; and a synthesis of waste audit data from local government authorities. The report estimates that 5% of alcohol entering UK homes (by purchase, gifted or home brew) is wasted (Quested et al., 2013). This new source of data suggests that the bias arising from waste could be approximately half of that previously calculated (0.53 l per adult in Scotland assuming equal waste in the on and off-trade (see also Meier et al. (2013)).

UPDATING BIASES

Travel and tourism

In both 2010 and 2013 there was a net loss of nights spent in Scotland as a result of travel and tourism (considering both visits into and out of Scotland), and a subsequent underestimation of population alcohol consumption. However, the net loss was smaller in 2013 leading to a reduction in the bias down from 0.04 l per adult to 0.003l (in both cases a negligible impact on population estimate).

More recently tourism in Scotland has benefited from hosting the Commonwealth games in 2014. Adding the 250,000 unique visitors who stayed on average 5.8 nights (Scottish Government, 2014) to the 2013 calculations creates a shift in the direction of bias from tourism from a source of underestimation to overestimation. However, the bias size is still very small at 0.01 l suggesting that even major tourism events in Scotland will have little impact on sales-based annual population consumption estimates.

Students

Students are counted in the census at their term-time address where they will spend at least 30 weeks of the year and potentially longer (National Records of Scotland, 2013). For this reason, it is not believed that students will create any meaningful bias on per adult estimates in Scotland. Where census methods differ, student numbers could be a source of overestimation in other countries.

Under 16s

Consumption of alcohol by under 16s in Scotland has fallen in recent years (National Services Scotland, 2014) and is reflected in a lower impact on per adult estimates: from a 0.06 l source of overestimation in 2010 (0.5%) to a 0.03 l source of overestimation in 2013 (0.3%).

Unrecorded alcohol

Based on expert opinion (Rehm et al., 2014), the World Health Organization (WHO) estimates that levels of unrecorded alcohol consumption in the UK have fallen from 1.7 l per adult (15%) to 1.2 l per adult (11%) (World Health Organization, 2014).

Non-inclusion of outlets

Vendors of alcohol not captured by Nielsen/CGA data are a potential source of underestimation. Nielsen off-trade estimates do not include sales from two discount supermarkets in Scotland (because they do not release their sales data to market research companies), whose combined market share of alcohol sales grew from 6% in 2010 to 11% in 2013. However, this bias is corrected for using a secondary data source from Nielsen prior to estimating population level consumption. This bias is therefore not presented in Fig. 1.

Alcohol can also be sold via channels other than retail outlets (including temporary venues, events and wholesalers). Replicating prior methods, approximate numbers of attendance at Scotland's five largest festivals show an increase since 2010 but attendees would still be required to consume the equivalent of 143 pints per day in order to impact the per adult estimate of alcohol consumption by 0.5 l (arbitrary value). Music festivals therefore remain unlikely to impact estimates of alcohol consumption.

Alcohol sold at other temporary events will not be captured in by Nielsen/CGA. In 2014 Scotland hosted the Commonwealth Games, selling 1.3 million tickets across all sporting events. However, even major events such as this are unlikely to have a notable impact on population alcohol estimates for a given year as 56 pints of beer or cider would need to be purchased with every ticket in order to impact the per adult estimates of alcohol consumption by 0.5 l.

UPDATED METHODS

Between 2010 and 2013 Nielsen/CGA increased their sample of impulse retailers but this had little impact upon sampling variation which remained constant at 4% for Scotland. Where changes in methods do impact measurement error this will affect the validity and reliability of retail sales data. An updated comparison of Nielsen/CGA retail sales in Great Britain with HMRC estimates of alcohol released for sale in the UK also demonstrated that the two sources continue to show consistent trends to 2013 (Henderson et al., 2015).

The use of a consistent wine percentage Alcohol by Volume (ABV) in converting retail sales data to volumes of pure alcohol, continues to show no notable difference in interpretation of trends over times or comparisons between Scotland and England/Wales when compared to variable ABV estimates derived from taxation data (see Fig. 2).

Fig. 2.

Impact of variable and standard ABVs on per adult alcohol consumption estimates in Scotland and England/Wales, 2002–2012.

DISCUSSION

This paper has demonstrated that when using sales data to estimate population levels of alcohol consumption, the size of biases that may lead to either under- or overestimation is subject to change over time. Furthermore, the accuracy of biases can be improved by identifying new sources of data.

Using Scotland as an example, the results show that there has been a decline in the size of all previously identified biases between 2010 and 2013 with less consumption of unrecorded alcohol, less consumption by those under 16 years and a smaller net loss in visits to the region. These decreases were small and the rank order of impact for each bias has been retained, suggesting some relative stability in biases. Quantifying new biases and, in particular, refining the estimate of alcohol waste, revealed that the net effect of biases in retail sales data is likely to result in a slightly higher underestimation of per adult calculation than previously calculated.

A key strength in the approach used to model biases here was to highlight the relative threat of potential biases. For example, unrecorded alcohol and waste account for a combined 90% of bias impact in 2013 and 99% in 2010, suggesting that these are the sources of bias which will be most important to consider and quantify when using sales data (retail or tax-based) to estimate alcohol consumption elsewhere. Although currently a small proportion of total alcohol sales, online alcohol sales are an important bias to consider as this part of the market develops.

This study presents a pragmatic approach to quantifying biases using available data; an approach that can be adapted by other countries using sales data to monitor alcohol consumption and evaluate alcohol control policies. However, there are some limitations to this study that will exist, to a lesser or greater extent, in other areas. For example, various bias estimates were only available at UK level, which may have masked important differences between constituent countries. The precision of bias estimates is entirely dependent on the robustness of the data sources used, which should be critiqued (Thorpe et al., 2012). In many cases these were necessarily crude approximations using sample-based estimates without consideration of the uncertainty around those estimates (although we do offer some critique of data sources in our original report). As such, adjustment of population alcohol consumption estimates should, in general, only be made for those biases that can be quantified using a sufficiently robust and reliable data source. Regardless, both unadjusted and adjusted estimates should be provided. The usefulness of the approach taken in this paper therefore lies in its ability to give a somewhat crude, but informed, indication of the direction of combined biases; and as a means of assessing if there is likely to have been a step-change in the size of under- or overestimation over time of sufficient magnitude to pose a serious threat to the use of sales data for monitoring and evaluation purposes.

In summary, whilst retail sales remain a robust source of data, the illustration of changing biases over time and the improved accuracy gleaned through new data highlight the importance in monitoring and updating the impact of biases when employing such data to estimate population level consumption.

AUTHORS' CONTRIBUTIONS

A.H. analysed the data and wrote the first draft of the manuscript under the supervision of M.R. and R.Mc. All authors commented on and contributed to further drafts, and read and approved the final manuscript.

FUNDING

Purchase of the sales data was funded by the Scottish Government as part of the wider Monitoring and Evaluating Scotland's Alcohol Strategy portfolio of studies. Funding to pay the Open Access publication charges for this article was provided by NHS Health Scotland.

CONFLICT OF INTEREST STATEMENT

All alcohol sales data are copyrighted to Nielsen and/or CGA Strategy. All authors are currently, or have recently been, involved in the evaluation of the Scottish Government's alcohol strategy. The views expressed in this paper are those of the authors and do not necessarily represent those of the Scottish Government or the author's employing organizations.