Alternative evidence and views on asymmetric loan loss provisioning

Abstract

Based on a linear provision/charge-off association and V-shaped scatterplots of these variables against nonperforming loan changes, Basu et al. (2020) argue that nonperforming loan changes mis-measure credit quality and linear provision models are mis-specified. They conclude that residual asymmetry controlling for charge-offs results from loan heterogeneity and the real estate crisis. Using additions to nonaccruals to measure credit quality, we find a linear association with provisions, that controlling for charge-offs induces misspecification, and no evidence of provision asymmetry. These results highlight the importance of basing hypotheses and causal models on theoretical underpinnings rather than on plots subject to known fallacies.

Highlights

• •We examine the Basu et al. (2020) conclusion that controlling for net charge-offs addresses measurement error in nonperforming loan changes as a measure of credit quality and that residual provision asymmetry results from loan heterogeneity and the real estate crisis.
• •Using additions to nonaccruals to measure credit quality, we find a linear association with provisions, that controlling for charge-offs induces misspecification, and no evidence of provision asymmetry.
• •Linear timeliness models without controlling for charge-offs produce results more highly correlated with models using additions to nonaccruals to measure credit quality suggesting these models are preferable absent the nonaccrual additions variable.
• •Controlling for charge-offs in provision models results in a substantially greater increase in false negatives than reduction in false positives in detecting accounting discretion.
• •Our results highlight the importance of basing hypotheses and causal models on theoretical underpinnings rather than on plots subject to known fallacies.

1. Introduction

Measurement error in changes in balance sheet measures used to capture flow variables is well known and acknowledged in the accounting literature. Hribar and Collins (2002) demonstrate the importance of this distinction in measuring overall accruals in non-financial firms. The importance of potential differences between changes in balances versus flows has also been incorporated in the banking literature by considering alternative measures absent flow variables. To address limitations from using changes in nonperforming loan balances to measure credit quality when estimating provision timeliness, Beatty and Liao (2011) validate their results using two alternative measures that do not rely on this construct. Basu et al. (2020) tackle this measurement error by allowing a non-linear relation between provisions and changes in nonperforming loans and controlling for charge-offs when measuring discretionary provisions and provision timeliness.

The Basu et al. (2020) approach is based on the premise that “large net loan charge-offs are frequently associated with large decreases in nonperforming loans and large increases in loan loss provisions.” Basu et al. (2020) argue that nonperforming loan changes are a poor measure of changes in credit quality and that using this measure leads to biased results. In addition, based on a flattening of the V-shape in the scatterplot of provisions unexplained by net charge-offs (versus provisions) against nonperforming loan changes, they argue the measurement error in nonperforming loan changes can be reduced by controlling for net allowance charge-offs (i.e., charge-offs that reduce the loan loss allowance balance). Although including a linear control for net allowance charge-offs shallows the V-shaped relation between provisions and nonperforming loan changes, an asymmetry remains in their plots and model estimates they attribute to loan heterogeneity and the 2007–2012 real estate crisis. They further support their claim of model bias using a simulation model to examine false positive rates in discretionary provision models used to detect earnings management.

To assess the appropriateness of the Basu et al. (2020) proposed approach to addressing measurement error in using a changes variable as a measure of credit quality, we employ a direct measure of additions to nonaccruals similar to the Hribar and Collins (2002) approach. This flow variable, which is free from the measurement error associated with using changes in balances highlighted by Basu et al. (2020), allows us to re-examine their claims that i) allowance net charge-offs effectively capture measurement error in nonperforming loan changes, ii) there is a remaining provision asymmetry after controlling for allowance charge-offs aimed to capture this measurement error and iii) loan heterogeneity and the real estate crisis drive that remaining asymmetry. Because the Bank COMPUSTAT database does not break out nonaccrual loans from loans past due 90 days but still accruing, research using that database (e.g., Beatty and Liao, 2011) examines nonperforming rather than nonaccrual loans. Since nonaccruals are the primary focus of regulatory attention, nonaccruals data is reported separately in the bank regulatory reports and research using that data typically examines nonaccrual loans. We focus on nonaccrual loans in most analyses since additions to this amount is the disclosure that regulators require arguing that it enhances their ability to assess credit quality (Federal Register, 2003).1

The additions to nonaccruals measure we use was not publicly disclosed until relatively recently but was added as a quarterly disclosure in bank regulatory reports during the 2003–2017 period to enhance credit quality assessments. Using this new measure allows us to directly examine the effects of measurement error induced by assuming that allowance charge-offs capture reductions to nonaccruals.2 Based on the additions to nonaccruals measure, we find that allowance net charge-offs do not reduce the measurement error in nonaccrual changes and there is no evidence of a residual provision asymmetry in pooled cross-sectional and time series provision models.

We also find that the R² from substituting the additions to nonaccrual measure in the bank-specific time-series provision timeliness model is more highly correlated with the R² from linear models using nonaccrual changes (that do not allow for provision asymmetry or include allowance net charge-offs) than the models proposed by Basu et al. (2020). In addition, we find that the Beatty and Liao (2011) conclusion that timely banks cut lending less during the financial crisis is robust to the timeliness measure estimated using the additions to nonaccruals variable, in contrast to the inference from the Basu et al. (2020) proposed models. Similarly, we find that the Bushman and Williams (2015) finding that bank downside tail risk (i.e., VaR, value at risk) increases with delayed loan loss recognition continues to hold based on the timeliness measure estimated using the additions to nonaccruals variable. This again differs from the Basu et al. (2020) inferences.

We further argue that assessing discretionary provision models’ efficacy requires consideration of both false positives and false negatives. While Basu et al. (2020) exclusively consider the likelihood of rejecting the no accounting misreporting null hypothesis too frequently (false positives), we also consider the likelihood of not rejecting the null when misreporting is present (false negatives). We document that, regardless of provision asymmetry restrictions, including net charge-offs in provision models increases false negatives substantially more than it decreases false positives. Given the importance of detecting misreporting, we argue this is unlikely to be a good trade in most cases.

Finally, while plots are one tool that can be used in a carefully executed research design and can provide useful insights as Basu et al. (2020) advocate, we argue that using plots can be a double-edged sword. Their paper highlights not only how plots can be affected by measurement error, but more generally demonstrates problems with using plots absent theoretical support to develop hypotheses and causal models.

2. Measurement error in nonaccruals changes as a measure of additions to nonaccruals

The problem of measurement errors that rise from using changes in balances to capture flow variables is not new to the accounting literature. Hribar and Collins (2002) examine this issue in the context of cash flows. Using data provided by the adoption of the Statement of Cash Flow Accounting they obtain direct measures of cash flows and accruals, and document biases in accrual measurement caused by using changes in balances. They note that correlation between the partitioning variable and the measurement error can cause erroneous inferences.

Obviously, having a flow variable without measurement errors is ideal, but that data is not always available and different approaches have been used to address the measurement problem when only changes in balances are available. For example, in the specific case of potential misspecification in provision timeliness models using changes in nonperforming loans, Beatty and Liao (2011) take the approach of using two alternative timeliness estimates not subject to the same potential measurement error in nonperforming loan changes. Specifically, they use the ratio of loan loss allowance balance to the nonperforming loan balance, which is a measure similar to the allowance to nonaccrual ratio used by bank supervisors and regulators to assess cumulative provision timeliness. They also use the Khan and Watts (2009) C-score conservatism measure (derived from the Basu (1997) asymmetric loss measure), which is a general measure of conservatism used in the accounting literature. The consistency in results across these three approaches provides comfort that the measurement error in the change in nonperforming loan balances is not driving the results.

Basu et al. (2020) use an alternative approach to deal with this potential measurement error. They use accounting identities to argue that the provision should be linearly related to new additions to nonaccruals and that the change in nonperforming loans mis-measures that construct because it is also affected by net nonperforming charge-offs. Specifically, Basu et al. (2020) equation (6)$NetNewNPl{L}_t=\Delta NP{L}_t+NC{O}_t$ states that changes in nonperforming loans are not equal to new additions to nonperforming loans and assumes the allowance charge-offs capture the difference. This equation introduces measurement error by substituting the allowance charge-offs for nonperforming charge-offs and other reductions to nonperforming loans (including repayments) that reconcile the difference between changes and additions in the accounting identity. This distinction between allowance charge-offs and nonperforming loan charge-offs is important due to substantial measurement differences between nonperforming loans and the loan loss allowance. Given data availability, our analyses focus on changes in nonaccruals and additions to nonaccruals except for Table 5 based on the regulatory focus on nonaccruals reflected in the requirement to disclose additions to nonaccruals in regulatory reports (but not additions to loans past due 90 days but still accruing).

Table 5.

Logit estimates of Restatements/SEC Comment Letters on the following model residuals. Restatement or Comment Letter = α₀+α₁ARES +ε_t.

	1) Linear	2) Linear_NCO	3) Asym	4) Asym_NCO
	Coefficients (z-statistics)	Coefficients (z-statistics)	Coefficients (z-statistics)	Coefficients (z-statistics)
Intercept	−1.0540 (−9.74)∗∗∗	−0.550 (−7.12)∗∗∗	−1.083 (−8.31)∗∗∗	−0.567 (−6.39)∗∗∗
ARES	468.673 (7.23)∗∗∗	190.517 (3.30)∗∗∗	543.42 (5.52)∗∗∗	211.716 (2.60)∗∗∗
Pseudo-R squared	0.0344	0.0063	0.0427	0.0076
Number of Banks	1394	1394	1394	1394
Naïve concordance rate assuming ARES = 0	59.11%	59.11%	59.11%	59.11%
Correct Prediction: ARES = 0	752/824 = 91.3%	791/824 = 96.0%	729/824 =  88.5%	787/824 = 95.5%
False Positive Rate	8.7%	4.0%	11.5%	4.5%
Correct Prediction: ARES = 1	120/570 = 21.1%	20/570 =  3.5%	147/570 =  25.8%	25/570 = 4.4%
False Negative Rate	78.9%	96.5%	74.2%	95.6%
Concordance Rate	62.6%	58.2%	62.6%	58.2%

In this table, we regress an indicator for banks that have made any provision restatement or received comment letters due to provisions in the entire sample period on the average absolute value of discretionary provisions over the sample period (1993–2012) from various loan loss provision models listed below. ∗∗∗ represents the 1% significance level.

1) Linear: LLP_t = α + β₁ΔNPL_t + Σ Controls + ε_t (see Beatty and Liao, 2014)

2) Linear_NCO LLP_t = α + β₁ΔNPL_t + γ NCO_t + Σ Controls + ε_t (see Beatty and Liao, 2014)

3) Asym: LLP_t = α₁ + β₁ΔNPL_t + β₂DΔNPL_t + β₃DΔNPL_t∗ ΔNPL_t + Σ Controls + ε_t

4) Asym_NCO LLP_t = α₁ + β₁ΔNPL_t + β₂DΔNPL_t + β₃DΔNPL_t∗ ΔNPL_t + γ NCO_t + Σ Controls + ε_t

LLP: Loan loss provision (COMPUSTAT “pllq”) scaled by lagged total loans (COMPUSTAT “lntalq”).

ΔNPL: Change in non-performing assets (COMPUSTAT “naptq”) scaled by lagged total loans (COMPUSTAT “lntalq”).

DΔNPL: An indicator for banks with negative ΔNPL.

NCO: Net charge off (COMPUSTAT “ncoq”) scaled by lagged total loans (COMPUSTAT “lntalq”).

Controls:SIZE: The natural log of total assets (COMPUSTAT “atq”).ΔLoan: Change in total loans (COMPUSTAT “lntalq”) divided by lagged total loans.ΔGDP: Change in GDP over the quarter.CSret: The return on the Case-Shiller Real Estate Index over the quarter.ΔUnemp: Change in unemployment rates over the quarter.

Nonaccrual loans (and those that are 90 days past due and still accruing) are measured at the full outstanding loan balance, not simply the delinquent payments. In contrast, the allowance is measured based on the impaired value measured by the i) present value of expected payments at effective rate, ii) loan's observable market price, or iii) collateral fair value.3 Nonaccrual charge-offs reflect a loan with a confirmed losses' outstanding balance while the allowance charge-offs reflect the uncollectible amount of a loan with a confirmed loss. Payments also affect nonaccrual loans and the allowance differently. Interest or principal payments directly reduce the nonaccrual loan balance but only reduce the allowance if the net loan balance does not fall below the present value of the future cash flows.4 Empirically, absent data on additions to nonaccruals, Basu et al. (2020) substitute allowance charge offs for nonaccrual (nonperforming loan) charge-offs and other reductions to nonaccruals (nonperforming loans) to back into the nonaccrual (nonperforming loan) additions variable. They note this substitution is also subject to measurement error problems given the differences in the definitions in the allowance versus nonaccrual measurements.

The direct data of additions to nonaccruals was not available during the time-series period examined in Beatty and Liao (2011) and Bushman and Williams (2015). However, on April 1, 2003 the Federal Reserve proposed adding the disclosure of additions to nonaccrual assets to the Y9–C “to enhance the Federal Reserve System's ability to assess portfolio credit quality, credit cycle trends, and approaches to problem asset resolution” (Federal Register, 2003). As of the fourth quarter of 2003 until the fourth quarter of 2017 this data was disclosed quarterly in bank regulatory reports.5 We use this variable, which is available for 14 of the 19 years in the Basu et al. (2020) sample period, to assess whether their proposed approach successfully addresses the measurement error problem. We use the disclosed new additions to nonaccruals for the period from 2003 to 2017 to calculate reductions to nonaccruals and the measurement error in the Basu et al. (2020) indirect approach to measuring new additions to nonaccruals. We also examine whether this measurement error results in different inferences from those arising from using the disclosed direct measure of new nonaccrual additions.

Consistent with the claim in Basu et al. (2020), in Panel A of Table 1, we find that as a percentage of total loans, reductions to nonaccrual loans including nonaccrual charge-offs (i.e., NCO_NAL, 0.39%) are significantly larger than allowance net charge-offs (i.e., NCO_ALL, 0.13%). That reductions to nonaccruals are nearly three times as large as allowance net charge-offs not only calls into question the approach of directly adding allowance net charge-offs to changes in nonaccruals, but also suggests that including allowance net charge-offs in the provision model can introduce consequential measurement errors. Specifically, the measurement error created by substituting allowance net charge-offs (NCO_ALL) for reductions to nonaccruals (NCO_NAL) is significantly correlated with the indicator variable for negative change in nonaccruals (i.e., 24% correlation between NCO_ME and DΔNAL). In addition, the scatter plot in Fig. 1 of measurement error (NCO_ME) against changes in nonaccruals has a V shape consistent with greater NCO_ME for negative than for positive changes in nonaccruals. Based on the Hribar and Collins (2002) argument that correlation between the measurement error and this partitioning variable may produce biased results, we contend that the models proposed by Basu et al. (2020) could cause erroneous inferences. In addition, whether the Basu et al. (2020) proposed models result in more or less erroneous inferences relative to models used in prior studies must be answered empirically.

Table 1.

This table presents summary statistics for the variables used in the main regression analyses in Table 2. The sample comprises 58,980 bank-quarter observations over the period 2004Q2 to 2017Q4. Panel A reports the descriptive statistics of the variables and Panel B reports the Pearson correlations between the variables. Bold face indicates significance level at the 1% level in two-tailed tests. Variable definitions are in the Appendix.

Panel A: Descriptive Statistics
	Mean		STD		25%		75%
LLP	0.0015		0.0029		0.0002		0.0014
ΔNAL	0.0002		0.0056		−0.0013		0.0011
NCO_ALL	0.0013		0.0026		0.0001		0.0013
NCO_NAL	0.0039		0.0065		0.0004		0.0045
NCO_ME	0.0026		0.0051		0.0001		0.0032
ADD_NAL	0.0041		0.0071		0.0002		0.0045
D ΔNAL	0.5297		0.4991		0		1
Panel B: Pearson Correlations
	ΔNAL	NCO_ALL		NCO_NAL	NCO_ME	ADD_NAL		DΔNAL
LLP	0.16834	0.80313		0.48776	0.20363	0.57489		−0.0885
ΔNAL		−0.01949		−0.32318	−0.39454	0.50882		−0.5477
NCO_ALL				0.62196	0.27753	0.54065		0.0158
NCO_NAL					0.91034	0.62062		0.1922
NCO_ME						0.48915		0.2359
ADD_NAL								−0.2581

Fig. 1.

Scatter plot of difference between nonaccrual charge-offs and allowance charge-offs versus changes in nonaccruals. Scatterplot of difference between reductions to nonaccruals and allowance charge-offs versus changes in nonaccruals both scaled by beginning loans. The solid blue line represents the locally weighted scatterplot smoothing (LOESS) curve that non-parametrically depicts the relationship. The dashed black line represents the OLS estimate for the same data. . (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

We examine this possibility by controlling for the net charge-off measurement error in the Basu et al. (2020) provision model. In the first model reported in Table 2 Panel A we find similar results to those reported in Basu et al. (2020) Table 2 model 5. Specifically, we find a significant negative coefficient on the negative change in nonaccruals offset, but we find provisions move in the same direction as nonaccrual loan changes, with loan loss provisions being more sensitive to nonaccrual loan increases than to decreases. In the second model reported in Table 2 Panel A we control for the measurement error in the NCO_ALL as a measure of NCO_NAL (i.e., NCO_ME). We find that provisions are no longer significantly less sensitive to decreases in nonaccrual loans than increases in nonaccrual loans. This suggests that the provision asymmetry is induced by measurement error arising from using NCO_ALL to capture NCO_NAL. In Table 2 Panel B we examine the association between provisions and the direct measure of additions to nonaccruals allowing the association to differ based on the sign of the change in nonaccruals like the models in Panel A. Consistent with the results for the second model in Panel A, we find no asymmetry in the association between the provision and additions to nonaccruals based on whether nonaccrual changes are increasing versus decreasing. Taken together, these results provide no evidence of provision asymmetry absent measurement error associated with substituting NCO_ALL for NCO_NAL to correct for measurement error in nonaccruals changes as a proxy for observable credit quality. This lack of evidence of provision asymmetry absent the effects of NCO_ME calls into question the use of the Basu et al. (2020) model to evaluate provision asymmetry.

Table 2.

Models of the relation between Basu et al. (2020) two accounting identities:

1) LLP_t = $\Delta$ALL_t–NCO_ALL_t and 2) ADD_NAL_t = $\Delta$NAL_t–NCO_NAL_t

Panel A: Provision model: 1) assuming NCO_ALL=NCO_NAL, and 2) controlling for NCO_NAL – NCO_ALL (i.e., NCO_ME)
	Model 1	Model 2
ΔNALt	0.0952 (12.55)∗∗∗	0.0938 (12.20)∗∗∗
DΔNALt	0.0000 (1.38)	0.0000 (1.54)
DΔNALt ∗ΔNALt	−0.0495 (−5.50)∗∗∗	−0.0142 (−1.36)
ΔNALt-1	0.0232 (10.82)∗∗∗	0.0215 (10.41)∗∗∗
ΔNALt-2	0.0211 (10.47)∗∗∗	0.0196 (9.62)∗∗∗
NCO_ALLt	0.7470 (41.42)∗∗∗	0.7520 (42.67)∗∗∗
NCO_MEt		0.0351 (9.13)∗∗∗
Sizet-1	0.0008 (4.40)∗∗∗	0.0011 (6.59)∗∗∗
ΔLoant	0.0001 (5.59)∗∗∗	0.0001 (4.38)∗∗∗
R2	53.36%	53.59%
Fixed Time and Bank Effects	Yes	Yes
Number of Observations	58,980	58,980

Panel B: Provision model using direct measure of ADD_NAL
ADD_NALt	0.1462 (18.00)∗∗∗
DΔNALt	0.0002 (4.99)∗∗∗
DΔNALt ∗ADD_NALt	−0.0001 (−0.02)
ADD_NALt-1	0.0558 (11.85)∗∗∗
ADD_NALt-2	0.0470 (11.07)∗∗∗
Sizet-1	−0.0012 (2.83)∗∗∗
ΔLoant	0.0001 (2.43)∗∗∗
R2	24.83%
Fixed Time and Bank Effects	Yes
Number of Observations	58,980

Results of estimating the three competing provision models. The sample comprises 58,980 bank-quarter observations over the period 2004Q2 to 2017Q4. Two tailed t-statistics are reported in parentheses. ∗∗∗ indicate statistical significance at the 1% level. Standard errors are clustered by quarter. Variable definitions are provided in the Appendix.

3. Sources of provision asymmetry

Two potential sources of provision asymmetry after controlling for allowance net charge-offs are explored in Basu et al. (2020). Specifically, they examine potential differences in impairment rules due to loan type and financial reporting regulation. The evidence provided in Basu et al. (2020) Table 6 suggests that the asymmetric response to increases in nonperforming loans is driven by construction loans and the 2007–2012 real estate crisis with a significantly lower response to increases in nonperforming loans absent construction loans and in the periods before and after the real estate crisis. Not surprisingly construction loans are significantly higher during the 2007–2012 real estate crisis period, so these results may be driven by similar factors.

To better understand what might be driving the flip in provision asymmetry between construction and other types of loans and between the 2007–2012 period and other periods, we first examine the association between the deciles of the proportion of construction loans in the bank and the allowance net charge-off measurement error. We also compare this measure error between crisis and non-crisis periods. We observe a monotonic increase in measurement error and construction loan deciles and find the measurement error is significantly higher nearly tripling during the real estate crisis period (untabulated). Specifically, the measurement error of allowance charge-offs is 0.43% during the 2007–2012 period versus 0.16% before and after that period. We next investigate whether we find similar results to those documented in Basu et al. (2020) using the direct measure of additions to nonaccruals. In Table 3 column 1, we do not find the association between provisions and new additions to accruals to vary with construction loans. More importantly, we do not find asymmetric relations between provision and new additions to accruals for banks with decreases in nonaccruals. Nor do we find this to vary with deciles of construction loans.

Table 3.

Provision model allowing coefficients to differ based on decile proportion of construction loans and during the 2007–2012 real estate loan crisis period and using the direct measure of additions to nonaccruals (ADD_NAL) to capture credit quality.

	PART=Construction Loan Deciles	PART=Real Estate Crisis
ADD_NALt	0.1360 (12.24)∗∗∗	0.1242 (19.08)∗∗∗
DΔNALt	0.0000 (0.05)	0.0001 (3.55)∗∗∗
DΔNALt ∗ADD_NALt	0.0059 (0.43)	−0.0116 (−0.88)
PART∗ADD_NALt	0.0018 (1.27)	0.0292 (3.43)∗∗∗
PART∗DΔNALt	0.0000 (3.70)∗∗∗	−0.0000 (−0.07)
PART∗DΔNALt ∗ADD_NALt	−0.0012 (−0.63)	0.01751 (1.07)
ADD_NALt-1	0.0554 (11.63)∗∗∗	0.0558 (11.51)∗∗∗
ADD_NALt-2	0.0470 (11.03)∗∗∗	0.0473 (10.89)∗∗∗
Sizet-1	0.0001 (2.65)∗∗	0.0001 (1.76)∗
ΔLoant	−0.0012 (−2.75)∗∗∗	−0.0013 (−2.95)∗∗∗
R2	24.87%	25.06%
Fixed Time and Bank Effects	Yes	Yes
Number of observations	58,980	58,980

This table examines the impact of the 2007–2012 real estate crisis periods and different loan types on loan loss provision asymmetry measuring credit quality using the additions to nonaccruals measure. In model (1) the partitioning variable PART represents the proportion of a bank's loans that is made up of construction loans transformed into a decile rank variable. In model (2), PART is an indicator for the 2007–2012 period. ∗∗∗, ∗∗ and ∗ represent significance at the 1%, 5%, and 10% levels respectively. Standard errors are clustered by quarter. Variable definitions are provided in the Appendix.

Further, we report in Table 3 column 2 that while the association between the provision and additions to nonaccruals during 2007–2012 is greater than during the periods before and after, there is no evidence of significant asymmetry during either of the two periods. These findings suggest that 1) there is no residual asymmetry when we use additions to nonaccruals to capture credit quality and 2) loan heterogeneity or the real estate crisis does not contribute to residual asymmetry. These results are also consistent with the observed flip in asymmetry reported in Basu et al. (2020) Table 6 being driven by measurement errors in the allowance net charge-off measure.

The lack of significant difference between the pre-1998 and post-1998 periods reported in Basu et al. (2020) Table 7 seems at odds with the findings in Table 6 that greater provision asymmetry is driven by the 2007–2012 real estate crisis. The results in Table 6 would seem to suggest that the pre-1998 period would exhibit a significantly lower response to decreases in changes in nonaccruals similar to what is documented outside of the real estate crisis period in Table 6. Although we cannot verify whether NCO_ME differences drive these inconsistencies absent additions to nonaccruals disclosures during the earlier period, the lack of asymmetry absent measurement errors suggests this is a possible reason for the seemingly inconsistent results.

4. Relative performance of R² from bank-specific models

Based on residual plots and simulations of false positive rejection rates, Basu et al. (2020) conclude that the best provision model allows for asymmetric coefficients on increases versus decreases in nonperforming loans and controls for allowance charge-offs. Consistent with the Hribar and Collins (2002) approach, we use the R² from bank-specific models based on the direct measure of additions to nonaccruals as the benchmark to assess these claims. Specifically, we compare this benchmark to the Basu et al. (2020) models that allow for asymmetric coefficients and control for allowance charge-offs versus the Beatty and Liao (2011) and Bushman and Williams (2015) linear provision timeliness model. In Table 4 Panel A, we report that the correlation between the R² from this direct benchmark model and the linear change in nonaccruals model is the highest at 48.4%, allowing for asymmetric change in nonaccruals coefficients drops the correlation to 43.7%, and adding the allowance net charge-offs decreases the correlation further to 30.4%. In addition, when we regress R² from the direct model on these three R² from alternative models, the linear model R² loads more positively than the other two models with a p-value of less than 0.01%.6 These findings suggest that using the direct additions to nonaccruals model as the benchmark, the linear model used by Beatty and Liao (2011) performs better than the models that allow for provision asymmetry and that adjust for allowance net charge-offs.

Table 4.

Panel A: Pearson Correlations of R2 calculated from different timeliness models.
	R2_Linear	R2_Asym	R2_Asym_NCO	R2_ADD
R2_Linear		0.77695	0.49695	0.49878
R2_Asym			0.62723	0.44107
R2_Asym_NCO				0.31137
R2_ADD

Panel B: Beatty and Liao (2011) change in loans model comparing timeliness measure based on R2 from bank-specific time series model using changes in nonaccruals versus additions to nonaccruals
Variables	ΔNAL	ADD_NAL
Intercept	0.0217 (5.03)∗∗∗	0.0117 (2.47)∗∗
<DELR	−0.0019 (−1.29)	−0.0012 (−0.82)
Recession	0.0358 (0.39)	0.0114 (1.19)
Recession∗<DELR	0.0052 (1.83)∗	0.0068 (1.97)∗∗
Capital R1	0.0386 (1.74)∗	0.0848 (3.76)∗∗∗
Capital R1∗<DELR	0.0101 (0.97)	0.0014 (0.14)
Capital R1∗Recession	0.0448 (1.25)	0.0080 (0.18)
Capital R1∗Recession∗<DELR	−0.0594 (−2.26)∗∗	−0.0671 (−2.26)∗∗
ΔUNRATE	−0.0148 (−1.79)∗	−0.0134 (−1.84)∗
Size	0.0010 (2.51)∗∗	0.0002 (0.56)
Deposits	−010,121 (−10.61)∗∗∗	−010,121 (−10.61)∗∗∗
ΔCapital R1	−1.2115 (−7.23)∗∗∗	−1.0770 (−6.06)∗∗∗
R2	0.0927	0.0902
N	37,177	30,050

Panel C: Bushman and Williams (2015) VAR model (Table 3, Column 3) comparing timeliness measure based on R2 from bank-specific time series model using additions to nonaccruals versus changes in nonaccruals in the bust period (i.e., the 2007–2009 financial crisis)
Variables	ΔNAL	ADD_NAL
DELR	−0.0709 (−2.79)∗∗∗	−0.0706 (−1.93)∗
Control Variables	YES	YES
Year Fixed Effects	YES	YES
R2	0.5649	0.5649
N	814	814

Panel A:

• 1)
  R²_Linear represents R² difference between the forward-looking bank-specific rolling regression model Linear (1) and backward looking model Linear (2). Similar to Beatty and Liao (2011), for each bank-quarter, we require 12 observations in the last three years to calculate R². We also exclude bank-quarter with a growth of non-loan assets greater than 10%. See Beatty and Liao (2011) for detailed discussions. There are 32,162 observations in this analysis. Other variables are defined in the Appendix.
• Linear (1): LLP_t = β₀ + β₁ΔNAL_t-2 +β₂ΔNAL_t-1+β₃ΔNAL_t +β₄ΔNAL_t+1 +β₅EBP_t +β₆Capital R1_t-1+ε.

  • (2): LLP_t = β₀ + β₁ΔNAL_t-2 +β₂ΔNAL_t-1+β₃EBP_t +β₄Capital R1_t-1+ε.
• 2)
  R²_Asym represents R² difference between the forward looking bank-specific rolling regression model Asym (1) and backward looking model Asym (2), where ΔNALPOS_t (ΔNALNEG_t) is equal to ΔNAL_t when ΔNAL_t is positive (negative), zero otherwise.
• Asym (1): LLP_t = β₀ + β₁ΔNAL_t-2 +β₂ΔNAL_t-1+β₃ΔNALPOS_t +β₄ΔNALNEG_t +β₅ΔNAL_t+1 +β₆EBP_t +β₇Capital R1_t-1+ε.

  • (2): LLP_t = β₀ + β₁ΔNAL_t-2 +β₂ΔNAL_t-1+β₃EBP_t +β₄Capital R1_t-1+ε.
• 3)
  R²_Asym_NCO represents R² difference between the forward looking bank-specific rolling regression model Asym_NCO (1) and backward looking model Asym_NCO (2), where ΔNALPOS_t (ΔNALNEG_t) is equal to ΔNAL_t when ΔNAL_t is positive (negative), zero otherwise.
• Asym_NCO (1): LLP_t = β₀ + β₁ΔNAL_t-2 +β₂ΔNAL_t-1+β₃ΔNALPOS_t +β₄ΔNALNEG_t +β₅ΔNAL_t+1 +β₆NCO_ALL_t +β₇EBP_t +β₈Capital R1_t-1+ε.

  • (2): LLP_t = β₀ + β₁ΔNAL_t-2 +β₂ΔNAL_t-1+β₃NCO_ALL_t +β₄EBP_t +β₅Capital R1_t-1+ε.
• 4)
  R²_ADD represents R² difference between the forward looking bank-specific rolling regression model ADD (1) and backward looking model ADD (2).
• ADD (1): LLP_t = β₀ + β₁ADD_NAL_t-2 +β₂ADD_NAL_t-1+β₃ADD_NAL_t +β₄ADD_NAL_t+1 +β₅EBP_t +β₆Capital R1_t-1+ε.

  • (2): LLPt = β₀ + β₁ADD_NAL_t-2 +β₂ADD_NAL_t-1+β₃EBP_t +β₄Capital R1_t-1+ε.

Panel B: This table replicates Beatty and Liao (2011)Table 5 flow measure analysis. In the first column, we regress percentage change in loans (bhck2122) on < DELR (an indicator for banks with Linear_R² from Panel A higher than the sample median by the quarter), Capital R1 and Recession (an indicator for the time period between 2008Q1 and 2009Q2). ΔUNRATE is measured as the change in unemployment rate. Deposits is measured as the sum of bhdm6631, bhfn6631, bhdm6636 and bhfn6636 scaled by bhck2122. In the second column, <DELR is defined based on ADD_R² instead. ∗∗∗, ∗∗, and ∗ represent significance at the 1%, 5% and 10% levels, respectively. Standard errors are clustered by quarter. Other variables are defined in the Appendix.

Panel C: This table replicates Bushman and Williams (2015) analysis in their Column 3, Table 3. In the first column, we regress VaR on DELR (an indicator for banks with Linear_R² from Panel A lower than the sample median by the quarter), along with control variables included in Bushman and Williams (2015) including maturity mismatch, revenue mix, trading assets, commercial loans, consumer loans, real estate loans, deposits, tier 1 capital ratio, market return beta, idiosyncratic return volatility, bank size, market-to-book ratio, illiquidity, prior period returns and unemployment rates. Please see Bushman and Williams (2015) for variable definitions. The dependent variable VaR is defined as the one percentile value-at-risk over the quarter, where the value-at-risk is computed over the market value of equity (see Bushman and Williams, 2015). In the second column, DELR is defined in a similar fashion but based on ADD_R² instead. ∗∗∗, ∗∗, and ∗ represent significance at the 1%, 5% and 10% levels, respectively. Standard errors are clustered by quarter. Other variables are defined in the Appendix.

We further re-estimate the Beatty and Liao (2011) pro-cyclicality analysis using this direct model that substitutes additions to nonaccruals for changes in nonaccruals. In column 1 of Table 4 Panel B, we re-estimate Beatty and Liao (2011) analysis based on the Y9C sample from 2003Q4 to 2017Q4 following the same data requirement as Beatty and Liao (2011). The inference of this column is very similar to Beatty and Liao (2011) who use a publicly traded company sample. That is, we continue to document a negative coefficient on the interaction between regulatory capital and the indicator for more timely banks during the recessionary period (Capital R1∗Recession∗<DELR) consistent with less pro-cyclicality for more timely banks. In column 2, we estimate provision timeliness based on this direct measure of new additions to nonaccruals. The results of these tests are very similar between these two models with a slightly larger effect estimated for the coefficient on the interaction of regulatory capital and provision timeliness during the recession for the new additions model. The two models also have similar statistical significance and explanatory power. This analysis suggests that, despite the measurement error, the Beatty and Liao (2011) linear model does not lead to incorrect inference. In addition, this analysis suggests that the positive coefficient in Basu et al. (2020) Table 5 row (5) based on the asymmetric model with the control for allowance net charge-offs and the insignificant coefficients in rows (3) and (4) based on the two asymmetric models are potentially incorrect inferences. These findings are consistent with R² from the linear model better resembles the direct model than the alternative models proposed by Basu et al. (2020).

In Panel C, we replicate the Bushman and Williams (2015) Table 3 Column 3, where they examine whether bank downside tail risk measured as the 1 percentile value at risk over a quarter (VaR) increases with delayed loss recognition in the bust period. Because in the 2003–2017 period, the 2007–2009 crisis is the only bust period, and because the merging between regulatory reports and COMPUSTAT/CRSP drops more observations, we end up with a smaller sample size compared to Bushman and Williams (2015). In the first column where we use the Beatty and Liao (2011) approach to define delayed recognition, we document evidence similar to Bushman and Williams (2015). That is, during the bust period, bank downside tail risk is exacerbated by delayed loan loss recognition. Our coefficient on DELR (indicator for less timely banks) and the statistical significance are similar to the Bushman and Williams (2015). More importantly, when we use new additions to nonaccruals to replace changes in nonaccruals in timeliness measurements, we find very similar results in the second column. These findings further suggest that the approaches proposed by Basu et al. (2020) can be erroneous.

5. Evaluation of both type 1 and type 2 errors in discretionary provision models

Basu et al. (2020) evaluate discretionary provision models by comparing each model's propensity to falsely reject the null of no abnormal accruals. The choice to consider only Type 1 errors is not articulated and the lack of consideration of Type 2 errors is a serious omission. Arguably there have been few times in the last 100 years that the importance of considering both false positives and false negatives has been more salient. Analogizing to COVID-19 tests, it is not difficult to think that false negatives might be equally or more costly than false positives. A failure to reject the null of accounting manipulation could lead to a spread of this malady to other banks that could cause systemic risk. In contrast falsely rejecting the null might create a disincentive for banks to use discretion in reporting their losses.

Basu et al. (2020) conduct an indirect test of the propensity of Type 1 error assuming that by randomly selecting bank-quarters it is reasonable there is not systematic earnings management among the randomly selected observations. In Table 5, Basu et al. (2020) use randomized trials to test each model's propensity to falsely reject the null of no discretionary provisions relying on this assumption. However, the validity of this assumption that there is no systematic earnings management obviously depends on the overall prevalence of earnings management since a truly random sample should contain the same proportion of earnings management as in the overall sample. They conclude that the Basu et al. (2020) models are superior because the “false” rejection rates are substantially higher for the linear model than for models that allow for asymmetric relation between provisions and changes in nonperforming loans with or without a control for allowance charge-offs. Given the prevalence of misstated provisions as discussed in Beatty and Liao (2014), this conclusion may not be appropriate based on the assumption that the null hypothesis of no earnings management “is true”. In addition, the conclusion may not be appropriate because Basu et al. (2020) do not consider the ability of these tests to detect earnings management when it is known to occur.

Direct evidence of the actual incidence of overall earnings management is provided in Beatty and Liao (2014). They use SEC comment letters on provisions and provision restatements to examine how the exclusion or inclusion of allowance net charge-offs in estimated provision affects the model's ability to detect earnings management. They identify banks' restatements and SEC comment letters from Audit Analytics 10 K wizard searches of SEC filings. They report that 41% of their sample of 1394 banks report a restatement or SEC comment letter related to loan loss provisioning during 1993Q4–2012Q2, with an average restatement period of 7 quarters. This is consistent with 30% of the sample bank-quarter observations are affected by provision management and calls into question an assumption of no provision management in a randomly chosen sample of bank-quarters in Basu et al. (2020). In their direct tests of the models' abilities to correctly reject both the presence and the absence of earnings management, Beatty and Liao (2014) find evidence that the false positive rate of 4% is lower when net charge-offs are included in the model compared to 8.7% when net charge-offs are excluded, which is consistent with the Basu et al. (2020) findings. Ignoring Type 2 errors this suggests that including net charge-offs improves the estimation. However, Beatty and Liao (2014) report that the Type 2 error rate is 17.6% lower when net charge-offs are excluded and that the overall concordance rate is 4.4% higher when net charge-offs are excluded. This suggests that the Basu et al. (2020) proposed models where net charge-offs are controlled for are less able to detect the Type 2 error and have lower concordance rates absent the non-linearity of nonperforming loans.

Because Beatty and Liao (2014) do not calculate the Type 1 and 2 errors for models that allow for non-linear relations between provisions and changes in nonperforming loans, in Table 5 we reproduce the results in Beatty and Liao (2014) in the first two models and augment their analysis by reexamining their data and introducing the Basu et al. (2020) negative change in non-performing loan non-linearity in the third and fourth models. The comparison of the first and third models indicates the relative performance of allowing for asymmetry in the coefficient on changes in nonperforming loans when net charge-offs are excluded. We see that the false positive rate increases by 3% while the false negative rate decreases by 3% when an asymmetry is allowed, resulting in the same concordance rates for these two models. Which model is preferred depends on the question being answered and the relative costliness of Type 1 versus Type 2 errors.

The comparison of the second and fourth models can be used to evaluate the exclusion versus the inclusion of net charge-offs when asymmetry in nonperforming loan changes is allowed. While the Type 1 error is 7% lower, the Type 2 error is 21.4% higher when net charge-offs are included and the concordance rate of 58.2% falls below the naïve rate of 59.1% based on the assumption that none of the banks engage in provision management. These results suggest that the conclusion that net charge-offs should be included in discretionary provision models when the asymmetric relation between provisions and nonperforming loan changes is allowed would be only appropriate if the cost of Type 1 errors is over 3 times greater than the cost of Type 2 errors.

6. Using plots to generate hypotheses

Distinguishing between association and causality is arguably the most important research challenge faced in observational empirical research. The associations that appear in bivariate plots can be driven by many factors other than a causal relation between the variables including but not limited to measurement error and correlated omitted variables. The direction of causality also cannot be determined from a bivariate plot such as the association between contemporaneous provisions and charge-offs. Basu et al. (2020) use a series of plots to generate their hypotheses and conclude that allowance net charge-offs are the source of measurement error in changes in nonperforming loans and that loan loss provisions vary asymmetrically with nonperforming loan changes. While plots are one tool that can be used in a carefully executed research design, we argue that their paper highlights that using plots can be a double-edged sword.

Problems with using plots to generate hypotheses are well established in the statistics literature and illustrated in the finance and accounting literature. Two well-known issues highlight potential problems in using plots to infer causal models. Simpson's paradox illustrates that trends that appear in groups of data can disappear or reverse when the groups are combined and the ecological fallacy arises when within group and between groups effects differ in sign or magnitude. Holderness (2016) illustrates these issues using country level versus firm-level cross-country comparisons of the effect of the rule of law on ownership concentration. He illustrates how the estimated sign can flip between these two units of analysis due to either differences in the number of observations or in the variance of the explanatory variable. He also illustrates that the estimated coefficient on pooled individual data has the opposite sign from the estimated coefficient on grouped individual observations.

Besides these subtle problems related to correlated omitted variables, measurement error in the plotted variables can lead to erroneous hypotheses generated based on these plots. Specifically, Basu et al. (2020) generate the asymmetric provision hypothesis based on three plots showing i) provisions and allowance net charge-offs are linearly related, ii) the relation between the allowance net charge-offs and changes in nonperforming loans is V-shaped, and iii) the provision unexplained by allowance net charge-offs and changes in nonperforming loans have a shallower V-shape relation. Based on the assumption that allowance net charge-offs capture the measurement error in changes in nonperforming loans, they argue that the remaining nonlinearity in the plot represents provision asymmetry.

Fig. 1 shows the measurement error that results from the difference between NCO_NAL and NCO_ALL has a similar V-shaped relation with nonaccrual changes. This suggests that the allowance net charge-offs do not address substantial measurement errors in changes in nonaccruals and may even lead to more serious measurement error problems. In contrast, Fig. 2 shows a linear relation between the provision and additions to nonaccruals, suggesting there is no residual provision asymmetry. These two plots also suggest that the Basu et al. (2020) use of plots can be misleading.

Fig. 2.

Scatterplot of Provision vs. Additions to Nonaccruals both scaled by total loans. With locally weighted scatterplot smoothing (LOESS) curve and OLS estimates Scatterplot of loan loss provisions against additions to nonaccruals both scaled by beginning loans. The solid blue line represents the locally weighted scatterplot smoothing (LOESS) curve that non-parametrically depicts the relationship. The dashed black line represents the OLS estimate for the same data. . (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Finally, despite their admonition not to rely solely on regression estimates, Basu et al. (2020) seem to do exactly that when they argue a residual asymmetry is implied by the finding that “controlling for net loan charge-offs eliminates the V-shaped pattern” and the regression finding “that loan loss provisions increase more when nonperforming loans increase than they drop when nonperforming loans decrease.” Even worse than having conflicting results between the scatterplots and the regression analysis is when the two pieces of evidence agree but are driven by spurious correlations. Absent a theoretic model, interpreting the results as anything else is at best mere storytelling.

7. Conclusion

Basu et al. (2020) focus on a well-known measurement error problem that arises when changes in balances are used to capture flow variables. While it is difficult to assess this measurement error absent an underlying flow measure, when this data becomes available it is possible to directly evaluate the measurement error and assess how this measurement error affects alternative approaches designed to mitigate potential bias. We use data on additions to nonaccruals disclosed quarterly in regulatory reports from 2003 to 2017 to evaluate the measurement error in the Basu et al. (2020) models and to compare relative bias in their models compared to those in the existing research both in panel data and in bank-specific time-series models.

We first find that the residual asymmetry when controlling for allowance charge-offs Basu et al. (2020) document disappears when we control for the net charge-off measurement errors. This is consistent with the linear relation we document between provisions and additions to nonaccruals. This finding also provides no evidence that loan heterogeneity or the real estate crisis plays a role in the relation between provisions and change in credit quality, i.e., additions to nonaccruals. The substantially larger allowance charge-off measurement error during the 2007–2012 real estate crisis is also consistent with the documented larger negative association of the provision with decreases in nonaccruals during that period compared to the remainder of the sample period.

Our analyses indicate that using additions to nonaccrual loans to measure provision timeliness based on bank specific regression R² generates very similar inferences to that using changes in nonaccrual loans in linear models without controlling for allowance net charge-offs. This finding suggests that the model that is linear in changes in nonaccrual loans excluding net charge-offs is less affected by measurement error than the models proposed by Basu et al. (2020). Based on our analyses, we suggest that future research use the “additions to nonaccruals” to measure provision timeliness when this variable is available. However, absent that data, we argue that a model that is linear in changes in nonaccruals or nonperforming loans that does not include allowance charge-offs produces results that are more highly correlated with this direct model.

We also examine both false positives and false negatives when assessing the ability of provision models to capture earnings management. We show that regardless of provision asymmetry restrictions, including net charge-offs in provision models increases false negatives substantially more than it decreases false positives. It is not difficult to think that false negatives might be equally or more costly than false positives. A failure to reject the null of accounting discretion could lead to a spread of accounting manipulation to other banks that could increase systemic risk. In contrast falsely rejecting the null might create a disincentive for banks to use discretion in reporting their losses. Given the importance of detecting reporting manipulation we argue this is unlikely to be a good trade in most cases. Finally, while the Basu et al. (2020) paper highlights several issues important to consider in designing structural models and drawing causal inferences from those models, we argue that using plots can cut both ways especially without sound theoretical support.

Appendix.

Variable	Definitions
Loans	Total loans (BHCK2122)
LLP	Loan loss provisions (BHCK4230) scaled by lagged loans
ΔNAL	Change in nonaccruals (BHCK5526 or BHCK1403 relevant to period) scaled by lagged loans
DΔNAL	Dichotomous variable equal to 1 if ΔNAL is negative
ADD_NAL	Additions to nonaccruals (BHCKc410) scaled by lagged loans
NCO_ALL	Net charge-offs (BHCK4635–BHCK4605) scaled by lagged loans
NCO_NAL	Additions to nonaccruals (ADD_NAL) less change in nonaccruals (ΔNAL)
NCO_ME	Difference between NCO_NAL and NCO_ALL
Size	Logarithm of lagged total assets (BHCK2170)
ΔLOAN	Change in loan (BHCK2122) scaled by lagged loans
EBP	Earnings before provisions (BHCK4301+BHCK4230) scaled by lagged loans (BHCK2122)
Capital R1	Tier 1 regulatory capital ratio (BHCK8274/BHCKa223)