Learning objectives: Identify and describe Type I and Type II errors in the context of a backtesting process. Explain the need to consider conditional coverage in the backtesting framework. Describe the Basel rules for backtesting.
Questions:
22.8.1. Bertha is analyzing the application of the Fundamental Review of the Trading Book (FRTB) toward her firm's regulatory capital. Compared to prior Basel regulations, she observes that the FRTB changed the measure for determining regulatory market risk capital to expected shortfall (ES). However, because it is difficult to backtest ES, the FRTB retains a backtest of the value at risk (VaR) measure. Specifically, the FRTB backtests both a one-day 95.0% VaR and one-day 97.5% VaR over a period of one year (assume 250 days per year).
Consider a 95.0% confident two-tailed (2T) backtest of a one-day 97.5% VaR over the prior 250 days. The number of expected exceptions is 2.5% * 250 = 6.25, or about 6. Bertha defines the acceptance region as 2 < X < 12. Put another way, the lower rejection region is X ≤ 2 and the upper rejection region is X ≥ 12. Her one-day 97.5% VaR is $1.40 million. Which of the following statements is TRUE?
a. The probability of a Type II (aka, Type 2) error is about 7.44%
b. Bertha can increase the power of her test by shrinking the acceptance region to 3 < X < 11
c. Bertha's test will be more powerful if she shifts, ceteris paribus, from a backtest of the 97.5% VaR to a backtest of the 99.0% VaR
d. If Bertha doubles the sample size to N = 500 days, the acceptance region will remain (i.e., 2 < X < 12) but the level of the one-day 97.5% VaR would approximately double to ~$2.80 million
22.8.2. Betty's firm wants to allocate an economic capital cushion based on a one-day 99.5% value at risk (VaR) model. Further, she will conduct a basic backtest (aka, exception or frequency test) where the frequency of tail losses fit a binomial distribution. Her backtest window is limited to two years, T = 500 days. In regard to this backtest, which of the following statements is TRUE?
a. To increase the power of her backtest, she can decrease the VaR confidence level and apply a higher (k) capital multiplier
b. At this high VaR confidence level and relatively short backtest window, the normal approximation is a better fit to test the model's bias than the binomial distribution
c. She can achieve a more powerful test of the same hypothesis, and without the need for any additional information, by utilizing any of several extensions approved by regulators
d. Although VaR is always a one-tailed quantile (e.g., the 95.0% standard normal VaR is always 1.645 and never 1.960), the backtest is always a two-tailed (2T) backtest such that Betty must have a lower and an upper cutoff
22.8.3. Peter conducts a set of backtests according to the Christoffersen approach by rephrasing a basic frequency (aka, coverage) test in likelihood ratio (LR) form. His joint test statistic is given by LR(cc) = LR(uc) + LR(ind) where cc = conditional coverage and und = unconditional coverage. His results are the following:
b. Mixed news, the VaR model is unbiased but there is clustering (aka, bunching)
c. Mixed news, there is clustering (aka, bunching) but the VaR model is unbiased
d. Bad news, the VaR model is biased and there is clustering (aka, bunching)
Answers:
Questions:
22.8.1. Bertha is analyzing the application of the Fundamental Review of the Trading Book (FRTB) toward her firm's regulatory capital. Compared to prior Basel regulations, she observes that the FRTB changed the measure for determining regulatory market risk capital to expected shortfall (ES). However, because it is difficult to backtest ES, the FRTB retains a backtest of the value at risk (VaR) measure. Specifically, the FRTB backtests both a one-day 95.0% VaR and one-day 97.5% VaR over a period of one year (assume 250 days per year).
Consider a 95.0% confident two-tailed (2T) backtest of a one-day 97.5% VaR over the prior 250 days. The number of expected exceptions is 2.5% * 250 = 6.25, or about 6. Bertha defines the acceptance region as 2 < X < 12. Put another way, the lower rejection region is X ≤ 2 and the upper rejection region is X ≥ 12. Her one-day 97.5% VaR is $1.40 million. Which of the following statements is TRUE?
a. The probability of a Type II (aka, Type 2) error is about 7.44%
b. Bertha can increase the power of her test by shrinking the acceptance region to 3 < X < 11
c. Bertha's test will be more powerful if she shifts, ceteris paribus, from a backtest of the 97.5% VaR to a backtest of the 99.0% VaR
d. If Bertha doubles the sample size to N = 500 days, the acceptance region will remain (i.e., 2 < X < 12) but the level of the one-day 97.5% VaR would approximately double to ~$2.80 million
22.8.2. Betty's firm wants to allocate an economic capital cushion based on a one-day 99.5% value at risk (VaR) model. Further, she will conduct a basic backtest (aka, exception or frequency test) where the frequency of tail losses fit a binomial distribution. Her backtest window is limited to two years, T = 500 days. In regard to this backtest, which of the following statements is TRUE?
a. To increase the power of her backtest, she can decrease the VaR confidence level and apply a higher (k) capital multiplier
b. At this high VaR confidence level and relatively short backtest window, the normal approximation is a better fit to test the model's bias than the binomial distribution
c. She can achieve a more powerful test of the same hypothesis, and without the need for any additional information, by utilizing any of several extensions approved by regulators
d. Although VaR is always a one-tailed quantile (e.g., the 95.0% standard normal VaR is always 1.645 and never 1.960), the backtest is always a two-tailed (2T) backtest such that Betty must have a lower and an upper cutoff
22.8.3. Peter conducts a set of backtests according to the Christoffersen approach by rephrasing a basic frequency (aka, coverage) test in likelihood ratio (LR) form. His joint test statistic is given by LR(cc) = LR(uc) + LR(ind) where cc = conditional coverage and und = unconditional coverage. His results are the following:
- Test statistic LR(uc) = 2.58 which is less than the chi^2(1) cutoff of 3.841 such that he cannot reject the null hypothesis
- Test statistic LR(ind) = 15.36 which is too high such that he rejects the null hypothesis
b. Mixed news, the VaR model is unbiased but there is clustering (aka, bunching)
c. Mixed news, there is clustering (aka, bunching) but the VaR model is unbiased
d. Bad news, the VaR model is biased and there is clustering (aka, bunching)
Answers:
