Learning objectives: Define backtesting and exceptions and explain the importance of backtesting VaR models. Explain the significant difficulties in backtesting a VaR model. Verify a model based on exceptions or failure rates.
Questions:
712.1. Sally the risk manager is backtesting her company's one-day 97.0% value at risk (VaR) model over a two-year horizon. Because there are 250 trading days in a year, the horizon is 500 days. For her two-tailed backtest, the desired confidence is 95.0% If she uses a normal distribution to approximate the binomial, then what is the maximum number of daily losses that can be observed (aka, the "cutoff") in order to conclude the model is calibrated correctly? (note: inspired by GARP's 2017 Part 2 Practice Exam, Question 12).
a. 15 exceptions
b. 19 exceptions
c. 22 exceptions
d. 25 exceptions
712.2. Peter the risk manager conducts a backtest of his company's 97.0% value at risk (VaR) model over a one-year horizon that includes 250 days. He observes only one exception; i.e., for the year the loss exceeded the VaR only once. If his backtest requires a one-tailed confidence level of 99.0%, should he conclude the model is correctly calibrated (note: because there are two "yes" and two "no" choices, please select the best answer)?
a. No, the model is probably bad because the cutoff (inclusive of the acceptance region) is two or more exceptions
b. No, the model is probably bad because the cutoff (inclusive of the acceptance region) is four or more exceptions
c. Yes, the model is probably good (ie, cannot be rejected) because the z-based cutoff extends to 0.55 exceptions and this straddles zero and one exception
d. Yes, the model is probably good (ie, cannot be rejected) because the the asymmetry of the binomial precludes left-tail rejection for this combination of probability and sample size
712.3. Charlie the risk manager has a small sample of only 60 days (three months each of 20 trading days) for which he wants to conduct a backtest. He observes zero exceptions over this short horizon. His VaR model is calibrated at 95.0% which, conditional on an accurate VaR model, occurs with a probability of only 0.95^60 = 4.61%. Assuming a one-tailed 95.0% confidence level, he considers the model bad, if barely. He really wished he could accept the model. Under each of the following alternative scenarios, ceteris paribus, he could accept the model EXCEPT which statement is false?
a. If zero exceptions had occurred for a 97.0% VaR model, then he could have accepted the model as good
b. If he had lowered his backtest confidence to 90.0%, then he could have accepted the model as good
c. If zero exceptions had occurred for a smaller sample of n = 50 days, then he could have accepted the model as good
d. If he switched to a two-tailed 95.0% backtest and used the normal approximation, then he could have accepted the model as good (even as his reliance on the normal approximation is dubious for this sample/probability scenario)
Answers here:
Questions:
712.1. Sally the risk manager is backtesting her company's one-day 97.0% value at risk (VaR) model over a two-year horizon. Because there are 250 trading days in a year, the horizon is 500 days. For her two-tailed backtest, the desired confidence is 95.0% If she uses a normal distribution to approximate the binomial, then what is the maximum number of daily losses that can be observed (aka, the "cutoff") in order to conclude the model is calibrated correctly? (note: inspired by GARP's 2017 Part 2 Practice Exam, Question 12).
a. 15 exceptions
b. 19 exceptions
c. 22 exceptions
d. 25 exceptions
712.2. Peter the risk manager conducts a backtest of his company's 97.0% value at risk (VaR) model over a one-year horizon that includes 250 days. He observes only one exception; i.e., for the year the loss exceeded the VaR only once. If his backtest requires a one-tailed confidence level of 99.0%, should he conclude the model is correctly calibrated (note: because there are two "yes" and two "no" choices, please select the best answer)?
a. No, the model is probably bad because the cutoff (inclusive of the acceptance region) is two or more exceptions
b. No, the model is probably bad because the cutoff (inclusive of the acceptance region) is four or more exceptions
c. Yes, the model is probably good (ie, cannot be rejected) because the z-based cutoff extends to 0.55 exceptions and this straddles zero and one exception
d. Yes, the model is probably good (ie, cannot be rejected) because the the asymmetry of the binomial precludes left-tail rejection for this combination of probability and sample size
712.3. Charlie the risk manager has a small sample of only 60 days (three months each of 20 trading days) for which he wants to conduct a backtest. He observes zero exceptions over this short horizon. His VaR model is calibrated at 95.0% which, conditional on an accurate VaR model, occurs with a probability of only 0.95^60 = 4.61%. Assuming a one-tailed 95.0% confidence level, he considers the model bad, if barely. He really wished he could accept the model. Under each of the following alternative scenarios, ceteris paribus, he could accept the model EXCEPT which statement is false?
a. If zero exceptions had occurred for a 97.0% VaR model, then he could have accepted the model as good
b. If he had lowered his backtest confidence to 90.0%, then he could have accepted the model as good
c. If zero exceptions had occurred for a smaller sample of n = 50 days, then he could have accepted the model as good
d. If he switched to a two-tailed 95.0% backtest and used the normal approximation, then he could have accepted the model as good (even as his reliance on the normal approximation is dubious for this sample/probability scenario)
Answers here:
