GARP 2016 Q61

[rohinjain]

Hi all,

I was just going over a past exam and stumbled upon this question. If you have already had the discussion on this question, could you please redirect me to the relevant forum (I can't seem to find this specific question in the past paper forum discussions). Here is the question:

You are backtesting a bank’s VaR model. Currently, the bank calculates a 1-day VaR at the 99% confidence level, and you are recommending that it switch to a 95% confidence level. Which of the following statements concerning this switch is correct?
a. The 95% VaR model is less likely to be rejected using backtesting than the 99% VaR model.
b. When validating with backtesting at the 90% confidence level, there is a smaller probability of incorrectly rejecting a 95% VaR model than a 99% VaR model.
c. The decision to accept or reject a VaR model based on backtesting results is more reliable with a 95% confidence level VaR model than with a 99% confidence level model.
d. When backtesting using a 90% confidence level, there is a smaller probability of committing a type I error when backtesting a 95% VaR model than with a 99% VaR model.
Correct answer: c
Explanation: The concept tested here is the understanding of the difference between the VaR parameter for confidence (here, namely 95% vs. 99%) and the validation procedure confidence level, and how they interact with one another. Using a VaR confidence level creates a narrower rejection region by allowing a greater number of exceptions to be generated. This in turn increases the power of the backtesting process and makes for a more reliable test

Why is (a) false? I thought that with a lower confidence interval, the 95% level would allow for more exceptions before rejecting the null hypothesis. Given the same confidence level for the backtest (e.g. 90%), shouldn't this be true?

Thanks
Ro

 

[David Harper CFA FRM]

Hi @rohinjain This question has been analyzed quite a bit due to the confusion it created. The primary problem is GARP's language, but I don't know if they've ever fixed it; see https://forum.bionicturtle.com/threads/garp-2020-p2-53-and-garp-2019-p2-53.22374/post-75487 (and maybe here too https://forum.bionicturtle.com/threads/practice-question-3-backtesting-var.8589/) i.e.,

Hi @aangermeyer Please see this previous discussion with @emilioalzamora1 https://forum.bionicturtle.com/thre...-backtest-significance-jorion.3604/post-59878 that includes my XLS visualization (see below) of the difference between 99.0% VaR (i.e., p = 1.0%) and 95.0% VaR (p = 5.0%) when the backtest confidence for both is the same 95.0%; this is just a visual extract from Jorion's Table 6-2:
@emilioalzamora1 I'm not following your calcs, sorry. Or maybe we get the same place. Here is my XLS https://www.dropbox.com/s/jp9qf6jt1arczui/0511-backtest-conf.xlsx?dl=0

The mean shifts (dark green). I'm using Table 6-2 acceptance regions (notice less than, not equal to or less than). Then my "total reject" = 100% - (sum of acceptance pmfs); i.e., total of all red cells. The shift to 95.0% appears to create a greater probability in the rejection region. But as before, I have a hard time interpreting this as wider/narrower: these values are pmf densities! So to me, the best description is "increase in power."

For those who wish to take a deep dive, you will see there is a salvageable (but highly unlikely) interpretation of GARP's previous statement that "Using a 95% VaR confidence level creates a narrower rejection region than using a 99% VaR confidence level" which, I can see in the 2019 Practice Exam, they have edited to "Using a 95% VaR confidence level creates a narrower non-rejection region than using a 99% VaR confidence level." As before, I suspect they do not understand (we did issue some guidance) the technical issues here. Using this language is should say instead "Using a 95% VaR confidence level creates a narrower rejection region than using a 99% VaR confidence level" because, put simply (see above where green is nonrejection and red is rejection), and keeping in mind that both refer to the same 95.0% confident backtest, the switch from 99.0% VaR (left) to 95.0% VaR (right column) increases the width of the nonrejection region from 6 to 13; i.e., N < 7 to 6 < N < 20. Although per our long discussion, this seems confusing even when correctly stated!

So my view is that the explanation contains a mistake and should read:
Using a 95% VaR confidence level creates a wider nonrejection region [d harper note: that's green above!] than using a 99% VaR confidence level by allowing a greater number of exceptions to be generated. This in turn increases the power of the backtesting process and makes for a more reliable test than using a 99% confidence level.

But the usage of increase in power is correct and technically the best. If I were to attempt a simplified explanation, it would be something like this:
Given a constant assumption of 95.0% confidence for the backtest, the choice here is between a 95.0% and 99.0% confident VaR. Keep in mind that, by definition, the 99.0% VaR is a higher loss threshold, so while the backtest confidence is the same, the VaRs are not! Notice that switching from 95.0% to 99.0% VaR greatly reduces the number of expected exceptions (during one year) from 12. 5 to 2.5. Related, this narrows the nonrejection (aka, acceptance region) from a width of 13 to only 6 (and they are only on one side). This is due to the fact that at the higher VaR there are fewer exceptions in the extreme tail and consequently less data with which to make the statistical inference. Technically, we can say there is less power in the 95% test of the 99.0% VaR and more power in the 95% test of the 95% VaR which is an increase in the probability of correctly rejecting a false null; i..e, more power is an increase in the probability of correctly rejection an inaccurate VaR model.

I hope that's helpful.