Backtesting VaR Calculations

[monsieuruzairo3]

Hello @David Harper CFA FRM CIPM
I am really struggling understanding the math behind the folowing BT Backtesting calculation problems.
At current, the Basel II IMA green zone refers to four or fewer exceptions, the red zone
refers to 10 or more, and the yellow zone refers to five through nine (i.e., green <=4, 5 <=
yellow <=9, red >=10). If the Committee increased the green zone from four to five (5)
exceptions and both yellow or red were to signify “reject the VaR model,” the probability of
a Type I error changes from 10.8% to what (assume 250 days)?
a) 4.1%
b) 8.2%
c) 10.8%
d) 12.6%

You have used BINOMDIST and calculated the probability. WOuld you be so kind to help me understand how I can do it on my calculator?

PS: Would you think these questions have a good chance of appearance on the exam

Thanks in advance
Uzi

 

[David Harper CFA FRM]

Hi @monsieuruzairo3

The source Q&A is here (that's question 60.2) at https://forum.bionicturtle.com/threads/l2-t5-60-basel-ima-backtest.3610 and the question is (clearly) more difficult than you'd find on an exam. You are right about that, exam-wise binomial applications will be more accessible ... the exam simply cannot ask for the Pr[X <= 5 | p = 1%, T = 250]. However, I think it's a good concept and, actually, if you fully understand the concept then there is a shortcut: by trimming the rejection region, we are shifting from (1 - Pr[X <= 4]) to (1 - Pr[X <= 5 ]); i.e., from the rejection tail "greater than 4" to rejection region "greater than 5". As Pr[X <= 5 ] = Pr[X <= 4] + P[X=5], we only need to subtract P[X=5] from 10.8%. P[X=5] is obtainable from the calculator: .99^245*.01^5*[250*249*248*247*246/fact(5)] = 6.63% and indeed 10.8% - 6.63% = 4.14%.

Don't get me wrong: per the discussion thread in the Q&A, the question I wrote is clearly "too difficult" for an exam. But, honestly, I like what it tests here, to grok the question is to grasp a key part of backtesting. I hope that helps!

 

[monsieuruzairo3]

Thanks @David Harper CFA FRM CIPM

I wish to clarify one thing with you. Does Probability of Type 1/Type 2 error decrease or increase as we increase the confidence level.

My reasoning(very shaky, no foundation): Since on increasing the confidence level the no of exceptions permitted under the LR (uc) test dramatically decrease so we will be rejecting more null hypothesis (null: model is correct) and since we are increasing Type 1 error, the Type 2 error decreases. Even in the above example, we are increasing the number of exceptions from 4 to 5 and we see probability of type 1 error decrease from 10.8% to 4.14%

Now as per Scheweser and I quote "Industry analysts have suggested lowering the VAR confidence level to 95% and compensating by using a greater multiplier. The 1 -year exception rate at 95% level would be 13 and with more than 17 exceptions, the probability of Type 1 error would be 12.5% (increase from 10.8% above) but the proability of Type 2 error at this level would fall to 7.4% (caompared to 12.8% at 97.5% confidence level). Thus inaccurate models are accepted less frequently.

Now I am really in a fix by comparing the above two contrary points.

Thanks
Uzi

 

[David Harper CFA FRM]

Hi @monsieuruzairo3 I am giving your question a star (enters into weekly drawing) because it's a nice illustration of a common (understandable!) confusion. You have isolated it very nicely with a keen observation, really, it really does look like a contradiction at first glance!

The resolution is that we are talking about two difference confidences:

1. There is a choice about the VaR confidence level. Schweser's totally correct statement refers to the argument for lowering market risk VaR confidence from 99.0% (expected exceptions = 1% * trading days) to 95.0% (expected exceptions = 5% * trading days). Think of this as informing the database upon which we conduct the backtest; i.e., given a VaR confidence level, for a given history, we will have a series of observed exceptions and non-exceptions
2. The backtest is a significance test, employing the binomial, which (like any significance test) can use any confidence/significance level. The typical is a 95% confident backtest of a 99% VaR.

Re: Does Probability of Type 1/Type 2 error decrease or increase as we increase the confidence level.
Confidence% = 1 - significance (aka, alpha) and significance = 1 - confidence%. If we increase the confidence level, we decrease the significance level (aka, alpha) and the significance level is the prob of a Type I error; visually, significance level is the area(s) in the reject tails. As there is a trade-off, given a fixed sample, if the decrease the Prob of a Type II error. So, virtually by definition: increase confidence --> decrease significance = decrease Pr[Type I] --> increase Pr[Type II].; visually, by truncating the reject tails, we increase the body region where the error is to accept a truly false null.

So when you write "Since on increasing the confidence level the no of exceptions permitted under the LR (uc) test dramatically decrease so we will be rejecting more null hypothesis (null: model is correct)" ... the first part is correct but means "Since on increasing the VaR confidence level the no of exceptions permitted under the LR (uc) test dramatically decrease" and this is so far a true statement but then you shift from VaR to backtest such that it does not follow that "so we will be rejecting more null hypothesis (null: model is correct)" I hope that resolves, thanks!

 

[siddharthmahanty]

Hi @David Harper CFA FRM
i am confused and seek some guidance . Acc to me (as the qs above )says expanding the green zone from 4 to 5 would mean increasing the Significance level and decreasing the CL, thereby increasing the prob of T1 error.is it not? why do you call it trimming. so we know P(<=5) is 10.8% and P(5) is 6.66% so why calculate P(<=4)?

also what does shrinking a zone mean? can logic of expanding/shrinking green zone be same for expanding/shrinking yellow/red zone?