Backtesting Var problem

[dymny]

I `ve got a problem with Hull example. We back test a var with 600 data. The var confidence is 99% with nine exceptions. Expected number of exceptions is 6. Should we reject model? The probability of 9 or more exceptions is 0,152.Should we reject model at 5% significance level? Answer is no but I don`t understand it. Can somebody explain me when I should accept or reject model using binomial distribution?
Thanks.

 

[brian.field]

If I understand correctly, increase the number of exceptions until you get to a quantile that reflects 1% significance or 99% confidence rather than 15%....say it is 15 or 20 exceptions.....then as long as your number of exceptions is below this level, you fail to reject the null. (I need to review this material too!)

The concern remains the potential for type I or type II errors! Which is a larger concern to you is a driving issue.

 

[QuantMan2318]

@dymny Well, use a shortcut for it, we can approximate the Binomial distribution as a Normal distribution and find out using the formula:

(x-p*n)/sqrt(n*p*(1-p)) where x is the currently recorded exceptions, p is the VaR significance, p*n is the expected exceptions for the VaR confidence level and n is the number of observations. If we substitute these values from your question, we get

(9-6)/sqrt(600*0.01*0.99) = 1.23; if you notice, the denominator is the sqrt of the Binomial distribution variance which serves as sigma here

The reference or critical value at a 95% confidence level for back testing is 1.96, our value is less, therefore we accept the VaR model

 

[QuantMan2318]

We can use another, albeit a more twisted way as well based on Logarithms; A chap named Kupiec found this by approximating the Binomial to a chi squared.

-2*ln((1-p)^(N-E)*p^E)+2*ln((1-E/N)^(N-E)*(E/N)^E), this is apparently a chi squared distribution, we know that chi squared is a squared of normal z
Therefore our reference value at 95% confidence is 1.96^2 = 3.84
Substitute the problem values in the formula, we get: 1.313 which is again less than 3.84, so we accept the VaR model

Coming to think of it, @brian.field expressed the above more elegantly because it is based on Exceptions/Total observations

 

[dymny]

Thanks guys. Best regards.Dymny.

 

[David Harper CFA FRM]

Hi @dymny can i ask you to avoid double-posting your question? Because I just answered your question (see below) because I work all of the outstanding questions in the forum and i didn't see that you already got help. Thanks


"Hi @dymny

Yes, given T = 600 and p = 1% = 100% - VaR confidence, the probability of 9 or more exceptions is 0.152 as 15.2% = 1 - BINOM.DIST(8, 600, 0.01, true). This also means that 15.2% is the p-value (we don't even need to consult a table, such a high lookup value tells us to avoid rejecting the null).

The graph below plots a binomial distribution, T=600, p = 0.01. This is the assumption embedded in the null hypothesis: that our backtest is characterized by a binomial distribution with p = 1.0%; if the VaR model is accurate, then we expect 1.0% exceptions. If the exceptions are truly each i.d.d., then we expect this binomial model conditional on an accurate model.

If we observe 9 exceptions, we are in the right tail (i.e., we observe more exceptions than we expected. We expected six). But it's only the 15.2% tail, it's not extreme at all; conditional on the truth of a binomial distribution with p = 1.0%, this outcome is quite plausible. The higher the p value, the less justification we have for rejecting the null, as the outcome is plausible to due randomness (random sampling). I hope that explains!

 

[bpdulog]

I `ve got a problem with Hull example. We back test a var with 600 data. The var confidence is 99% with nine exceptions. Expected number of exceptions is 6. Should we reject model? The probability of 9 or more exceptions is 0,152.Should we reject model at 5% significance level? Answer is no but I don`t understand it. Can somebody explain me when I should accept or reject model using binomial distribution?
Thanks.

Can I ask how you arrived at the probability of 9 or more exceptions as 15.2%? Is that provided in the question?

 

[ShaktiRathore]

Hi,
The probability of an exception=1%=0.01
probability of no exception=1-The probability of an exception=1-.01=.99
The variable of no either no exception of an exception is binomial distributed ,
the probability of 9 or more exceptions out of a sample size of 600
=1-probability of 8 or less exceptions out of a sample size of 600
=1-[600 C 0 *(.01)^0*(.99)^600+600 C 1 *(.01)^1*(.99)^599+600 C 2 *(.01)^2*(.99)^598+600 C 3 *(.01)^3*(.99)^597+600 C4 *(.01)^4*(.99)^596+600 C 5 *(.01)^5*(.99)^595+600 C 6 *(.01)^6*(.99)^594+600 C 7 *(.01)^7*(.99)^593+600 C 8 *(.01)^8*(.99)^592]
=1-[(600!/600!*0!)*(.01)^0*(.99)^600+(600!/599!*1!)*(.01)^1*(.99)^599+(600!/598!*2!)*(.01)^2*(.99)^598+(600!/597!*3!)*(.01)^3*(.99)^597+(600!/596!*4!)*(.01)^4*(.99)^596+(600!/595!*5!)*(.01)^5*(.99)^595+(600!/594!*6!)*(.01)^6*(.99)^594+(600!/593!*7!)*(.01)^7*(.99)^593+(600!/592!*8!)*(.01)^8*(.99)^592]
=1-((0.01)^0*(0.99)^600+600*(0.01)^1*(0.99)^599+(600*599/2)*(0.01)^2*(0.99)^598+(600*599*598/6)*(0.01)^3*(0.99)^597+(600*599*598*597/24)*(0.01)^4*(0.99)^596+(600*599*598*597*596/120)*(0.01)^5*(0.99)^595+(600*599*598*597*596*595/720)*(0.01)^6*(0.99)^594+(600*599*598*597*596*595*594/5040)*(0.01)^7*(0.99)^593+(600*599*598*597*596*595*594*593/40320)*(0.01)^8*(0.99)^592)
=.1517=~.152=15.2%
or you can get the the probability of 9 or more exceptions using the excel function BINOM.DIST(8, 600, 0.01, true).
BINOM.DIST(8, 600, 0.01, true) shall give the value of probability of 8 or less exceptions out of a sample size of 600 or the value of 1-[600 C 0 *(.01)^0*(.99)^600+600 C 1 *(.01)^1*(.99)^599+600 C 2 *(.01)^2*(.99)^598+600 C 3 *(.01)^3*(.99)^597+600 C4 *(.01)^4*(.99)^596+600 C 5 *(.01)^5*(.99)^595+600 C 6 *(.01)^6*(.99)^594+600 C 7 *(.01)^7*(.99)^593+600 C 8 *(.01)^8*(.99)^592].
the probability of 9 or more exceptions out of a sample size of 600=1-probability of 8 or less exceptions out of a sample size of 600=1-BINOM.DIST(8, 600, 0.01, true)=1-.8483=.1517=~.152=15.2%
thanks