Backtesting Exceptions

[ashm07]

Hi David,
Generally in 95% VaR, it is said that it can cause 5% exceptions say in 252 tests ~12.6 or 13. Normally we take 1.65 normal deviate and check it out. In the calculations in Jorion table 6.2 calculates this using the scheme to be less than 20. The number is actually 19.4.
1. why is the difference between counting off 5% of n. like 13, versus calculating, 19.4, based on confidence?
2, Also, the calculation is based on 2 tails of 1.96. The 95% confidence in VaR is one tail. Is this being done because the z score is cumulative so should it not use one tail of 1.65? When should we use one tail and when 2 tails? is this because z score estimation of the binomial distribution is not accurate?
Will really appreciate your clarification.
Thanks.

 

[David Harper CFA FRM]

Hi seaTurtle,

Please note I have XLS, which may be helpful @
http://www.bionicturtle.com/premium/spreadsheet/5.c.1._jorion_backtest/
(2nd tab replicates the JP morgan example)

...and i think this is an instructive practice question: http://forum.bionicturtle.com/viewthread/1409/

1. Difference is not confidence, if it were then the cutoff of 19.4 (as a two-tailed 95%) would equal the one-tailed 90% VaR (= 10% of 252 = 25). If only matter of confidence, 19.4 would match the 25.2. Rather, the cutoff here is an application of the significance test of the mean of a binomial distribution (using normal to approximate): is the observed number of exceptions statistically different from the mean (expected) number of exceptions. So, the analogy here is the difference between the "naked" distribution and the (sampling) distribution of sample means which operates per CLT.

2. If you see the other thread, this is understandable confusion re: the backtest. Note there are two distributions involved:

a. The underlying loss distribution upon which Jorion, as usual, lists different confidence levels; e.g., 1% significance, 5% significance. VaR is always one-tailed.
b. The model backtest which is a significance test of the observed number of exceptions (which, here i think the key is, to view this as: a test of the observed exceptions against the hypothesized sample mean). Like our default signficance tests in Gujarati (e.g., is slope coefficient significant?) where the null is "=" and the alternative is "< or >", this is two-tailed and happens to be 95%.

Hope that helps, David

 

[Vaishnevi]

Hi David, To convert a z of 0.69 when number of VaR exceptions is 15, why is the formula in excel like NORM.S.DIST(0.69,TRUE)*2-1. Could you please let me know the significance of multiplying by 2 and subtracting by 1?

 

[David Harper CFA FRM]

Hi @Vaishnevi I don't grok the exact numerical context (ie, where the 0.69 comes from) but I'm familiar with the formula: it's translating a one-sided acceptance region into a two-sided acceptance region (and the backtest is typically two-sided, unlike VaR which is always one-sided).

What I mean is:

• NORM.S.DIST(z, true) returns the area to the left of the tail; e.g., NORM.S.DIST(0.69, true) = N(0.69) = 74.49% is the area to the left of Z = 0.69 in the standard normal distribution
• If we want the area (under the standard normal probability density function) between -0.69 < Z < + 0.69 then we want N(0.69) - N(-0.69) but per the symmetry of the normal N(-0.69) = 1-N(0.69) such that N(0.69) - N(-0.69) = N(0.69) - [1- N(-0.69)) = N(0.69)*2 - 1 = NORM.S.DIST(0.69, true)*2-1 = 50.98% which is the area "inside the acceptance region;" or, we can visualize it as the entire area then "subtracting" the 24.5% area on both the left and right. I hope that helps!

 

[Vaishnevi]

Thanks a lot. Yes it really helped.