VaR Backtesting (Jorion vs. Hull) - testable concept

[emilioalzamora1]

Hi David,

I would like to address another topic (from a computational perspective as well as from a testable perspective on exam day):

First, perhaps you can give me a bit of help with this: Jorion in his book 'VaR' gets the values for prob (X = N) in the fifth column of the attached screenshot (Basel Rules for Backtesting): 0.0; 0.4; 1.5; 3.8; 7.2............I tried several BINOMDIST options in Excel, but did not these results. Any ideas how he gets to these values and what they imply? I would be very happy to get either the formula in Excel or the formula for calculation by hand.

Second, is it likely that we need to compute Type I and II errors on the exam? If so, for what 'range' of exceptions? Anything beyond 5 exceptions would be computationally burdensome, right?
Does the exam clearly indicate whether they want to see for example: 5 or more exceptions, greater than 5 exceptions?
Your example 60.2 is not quite clear about this as it says 'if the committee increased the green zone from 4 to five exceptions' it does not say whether it is more than 4 exceptions or 4 exceptions or more.

Hull (Risk Management and Fin. Institutions, 1st Edition) in his examples 8.6 and 8.7 is a bit confusing as well, because he mentions 'we observe nine exceptions while the expected number of exceptions is six'. And in the next sentence he says 'the prob. of nine or more exceptions can be calculated....'

Another one, do we need to memorise the Kupiec formula for backtesting or perhaps the more tedious one by Christoffersen?

Thank you!

 

[David Harper CFA FRM]

Hi @emilioalzamora1

1. Re: Jorion's Table 6-4, please see tab "Table6-4" of this spreadsheet to see the binomial calcs (I recreated his exhibits to understand myself): https://www.dropbox.com/s/dhqyr4vza6kmqtl/5.c.1.var_backtest_jorion.xlsx?dl=0
2. Re: is it likely that we need to compute Type I and II errors on the exam? For the reasons you imply, it is (very) unlikely you would be asked to compute P(Type I errors) or P(Type II errors); I cannot recall such a question being asked. If it were asked, it would need to be on a small/truncated binomial subset (ie, it could only be asked if the binomial were not tedious). Instead, it is much more likely to be asked a conceptual/interpretation question about Type I/Type II errors
3. The green zone is zero to four inclusive; i.e., five to nine inclusive is the yellow zone. So, for example, four exceptions qualifies as "green." The confusion might be due to the binomial calculation. For example in Jorion Table 6-4, with respect to 4 exceptions (i.e., green), the P(X ≥ 4) = 1 - BINOM.DIST(3, 250, 1%, TRUE). Note this is essentially similar to the Hull question that you mention (copied below).
   
4. Re: "do we need to memorise the Kupiec formula for backtesting or perhaps the more tedious one by Christoffersen?" No, absolutely not. These would never be tested. I can't imagine the pushback GARP would get I hope that clarifies! Good job drilling down to understand this stuff!

Here is Hull's Ex 9-14:

"EXAMPLE 9.13 Suppose that we back-test a VaR model using 600 days of data. The VaR confidence level is 99% and we observe nine exceptions. The expected number of exceptions is six. Should we reject the model? The probability of nine or more exceptions can be calculated in Excel as 1− BINOMDIST(8,600,0.01,TRUE). It is 0.152. At a 5% confidence level we should not therefore reject the model. However, if the number of exceptions had been 12 we would have calculated the probability of 12 or more exceptions as 0.019 and rejected the model. The model is rejected when the number of exceptions is 11 or more. (The probability of 10 or more exceptions is greater than 5%, but the probability of 11 or more is less than 5%.)" - Hull. Risk Management and Financial Institutions (Wiley Finance) (Kindle Locations 6180-6186). Wiley. Kindle Edition.