De Laurentis, Validating Rating Models

[Roddefeller]

Hi @David Harper CFA FRM,

I was just reading through the "Validating Rating Models" Chapter and I'm left slightly confused with the definition De Laurentis uses for Type 1 and Type 2 errors with regard to the graph on p16 on the slides or p160 figure 10-5 in the original readings. What De Laurentis calls a Type 2 error (credit restriction for non-default), isn't this a clear case of a Type 1 error according to our understanding, as we reject a "good" one?

Many thanks for clarification.

Updated by Nicole to include graph that is being referenced:

 

[David Harper CFA FRM]

Hi @Roddefeller Great question, this isn't easy. I think it depends on the definition of the null hypothesis. The chart itself doesn't offend me, so to speak. Let's work backwards from what we know to be factual: to commit a Type I error is to mistakenly reject the null (when actually it is true), and to commit a Type II error is to mistakenly accept the null as true, when actually it is false (https://en.wikipedia.org/wiki/Type_I_and_type_II_errors). In the chart above, De Laurentis labels Type I errors (on the left) as lending to borrowers who actually default, and Type II errors (on the right) as rejecting borrowers who actually do not default. Therefore, the null hypothesis is "borrower will default," because, the Type II error (on the right) is mistaken acceptance of this null. So, this is not itself troubling: Altman's Z similarly defines the null as "borrower will default" (https://en.wikipedia.org/wiki/Altman_Z-score) and, btw, so does De Servigny in Chapter 3's mention of Neyman-Pearson where the Type I error (i.e., classification of a good borrower who will actually default) is the more damaging to the bank (although this has caused confusion, too). I do notice this is consistent with his earlier plot, see below, where the Type I error of "Predicted = Good, but Actual = Bad" implies the null is "borrower is bad." Although, the setup does lead to identifying the Type II are as a "false positive" and Type I error as a "false negative" which is the opposite of convention; and why we probably should be mindful that Type I/II errors are defined in relationship to the stated null. (statistically, this is not always arbitrary. Sometimes you can switch the null and the alternative, but sometimes you can't: the null hypothesis alone contains the "=" so it requires an affirmative definition). I hope that's helpful!

 

[Roddefeller]

Hi @David Harper CFA FRM,

thanks a lot for clarifying this. I get it know, key is how the H0 Hypothesis is stated and Type 1 Error is rejecting a good Null while Type 2 is mistakenly accept a bad Null.

Thanks a lot!