FRM Fun 7 (Tue)

[Suzanne Evans]

FRM Fun 7.

GARP just released the pass rates for the May 2012 FRM: 47.3% for P1 and 61.1% for P2.

These results are somewhat in line with a recent historical trend in which the pass rate for P1 is below 50% and the pass rate for P2 is above 50%.

Here are the actual pass rates:

P1: 55.20%, 52.50%, 39.30%, 53.10%, 46.60%, 47.30%
P2: 54.0%, 54.9%, 61.6%, 57.0%, 61.1%

Therefore:

• P1: n = 6 with mean of 49.0%
• P2: n = 5 with mean of 57.7% (P2 was not offered when the test split into two parts in 2009)

Question #1: The average pass rates appear to be different, but on the other hand, these are very small samples. Statistically, is the P1 pass rate different than the P2 pass rate; i.e., is there a difference between the means with confidence?

Question #2: Can we conclude that P2 is an easier test, since a greater percentage pass it?

 

[Aleksander Hansen]

Don't have time to get my calculator out to do Q1 right now and it's definitely something P1 takers should know and give a try, but since Q2 is a low-hanging fruit and I'm a little lazy/busy I'll give it a go.

Answer: No, there is no basis for concluding that P2 is an easier test. From a statistics standpoint we need to think in terms of general selection bias, with the particular case in this category being survivorship bias.
There can be many plausible explanations for the higher passing rate for P2, but building on this theme, it does not take a leap of imagination to think along the lines that those that passed P1 are the "more qualified" candidates [this can be in any number of ways], and thus are more likely to pass P2 as well.

 

[ShaktiRathore]

Q1) m1=49% ; m2=57.72%
n1=6;n2=5
calculated standard deviations of samples is: s1=5.84% and s2=3.49%
Now establishing a two tail test:
Null Hypothesis: m1=m2 and Alt.Hypothesis: m1!=m2
t-stat for the two sample test is given by:
t-stat=(m2-m1)/s where s=sqrt((s1^2/n1)+(s2^2/n2))=2.85%
t-stat=(57.72%-49%)/2.85%=3.061 and t-crit at 95% confidence at 6+5-2=9 df is .0644
Here t-stat>t-crit implies that we reject the null hypothesis that means are equal and which suggest that we are 95% confident that the means of pass rates for the two parts that is partI and part II are different.This also suggest that m2>m1 or that the mean pass rates of part II exam are generally higher than part I exam.
Q2)Regarding 2nd question a higher pass rate does not necessarily mean a easier test.
from conditional probability: P(P2/P1)=P(P1&P2)/P(P1) where P1 and P2 are passing rates for part I and part II resp.
Now P2 and P1 are not independent events there is some correlation between the two.
Suppose if P1=50% then P(P2/P1)=2*P(P1&P2) which suggest probability of passing P2 given a candidate has passed P1 is twice the probability of passing P2 and P1 simultaneously. Now considering P(P1&P2) as .3 since candidates qualified in P1 are intelligent enough to pass P2 as well or say they are more hard working and that GARP wants minimum of atlest 30% of those who sit for the exam must finally should become frm to increase the scope of FRM across the world and ths increase risk profession across the world and that number cant surely be less than 25 to 30% of total candidates. So P(P2/P1)=2*.3=.6. So there is 60% chance of passing P2 given a candidate has passed P1 which is 20% higher probability than of simply passing Part II.If some day in future the Garp people decrease the overall passing rate then that can affect part 2 rates as well. So without considering the difficulty level of test we can reason out that pass rates for part 2 are generally on the higher side than part I. Hope this satisfies your 2nd part.

 

[troubleshooter]

Question 1: Without doing any calculations, I would think the difference between two means are not statistically significant mainly due to the small sample set that leads to higher t-stat.
Quesion 2: As Aleks pointed out, survivorship bias would mean that we should expect pass rate for L2 higher than L1. If they were equal, then that would really mean L2 is actually more difficult.

 

[David Harper CFA FRM]

@ShaktiRathore, you win the star for this (and one entry into the weekly drawing, of course!). Thank you for elaborating on the full calculation, which in my opinion, correctly employs a comparison test for the difference between two means (see Stock and Watson 3.4).

Your result matches my calculations exactly, here is my spreadsheet: https://www.dropbox.com/s/aqcxdhhr4whuy4p/0712_frm_fun7.xlsx
i.e., a computed t (test) statistic of 3.06 which is greater than a critical value of 2.262 @ 95%

I get a p-value of 1.35%. Therefore, owing essentially to the small sample, we can either conclude these means are different with 95% confidence; or we can fail to conclude they are different with 99%. I say "fail to conclude" because the null is that they are the same (null = difference of means is zero), such that we can reject the null (i.e., the means are not the same) or fail to reject the null.
(... at some future point, I will not be surprised if this analysis is challenged on the grounds that it is not strictly a test of the difference of two means given the two "populations" are varying over time and therefore the underlying populations are not two but many. I'd agree the exercise may not be a perfect fit.)

Your answer to (b) is very interesting, I had not thought to try and formalize based on conditional probabilities. In my humble opinion, I might challenge your argument because it appears to "assume the conclusion." True, if GARP desires a joint probability P(P1,P2) = 30%, then it seems to me that for any P(P1) < 55%, P(P2|P1) must exceed 55% (i.e., SQRT(30%) = 55%). For example, if P1 = 56%, then the necessary P(P2|P1) is a lower 53.6%. However, to be honest, I could still believe your math could plausibly reflect their thought process!

I was thinking survivorship bias myself, fwiw, here is my FB exchange with GARP:

@David Harper @GARP FRM: do you share numbers of passers? my hypothesis re the higher P2 pass rate includes "survivorship bias" (the possibility that fewer, better prepared candidates sit for P2 ... contra the notion that P2 is easier)
@GARP FRM ‎@David Harper: We only score Part II exams of candidates who have passed Part I (either earlier that same day or on another exam date), so we have more Part I candidates than Part II candidates.

Please note the point from GARP, I forgot this until they reminded me: the P2 pass rate does not include the dual P1-P2 candidates who fail P1. This further supports the explicit survivorship by nullifying a certain percentage of sitting P2 candidates, who are filtered (with bias) out. In other words, the P2 pass rate is fully conditional (even simultaneously) such that we should not expect the candidate populations to be equivalent.

Thank you, I hope you enjoyed the exercise as much as I did. At the very least, I am not somewhat confident (specifically, i am 1 - 1.35% or 98.7% confident!) that the P2 pass rates are higher than P1 pass rates. Cheers,