FAQ After Exam Quartiles and weights - Exam results

[sipanivishal]

Hi David,

I have heard that they provide quantiles for each section,so if the questions are mixed then how do they evaluate individual quantile ? Moreover pass and fail depends on overall marks or one has to clear cut offs in each section ?

Thanks
Sipani

 

[hellohi]

I asked GARP about quartiles and weights in the exam result,,,, and my question was as follow:

according to my notes from the last exams results, when I were look at the exam quartiles results ... I saw 4 quartiles (FRM part 1)....if we look at two candidates quartile results for example 1233 and 1332, the frist candidate faild and the second who got the second result passed, if they got the same frequancy quartiles why one passed and the other faild, me and many agree that the weight of the book (30% or 20%) and the better quartile result related to this book that has more weight effect the final result (pass or fail).
for clarification if we took the above example we see that the first candidatie took 1233 thats mean 33 in the third and fourth more weight books (30% not the 20%)so he is faild despit he got the same four quartiles as the one who passed but in different orderlines?????

and they answerd me three times by three emails as follow:

the first email

A 20% subject area is worth 20 Pts of the total exam
a 30% subject area is worth 30 Pts of the total Exam.
A 3rd quartile (vertical is worth between .26 and .50.
Therefore a 20 % subject area is worth between 5.2 pts and 10 points
Therefore a 30% subject are is worth between7.8 and 15 points.
Sincerely,
Frank Weber

the second email

In addition to your calculation with regard to horizontal value of the subject areas, you must remember the vertical differential in each quartile has a range of 25 points.
Your comparison is not valid.
Sincerely,
Frank Weber
Vice President - MemberServices


the third email
Quartile Explanation from Chris Donohue


Thank you for your interest in and commitment to the FRM program. As you know, your exam results are released only with a pass or fail grade. Quartile data is provided only as a broad guideline to show you what your performance was relative to your peers taking the exam on the same day. This is valuable information if you did not pass the exam in that it will direct you to the areas that need improvement.

Please note that, by definition, each quartile number covers a 25% vertical range and your performance could be anywhere within that range. Additionally, subject area values are not evenly distributed. Performing well in smaller subject areas. Subject areas also have a different % value. Taken in context, this means many candidates may have the same or very similar quartiles with some passing and some failing because their scores vary by a sizeable margin. The quartiles are issued to provide you with a more in depth view of your performance so you may know what path you need to take in your studies; they cannot be used to make comparisons to other candidates ‘overall score.

may the memebers of this forum discuss and explain what GARP says related to my question above?

thanks
Nabil

 

[kevolution]

I think what they're trying to say is that there's a lot of variability not only with weighted percentage of each section, but also within the range of tthe Quartiles.

For example:

Student 1: Q2, Q2, Q3, Q3
Student 2: Q2, Q2, Q3, Q3

Perhaps Student 1 passes and Student 2 fails even though they have equivalent Quartile results because Student 1 was Top 51% for his Q3 results (top of Q3 Quartile) while Student 2 was Top 74% (bottom of Q3 Quartile but still in Q3). So if they are borderline to the cutoff, then Student 1 would have the statistical advantage over Student 2 in passing.

 

[hellohi]

I think what they're trying to say is that there's a lot of variability not only with weighted percentage of each section, but also within the range of tthe Quartiles.

For example:

Student 1: Q2, Q2, Q3, Q3
Student 2: Q2, Q2, Q3, Q3

Perhaps Student 1 passes and Student 2 fails even though they have equivalent Quartile results because Student 1 was Top 51% for his Q3 results (top of Q3 Quartile) while Student 2 was Top 74% (bottom of Q3 Quartile but still in Q3). So if they are borderline to the cutoff, then Student 1 would have the statistical advantage over Student 2 in passing.

thanks a lot for your great effort,,,,is there any ideas else? or it is the only one?

 

[David Harper CFA FRM]

@hellohi

Okay I built this small spreadsheet (because trying to talk about this soon becomes a word soup imo), please see https://www.dropbox.com/s/ikhbkm0571fdchh/1019-garp-frm-scoring-exam.xlsx?dl=0
... and below is a snapshot. This is just for P1 and you only input (change) the yellow cells, the rest is calculated.

Notice how I input an extreme version of the one you referenced in your email; i.e., Student #1 earns 3/3/2/2 and passes while Student #2 earns 2/2/3/3 but fails. I was deliberately provocative: notice how the seemingly subtle difference can lead to a difference between a final score of 62 and 36 (wow!). Caveat: I'm not sure my quantiles are exactly calibrated, but they can't be too far off. I hope that clarifies!

 

[hellohi]

dear @David Harper CFA FRM

thanks .... great explanation

 

[Elnur1]

Hi @David Harper CFA FRM .There is something which i cant understand.The Peak level of normal distribition which GARP send to show results is 1200 in my distribition but 1500 in the other members i wont to know what is that .I gave fist level FRM in 19 Novermber 2016 and i failed results 2334 .Please if you have any information about this write me .

Thansk

 

[David Harper CFA FRM]

Hi @Elnur1 I admit that I am not familiar with this "normal distribution" reporting (peak at 1200 versus 1500) to which you refer; apologies, but I am only familiar with GARP's quantile scoring (eg, 2334). cc @Nicole Seaman I haven't analyzed the feedback thread, have you seen reference to a "normal distribution" reported by GARP along with the results?

 

[Elnur1]

Hİ@David Harper CFA FRM .İ add attachment you can see the difference .

 

[David Harper CFA FRM]

Hi @Elnur1 I'm baffled by the y-axis units of the displayed normal distribution; given the density, I believe the max y is 1/[sqrt(2*pi)*σ]; i.e., the pmf density function without the effectively discounting exp(). I don't see how the axis can get that high. I suspect it is merely visual. Please note GARP's methodology is based on quartiles (quantiles): there is no need to assume or impose normality. GARP's method does not impose normality, as far as I understand (it already quite sufficiently "grades on a curve"). The graphs may just be visual signals. And perhaps they varied the y-axis values because according to the topic weights. @Nicole Seaman If @Elnur1 really prefers, do you mind reaching out to confirm my assumption that "the normal graphs displaying along with the quartile reporting are merely visual indicators to convey quartile status. Specifically, the y-axis values are not exactly meaningful, nor do the pictorial normals imply any necessary normality; ie, normality would be coincident, as our understanding is that GARP does not actually impose normality on the score distribution." Thanks!

 

[Nicole Seaman]

Hi @Elnur1 I'm baffled by the y-axis units of the displayed normal distribution; given the density, I believe the max y is 1/[sqrt(2*pi)*σ]; i.e., the pmf density function without the effectively discounting exp(). I don't see how the axis can get that high. I suspect it is merely visual. Please note GARP's methodology is based on quartiles (quantiles): there is no need to assume or impose normality. GARP's method does not impose normality, as far as I understand (it already quite sufficiently "grades on a curve"). The graphs may just be visual signals. And perhaps they varied the y-axis values because according to the topic weights. @Nicole Seaman If @Elnur1 really prefers, do you mind reaching out to confirm my assumption that "the normal graphs displaying along with the quartile reporting are merely visual indicators to convey quartile status. Specifically, the y-axis values are not exactly meaningful, nor do the pictorial normals imply any necessary normality; ie, normality would be coincident, as our understanding is that GARP does not actually impose normality on the score distribution." Thanks!

@David Harper CFA FRM and @Elnur1

I emailed GARP about this, and received this response today,

"David is right. The graphs are only from a visual perspective, so candidates can see their strengths and weakness. That’s it."

I hope this helps to clear up any confusion

Nicole

 

[David Harper CFA FRM]

@Nicole Seaman Thank you, good to know!

 

[Elnur1]

@Nicole Seaman thank you .David is always right)

 

[Jash]

Dear @David Harper CFA FRM @Nicole Seaman

This is more of off topic question, I guess
But if i have passed 3 section out of 4 section in frm and failed in one section..
So should i able to clean part 1 or i need to pass separately in all individual section.

 

[Nicole Seaman]

Dear @David Harper CFA FRM @Nicole Seaman

This is more of off topic question, I guess
But if i have passed 3 section out of 4 section in frm and failed in one section..
So should i able to clean part 1 or i need to pass separately in all individual section.

Hello @Jash

I moved your post to this thread, which discusses the exam results and quartiles. The discussion in this thread should help to answer your questions about how GARP determines a pass/fail.

 

[Jash]

Hi @David Harper CFA FRM @Nicole Seaman
Can you just tell me what are the minimum score that i have to score in each topic.

 

[David Harper CFA FRM]

@Jash There is not a minimum in each topic. Can you try and read the thread first, then you can benefit from what's already been written. Above I shared a concrete (and counterintuitive) example which should be clarifying. Thanks,

 

[dtammerz]

@hellohi

Okay I built this small spreadsheet (because trying to talk about this soon becomes a word soup imo), please see https://www.dropbox.com/s/ikhbkm0571fdchh/1019-garp-frm-scoring-exam.xlsx?dl=0
... and below is a snapshot. This is just for P1 and you only input (change) the yellow cells, the rest is calculated.

Notice how I input an extreme version of the one you referenced in your email; i.e., Student #1 earns 3/3/2/2 and passes while Student #2 earns 2/2/3/3 but fails. I was deliberately provocative: notice how the seemingly subtle difference can lead to a difference between a final score of 62 and 36 (wow!). Caveat: I'm not sure my quantiles are exactly calibrated, but they can't be too far off. I hope that clarifies!

@David Harper CFA FRM would you have the same spreadsheet for the P2 exam? I was trying to recreate it for the second part but dont quite understand the grey box how to get the values in the grey boxes (specifically the row "Index")

Thank you

 

[David Harper CFA FRM]

Hi @dtammerz I didn't create a matching version for P2, but it won't take me long ( i will try if i can catch any break in the forum here in the near term). Thanks,

 

[dtammerz]

Hi @dtammerz I didn't create a matching version for P2, but it won't take me long ( i will try if i can catch any break in the forum here in the near term). Thanks,

@davidharper Thank you for your response; no rush