How to Fix the FRM: David's 2018 Memo to the FRM Committee and GARP's Board of Trustees

[David Harper CFA FRM]

In May 2018, I sent the attached 18-page memo called "How to Fix the FRM" to GARP's leadership (including the Board of Trustees). As host of this forum for over a decade, I have a front-row seat into candidates' experiences and their feedback; in particular, I've spent several hundred hours debugging syllabus material and practice questions. The primary motivation for the memo was the high error rate we've observed in the source material, in particular sample practice questions. Related, the anthology approach (i.e., differing authors) of the exam has created certain inconsistencies and ambiguities, many of which are detailed in the memo. As I wrote in the memo:

On behalf of my customers and myself, this memo has three goals. One, to warn you of reputational risks that are accruing to the FRM product, at least online. Two, to convince you there is an opportunity for improvement in the FRM product. Three, if those arguments are convincing, to learn that the Committee will take some concrete steps to make improvements. With that agenda, at the end of this memo, after asking important questions, I offer recommendations and suggestions.

The irony of these quality problems is that they are mostly unnoticed by the majority of candidates who are too busy digesting the material in a first pass effort at comprehension. Rather, these issues have been noticed only by the more experienced and proficient candidates, who tend to arrive with prerequisites that enable them to discern defects. What I mean is, most candidates are just trying to learn how to calculate value at risk (VaR), for example, or credit value adjustment (CVA), it's after you master them are you likely to noitce that GARP's own approaches can be inconsistent. A key point that I have repeatedly emphasized to management, with actual examples, is that some inconsistencies ironically bias against better-prepared candidates; that is, some submitted questions have a perceived correct answer that is easier to identify if you know less (candidates who know more often tend to identify an alternative)! In any case, I assume that I do not need to defend high standards especially in the context of how much time and money must be invested by an FRM candidate.

This memo summarizes, on a certain level, my years long effort to contribute to the improvement of syllabus. I did receive a careful, thoughtful, and encouraging reply from Bill May (Global Head of Certifications and Education Program). My reply to his reply is appended, but due to privacy, his reply to me is not included. I'm sharing here now only because this phase has long-since concluded (months ago) and I am very much looking forward to seeing GARP's release of the 2020 syllabus.

The file is attached but same is here at https://www.dropbox.com/s/1vx96sckabn2qbv/howto-fix-frm-v4.pdf?dl=0

 

[Stuart D Moncrieff]

Inspiring to have people of your calibre pushing for these improvements. I doubt there are many people capable of finding errors in garp's material or areas for improvement - let alone take the time to raise these issues and comments . Thank you. I've always admired and appreciated your dedication to those of us trying to better themselves through the FRM programme.

 

[Champaklal Chodoo]

So finally someone showed an intent to Bell the CAT and I would like to congratulate David for that. I hope GARP will take it positively and take steps to develop its own study material content (which will be [hopefully] error free and as interesting as CFA institute study material )

 

[JamesVU2000]

Wow. I just read this letter and I am in complete agreement with David. I think the FRM is a possibly a scam.

 

[JamesVU2000]

I would also like to point out that I know Nassim Taleb personally and he thinks people involved in the academic risk industry are incompetent. This means people like Jorion and the people behind the FRM.

 

[JamesVU2000]

Plus, the constant turnover of the syllabus smells like a desperate attempt to unfairly take in fees. Things are not changing that often

 

[amit.m.sharma]

That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?

 

[Nicole Seaman]

That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?

Hello @amit.m.sharma

We are currently working on updating the materials in the study planner for 2020 so there will be limited forum support at this time. The two threads below may help to answer your questions. Please feel free to discuss any further questions that you have in those threads after you have read through the discussions. Thank you.

• Quartiles and Weights: https://forum.bionicturtle.com/threads/quartiles-and-weights-exam-results.9883/
• Pass Rate: https://forum.bionicturtle.com/threads/what-is-the-pass-rate-for-the-frm.10093/

 

[ami44]

That's an excellent letter to GARP.

@David Harper CFA FRM - if you could please share your thoughts on the following statement by GARP on their website:

"The exam result is determined by many factors and is not predetermined nor is it a specific percentage of the points either for individual modules areas or for the entire exam. The exam results are determined by the FRM/ERP Committee(s). "

1. What are these factors? How do we as candidates benefit from not knowing these factors?
2. Do we get penalized for guessing? Is there a statistical test on similar questions to determine if we guessed or actually understand the material? If yes, then it's not right because people do run out of time and have to circle in something quickly or risk not reading the last few questions as time is running out.
3. Do they weight certain questions differently?
4. Is it possible that a candidate who passed may have scored less than someone who failed the exam?

1.+4. is answered by Nicoles links
2. no penalizing. You should guess in case you don‘t know the answer or your time runs out.
3. All questions have the same weight

 

[Champaklal Chodoo]

I would also like to point out that I know Nassim Taleb personally and he thinks people involved in the academic risk industry are incompetent. This means people like Jorion and the people behind the FRM.

I don't know Mr.Taleb personally but I've read his books. Although the books are quite decent, (from his writing) Mr.Taleb appears to be an oversmart person. According to him not only Risk professionals but many other professionals (stock Traders, Investment professionals, Dentists) are incompetent. It seems that according to him only he is the most intelligent and most competent professional on this planet earth. Of course, Mr.Taleb has right to hold his opinions and express them freely. But now after listening to Taleb, should I stop seeing a Dentist or Stop Investing in Mutual fund (Because of incompetent dentist and fund managers) ??? I WILL NOT.
As far as FRM exam is concerned, although there a some problems, overall its a good course and do lot of value addition, Provided, you approach the syllabus correctly and use good resources Like Bionic Turtle. Thanks
PS : This is what I feel, I may be wrong/ I may be right and others need not agree with what I said

 

[JamesVU2000]

If 30% of the questions are wrong it’s a good exam? If it has zero transparency and test is not electronic it’s a good exam? GARP doesn’t know how to test risk management. Most of the things are testing would be better on a computer in an applied setting.

 

[JamesVU2000]

For example David’s videos are vastly more useful than the exam and questions. Incorporating questions with applied examples is the way to go.

 

[Champaklal Chodoo]

If 30% of the questions are wrong it’s a good exam? If it has zero transparency and test is not electronic it’s a good exam? GARP doesn’t know how to test risk management. Most of the things are testing would be better on a computer in an applied setting.

I agree with some of the points and disagree with some points mentioned in your 2 replies.
First let's speak about the points I agree with 1) David's videos are useful- I agree with that. However, you are saying videos are More important than exam and questions. Sorry, I didn't get your point. I don't know much about coaching industry but in my opinion most of the students are subscribing to David's services (BT) for understanding difficult concepts of risk management and TO PASS FRM EXAM. If there is no FRM exam, then I wonder how many them would join BT just for the sake of enhancement of their knowledge. At least, people from India/China (Major mkt for FRM) cant afford to spend $799 just for intellectual stimulation.
2) Number of Practical/calculation-based questions should be increased ( I assume this is what you mean when you say "Incorporating questions with applied examples "). There are lot of theory questions in actual exam. In my opinion this number should be reduced.

Now points I Disagree with
1) 30% questions are wrong : 30% wrong questions in MOCK. We can not say that about actual exam.

2) Zero transparency : The level of transparency in FRM exam is similar to CFA exam. Even, CFA institute doesn't publish its past exam papers. So now are you going to say that even CFA is scam ?

3) Electronic test : How can you say electronic test is better than paper format test ?? In fact, in case electronic tests there is more possibility of manipulation, hacking and even cyber attack.

 

[Eustice_Langham]

I've just been reading the letter that David sent to GARP and wonder if the major(?) changes that have been made to the 2020 Part 1 syllabus were the result of David's letter?

I am also wondering if the CFA suffers from the same shortcomings.

 

[Sjegathe]

I've just been reading the letter that David sent to GARP and wonder if the major(?) changes that have been made to the 2020 Part 1 syllabus were the result of David's letter?

I am also wondering if the CFA suffers from the same shortcomings.

I can confidently say no, having gone through all the levels i can attest that the standards are top notch. I believe they've
integrated machine learning and the latest fin tech technologies in to the curriculum as well. There are people who would say level
2 and level 3 questions are tough but what that means is you would need solid understanding of the concepts not by means of flawed questions and answers. This is my opinion though.

 

[shuairan82]

I don't want to unburry an older thread, but I came here from the October 2020 Exam Feedback Thread and wanted to share some of my thoughts.

I think the lack of actual excercises in the curriculum makes it quite hard to pass the exam just with the GARP provided materials - frankly, it is beyond me how one could pass just with the GARP books, the online questions and the one or one-and-a-half mock exams provided with an acceptable effort.

I do firmly belive in free markets and this way it might be possible that you get a better outcome by letting external providers do the job
and obviously BT does deliver a real good service given the pass rates and feedback by subscribers (I did recomend it on Reddit as well, David also commented in this thread - and of course I did not receive any payments for my recommendation).

I also really appreciate Davids effort to cooperate with GARP and give some constructive feedback. While a good reputation of the FRM will help th BT business modell, all the lacks described in the letter are probably even better for BT ;-) So I think it is safe to say that David did take the effort for all the current and future FRM candidates.

I think the points are very valid and as we saw in the process of the exam postponements and the rather chaotic publication of exam results, GARP has also some serious issues with their communication.
A timely, friendly and empthatic communication could take a lot of the edges plus it would look much more professional - a level of professionalism I would expect from an international organisation with that reputation.

The whole curriculum is very extensive and challenging and it takes a lot of effort. As David quoted “lack of quality time spent with my kids while they're still young enough to think I’m cool.” - I totally feel that.

That said, I think the curriculum does cover the field well and I fell that I have learnt a lot of relevant knowledge - so I don't regret to have earned the FRM. I also think, that it is ok to have different opinions and approaches in the literature, as it good to know that there might be divergent opinions amongs professionals if you want to be able to discuss and evaluate the approaches in daily business. However, it would be extermly helpful if that fact would be pointed out by GARP, rather than just put them next to eachother in the sylabus.

Another point is that I am not against challenge or difficulty. In oder to have a real value, the designation should not be given away easily. But the challenge should be fair - if you know the right answer, you should be able to pass. To some extent, red herrings are ok too, if they are used to test the understanding of a candidate. But cheapshots, unnecessary tedious calculations (like adding up 20 random figures to calculate an Expected Shortfall, which is just a test of concentration and how well you can use the calculator) and distraction long descriptions in the questions (Like: John is a Hedge Fund Manger based in the USA. He likes to play Golf and his Grand-Grandmothers Name was .... and in the end they ask something about volatility, just to waste your time) should not be used just to provoke errors or put people under time pressure.

So long story short: Thanks again David, especially the BT question were more than helpful. Whitout them, I would not have passed the exam.
For all future candidates: The mock exam from GARP might help you with the general format of the questions, but BT will prepare you better. And be prepared / watch out for cheap tricks in the exam questions ;-)

 

[shuairan82]

Addition, some examples how it is not supposed to be:

- as a non native English Speaker, I found in some cases in the exam, they used synonyms for words like "good", "acceptable" or "appropriate" which are not that common - like someone said, "we used "best approach" already twice in the exam, lets get a thesarus and look for the most uncommon word for "best" to replace it".

- They should test, if the candidate understands the problem, not how well he or she can handle the calculator under pressure. E.g. if you have to calulate an Expected Shortfall over a Period of 300 days, 1% confidence is enough. The candidate should know, wether to take the biggest 4, 3 or 2 losses and how to average them. They shouldn't ask for 10% or even 5% confidence, because adding up over 10 figures, at best with completely random number over 1bn has nothing to do with risk skills or knowledge.

- Red Herrings are ok, e.g. there was one case where there was an output of statistical software, were the figure could just be read from the table. It could also be calculated by the number given, but it was more tedious. So fair enough, if you know shortcuts, that's a skill.
But putting in an endless story with all the information hidden in it, which just wastes you time, is just not fair.
I always read the question first before reading the backstory, in order to pick up the right information on the first go.
But in a very time constraint exam, it is unfair to uneccessarily waste the candidates times.

 

[Garbanzo]

But putting in an endless story with all the information hidden in it, which just wastes you time, is just not fair.
I always read the question first before reading the backstory, in order to pick up the right information on the first go.
But in a very time constraint exam, it is unfair to uneccessarily waste the candidates times.

I disagree. It is a very important skill for a risk manager (or anyone really) to quickly discern what is important. Real world is messy, you don't get stuff served on a plate with 3-4 inputs and a template. In fact, I think FRM should take the approach of CFAI and incorporate vignettes with multiple questions (4-6) for each.
They should also incorporate structured response questions (aka "essays", but they really are at most 4-5 bullet points or numerical answers to calculation questions), because that way you cannot just (partly) guess your way to an FRM, which I am sure is currently happening in Part II. It's much harder to prepare for an exam where you cannot reverse-engineer what's being asked by looking at the answers. You need to hedge by knowing the curriculum in and out and that's what this should be about, not pumping out 50-60% of test-takers as FRMs each cycle. What is the value of the FRM going to be in a few years this way? CFAI took the hint and drastically lowered the pass rates (even though they coat it as "the pandemic caused candidates to be less well-prepared"). But this will be better for the "product" in the long-term. The fees should definitely be lower, but the bar should be raised so it truly is a distinction.

Given the many errors in the questions they provide either as practice or at the exam, GARP should massively invest in true experts who create their curriculum/question sets and do a massive audit.

However, given the low reputation of GARP, I am sure they are purely optimizing for the highest revenue so if they don't see any potential for these to raise the number of candidates applying and bringing in fees, they won't do anything extra that would temporarily hurt them in the short-medium term.

 

[Garbanzo]

You deserve a medal for challenging (rightly so) GARP this way @David Harper CFA FRM

It's unacceptable for them to be raking in tens of millions of dollars every year and not even properly reviewing or improving their own work.

Is there any follow up to this? Like a reconciliation of what has/has not yet been corrected and improved - e.g. the Part I proprietary syllabus (which in your view is actually just an inferior compilation/reformulaion ("rip-off" ?! ) of the original readings), fixed practice exam questions, and so on?
On reddit I've noticed you mentioning having a tough time with GARP's CEO because of this. Has there been any development to this?

 

[David Harper CFA FRM]

Hi @Garbanzo Thank you for your attention here. There have been no significant, public developments, unfortunately. Behind the scenes, I've repeatedly emphasized individual items in the memo. In GARP's defense, some of the egregious inconsistencies have been "fixed;" e.g., I've sent back so many issues related to the default probability terminology (or CVA is another example) cluster, and they've repeatedly revised, such that I'm confident candidates will see proper, consistent terms in those clusters.

And, I think maybe they added better help on their exam review team, I'm not concretely sure. In terms of major, thematic improvements, I think the reality is that they just have not had a business imperative to do so. It will be interesting to see if the pandemic, which somewhat exposed some of these weaknesses, has any effect. I was a management consultant to Boards for eight years and my high-level assessment is: there would need to be a market event (to create urgency) and/or a leadership change. As long as a business is profitable and "working" (and they have a fabulous business model), it usually requires some kind of exogenous shock to force change; e.g.., activist investor, sudden switch from growing revenues to declines. What has not changed is my belief that candidates deserve better. I'm glad the some improvements have been made, and I'm grateful our contacts listen to us, but I want (i) to be proud of the designation and profession and (ii) an association should be a role model and eat its own dog, including living up to whatever it asks of its own members. Thanks