May 2012 Survey Feedback

[Suzanne Evans]

After the May FRM Exam results were released, we sent our customers a survey to discover how we might improve. After reviewing the survey feedback in depth, I wanted to provide a follow-up.

We considered all of the suggestions and feedback. While we are unable to implement everything, we decided to implement the following:

• We are working on simplifying the site as well as the study planner to create easier usability.
• We are working to add more mock exams.
• We plan to add a formula sheet (both Part 1 & 2) prior to the November exam.
• We already started an 8-step focus review (for each Part) ahead of the November exam. See our target schedule here: http://www.bionicturtle.com/news-and-announcements/2012-frm-calendar
• David and I have discussed an automatic email notification when a paid content has been published. We are working with our developers to implement this.
• David and I have discussed linking the practice questions in the PDF documents with the forum link. We feel this would be a great addition and helpful to our paying customers as it provides a direct link for discussion purposes. We plan to implement this going forward. To start, we are currently revising the existing practice question to include, in many cases, direct links to the relevant forum discussions.

One area of feedback made us realize that we have a few areas of usability that can be improved. One of the suggestions that we received on multiple surveys is:

The website is not easy to navigate and the contents are not easy to find. Better organization of the site would go a long way.

Comments: We currently have all of our core documents located within our study planner. Our intention is to send the paid members to the study planner which contains all of the core contents for the FRM curriculum. The study planner can be found here: https://www.bionicturtle.com/my-account/study-planner. The study planner can be sorted by topic and/or most recent. It is recommended to sort by topic as it places all of the content in a recommended sequence. Essentially, the study planner is the place to go for FRM core content.

Importantly, we are currently working with our designers and developers to both (i) make access to the Study Planner more obvious and (ii) make the Study Planner itself even more intuitive and usable.

Unfortunately a few areas that were suggestions that we are unable to implement at this time are:

• Video Tutorials are too long.
  Comments: Our video tutorials are already segmented into parts (a,b,c, etc.) for each topic of the FRM. We do our best to keep them as short as possible without creating additional workload on recording as well as additional unnecessary files.
  
  However, the Focus Reviews are meant to supplement with brief videos such that some candidates, namely more experienced candidates, can probably rely mostly on the Focus Reviews instead of sitting through the entire regular video series.
  
  
• Free trials
  Comments: Unfortunately, we don't find a need for free trials because we already offer samples for all of our contents. If you click on the content entries, you should see a green "view sample" button. If you click the sample button, you will be directed to the sample document.

• I think that adding live Webinars or live on-line class per topic would be a definitely asset, through this way we would be able to ask question live.
  Comments: We had implemented live webinar reviews in previous years. In early years participation was high. However, participation rates steadily fell, as time-pressed customers increasingly accessed recorded video. It was decided to best focus on other areas where there was higher interest and would better benefit our customers.

• The ability to buy a set of practice questions instead of the whole package of products.
  Comments: Unfortunately, at this time we do not have products available individually. We have structured our products within tier levels to give customers an opportunity to purchase a lower tier if they are not interested in all contents. You can view our products here: https://www.bionicturtle.com/products/financial-risk-management/

• If there was a mentor who we can talk and meet with persionally 2 - 3 times between the registration and test date, I think the experience would be much better.
  Comments: Unfortunately, Bionic Turtle is only a team of 2. We do not provide individual email support outside of basic questions about our product. All of the customer support is provided within the forum. David provides individual support to inquiries within the forum here: https://forum.bionicturtle.com/

 

[David Harper CFA FRM]

Suzanne, thank you for conducting the survey, analyzing all of the feedback, starting the above initiatives, and sharing back the findings (including even, your no-spin zone on the requests that we cannot act on!). It's really great work, I am very excited to see through these improvements. I would award you a star (or several) into the weekly drawing, if only you were eligible

 

[vallerano]

Dear Suzanne,

It is true customer service that you are providing! What was the pass rate among your customers?

Thanks,

vallerano

 

[David Harper CFA FRM]

Hi Vallerano,

This survey wasn't for that purpose, we do that after the November exam. This purpose here was specifically site usability and product improvement, so the data includes many non-customers (free visitors), including even many users of competitive products who just visit our forum, such that the rate (owing to the larger sample) is nearer to the population average, a statistically expected outcome given the large number of respondents.

Currently, the cumulative pass rate, over the three surveys conducted where the sample is BT customers who responded is 72%. This does not parse into L1/L2 and is not adjusted for any self-selection bias; i.e., would customers who choose to respond by more or less likely to report?

In truth, our cumulative pass rate, given it's high initial value, will decline as we grow (we lead the market). We will survey, again, after November to include attributable pass rates; our pass rate is so much significantly above the average that, given our growth, I expect it to decline ... I hope we can stay above 70% but as our share increases, this will be difficult.

Thanks,