# Planned benchmarks

**URL:** <https://datachallenge.cfm.fr/t/planned-benchmarks/46>\
**Category:** CFM\
**Created:** [February 9, 2018, 8:44am UTC](https://datachallenge.cfm.fr/t/planned-benchmarks/46 "2018-02-09T08:44:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Remjez](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@Remjez](https://datachallenge.cfm.fr/u/Remjez)\
**Post date:** [February 9, 2018, 8:44am UTC](https://datachallenge.cfm.fr/t/planned-benchmarks/46/1 "2018-02-09T08:44:45Z")

</div>

Hi,

Do you plan to release new benchmarks? If so, when do you plan to do it?  
Even if you won’t, have you developed a model internally? Do you have an idea of the type of performance that could be achieved?

Thanks

---

<div class="post-metadata">

**Author:** ![lebigot](https://yyz1.discourse-cdn.com/flex029/user_avatar/datachallenge.cfm.fr/lebigot/32/10_2.png) [@lebigot](https://datachallenge.cfm.fr/u/lebigot)\
**Post date:** [February 9, 2018, 12:12pm UTC](https://datachallenge.cfm.fr/t/planned-benchmarks/46/2 "2018-02-09T12:12:03Z")

</div>

We are not planning to release any additional benchmark, as we have no other internal model.

We are expecting this challenge to be… challenging!

However, the current top scores in the leader board provide alternative benchmarks and do show that it is possible to significantly improve the benchmark score. How far is it possible to go? We do not know but are looking forward to being impressed!

---

<div class="post-metadata">

**Author:** ![AiOpH](https://yyz1.discourse-cdn.com/flex029/user_avatar/datachallenge.cfm.fr/aioph/32/13_2.png) [@AiOpH](https://datachallenge.cfm.fr/u/AiOpH)\
**Post date:** [February 12, 2018, 10:17am UTC](https://datachallenge.cfm.fr/t/planned-benchmarks/46/3 "2018-02-12T10:17:14Z")

</div>

Hi Remjez,  
I can propose to you another benchmark of the same type as the one CFM submitted.  
If you take the median of each series, you get a accuracy of 26.486% whereas, with the mean you get only 28.207%. Maybe understanding this difference can give some informations or not :p.

---

<div class="post-metadata">

**Author:** ![Remjez](https://avatars.discourse-cdn.com/v4/letter/r/eb9ed0/32.png) [@Remjez](https://datachallenge.cfm.fr/u/Remjez)\
**Post date:** [February 12, 2018, 2:45pm UTC](https://datachallenge.cfm.fr/t/planned-benchmarks/46/4 "2018-02-12T14:45:45Z")

</div>

Thank you for your response to both of you.  
It’s interesting to see that the median significantly improves performance.  
But I have a score of 21.5 and I wanted to know if it was possible to do better. But I guess you just have to wait.

---

<div class="post-metadata">

**Author:** ![lebigot](https://yyz1.discourse-cdn.com/flex029/user_avatar/datachallenge.cfm.fr/lebigot/32/10_2.png) [@lebigot](https://datachallenge.cfm.fr/u/lebigot)\
**Post date:** [February 12, 2018, 4:20pm UTC](https://datachallenge.cfm.fr/t/planned-benchmarks/46/5 "2018-02-12T16:20:26Z")

</div>

Nobody knows yet if it is possible to do better in a robust way (i.e. that would yield good performance on a different test set). That said, I would not be surprised to see improvements over this score in the coming months, as participants will have many interesting new ideas.