CiML 2018‎ > ‎


2018 Challenges in Machine Learning workshop@NIPS
Machine Learning Challenges "in the wild"

Morning session (8:00 am-12:00 pm): New platforms and innovative designs

8:00 am (10 min) - Morning welcome and introduction 

8:10  (30 min) - Esteban Arcaute and Umut Ozertem - Facebook project on developing benchmarks of algorithms in realistic settings

8:40 (30 min) - Laura Seaman - Project Alloy – Machine Learning Challenges for Researching Human-Machine Teaming

9:10  (90 min)  - Live competition: Pommerman

10:40 - 11:00  (20 min)  Break, poster viewing

11:00  (30 min)  - Julien Hay, Bich-Liên Doan, Fabrice Popineau  - Renewal news recommendation platform 

11:30 - 12:00  (30 min)  Panel discussion. Design principles and implementation issues. (3 panelists TBA + 1 moderator)

Lunch break (12:00 pm-1:00pm)

Lunch served in the room for participants.

Discussions and poster viewing.

Afternoon session (1:00 pm-6:30 pm): Robotics, RL, and interactive challenges

1:00 - Afternoon welcome and announcements (organizers)

1:10  (30 min) - Mikhail Burtsev and Varvara Logacheva - Wild evaluation of chat-bots

1:40  (30 min) - Larry Jackel - Measuring Progress in Robotics

2:10  (30 min) - Daniel Polani - Competitions to Challenge Artificial Intelligence: from the L-Game to RoboCup 

2:40  (30 min) - Antoine Marot - Learning to run a power network 

3:10 - 3:30  (20 min)  Break, poster viewing

3:30  (90 min) - Live competition: The driving AI Olympics

5:00 - 5:30  (30 min) - Panel discussion: Opportunities to organize new impactful challenges. (3 panelists TBA + 1 moderator)

5:30 - Adjourn.

6:30 - Closing banquet.


Posters dimensions are: 36 x 48 in. (91cm x 122cm); masking tape will be provided. All posters can be presented at all poster sessions.

New platforms and innovative designs:

Beyond the Leaderboard, Adriënne M. Mendrik, Stephen R. Aylward, James A. Meakin Bram van Ginneken

Corpus for AutoML Pipelines, Richard Lippmann, Swaroop Vattam, Pooya Khorrami, and Cagri Dagli

How to fail hosting data science contests with images, Evgeny Nizhibitsky, Moscow State Univ. Artur Kuzin, Dbrain, Moscow Institute of Physics and Technologies, Russia

ML Benchmark Tools Package, Ryan Turner, Uber AI Labs

New impactful challenges:

NASA Frontier Development Lab 2018A. Bell, A. Chopra, W. Fawcett, R. Talebi, D. AngerhausenA. Berea, N.A. Cabrol, C. Kempes, M. Mascaro

AutoDL challenge design and beta tests, Zhengying Liu, Olivier Bousquet, Andre Elisseeff, Isabelle Guyon, Adrien Pavao, Lisheng Sun-Hosoya, and Sebastien Treguer

TrackML, a Particle Physics Tracking Machine Learning Challenge, Jean-Roch Vlimant (Caltech), Vincenzo Innocente, Andreas Salzburger (CERN), Isabelle Guyon (ChaLearn), Sabrina Amrouche, Tobias Golling, Moritz Kiehn (Geneva University),David Rousseau∗, Yetkin Yilmaz (LAL-Orsay), Paolo Calafiura, Steven Farrell, Heather Gray (LBNL), Vladimir Vava Gligorov (LPNHE-Paris), C ́ecile Germain, Victor Estrade(LRI-Orsay),Edward Moyse (University of Massachussets), Mikhail Hushchyn, Andrey Ustyuzhanin (Yandex, HSE)

L2RPN: Learning To Run a Power Network Competition, Antoine MarotBalthazar Donon, Isabelle GuyonBenjamin Donnot.

Multi-Agent RL in MalmÖ(MARLÖ) Competition, Diego Perez-Liebana Katja Hofmann Sharada Prasanna Mohanty Noburu Kuno Andre Kramer Sam Devlin Raluca D. Gaina Daniel Ionita 

Live competitions:


Competition summary: Train a team of communicative agents to play Bomberman. Compete against other teams.

Cinjon Resnick, NYU,
David Ha, Google Brain,
Denny Britz, Prediction Machines,
Jakob Foerster, Oxford,
Jason Weston, Facebook FAIR,
Joan Bruna, NYU,
Julian Togelius, NYU,
Kyunghyun Cho, NYU,

The AI diving Olympics

Competition summary: Machine Learning (ML), deep learning, and deep reinforcement learning have shown remarkable success on a variety of tasks in the very recent past. However, the ability of these methods to supersede classical approaches on physically embodied agents is still unclear. In particular, it remains to be seen whether learning-based approached can be completely trusted to control safety-critical systems such as self-driving cars. This live competition, presented by the Duckietown Foundation, is designed to explore which approaches work best for what tasks and subtasks in a complex robotic system. The participants will need to design algorithms that implement either part or all of the management and navigation required for a fleet of self-driving miniature taxis. There will be a set of different trials that correspond to progressively more sophisticated behaviors for the cars. These vary in complexity, from the reactive task of lane following to more complex and “cognitive” behaviors, such as obstacle avoidance, point-to-point navigation, and finally coordinating a vehicle fleet while adhering to the entire set of the “rules of the road”. We will provide baseline solutions for the tasks based on conventional autonomy architectures; the participants will be free to replace any or all of the components with custom learning-based solutions.The competition will be live at NIPS, but participants will not need to be physically present—they will just need to send their source code packaged as a Docker image. There will be qualifying rounds in simulation and we will make available the use of “robotariums,” which are facilities that allow remote experimentation in a reproducible setting.


Andrea Censi, nuTonomy and ETH Zürich,
Liam Paull, Université de Montréal,
Jacopo Tani, ETH Zürich,
Scott Livingston,
Julian Zilly, ETH Zürich,
Ruslan Hristov, nuTonomy,
Oscar Beijbom, nuTonomy,
Eryk Nice, nuTonomy,
Sunil Mallya, Amazon,
Justin De Castri, Amazon,
Hsueh-Cheng (Nick) Wang, National Chiao Tung University,
Qing-Shan Jia, Tsinghua,
Tao Zhang, Tsinghua ,
Stefano Soatto, UCLA and Amazon,
Magnus Egerstedt, Georgia Tech,
Yoshua Bengio, Université de Montréal,
Emilio Frazzoli, ETH Zürich and nuTonomy,


Dinner information:

The speakers and organizers are invited for dinner at:

The dinner will be at 8 pm the night before the workshop, Friday December 7:

Restaurant ChinaTown Kim Fung, 1111 Rue Saint-Urbain, Montréal, QC H2Z 1Y6, Canada.

Isabelle Guyon,
Oct 18, 2018, 1:23 PM
Isabelle Guyon,
Oct 18, 2018, 1:23 PM
Isabelle Guyon,
Oct 18, 2018, 1:23 PM
Isabelle Guyon,
Oct 18, 2018, 1:23 PM
Isabelle Guyon,
Oct 18, 2018, 1:24 PM
Isabelle Guyon,
Oct 19, 2018, 10:09 AM
Isabelle Guyon,
Oct 18, 2018, 1:24 PM
Isabelle Guyon,
Oct 18, 2018, 1:23 PM
Isabelle Guyon,
Oct 18, 2018, 1:23 PM
Isabelle Guyon,
Oct 19, 2018, 11:14 AM