CiML 2015

CiML2015
Challenges in Machine Learning: 
"Open Innovation" and "Coopetitions" [slides/papers/photos]
NIPS 2015 workshop
Saturday December 12, 2015, Montreal, Canada
Palais des Congrès de Montréal
Convention and Exhibition Center
[AutoML challenge][Schedule][Talk abstracts][Committee][Organizers]

Motivations

Challenges in Machine Learning have proven to be efficient and cost-effective ways to quickly bring to industry solutions that may have been confined to research. In addition, the playful nature of challenges naturally attracts students, making challenge a great teaching resource. Challenge participants range from undergraduate students to retirees, joining forces in a rewarding environment allowing them to learn, perform research, and demonstrate excellence. Therefore challenges can be used as a means of directing research, advancing the state-of-the-art or venturing in completely new domains. 
Because challenges have become stream line in the execution of Machine Learning projects, it has become increasingly important to regularly bring together workshop organizers, platform providers, and participants to discuss best practices in challenge organization and new methods and application opportunities to design high impact challenges. Following the success of last year's workshop (http://ciml.chalearn.org/), in which a fruitful exchange led to many innovations, we propose to reconvene and discuss the new avenues that have been explored and lay the basis for further developments. We are particularly interested in following progresses made in two conceptually important directions:
1) Open innovation: Organization of contests in which data are made available and the contestants must both formalize and solve a problem (with some constraints), leaving more freedom to creativity, while giving more difficulty to the organizers to objectively assess the results.
2) Coopetitions: Organization of contests encouraging both collaboration and competition, in an effort to make possible the contributions of many towards a the grand goal of solving the overall problem; this poses to the organizers the problem of rewarding partial contributions.

We also want to closely follow more technical, albeit important aspects:
3) Platforms: New developments including "code submission" (platforms and protocols permitting code submission, as opposed to result submission, allowing fairer standardized comparisons in terms of hardware utilization and easier reproducibility) and " worksheets" or "scripts" facilitating code sharing.
4) Sharing, dissemination, and recognition: Facilitate sharing resources, including data, means of data collection and annotation, challenge announcements, best practices, challenge templates, publication channels, etc.; creation of awards to recognize academic services rendered by the various actors of challenge organization.		 News
 
We have posted the slides and paper preprints on the schedule page. 

Invited speakers



Frank Hutter

Univ. Freibug, Germany
 AutoML challenge: successes and challenges



Radboud University, The Netherlands
Challenges in medical image analysis
  
Mary Larson

 		 
Chris Williams


Univ. Edinburgh, UK
Pascal VOC challenges and
Improving the Data Analytics Process

Discussion leaders

 Balasz Kegl
U. Paris-Saclay, France		 Ben Hamner
Kaggle, USA		 Evelyne Viegas
Microsoft Research, USA		 Isabelle Guyon 
ChaLearn, USA
 balasz Kegl  
Evelyne Viegas
 

Format

We want to give a large space to discussion, by organizing  four discussions monitored by the organizers, who will first give a  brief introduction to the selected topics: open innovation, coopetitions, platforms, and resource sharing. In addition, The invited speakers will be asked to reflect in their presentations upon the four main topics of discussion.

Call for abstract: We welcome 2-page extended abstracts on one of the topics of the workshop. Selected papers will be presented primarily as posters, but exceptional contributions will be given oral presentations. Abstract should be submitted by October 10th, 2015 by sending email to nips2015@chalearn.org.

Topics of interest

Methods:
- Novel or atypical challenge protocols, particularly to tackle complex tasks with very large datasets, multi-modal data, and data streams.
- Methods and metrics of entry evaluation, quantitative and qualitative challenges.
- Methods of data collection, "ground-truthing", and preparation including bifurcation/anonymization, data generating models.
- Teaching challenge organization.
- Hackatons and on-site challenges.
- Challenge indexing and retrieval, challenge recommenders.

Theory:
- Experimental design, size data set, data split, error bounds, statistical significance, violation of typical assumptions (e.g. i.i.d. data).
- Game theory applied to the analysis of challenge participation, competition and collaboration among participants.
- Diagnosis of data sanity, artifacts in data, data leakage.

Implementation:
- Re-usable challenge platforms, innovative software environments.
- Linking data and software repositories to challenges.
- Security/privacy, intellectual property, licenses.
- Cheating prevention and remedies.
- Issues raised by requiring code submission.
- Challenges requiring user interaction with the platform (active learning, reinforcement learning).
- Dissemination, fact sheets, proceedings, crowsourced papers, indexing post-challenge publications.
- Long term impact, on-going benchmarks, metrics of impact.
- Participant rewards, stimulation of participation, advertising, sponsors.
- Profiling participants, improving participant professional and social benefits.

Applications:
- Where to venture next: opportunities for challenge organizers to organize challenges in new domains with high societal impact.
- Successful challenge leading to significant breakthrough or improvement over the state-of-the-art or unexpected interesting results.
- Rigorous study of the impact of challenges, analyzing topics and tasks lending themselves to high impact machine learning challenges.
- Challenges as an educational tool.
- Challenges organized or supported by Government agencies, funding opportunities.

Related workshops:
We are connected to the Bayesian Optimization workshop and the Black Box Learning and Inference workshop, because they both treat in some way the "Automatic Machine Learning" problem.