Challenges in Machine Learning:
"Open Innovation" and "Coopetitions" [slides/papers/photos] NIPS 2015 workshop
Saturday December 12, 2015, Montreal, Canada
Palais des Congrès de Montréal
Convention and Exhibition Center
Invited speakers
Discussion leaders
Format We want to give a large space to discussion, by organizing four discussions monitored by the organizers, who will first give a brief introduction to the selected topics: open innovation, coopetitions, platforms, and resource sharing. In addition, The invited speakers will be asked to reflect in their presentations upon the four main topics of discussion. Call for abstract: We welcome 2-page extended abstracts on one of the topics of the workshop. Selected papers will be presented primarily as posters, but exceptional contributions will be given oral presentations. Abstract should be submitted by October 10th, 2015 by sending email to nips2015@chalearn.org. Topics of interest Methods: - Novel or atypical challenge protocols, particularly to tackle complex tasks with very large datasets, multi-modal data, and data streams. - Methods and metrics of entry evaluation, quantitative and qualitative challenges. - Methods of data collection, "ground-truthing", and preparation including bifurcation/anonymization, data generating models. - Teaching challenge organization. - Hackatons and on-site challenges. - Challenge indexing and retrieval, challenge recommenders. Theory: - Experimental design, size data set, data split, error bounds, statistical significance, violation of typical assumptions (e.g. i.i.d. data). - Game theory applied to the analysis of challenge participation, competition and collaboration among participants. - Diagnosis of data sanity, artifacts in data, data leakage. Implementation: - Re-usable challenge platforms, innovative software environments. - Linking data and software repositories to challenges. - Security/privacy, intellectual property, licenses. - Cheating prevention and remedies. - Issues raised by requiring code submission. - Challenges requiring user interaction with the platform (active learning, reinforcement learning). - Dissemination, fact sheets, proceedings, crowsourced papers, indexing post-challenge publications. - Long term impact, on-going benchmarks, metrics of impact. - Participant rewards, stimulation of participation, advertising, sponsors. - Profiling participants, improving participant professional and social benefits. Applications: - Where to venture next: opportunities for challenge organizers to organize challenges in new domains with high societal impact. - Successful challenge leading to significant breakthrough or improvement over the state-of-the-art or unexpected interesting results. - Rigorous study of the impact of challenges, analyzing topics and tasks lending themselves to high impact machine learning challenges. - Challenges as an educational tool. - Challenges organized or supported by Government agencies, funding opportunities. Related workshops: We are connected to the Bayesian Optimization workshop and the Black Box Learning and Inference workshop, because they both treat in some way the "Automatic Machine Learning" problem. |