CiML2019‎ > ‎


"Machine Learning Competitions for All" (Friday December 13th)
West 215 + 216

08:00  - 08:15 Welcome and Opening Remarks

 08:15 - 09:00Invited Talk by Amir Banifatemi (XPrize)  
"AI for Good via Machine Learning Challenges"

 09:00 - 09:45Invited Talk by Emily Bender (University of Washington)
"Making Stakeholder Impacts Visible in the Evaluation Cycle: Towards Fairness-Integrated Share
d Tasks and Evaluation Metrics"

 09:45 - 10:30Coffee Break
 10:30 - 11:15Invited Talk by Dina Machuve (Nelson Mandela African Institution of Science and Technology)
"Machine Learning Competitions: The Outlook from Africa”

 11:15 - 11:30"Dog Image Generation Competition on Kaggle" 
Wendy Kan (Kaggle / Google)

 11:30 - 11:45"Learning To Run a Power Network Competition"
Benjamin Donnoty (RTE France)

 11:45 - 12:00 "The AI Driving Olympics: An Accessible Robot Learning Benchmark"
Matthew R. Walter (Toyota Technological Institute at Chicago)

 12:00 - 12:15"Conclusion on TrackML, a Particle Physics Tracking Machine Learning Challenge Combining Accuracy and Inference Speed"
David Rousseau (LAL-Orsay)

 12:15 - 14:00Poster Session and Lunch Break (lunch is provided in the room)

 14:00 - 14:45Invited Talk by Frank Hutter (University of Freiburg) 
"A Proposal for a New Competition Design Emphasizing Scientific Insights
 14:45 - 15:00"Design and Analysis of Experiments: A Challenge Approach in Teaching"
Adrien Pavao (Université Paris-Saclay (UPSud/INRIA))

 15:00 - 15:15"The model-to-data paradigm: overcoming data access barriers in biomedical competitions"
Justin Guinney (Sage Bionetworks)

 15:15 - 15:30"The Deep Learning Epilepsy Detection Challenge: Design, Implementation, and Test of a New Crowd-Sourced AI Challenge Ecosystem"
Isabell Kiral (IBM)

 15:30 - 16:15Coffee Break
 16:15 - 18:00Open Space Session on “The Organization of Challenges for the Benefit of More Diverse Communities”

 19:30 Dinner at Banana Leaf at 1779 Robson street [for invited speakers and organizer; contact us if you want to join at your own expenses]

Invited Talks

AI for Good via Machine Learning Challenges
Talk by Amir BanifatemiXPrize

"AI for Good" efforts (e.g., applications work in sustainability, education, health, financial inclusion, etc.) have demonstrated the capacity to simultaneously advance intelligent system research and the greater good. Unfortunately, the majority of research that could find motivation in real-world "good" problems still center on problems with industrial or toy problem performance baselines. 

Competitions can serve as an important shaping reward for steering academia towards research that is simultaneously impactful on our state of knowledge and the state of the world.  This talk covers three aspects of AI for Good competitions. First, we survey current efforts within the AI for Good application space as a means of identifying current and future opportunities. Next we discuss how more qualitative notions of "Good" can be used as benchmarks in addition to more quantitative competition objective functions. Finally, we will provide notes on building coalitions of domain experts to develop and guide socially-impactful competitions in machine learning.

Making Stakeholder Impacts Visible in the Evaluation Cycle: 
Towards Fairness-Integrated Shared Tasks and Evaluation Metrics

Talk by Emily M. Bender, University of Washington

Joint work with 
  • Hal Daumé III, University of Maryland
  • Bernease Herman, University of Washington
  • Brandeis Marshall, Spelman College
In a typical machine learning competition or shared task, success is 
measured in terms of systems' ability to reproduce gold-standard labels. The potential impact of the systems being developed on stakeholder populations, if considered at all, is studied separately from system `performance'. Given the tight train-eval cycle of both shared tasks and system development in general, we argue that making disparate impact on vulnerable populations visible in dataset and metric design will be key to making the potential for such impact present and salient to developers.  We see this as an effective way to promote the development of machine learning technology that is helpful for people, especially those who have been subject to marginalization. This talk will explore how to develop such shared tasks, considering task choice, stakeholder community input, and annotation and metric design desiderata.

Machine Learning Competitions: The Outlook from Africa

Talk by Dina Machuve
Nelson Mandela African Institution of Science and Technology (NM-AIST)

The current AI landscape in Africa mainly focuses on capacity building. The ongoing efforts to strengthen the AI capacity in Africa are organized in summer schools, workshops, meetups, competitions and one long-term program at the Masters level. The main AI initiatives driving the AI capacity building agenda in Africa include a) Deep Learning Indaba, b) Data Science Africa, c) Data Science Nigeria, d) Nairobi Women in Machine Learning and Data Science, e) Zindi and f) The African Master's in Machine Intelligence (AMMI) at AIMS. The talk will summarize our experience on low participation of African AI developers at machine learning competitions and our recommendations to address the current challenges.

Dina Machuve is a Lecturer and Researcher at the Nelson Mandela African Institution of Science and Technology (NM-AIST) in Arusha, Tanzania. She holds a PhD in Information and Communication Science and Engineering from NM-AIST. She co-organizes the Data Science Africa (DSA), an organization that runs an annual data science and machine learning summer school and workshop in Africa. She is an awardee of the 2019 Organization of Women in Science for the Developing World (OWSD) Early Career Fellowship.

Afbeeldingsresultaat voor frank hutter challenges

A Proposal for a New Competition Design Emphasizing Scientific Insights
Talk by Frank HutterUniversity of Freiburg

The typical setup in machine learning competitions is to provide one or more datasets and a performance metric, leaving it entirely up to participants which approach to use, how to engineer better features, whether and how to pretrain models on related data, how to tune hyperparameters, how to combine multiple models in an ensemble, etc. The fact that work on each of these components often leads to substantial improvements has several consequences: (1) amongst several skilled teams, the one with the most manpower and engineering drive often wins; (2) it is often unclear *why* one entry performs better than another one; and (3) scientific insights remain limited.

Based on my experience in both participating in several challenges and also organizing some, I will propose a new competition design that instead emphasizes scientific insight by dividing the various ways in which teams could improve performance into (largely orthogonal) modular components, each of which defines its own competition. E.g., one could run a competition focussing only on effective hyperparameter tuning of a given pipeline (across private datasets). With the same code base and datasets, one could likewise run a competition focussing only on finding better neural architectures, or only better preprocessing methods, or only a better training pipeline, or only better pre-training methods, etc. One could also run multiple of these competitions in parallel, hot-swapping better components found in one competition into the other competitions. I will argue that the result would likely be substantially more valuable in terms of scientific insights than traditional competitions and may even lead to better final performance.

Meet at Banana Leaf at 1779 Robson street at 7:30 pm.