CiML2019‎ > ‎


Invited Speakers

Afbeeldingsresultaat voor frank hutter challenges

Amir Banifatemi


Emily M. Bender (University of Washington)

Dina Machuve 


Frank Hutter (University of Freiburg)

Speaker Bios

 Amir Banifatemi
Amir Banifatemi is a thought leader, entrepreneur, and investor. He currently leads the global AI XPRIZE while managing two venture funds in US and Europe. He has more than 25 years of experience in development and growth of emerging and transformative technologies and focuses on working with growth-oriented companies on products and initiatives that could trigger significant breakthrough with strong economic and societal impact.

AI for Good via Machine Learning Challenges
Talk by Amir BanifatemiXPrize

"AI for Good" efforts (e.g., applications work in sustainability, education, health, financial inclusion, etc.) have demonstrated the capacity to simultaneously advance intelligent system research and the greater good. Unfortunately, the majority of research that could find motivation in real-world "good" problems still center on problems with industrial or toy problem performance baselines. 

Competitions can serve as an important shaping reward for steering academia towards research that is simultaneously impactful on our state of knowledge and the state of the world.  This talk covers three aspects of AI for Good competitions. First, we survey current efforts within the AI for Good application space as a means of identifying current and future opportunities. Next we discuss how more qualitative notions of "Good" can be used as benchmarks in addition to more quantitative competition objective functions. Finally, we will provide notes on building coalitions of domain experts to develop and guide socially-impactful competitions in machine learning.

Emily M. Bender
Professor dr. Emily M. Bender is the director of the Computational Linguistics Laboratory at the University of Washington (USA). Her primary research interests are in multilingual grammar engineering, the study of variation, both within and across languages, the relationship between linguistics and computational linguistics, and practical methods for promoting engagement with ethical issues in NLP. She is involved in Shared Task Evaluation Campaigns (STECs) that aim to bridge the gap between available speech and language processing tools and challenges in endangered language documentation. This is done by designing tasks (challenges) that are compelling to speech and natural language processing researchers while addressing technical challenges in language documentation and exploiting growing archives of endangered language data.

Making Stakeholder Impacts Visible in the Evaluation Cycle: 
Towards Fairness-Integrated Shared Tasks and Evaluation Metrics

Talk by Emily M. Bender, University of Washington

Joint work with 
  • Hal Daumé III, University of Maryland
  • Bernease Herman, University of Washington
  • Brandeis Marshall, Spelman College
In a typical machine learning competition or shared task, success is measured in terms of systems' ability to reproduce gold-standard labels. The potential impact of the systems being developed on stakeholder populations, if considered at all, is studied separately from system `performance'. Given the tight train-eval cycle of both shared tasks and system development in general, we argue that making disparate impact on vulnerable populations visible in dataset and metric design will be key to making the potential for such impact present and salient to developers.  We see this as an effective way to promote the development of machine learning technology that is helpful for people, especially those who have been subject to marginalization. This talk will explore how to develop such shared tasks, considering task choice, stakeholder community input, and annotation and metric design desiderata.

Dina Machuve

Dina Machuve is a Lecturer and Researcher at the Nelson Mandela African Institution of Science and Technology (NM-AIST) in Arusha, Tanzania. She holds a PhD in Information and Communication Science and Engineering from NM-AIST in 2016, and a MS in Electrical Engineering from Tennessee Technological University (TTU), USA in 2008 and BSc Electrical Engineering degree from University of Dar es Salaam (UDSM) in 2001. She co-organizes the Data Science Africa (DSA), an organization that runs an annual data science and machine learning summer school and workshop in Africa. She is a member of a research group on Mechanism Design for Social Good (MDSG) and an awardee of the 2019 Organization of Women in Science for the Developing World (OWSD) Early Career Fellowship.

Machine Learning Competitions: The Outlook from Africa

Talk by Dina Machuve
Nelson Mandela African Institution of Science and Technology (NM-AIST)

The current AI landscape in Africa mainly focuses on capacity building. The ongoing efforts to strengthen the AI capacity in Africa are organized in summer schools, workshops, meetups, competitions and one long-term program at the Masters level. The main AI initiatives driving the AI capacity building agenda in Africa include a) Deep Learning Indaba, b) Data Science Africa, c) Data Science Nigeria, d) Nairobi Women in Machine Learning and Data Science, e) Zindi and f) The African Master's in Machine Intelligence (AMMI) at AIMS. The talk will summarize our experience on low participation of African AI developers at machine learning competitions and our recommendations to address the current challenges.

Frank Hutter

Afbeeldingsresultaat voor frank hutter challengesProfessor dr. Frank Hutter is head of the machine learning lab at the University of Freiburg (Germany). He and his lab won both the first and second AutoML (automatic machine learning) challenge. His research is mostly of a methodological nature on the meta-level, such as machine learning methods that model & optimize the performance of other machine learning methods. He co-organized the NeurIPS workshops on Meta-Learning in 2017 and 2018.

A Proposal for a New Competition Design Emphasizing Scientific Insights

Talk by Frank HutterUniversity of Freiburg

The typical setup in machine learning competitions is to provide one or more datasets and a performance metric, leaving it entirely up to participants which approach to use, how to engineer better features, whether and how to pretrain models on related data, how to tune hyperparameters, how to combine multiple models in an ensemble, etc. The fact that work on each of these components often leads to substantial improvements has several consequences: (1) amongst several skilled teams, the one with the most manpower and engineering drive often wins; (2) it is often unclear *why* one entry performs better than another one; and (3) scientific insights remain limited.

Based on my experience in both participating in several challenges and also organizing some, I will propose a new competition design that instead emphasizes scientific insight by dividing the various ways in which teams could improve performance into (largely orthogonal) modular components, each of which defines its own competition. E.g., one could run a competition focussing only on effective hyperparameter tuning of a given pipeline (across private datasets). With the same code base and datasets, one could likewise run a competition focussing only on finding better neural architectures, or only better preprocessing methods, or only a better training pipeline, or only better pre-training methods, etc. One could also run multiple of these competitions in parallel, hot-swapping better components found in one competition into the other competitions. I will argue that the result would likely be substantially more valuable in terms of scientific insights than traditional competitions and may even lead to better final performance.