We welcome 2-page extended abstracts on one of the topics of the workshop.
The abstracts should be submitted before October 9th, 2014 by email to email@example.com.
Topics of interest: Methods: - Novel or atypical challenge protocols, particularly to tackle complex tasks with very large datasets, multi-modal data, and data streams. - Methods and metrics of entry evaluation, quantitative and qualitative challenges. - Methods of data collection, "ground-truthing", and preparation including bifurcation/anonymization, data generating models. - Teaching challenge organization. - Hackatons and on-site challenges. - Challenge indexing and retrieval, challenge recommenders. Theory: - Experimental design, size data set, data split, error bounds, statistical significance, violation of typical assumptions (e.g. i.i.d. data). - Game theory applied to the analysis of challenge participation, competition and collaboration among participants. - Diagnosis of data sanity, artifacts in data, data leakage. Implementation: - Re-usable challenge platforms, innovative software environments. - Linking data and software repositories to challenges. - Security/privacy, intellectual property, licenses. - Cheating prevention and remedies. - Issues raised by requiring code submission. - Challenges requiring user interaction with the platform (active learning, reinforcement learning). - Dissemination, fact sheets, proceedings, crowsourced papers, indexing post-challenge publications. - Long term impact, on-going benchmarks, metrics of impact. - Participant rewards, stimulation of participation, advertising, sponsors. - Profiling participants, improving participant professional and social benefits. Applications: - Where to venture next: opportunities for challenge organizers to organize challenges in new domains with high societal impact. - Successful challenge leading to significant breakthrough or improvement over the state-of-the-art or unexpected interesting results. - Rigorous study of the impact of challenges, analyzing topics and tasks lending themselves to high impact machine learning challenges. - Challenges as an educational tool. - Challenges organized or supported by Government agencies, funding opportunities.