IBM Computational Biology Center
Icahn School of Medicine at Mount Sinai
Seeking the Wisdom of Crowds in Biomedical Research: The DREAM Challenges
DREAM (Dialogue for Reverse Engineering Assessment and Methods) Challenges are community-based collaborative competitions that engage diverse communities of scientists to solve specific problems in biomedicine in a fixed time period. Since its founding in 2006, DREAM’s distributed community of Challenge organizers has launched 32 successful Challenges, attracted over 8,000 Challenge registrants and evolved a powerful approach to democratize data and accelerate research. The process of curating a DREAM challenge consists of several well defined steps including the definition of a scientific question of fundamental clinical and/or basic research importance, the procurement of a Gold Standard, the definition of incentives for participation, an open Challenge competitive phase (which can be followed by a collaborative phase), the scoring of the submissions and the writing of papers documenting the insights gained from the Challenges. The recent partnership between DREAM and Sage Bionetworks led to a new generation of Challenges that leverage collaborative data hosting and analysis tools, such as real-time leaderboards and shared project spaces using Synapse, Sage Bionetworks IT platform, which handles the growing computational demands of these open collaborative efforts. In this talk I will discuss some of the lessons learned from the curation of DREAM Challenges, the potential difficulties that may arise in the course of a Challenge, and important scientific insights gained in many of the DREAM Challenges. Challenge-based research will play an increasingly important role in translational systems biology, and its broader adoption will drive progress in both algorithm development and biological discovery.