Designing, Deploying, and Analyzing Adaptive Educational Field Experiments

Digital experiments can be used in CSedu to test hypotheses about interventions and conditions' efficacy (or inefficacy). This workshop will discuss and deconstruct the design process and analysis for various experiments conducted in CS1. E.g., experiments testing which explanations students find helpful, which emails get them to start homework early, or which webpages effectively encourage and motivate students. This workshop teaches participants how to conduct, interpret, and analyze adaptive field experiments. These adaptive experiments employ machine learning algorithms to analyze experiments during deployment and dynamically shift the allocation of arms/conditions to give future students better conditions more rapidly. Adaptive field experiments can accelerate scientific discovery by enabling more complex experimental designs and increasing statistical power by phasing conditions in and out more efficiently. The workshop is supported by a 5-year NSF grant to build software tools and a digital community, gathering instructors, domain scientists and methodologists to teach them how to run adaptive experiments. The methodological focus includes understanding: (1) which algorithms are best for adaptive experiments that meet domain scientists' needs in specific experimental designs and data sets; (2) which hypothesis tests and Bayesian analyses to choose. Software companies use these innovative methodologies extensively to continuously improve product design. This workshop demonstrates how the same methods can be used in CSedu to improve research rigor and accelerate educational research implementation, ultimately improving student outcomes.