Bayesian Optimization Objective-Based Experimental Design Mahdi Imani and Seyede Fatemeh Ghoreishi Abstract— Design has become a salient part of most of the scientific and engineering tasks, embracing a wide range of domains including real experimental settings (e.g., material discovery or drug design), simulation-based design, and hyper- parameter tuning. Model-based experimental design refers to a broad class of techniques, applicable to domains that a partial knowledge about the underlying process exists. Unlike entropy- based techniques which aim to reduce the whole uncertainty in the process, the mean objective cost of uncertainty (MOCU) is a rigorous statistically-oriented experimental design framework which takes the main objective into account during the decision making. However, the lack of scalability of this framework has restricted its application to domains with very small design spaces. This paper proposes a framework using the combination of Bayesian optimization and MOCU policy, which enables experimental design to much larger design spaces and systems. The reliability, scalability and efficiency of the proposed frame- work are investigated through experimental design for optimal structural intervention in gene regulatory networks. I. I NTRODUCTION Design is a crucial component in most of the science and engineering tasks. This is, in particular, important in contemporary applications, where the advancements in technology have highly expanded the size of systems and phenomena that are under study. Experimental design aims to systematically select or plan experiments for achieving the desired outcomes. The domains that experimental design have impacted significantly comprise real experiments, such as drug design [1], material discovery [2] and aerospace en- gineering [3]; simulation-based experimental design, such as design in robotics [4], materials science [5], and transporta- tion [6], [7]; and parameter learning or hyper-parameter tuning, including efficient tuning of parameters of deep belief networks [8], Markov chain Monte Carlo (MCMC) methods [9]–[12], and convolutional neural networks [13]. Model-based experimental design is a broad class of experimental design that has received a lot of attention in statistics community as well as in many science and engineering applications [14]–[20]. The main goal in model- based experimental design is to make good designs by taking into account the knowledge about the underlying physical processes in decision-making process. Applications of these techniques include parameter inference, prediction, or model discrimination [21]–[27] as well as some real experimen- tal processes such as materials science [28], biomedical engineering [29], and aerospace engineering [10], [30]– [35]. Information theoretic-based [36]–[38] and objective- M. Imani is with the Department of Electrical and Computer Engineering, George Washington University, and S. F. Ghoreishi is with Institute for Systems Research (ISR) at University of Maryland mimani@gwu.edu, sfg@umd.edu based [15], [39], [40] techniques are two well-known classes of model-based experimental design. The former techniques aim to reduce the uncertainty in the whole systems without taking the main objective into account, whereas the latter ones are capable of reducing the uncertainty that matters the most with regards to the main objective. This feature makes the class of objective-based experimental design a suitable choice for many realistic domains, as experiments are often very expensive and are performed for achieving specific goals. The main focus of this paper is objective-based experi- mental design, and in particular, the well-known policy in this category which is the mean objective cost of uncertainty (MOCU) [15], [39], [40]. Despite the success of MOCU policy in some domains, its lack of scalability has prevented its application to a wide range of practical problems. In fact, the existing MOCU techniques are applicable to small systems with finite and discrete design spaces, while practical systems are often large with possibly continuous design spaces. In this paper, we enable large-scale experimental design using the combination of Bayesian optimization and MOCU policy. The Bayesian nature of the proposed framework enables correlation consideration over the design space and making an efficient balance between exploration and ex- ploitation during the design process, leading to scalable and efficient experimental design process. The performance of the proposed framework is investigated using the optimal structural intervention in gene regulatory networks, which is critical to alter the regulatory logic in order to maximally reduce the long-run likelihood of being in a cancerous state. II. MODEL-BASED EXPERIMENTAL DESIGN:MEAN OBJECTIVE COST OF UNCERTAINTY (MOCU) In some domains, we might have access to a partial knowl- edge about the underlying process of systems, and would like to take advantage of this knowledge during the design pro- cess. The term “partial” refers to the uncertainty in systems, which needs to be reduced for achieving a proper perfor- mance regarding the main objective of the process. However, since experiments can be costly and time-consuming, it is desirable to determine the experiments which provide the most useful information. A classical approach is to maxi- mally reduce the overall uncertainty in the model, meaning maximal entropy reduction [16]. By contrast, the objective- based experimental design techniques aim to reduce the part of uncertainty that matters the most with respect to the main objective function. The most popular objective-based experimental design is called the mean objective cost of