©EDA Publishing/THERMINIC 2011 ISBN:978-2-35500-018-8 27-29 September 2011, Paris, France System-Level Thermal-Aware Design of 3D Multiprocessors with Inter-Tier Liquid Cooling Arvind Sridhar Mohamed M. Sabry David Atienza Embedded Systems Laboratory (ESL), École Polytechnique Fédérale de Lausanne (EPFL), ELG 130 (Bâtiment ELG), Station 11, 1015 Lausanne, Switzerland. {arvind.sridhar, mohamed.sabry, david.atienza} @epfl.ch Abstract- Rising chip temperatures and aggravated thermal reliability issues have characterized the emergence of 3D multiprocessor system-on-chips (3D-MPSoCs), necessitating the development of advanced cooling technologies. Microchannel based inter-tier liquid cooling of ICs has been envisaged as the most promising solution to this problem. A system-level thermal-aware design of electronic systems becomes imperative with the advent of these new cooling technologies, in order to preserve the reliable functioning of these ICs and effective management of the rising energy budgets of high-performance computing systems. This paper reviews the recent advances in the area of system- level thermal modeling and management techniques for 3D multiprocessors with advanced liquid cooling. These concepts are combined to present a vision of a green data-center of the future which reduces the CO 2 emissions by reusing the heat it generates. Keywords- 3D Integration, Liquid Cooling, System-Level Thermal Aware Design, Green Data-Centers. I. LIQUID COOLING OF MICROELECTRONIC SYSTEMS The economic and technological drivers pushing the trend of shrinking CMOS feature size and the increasing die size to incorporate larger functionality in Integrated Circuits (ICs) have slowed down in the recent years, but the demand for faster and more versatile electronic products remains insatiable [1]. Multiprocessor system-on-chips (MPSoCs) have helped meeting this demand via the integration of diverse functionalities on a single silicon die, and have revolutionized the electronics industry. However, increasing the size of the die in two dimensions for this purpose has depreciating returns in terms of performance enhancements due to increasing interconnect length and the resulting delays. Hence, even the MPSoCs are quickly approaching the limits of their computing throughput capacity. In this context, 3D integration of multiprocessor ICs opens up a new dimension in design space for VLSI engineers. On one hand, 3D integration enables shorter interconnections, handling a larger IC design complexity and the possibility of heterogeneous integration [2]. On the other hand, it also brings compounded heat dissipation and larger thermal resistances to heat sinks, which results in chip temperatures well beyond the safe operating levels, thus severely undermining the already aggravated thermal reliability of MPSoC designs and lifetimes. As a result, conventional air- Fig. 1: On chip power density during the last two decades (Courtesy: [6], IEEE Special Issue on Thermal Engineering) cooled heat sinking has become insufficient in the context of 3D-MPSoC integration [3,4]. Thirty years ago, Tuckerman and Pease [5] published a seminal paper studying the merits of liquid cooling in ICs via microchannels etched directly on the substrate, and formulated the main determinants for the design of microchannel heat sinks that maximize cooling efficiency. With on-chip heat flux densities approaching 100 W/cm 2 in 2D processor chips- and twice or even thrice that value in 3D chips- in the recent years (Fig. 1), this work has inspired a renewed interest in the development of liquid cooled package design for ICs [4, 7-13]. The microchannel single- phase liquid cooling of 3D ICs is now envisaged to be the most promising short to medium solution to the problem of rising chip thermal reliability issues [4], while two-phase cooling is seen as the long term solution to meet the growing demands for energy in the high-performance computing installations and data-centers of the future [13]. A considerable amount of research effort has been invested in the last two decades towards the design of efficient microchannel liquid cooled heat sinks, especially single-phase cooling, of ICs. However, the electronics industry hasn’t seen a large-scale acceptance of this technology. While this is partly due to the fact that the advent of CMOS circuits postponed the anticipated 1