Journal of Parallel and Distributed Computing 62, 1069–1103 (2002) doi:10.1006/jpdc.2001.1817 Scheduler-Activated Dynamic Page Migration for Multiprogrammed DSM Multiprocessors 1 Dimitrios S. Nikolopoulos 2 and Constantine D. Polychronopoulos Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, 1308 West Main St., Urbana, Illinois 61801-228 E-mail: dsn@csrd.uiuc.edu, cdp@csrd.uiuc.edu Theodore S. Papatheodorou Department of Computer Engineering and Informatics, University of Patras, Rion 26 500, Patras, Greece E-mail: tsp@hpclab.ceid.upatras.gr and Jes ! us Labarta and Eduard Ayguad ! e Department d’ Arquitectura de Computadors, Universitat Politecnica de Catalunya, c. Jordi Girona 1–3, 08034 Barcelona, Spain E-mail: jesus@ac.upc.es, eduard@ac.upc.es Received October 23, 2000; revised December 27 2001; accepted 28 December 2001 The performance of multiprogrammed shared-memory multiprocessors suffers often from scheduler interventions that neglect data locality. On cache- coherent distributed shared-memory (DSM) multiprocessors, such scheduler interventions tend to increase the rate of remote memory accesses. This paper presents a novel dynamic page migration algorithm that remedies this problem in iterative parallel programs. The purpose of the algorithm is the early detection of pages that will most likely be accessed remotely by threads associated with them via a thread-to-memory affinity relation. The key mechanism that enables timely identification of these pages is a communica- tion interface between the page migration engine and the operating system scheduler. The algorithm improves previously proposed competitive page migration algorithms in many aspects, including accuracy, timeliness and cost amortization. Most notably, the algorithm is not biased by obsolete memory access history that may be accumulated in the page access counters at runtime. 1 This work was supported by the European Commission under TMR Grant ERBFMGECT-950062 and in part by ESPRIT IV Grant 21907, Greek Secretariat of Research and Technology Grant E.D.-99- 566, Ministry of Education of Spain Grants TIC-98-511 and TIC97-1445CE, NSF Grant EIA-99-75019, a research Grant from NSA and a research Grant from Intel Corporation. The experiments presented in this paper were conducted with resources provided by the European Center for Parallelism of Barcelona (CEPBA). 2 To whom correspondence should be addressed. 1069 0743-7315/02 $35.00 # 2002 Elsevier Science (USA) All rights reserved.