Copyright © by SIAM. Unauthorized reproduction of this article is prohibited. SIAM J. MATRIX ANAL. APPL. c  2008 Society for Industrial and Applied Mathematics Vol. 30, No. 1, pp. 324–345 OPTIMIZING THE COUPLING BETWEEN TWO ISOMETRIC PROJECTIONS OF MATRICES ∗ CATHERINE FRAIKIN † , YURII NESTEROV ‡ , AND PAUL VAN DOOREN † Abstract. In this paper, we analyze the coupling between the isometric projections of two square matrices. These two matrices of dimensions m × m and n × n are restricted to a lower k- dimensional subspace under isometry constraints. We maximize the coupling between these isometric projections expressed as the trace of the product of the projected matrices. First we connect this problem to notions such as the generalized numerical range, the field of values, and the similarity matrix. We show that these concepts are particular cases of our problem for special choices of m, n, and k. The formulation used here applies to both real and complex matrices. We characterize the objective function, its critical points, and its optimal value for Hermitian and normal matrices, and, finally, give upper and lower bounds for the general case. An iterative algorithm based on the singular value decomposition is proposed to solve the optimization problem. Key words. trace maximization, generalized numerical range, isometry, singular value decom- postion AMS subject classifications. 15A60, 47A12, 65F30, 65K10 DOI. 10.1137/050643878 1. Introduction. The problem of projection of matrices in lower-dimensional subspaces is of great interest for a large field of applications. The projection of matrices provides an easier visualization and comprehension of the initial problem and is often used to reduce the complexity of some computational problems. Moreover the coupling between these projections can reveal some particularities inherent to the data which can be analyzed and interpreted. We consider the coupling or similarity between two “projected” matrices A and B, respectively, of dimensions m × m and n × n, expressed as the real part of the trace of the product of the isometric projections U ∗ AU and V ∗ BV :  tr(U ∗ AUV ∗ B ∗ V ) (1.1) under the constraint that U ∗ U = V ∗ V = I k , where I k denotes the identity matrix of dimension k, with k ≤ min(m, n). In this paper, we will consider both real and complex matrices. The notation will be different for the real and complex cases, i.e., T and ∗ , respectively for the transpose and complex conjugate transpose, the real inner product and real-valued inner product, respectively, for the real and complex case (see the notations in section 2.1). In particular, for real matrices, the coupling we consider is the following: tr(U T AUV T B T V ). (1.2) ∗ Received by the editors October 31, 2005; accepted for publication (in revised form) by M. L. Overton October 16, 2007; published electronically March 19, 2008. This paper presents research results of the Belgian Programme on Interuniversity Attraction Poles, initiated by the Belgian Fed- eral Science Policy Office, and a grant Action de Recherche Concert´ ee (ARC) of the Communaut´ e Fran¸caise de Belgique. The scientific responsibility rests with its authors. http://www.siam.org/journals/simax/30-1/64387.html † Department of Mathematical Engineering, Universit´ e catholique de Louvain (UCL), Bˆatiment Euler, 4 avenue Georges Lemaˆ ıtre, B-1348 Louvain-La-Neuve, Belgium (fraikin@inma.ucl.ac.be, vdooren@inma.ucl.ac.be). ‡ Bˆatiment CORE, 34 voie du Roman Pays, B-1348 Louvain-La-Neuve, Belgium (nesterov@ inma.ucl.ac.be). 324