Journal of Mathematical Sciences, Vol. 224, No. 2, July, 2017 SPECIAL REPRESENTATIONS OF THE IWASAWA SUBGROUPS OF SIMPLE LIE GROUPS A. M. Vershik ∗ and M. I. Graev † UDC 517.986 In the paper, a family of representations of maximal solvable subgroups of the simple Lie groups O(p, q), U (p, q), and Sp(p, q), where 1 ≤ p ≤ q, is introduced. These subgroups are called the Iwasawa subgroups of thecorresponding simple groups. The main property of these representations is the existence of nontrivial 1-cohomology with values in the representations. For groups of rank 1, the representations from this family are unitary; for ranks greater than 1, they are nonunitary. The paper continues a series of our previous papers and serves as an introduction to the theory of nonunitary current groups. Bibliography: 9 titles. 1. Some subgroups of simple Lie groups (Z, N , N ∗ , Heis, S , and P ) 1.1. Notation. We will consider certain subgroups of the simple Lie groups of real, complex, and quaternion matrices – O(p, q), U (p, q), and Sp(p, q), respectively. Let us denote by ∗ the conjugation in the field of complex numbers C and in the division algebra of quarternions H. When applied to matrices, this symbol will denote the operation of simultaneously conjugating the matrix entries and transposing the matrix itself. With each pair of positive integers p, q, where p ≤ q, we associate the additive group Z = Z p,q of all (p, q)-matrices over R, C, or H, and denote by N = N p the additive group of skew-Hermitian p-matrices over R, C, or H satisfying the condition n ∗ = −n. Observe that for p = 1, this is the zero subgroup, the group of imaginary numbers, and the three-dimensional group of imaginary quaternions, respectively. Denote by N ∗ ≃ N the dual group of N , whose elements will be denoted by m. The pairing between the groups N ∗ and N will be denoted by 〈m, n〉, i.e., 〈m, n〉 = exp[i Re Tr(m, n)], (1) where Re is the real part of a quaternion z. 1.2. The groups Heis = Heis(p, q) Definition 1. The generalized Heisenberg group Heis(p, q) is the rank 2 nilpotent group whose elements are pairs (n, z) where n ∈ N p , z ∈ Z p,q with the multiplication law (n 1 ,z 1 )(n 2 ,z 2 )=(n 1 + n 2 − 1 2 (z 1 z ∗ 2 − z 2 z ∗ 1 ),z 1 + z 2 ) =(n 1 + n 2 − Im z 1 z ∗ 2 ,z 1 + z 2 ), (2) where Im z 1 z ∗ 2 is the natural sesquilinear 2-cocycle of the additive group Z p,q + Z p,q with values in the group N p of skew-Hermitian matrices of order p. In particular, for p = q = 1 this group is isomorphic to the Abelian group R in the real case, to the classical complex Heisenberg group of real dimension 3 (= 2 + 1) in the complex case, and to the rank 2 nilpotent group with a three-dimensional center of real dimension 7 (= 4 + 3) in the quaternionic case. * St.Petersburg Department of Steklov Institute of Mathematics; St.Petersburg State University, St. Peters- burg, Russia; Institute for Information Transmission Problems, Moscow, Russia, e-mail: vershik@pdmi.ras.ru. † Institute for System Studies, Moscow, Russia, e-mail: graev 36@mtu-net.ru. Translated from Zapiski Nauchnykh Seminarov POMI, Vol. 448, 2016, pp. 99–106. Original article submitted September 27, 2016. 1072-3374/17/2242-0231 ©2017 Springer Science+Business Media New York 231 DOI 10.1007/s10958-017-3408-2