arXiv:1404.1171v2 [math.RT] 17 Mar 2016 THE COMPACT PICTURE OF SYMMETRY BREAKING OPERATORS FOR RANK ONE ORTHOGONAL AND UNITARY GROUPS JAN M ¨ OLLERS AND BENT ØRSTED Abstract. We present a method to calculate intertwining operators between the underly- ing Harish-Chandra modules of degenerate principal series representations of a semisimple Lie group G and a semisimple subgroup G ′ , and between their composition factors. Our method decribes the restriction of these operators to the K ′ -isotypic components, K ′ ⊆ G ′ a maximal compact subgroup, and reduces the representation theoretic problem to an infinite system of scalar equations of a combinatorial nature. For rank one orthogonal and unitary groups and spherical principal series representations we calculate these relations explicitly and use them to classify intertwining operators. We further show that in these cases auto- matic continuity holds, i.e. every intertwiner between the Harish-Chandra modules extends to an intertwiner between the Casselman–Wallach completions, verifying a conjecture by Kobayashi. Altogether, this establishes the compact picture of the recently studied symme- try breaking operators for orthogonal groups by Kobayashi–Speh, gives new proofs of their main results and extends them to unitary groups. Applications of our classification for orthogonal groups include the construction of discrete components in the restriction of certain unitary representations, a Funk–Hecke type formula and the computation of the spectrum of Juhl’s conformally invariant differential operators. 1. Introduction Representation theory of semisimple Lie groups consists to a large extent of the study of the structure of standard families of representations, for example principal series representations. Here intertwining operators, such as the classical Knapp–Stein operators, play an important role, and they also provide important examples of integral kernel operators appearing in clas- sical harmonic analysis. Recently [6, 9] similar operators have been introduced in connection with branching laws, i.e. the study of how representations behave when restricted to a closed subgroup of the original group. Again these are integral kernel operators, now intertwining with respect to the subgroup, and they appear to be very natural objects, not only for the problem of restricting representations (see [12]), but also for questions in classical harmonic analysis and automorphic forms (see [8, 10]). In this paper we shall give a novel approach to this new class of symmetry-breaking op- erators, namely one based on the Harish-Chandra module, i.e. the K-finite vectors in the representation, in analogy with the idea of spectrum generating operators [1]. This gives new proofs of the main results of [6], and generalizes these results to unitary groups. Moreover, our more algebraic framework provides an alternative proof of the discrete spectrum in certain unitary representations, as well as (seemingly new) explicit Funk–Hecke type identities. The approach is quite general and discussed in the first part of the paper (see Sections 2 and 3), 2010 Mathematics Subject Classification. Primary 22E46; Secondary 17B15, 05E10. Key words and phrases. symmetry breaking operators, intertwining operators, Harish-Chandra modules, principal series, spectrum generating operator, Funk-Hecke formula. 1