Proceedings of the Edinburgh Mathematical Society (1990) 33, 337-366 © A SCHUR-COHN THEOREM FOR MATRIX POLYNOMIALS by HARRY DYM and NICHOLAS YOUNG (Received 14th July 1988) Let N(X) be a square matrix polynomial, and suppose det N is a polynomial of degree d. Subject to a certain non-singularity condition we construct a d by d Hermitian matrix whose signature determines the numbers of zeros of N inside and outside the unit circle. The result generalises a well known theorem of Schur and Cohn for scalar polynomials. The Hermitian "test matrix" is obtained as the inverse of the Gram matrix of a natural basis in a certain Krein space of rational vector functions associated with N. More complete results in a somewhat different formulation have been obtained by Lerer and Tismenetsky by other methods. 1980 Mathematics subject classification (1985 Revision): 30C15, 46D05, 93D20. 1. Introduction A celebrated paper of I. Schur [11] contains, among much else, a criterion for a polynomial p(X) to have all its zeros in the open unit disc D: one forms a certain Hermitian matrix H from the coefficients of p, and the assertion is that the zeros of p lie in D if and only if H is positive definite. The theorem was generalized by A. Cohn [4]: if the zeros of p are all non-conjugate with respect to the unit circle T then the numbers of zeros inside and outside the unit circle are equal to the numbers of positive and negative eigenvalues of H respectively. The corresponding problem for polynomial matrices holds considerable interest, both for its own sake and for application to multivariable systems theory: the stability of a linear system with many inputs and outputs depends on the zeros of an associated matrix polynomial lying in D. Now the zeros of a square matrix polynomial N(X) (also known as the eigenvalues of N) are defined to be the zeros of the scalar polynomial det N(X), and so one way of testing N is to apply Schur's criterion to det N. However, since the calculation of the determinant of a matrix polynomial is lengthy and numerically somewhat ill behaved, it is natural to look for a more direct way of associating with N(X) a Hermitian matrix whose signature indicates the location of the zeros of N with respect to T. One way to approach this question is through Bezoutian matrices. In the scalar case the Schur matrix can be regarded as the Bezoutian of two polynomials, and there are several extensions of the notion of Bezoutian to matrix polynomials. The most successful one appears to be that due to Anderson and Jury [2], who asked whether their form of Bezoutian could be used to prove an analogue of the Schur-Cohn 337