Siberian Mathematical Journal, Vol. 53, No. 3, pp. 404–418, 2012 Original Russian Text Copyright c  2012 Bilalov B. T. ON SOLUTION OF THE KOSTYUCHENKO PROBLEM B. T. Bilalov UDC 517.5 Abstract: Under study is the basis property in L 2 of a system of Kostyuchenko type. In particular, some criterion is established for the basis property of the Kostyuchenko system under the natural constraints on a parameter in this system. Keywords: basis property, completeness, minimality, Kostyuchenko system Introduction In the spectral theory of pencils of differential operators, well-known since 1969 has been the problem by A. G. Kostyuchenko: Prove completeness of the system S + α ≡{e iαnt sin nt} n≥1 in L 2 (0,π) by function- theoretic methods and study the basis properties of this system. The first results in this direction were obtained by B. Ya. Levin in 1971 (cf. [1]). He proved completeness of the system S + α in L 2 (0,π) for all α ∈ iR. The same result was established earlier in [2] by another method with the use of twofold completeness of the system S α ≡{e iαnt sin nt} +∞ −∞ in L 2 (0,π). The system S α is the collection of eigenfunctions of the quadratic pencil −y  (t)+2iαλy  (t) + (1 − α 2 )λ 2 y(t)=0,t ∈ (0,π), y(0) = y(π)=0. It is twofold complete in L 2 (0,π) which fact follows from the celebrated article [3] by M. V. Keldysh. Multiple completeness of a half of eigenfunctions of differential pencils was examined in [4–6]; in particular, the results of these articles ensure completeness of the system S + α in L 2 (0,π) for imaginary α (with some restrictions). The interest in the study of basis properties of the system S + α was considerably aroused in this connection. The general approach to the study gave an impetus to examining the systems of the form  υ ± n  n≥m ≡{a (t)ϕ n (t) ± b(t)ψ n (t)} n≥m , (1) where a(t), b(t), ϕ(t), and ψ(t) are complex-valued functions in general. In turn, the latter results in creating a distinguished direction in approximation theory for studying the basis properties of systems of the form (1). The case of ψ(t) ≡ ϕ(t) (the overline means complex conjugation) is most studied (see [7– 11]). This case for nonreal parameters α does not encompass the Kostyuchenko system. The system (1) seemed to be difficult for studying since its basis properties are reduced to the solvability of the Riemann problems with shift on the boundary of the corresponding domain in the Hardy space H ± p or the Smirnov spaces E p (D). The resulting boundary value problems have essential difficulties in comparison with those of the literature (for instance, see [12, 13]). Hence, the basis properties of systems of the form (1) were not completely investigated. Various authors develop their own approaches to solving the problem under various conditions on the functions in (1). Despite this fact, many articles are devoted to the systems of the form (1) (see [14–19] and the survey [18]). As applied to the Kostyuchenko system S + α , the summary is as follows: for α ∈ R and |α| < 1, the system S + α is complete in L p (0,π) for every p ∈ [1, +∞). In general, for α/ ∈ (−∞, −1] ∪ [1, ∞), the system S + α is overfilled in L 2 (0,π). The excess of S + α grows infinitely as α tends to [1, ∞) (or to (−∞, −1]). If α ∈ C\{(−∞, −1] ∪ [1, +∞)} is an arbitrary complex number then the system is complete in L 2 (0,π); for α ∈ iR, the system S + α is also minimal in L 2 (0,π). If α/ ∈ R then S + α is not uniformly minimal in L 2 (0,π) [20] and so does not form a basis for this space. Baku. Translated from Sibirski˘ı Matematicheski˘ı Zhurnal, Vol. 53, No. 3, pp. 509–526, May–June, 2012. Original article submitted May 13, 2009. Revision submitted December 7, 2011. 404 0037-4466/12/5303–0404 c 