Sbornik : Mathematics 202:11 1593–1615 c ⃝ 2011 RAS(DoM) and LMS Matematicheski˘ ı Sbornik 202:11 31–54 DOI 10.1070/SM2011v202n11ABEH004200 Extremal trajectories in a nilpotent sub-Riemannian problem on the Engel group A. A. Ardentov and Yu. L. Sachkov Abstract. On the Engel group a nilpotent sub-Riemannian problem is considered, a 4-dimensional optimal control problem with a 2-dimensional linear control and an integral cost functional. It arises as a nilpotent approximation to nonholonomic systems with 2-dimensional control in a 4-dimensional space (for example, a system describing the navigation of a mobile robot with trailer). A parametrization of extremal trajectories by Jacobi functions is obtained. A discrete symmetry group and its fixed points, which are Maxwell points, are described. An estimate for the cut time (the time of the loss of optimality) on extremal trajectories is derived on this basis. Bibliography: 25 titles. Keywords: optimal control, sub-Riemannian geometry, geometric meth- ods, Engel group. § 1. Introduction This paper is concerned with the analysis of a nilpotent sub-Riemannian problem on the Engel group, a 4-dimensional optimal control problem with a 2-dimensional linear control and an integral performance functional. Nilpotent sub-Riemannian problems are fundamental for sub-Riemannian geometry since they provide a local quasi-homogeneous approximation to general sub-Riemannian problems (see [1]–[4]). For instance, a nilpotent sub-Riemannian problem on the 3-dimensional Heisenberg group (see [5]) is a cornerstone of the entire sub-Riemannian geometry. Invariant sub-Riemannian problems on Lie groups have intensively been investigated through the last 10 years by means of geometric control theory (see [6]–[12]). The invariant sub-Riemannian problem on the Engel group has several important properties which underline its special role in sub-Riemannian geometry. First, this is the simplest sub-Riemannian problem with nontrivial abnormal extremal trajec- tories (it is known that in 3-dimensional contact problems abnormal extremal tra- jectories are constant [13]). Second, this problem projects onto the sub-Riemannian problem in the Martinet flat case [14], so the problem on the Engel group is the simplest invariant sub-Riemannian problem on a nilpotent Lie group with nonsub- analytic sub-Riemannian sphere. Third, the vector distribution in this problem This research was carried out with the support of the Russian Foundation for Basic Research (grant no. 09-01-00246-a), and the “Mathematical Control Theory” programme of the Presidium of the Russian Academy of Sciences. AMS 2010 Mathematics Subject Classification. Primary 53C17, 95B29; Secondary 49K15.