Rotonlike Fulde-Ferrell Collective Excitations of an Imbalanced Fermi Gas in a Two-Dimensional Optical Lattice Zlatko Koinov, 1, * Rafael Mendoza, 2 and Mauricio Fortes 2 1 Department of Physics and Astronomy, University of Texas at San Antonio, San Antonio, Texas 78249, USA 2 Instituto de Fı ´sica, UNAM, Apartado Postal 20-364, 01000 Me ´xico, D.F., Me ´xico (Received 4 November 2010; revised manuscript received 23 December 2010; published 7 March 2011) We address the question of whether superfluidity can survive in the case of fermion pairing between different species with mismatched Fermi surfaces using as an example a population-imbalanced mixture of 6 Li atomic Fermi gas loaded in a two-dimensional optical lattice at nonzero temperatures. The collective mode is calculated from the Bethe-Salpeter equations in the general random phase approxi- mation assuming a Fulde-Ferrell order parameter. The numerical solution shows that, in addition to low- energy (Goldstone) mode, two rotonlike minima exist, and therefore, the superfluidity can survive in this imbalanced system. DOI: 10.1103/PhysRevLett.106.100402 PACS numbers: 67.85.d, 03.75.Ss, 71.10.Pm, 73.20.Mf Introduction.—Although the Fulde and Ferrell (FF) [1] and the Larkin and Ovchinnikov (LO) [2] phases were introduced quite a long time ago, they are still of very high interest because the question of whether the super- conductivity or superfluidity can survive in polarized sys- tems remains unanswered. In the FFLO phase, Cooper pairing occurs between a fermion (or up and down quarks) with momentum k þ q and spin " and a fermion with momentum k þ q, and spin # . As a result, the pair- momentum is 2q and the order parameter becomes spa- tially dependent. The mean-field treatment of the FFLO phase in a variety of systems, such as superconductors with Zeeman splitting and heavy-fermion superconductors [3], atomic Fermi gases with population imbalance loaded in optical lattices [4–6] and harmonic traps [7], and dense quark matter [8] shows that the FFLO state competes with a number of other states, such as the Sarma (q ¼ 0) states [9], but in some regions of momentum space the FFLO phase provides the minimum of the mean-field expression of the Helmholtz free energy. Since it is known that with decreasing dimensionality, the pair fluctuation becomes increasingly important, and therefore, there is no a priori justification for applying the mean-field calculations in one-dimensional (1D) systems. Thus, in what follows we consider an imbalanced mixture of a 6 Li atomic Fermi gas of two hyperfine states j "i and j #i with contact interaction loaded into a 2D square optical lattice. The total number of atoms is M ¼ M " þ M # , and they are distributed along N sites. The FFLO state is expected to occur on the BCS side of a Feshbach resonance, where the effective attractive interaction between fermion atoms leads to BCS type pairing. We also assume that the lattice potential is suffi- ciently deep such that the tight-binding approximation is valid and the system is well described by the single-band attractive Hubbard model (on the BCS side the Hubbard parameter U is negative, but in what follows U denotes its absolute value). The tight-binding form of the electron energy is  ";# ðkÞ¼ 2Jð1  P  cosk  aÞ  ";# , where  ";# is the corresponding chemical potential, J is the tunneling strength of the atoms between nearest-neighbor sites, and the lattice constant a ¼ =2 ( is the laser wavelength and in our numerical calculations we use  ¼ 1032 nm). The order parameter is assumed to vary as a single plane wave  q ¼  expð2{q:rÞ. Unlike the population-balanced sys- tems, for which the spectrum of the collective excitations has been obtained by linearizing the Anderson-Rickayzen equations [10], by the Kadanoff and Baym approach [11], and by the Bethe-Salpeter (BS) formalism [12], to the best of our knowledge the FFLO collective modes have been studied in (i) a 1D population-unbalanced trapped system [7] by using the linear response of the equilibrium system by supplementing the Bogoliubov–de Gennes (BdG) equa- tions with a self-consistent random phase approximation, (ii) a 1D superconductor [13] by transforming slow defor- mations of the order parameter into small corrections to the BdG Hamiltonian, and (iii) a cold-atom rotated system [14] by locating the poles of the many-body scattering function. We present here a theory which is the first calculation to find the spectrum of the collective excitations in the pres- ence of FF phase which goes beyond the mean-field gap, number, and pair-momentum equations by solving the BS equations for the spectrum of the collective excitations in the general random phase approximation. Since at a finite temperature the FF states compete with the Sarma states, before calculating the collective modes we have obtained the phase diagram. In Fig. 1, we show the phase separation between the FF, Sarma, and normal states for a total filling factor f ¼ f " þ f # ¼ 0:5 (f ";# ¼ M ";# =N) and an interac- tion strength U=J ¼ 2:64, which is similar to the phase diagram in the case of 3D optical lattice [5]. The polariza- tion P ¼ðf "  f # Þ=ðf " þ f # Þ that we shall use in collective-mode calculations is P ¼ 0:1. In this case, the FF states lower the system free energy compared to the corresponding Sarma states at low temperatures. As PRL 106, 100402 (2011) PHYSICAL REVIEW LETTERS week ending 11 MARCH 2011 0031-9007= 11=106(10)=100402(4) 100402-1 Ó 2011 American Physical Society