Fulde-Ferrell-Larkin-Ovchinnikov vortex lattice states in fermionic cold-atom systems Y.-P. Shim, 1 R. A. Duine, 1,2 and A. H. MacDonald 1 1 Department of Physics, University of Texas at Austin, Austin, Texas 78712, USA 2 Institute for Theoretical Physics, Utrecht University, Leuvenlaan 4, 3584 CE Utrecht, The Netherlands Received 11 August 2006; published 2 November 2006 Condensation of atom pairs with finite total momentum is expected in a portion of the phase diagram of a two-component fermionic cold-atom system. This unusual condensate can be identified by detecting the exotic higher-Landau-level HLL vortex lattice states it can form when rotated. With this motivation, we have solved the linearized gap equations of a polarized cold-atom system in a Landau-level basis to predict experimental circumstances under which HLL vortex lattice states occur. DOI: 10.1103/PhysRevA.74.053602 PACS numbers: 03.75.Ss, 71.10.Ca, 32.80.Pj I. INTRODUCTION Polarized two-component fermion systems tend toward finite-pair-momentum condensates because of the Fermi ra- dius mismatch between majority and minority components. In superconductors, electron spin polarization can be induced by the application of an external field or by proximity cou- pling to a ferromagnet. Finite-momentum Cooper pair con- densates in spin-polarized superconductors, Fulde-Ferrell- Larkin-Ovchinnikov FFLO states, were first proposed in the early 1960s 1,2. One important consequence of finite- momentum pairing in an isolated superconductor is a spa- tially inhomogeneous order parameter. There have been many efforts in various solid-state systems to detect this ex- otic state, including recent ones 3,4, but its definitive iden- tification has remained elusive. The disorder that is inevita- bly present in a solid-state system may have played a role in the absence of a conclusive FFLO-state identification in studies of spin-polarized superconductors. Experimental progress 5–8 with fermionic cold-atom systems has given rise to a strategy for realizing the FFLO state or the related Sarma state 9 and has stimulated a great deal of theoretical activity 10–24. The tunability of the in- teraction between atoms via a Feshbach resonance 25,26 has made it possible to increase the strength of fermion pair- ing and has even made the BEC-BCS crossover 27–29 ex- perimentally accessible. On the Bose-Einstein condensate BEC side of a Feshbach resonance fermionic atoms form bosonic molecules which condense at low temperatures. On the Bardeen-Cooper-Schrieffer BCS side, the effective at- tractive interaction between fermion atoms leads to BCS- type pairing. In between lies the so-called unitarity limit 30 in which no weakly interacting particle description applies. Easy control over the population of two hyperfine states in a trapped-atom cloud makes cold-atom systems a promis- ing candidate for FFLO-state realization. The FFLO state competes 10–24 with a number of other states, including in cold-atom systems states with phase-separated regions that are respectively unpolarized and unpaired. The FFLO state is expected to occur on the BCS side of the BEC-BCS cross- over, at temperatures and pressures close to the normal- superfluid phase boundary. Population imbalance in cold at- oms plays essentially the same role as a Zeeman or exchange field in a superconductors since pairing is dependent on en- ergy measured from the Fermi energy for each species of fermion. In both cases the Fermi radius of the majority spe- cies exceeds the Fermi radius of the minority species and pairs at the Fermi energy necessarily have nonzero total mo- mentum. One of the most obvious signatures of superfluidity in fermionic cold-atom systems is the appearance of vortices and vortex lattices when the system is rotated 31. Indeed recent experiments 5 have observed vortex lattice struc- tures in fermionic cold-atom systems close to the BEC-BCS crossover region. For this reason an obvious potential signa- ture of an FFLO state is the appearance of the exotic vortex lattice structures they are expected to form 32–34. FFLO vortex lattices can be wildly different from the usual hexago- nal Abrikosov vortex lattice. The structure of the vortex lat- tice is determined mainly 32–34 by the Landau-level index of its condensed fermion pairs; the Abrikosov lattice forms when the Landau level index j =0, which is the closest ap- proximation to zero-total-momentum pairing allowed in a system that has come to equilibrium in a rotating frame. FFLO states in the absence of rotation can imply j  0 fer- mion pair condensation in rotated systems. Vortices have been observed in systems with population imbalance 6, but so far no unusual vortex structures have been observed. This could be due to the fact that these experiments realize the gapless Sarma phase 24, and another reason could be that the FFLO state is predicted by weak-coupling theory while all experiments are in the unitary limit. With this motivation, we report on a study of the polar- ization and interaction strength regime over which nonzero j pairing is expected in a rotating two-component fermion sys- tem. We consider only the BCS side of the Feshbach reso- nance on which FFLO physics occurs. We consider three- dimensional systems for the sake of definiteness, although two-dimensional systems could also be interesting experi- mentally. Working in the corotating reference frame, rotation is equivalent to an external magnetic field and a reduction in radial confinement strength. All our explicit calculations are for a uniform three-dimensional system and do not account for confinement. In typical experiments the atomic Landau- level splitting, equal to 2 where  is the rotation fre- quency, is much smaller than the Fermi energy. In this limit the Landau-level index of the condensate could be deter- mined by finding the optimal pairing wave vector on the BCS superfluid-normal phase boundary in the absence of PHYSICAL REVIEW A 74, 053602 2006 1050-2947/2006/745/05360210 ©2006 The American Physical Society 053602-1