LETTERS PUBLISHED ONLINE: 21 NOVEMBER 2010 | DOI: 10.1038/NPHYS1861 Two-dimensional surface state in the quantum limit of a topological insulator James G. Analytis 1,2 * , Ross D. McDonald 3 , Scott C. Riggs 4 , Jiun-Haw Chu 1,2 , G. S. Boebinger 4 and Ian R. Fisher 1,2 The topological insulator is a unique state of matter that possesses a metallic surface state of massless particles known as Dirac fermions, which have coupled spin and momentum quantum numbers. Owing to the preservation of time-reversal symmetry, this coupling protects the wavefunctions against disorder 1–3 . The experimental realization of this state of matter in Bi 2 Se 3 and Bi 2 Te 3 has sparked considerable interest owing both to their potential use in spintronic devices and in the investigation of the fundamental nature of topologically non- trivial quantum matter. However, the conductivity of these compounds tends to be dominated by the bulk of the material because of chemical imperfection, making the transport properties of the surface nearly impossible to measure. We have systematically reduced the number of bulk carriers in Bi 2 Se 3 to the point where a magnetic field can collapse them to their lowest Landau level. Beyond this field, known as the three-dimensional (3D) ‘quantum limit’, the signature of the 2D surface state can be seen. At still higher fields, we reach the 2D quantum limit of the surface Dirac fermions. In this limit we observe an altered phase of the oscillations, which is related to the peculiar nature of the Landau quantization of topological insulators at high field. Furthermore, we observe quantum oscillations corresponding to fractions of the Landau integers, suggesting that correlation effects can be observed in this new state of quantum matter. The linear dispersion and energy quantization in field of the Dirac surface state of the archetypal topological insulators Bi 2 Se 3 and Bi 2 Te 3 have been revealed by angle-resolved pho- toemission spectroscopy and scanning tunnelling spectroscopy respectively 4–9 . However, the transport properties of the surface states, which are arguably their most technologically useful as well as fundamentally interesting characteristics, have proved to be especially difficult to measure owing to the overwhelming effect of bulk conduction channels 10–14 . The materials challenge is to find a way to cleanly eliminate the bulk conductivity so that the properties of the surface can be observed. By partially substituting 15 Sb for Bi in Bi 2 Se 3 we systematically reduce the bulk carrier density to n ∼ 10 16 cm −3 . We study samples in which the Fermi surface of the Dirac fermion is small enough that at pulsed fields of up to 60 T we can access, for the first time, the 2D quantum limit (see Supplementary Information SA)—a regime where correlation effects are likely to arise 16–20 . In addition, we observe a distinguishing signature of topological insulators at high field whereby the phase of the oscillations is affected by the Zeeman energy. 1 Stanford Institute for Materials and Energy Sciences, SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, California 94025, USA, 2 Geballe Laboratory for Advanced Materials and Department of Applied Physics Stanford University, California 94305, USA, 3 Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA, 4 National High Magnetic Field Laboratory and Department of Physics,1800 E. Paul Dirac Drive, Florida State University, Tallahassee Florida 32310, USA. *e-mail: analytis@slac.stanford.edu. In Fig. 1a we illustrate the dependence of the resistivity on temperature for representative single crystals with bulk carrier densities ranging from ∼10 19 cm −3 down to ∼10 16 cm −3 . The carrier density was determined by the (low field) Hall effect, which matches the size of the Fermi surface measured by quantum oscillations, known as Shubnikov–de Haas oscillations (SdHO) (see Fig. 1b–d). These oscillations are periodic in inverse field and their period ϒ can be related to the extremal cross-sectional area A k of the Fermi surface in momentum space through the Onsager relation 1/ϒ = ( ¯ h/2π e )A k . When the field is rotated away from the crystal trigonal axis, the frequency of the SdHO changes according to the morphology of the Fermi surface. In every one of our samples, the bulk Fermi surface is a closed 3D ellipsoid 21–23 . The effective mass m ∗ ∼ 0.12m e varies weakly with doping for these carrier densities, as previously reported 21,22 . For our lowest carrier density sample, the pocket is expected to have an orbitally averaged Fermi wavenumber of k F = 0.0095 Å −1 . This is small enough that we are able to exceed the bulk (3D) quantum limit with moderate fields ∼4 T. This allows us to directly measure all of the relevant transport properties of the bulk, considerably simplifying our analysis, because there is no need to rely on surface-sensitive probes, which are inadequate for determining the bulk Fermi energy 23 . The remainder of this study is concerned with the 2D properties of the lowest carrier density samples at high fields. In Fig. 2a,b we illustrate the longitudinal and transverse (Hall) resistances, R xx and R xy respectively, taken at 1.5 K on sample 1 with n ∼ 4 × 10 16 cm −3 . Strong features appear in R xy and R xx at similar fields in the bulk ultra-quantum limit 24 . To investigate the dimensionality of these features, we rotate the crystal in the field. For the 2D surface state of a topological insulator, quantum oscillatory phenomena depend only on the perpendicular component of the field B ⊥ and are periodic in 1/B. In Fig. 2c we show the background-subtracted signal, illustrating that the oscillatory features grow with increasing field, as one might expect of SdHOs (for details of the background subtraction for all samples 1–3, see Supplementary Information SC). In Fig. 2d,e we have plotted the dR xy /dB and −d 2 R xx /dB 2 as a function of B ⊥ = B cos θ —the usual procedure to determine whether minima in R xx (equivalent to minima in the negative second derivative) fall on top of the Hall plateaux (appearing as minima in the first derivative). Pronounced dips that are periodic in 1/B in both the Hall derivative and the magnetoresistance align at all angles, providing unambiguous evidence that the plateau-like features in the Hall and minima in the SdHOs originate from a 2D metallic state. 960 NATURE PHYSICS | VOL 6 | DECEMBER 2010 | www.nature.com/naturephysics © 2010 Macmillan Publishers Limited. All rights reserved.