Magnetic phase diagram of the two-dimensional antiferromagnet Ni 5 (TeO 3 ) 4 Br 2 Matej Pregelj and Andrej Zorko Institute Jožef Stefan, Jamova 39, 1000 Ljubljana, Slovenia Oksana Zaharko Laboratory for Neutron Scattering, ETHZ and Paul Scherrer Institute, CH-5232 Villigen, Switzerland Rodolphe Bousier LCMI, CNRS, 25 rue des Martyrs, 38042 Grenoble, France Helmuth Berger Institute of Physics of Complex Matter, EPFL, 1015 Lausanne, Switzerland Hiroko A. Katori Magnetic Materials Laboratory, Wako Institute, RIKEN, 351–0198 Saitama, Japan Denis Arčon Institute Jožef Stefan, Jamova 39, 1000 Ljubljana, Slovenia and Faculty of Mathematics and Physics, University of Ljubljana, Jadranska 19, 1000 Ljubljana, Slovenia Received 23 June 2008; revised manuscript received 9 December 2008; published 9 February 2009 Phase diagram of the two-dimensional antiferromagnet Ni 5 TeO 3 4 Br 2 with triangular arrangement of Ni 2+ S =1magnetic moments within the Ni 5 O 17 Br 2 subunits has been investigated by temperature and magnetic field dependent heat-capacity, magnetization, and magnetic-torque measurements down to 1.5 K and up to 23 T. A nonzero magnetic contribution to the heat capacity observed up to 2.3T N is consistent with short-range magnetic ordering and the two-dimensional nature of the system. Below the Néel temperature T N =29 K several antiferromagnetic phases were identified. The zero-field phase is characterized by a planar antiferro- magnetic arrangement of the two in-layer neighboring Ni 5 O 17 Br 2 magnetic clusters within the magnetic unit cell. When the magnetic field is applied along the a crystal axis, a spin-flop-like transition to a phase with a complex out-of-plane arrangement of Ni 2+ S =1magnetic moments occurs at 10 T. Using a molecular-field approach we predict that this transition will shift to higher fields with increasing temperature and that a magnetic phase with ferromagnetic ordering of Ni 5 O 17 Br 2 magnetic clusters will occur above 24 T. We ascribe the richness of the magnetic phases to strongly exchange-coupled clusters, being the basic building blocks of the investigated layered system. DOI: 10.1103/PhysRevB.79.064407 PACS numbers: 75.50.Ee, 81.30.Bx I. INTRODUCTION Two-dimensional 2Dantiferromagnetic AFMsystems represent a fertile playground for experimental and theoreti- cal investigations of phase transitions and critical phenom- ena. In recent years, they were intensively studied in relation to high-temperature superconductivity in cuprates, 1 colossal magnetoresistance in manganites, 2 and various exotic phases in geometrically frustrated lattices, with a kagome lattice as an archetype. 3 Phase diagrams of these systems are very rich and sensitively depend on a particular spin arrangement, magnetic anisotropy, and external perturbations, such as ap- plied magnetic field or pressure. Lately, 2D AFM systems that are built of strongly exchange-coupled magnetic clusters instead of individual magnetic moments have become accessible. Their phase diagrams, influenced by the competition between intracluster and intercluster interactions, are expected to be even more complex. The recently discovered compound, 4 Ni 5 TeO 3 4 Br 2 , can be regarded as a novel representative of such systems. Here, five magnetic Ni 2+ S =1moments con- stitute the Ni 5 O 17 Br 2 basic building block in which they are arranged into two Ni 2+ triangles connected with a common central Ni site. 5 Ni 5 O 17 Br 2 entities are linked via eight cor- ners to four nearest neighbors, which then form a layered structure. The triangle-based structure of the Ni 5 O 17 Br 2 unit suggests a geometrical frustration if AFM interactions between nearest-neighbor Ni 2+ moments are assumed. These moments are situated in three different severely distorted oc- tahedral surroundings, implying strong magnetic anisotropy. 6 The dominant magnetic interactions between the Ni 2+ are indeed antiferromagnetic, as indicated by high-temperature susceptibility data, implying a negative Curie temperature -50 K. The system exhibits a long-range Néel ordered state below T N =29 K. 48 Consequently, the empirical frus- tration parameter, / T N = 1.7, is rather small. Neutron mag- netic diffraction and unusual temperature dependence of the antiferromagnetic resonance AFMRfrequency, 6,9 however, suggest a very complicated temperature dependence of the Ni 2+ sublattice magnetizations in the ordered phase. This re- flects the importance of competition between various mag- netic terms, possibly leading to intriguing magnetic struc- tures as a function of temperature and external magnetic field. PHYSICAL REVIEW B 79, 064407 2009 1098-0121/2009/796/0644077©2009 The American Physical Society 064407-1