1132 J. Med. Chem. 1995,38, zyxwvu 1132-1139 Mapping the Melatonin Receptor. 3. Design and Synthesis of Melatonin Agonists and Antagonists Derived from 2-Phenyltryptamines Peter J. Garratt," Rob Jones, and Derek A. Tocher Department zyxwvutsrqp of Chemistry, University College London, 20 Gordon Street, London WClH zyxwv OAJ, U.K. David Sugden Physiology Group, Biomedical Sciences Division, King's College London, Campden Hill Road, London W8 7AH, U.K. Received February 3, 1995@ Three series of 2-phenyltryptamides were prepared as melatonin analogues to investigate the nature of the binding site of the melatonin receptor in chicken brain and in Xenopus Zaeuis melanophore cells. The 5-methoxy-2-phenyltryptamides (6a-j) have high binding affinities for the chicken brain receptor, in some cases (6a-d) greater than that for melatonin, confirming and extending the work of Spadoni et aZ., and act as agonists in the Xenopus melanophore assay. Analogues lacking the 5-methoxyl group (2a-n) had a considerably lower affinity for the chicken brain receptor. In the Xenopus melanophore assay the compounds acylated on nitrogen by an alkyl group (2a-d) were agonists whereas the compounds acylated on nitrogen by an alicyclic group (2f-i) were antagonists. Introducing a methyl group at N' zyx (7) led to an increase in binding affinity in the chicken brain assay, whereas introducing an ethyl group (13) led to a decrease in binding affinity. A methyl substituent at the ,&position of the 3-amidoethane side chain (8, zyxwvut 11) also led to an increase in the binding affinity. The only analogue acylated on nitrogen with an alkyl group (acetyl) which showed antagonist activity was 9, which has a P-methoxymethyl side chain. In the absence of the 5-methoxyl group the methoxymethyl function may cause the molecule to bind in a different configuration so that it is no longer able to activate the receptor. All of these observations are in agreement with a model of melatonin at the receptor site in which the 3-amidoethane side chain is in a conformation close to the 5-methoxyl group. Introduction The pineal hormone melatonin (la), first isolated by Lerner et al.' in 1958 from bovine pineal tissue, has a central role in the regulation of daily rhythms and seasonal cycles in vertebrates. Quay2 and Wurtman et aL3 showed that the pineal gland produces and releases melatonin during the hours of darkness, and Hoffman and Reitel.4 found that the endocrine functions of the pineal are governed by changes in the duration of the photoperiod. Many types of seasonal behavior, such as reprod~ction~-~ and the accumulation of fat reserves in autumn,8 in photoperiodic mammals appear to be controlled by the pineal gland. Melatonin has found practical use to induce seasonally breeding animals, such as sheep, to breed out of seas~n.~,~ In humans it has been suggested that melatonin might have a variety of clinical uses, for example in jet-lag and shift work disturbances,1° and for circadian rhythm control in the blind. Melatonin has hypnotic properties in animals and humans, and it has been reported to have an oncostatic action and modulate the immune resp0nse.l' Despite the practical and clinical interest in melato- nin, very little is known about its mode of action or of the way in which it interacts with its receptor. High- affinity melatonin binding sites have been identified in central and, more recently, in peripheral tissues,12 and very recently the receptor in has been cloned from Xenopus melanophores, sheep, and humans.13 Advan- tage can now be taken of these advances in our knowledge of the receptor characteristics and structure @ Abstract published in Advance ACS Abstracts, March 1, 1995. 0022-2623/95/1838-1132$09.00/0 to attempt to model the binding of melatonin to its receptor and to use this model to design melatonin agonists and antagonists. rJN HCo Me la R=H lb R=l From experiments with melatonin and tryptamine derivatives on melanophores in patches of frog skin (Ranapipiens), Heward and Hadley14proposed that the melatonin molecule could be divided into three re- gions: the indole nucleus, which acts as a spacer, probably with some van der Waals binding affinity, the C-3 amidoethane side chain acting as the major binding site, and the 5-methoxyl group as the biological trigger for the response. Following this model, a number of compounds with improved potency have been ob- tained15J6 and some compounds with antagonistic prop- erties have been developed.lZa It had also been found that 2-iodomelatonin (lb) binds to the receptor more strongly than melatonin itself, and this finding has allowed [2-1251]-lb to be used in binding studies. Varia- tions of the acyl group on the C-3 side chain had also shown that acetyl was not the optimum group and that replacement by propanoyl or butanoyl effected an improvement in binding As little is known 0 1995 American Chemical Society