APPLICATIONES MATHEMATICAE 40,4 (2013), pp. 447–457 Małgorzata Grabińska, Paweł Błażej and Paweł Mackiewicz (Wroclaw) TWO ALGORITHMS BASED ON MARKOV CHAINS AND THEIR APPLICATION TO RECOGNITION OF PROTEIN CODING GENES IN PROKARYOTIC GENOMES Abstract. Methods based on the theory of Markov chains are most com- monly used in the recognition of protein coding sequences. However, they require big learning sets to fill up all elements in transition probability ma- trices describing dependence between nucleotides in the analyzed sequences. Moreover, gene prediction is strongly influenced by the nucleotide bias mea- sured by e.g. G+C content. In this paper we compare two methods: (i) the classical GeneMark algorithm, which uses a three-periodic non-homogeneous Markov chain, and (ii) an algorithm called PMC that considers six indepen- dent homogeneous Markov chains to describe transition between nucleotides separately for each of three codon positions in two DNA strands. We have tested the efficiency (in terms of true positive rate) of these two Markov chain methods for the model bacterial genome of Escherichia coli depending on the size of the learning set, uncertainty of ORFs’ function annotation, and model order of these algorithms. We have also applied the methods with dif- ferent model orders for 163 prokaryotic genomes that covered a wide range of G+C content. The PMC algorithm of different chain orders turns out to be more stable in comparison to the GeneMark algorithm. The PMC also outperforms the GM algorithm giving a higher fraction of coding sequences in the tested set of annotated genes. Moreover, it requires much smaller learning sets than GM to work properly. 1. Introduction. DNA contains genetic information about coded pro- teins and consists of two strands, which are long chains of four simpler units 2010 Mathematics Subject Classification : Primary 92D20, 60J10; Secondary 60J20, 62P10. Key words and phrases : Markov chains, DNA sequence, recognition of protein coding sequences, open reading frame. DOI: 10.4064/am40-4-5 [447] c  Instytut Matematyczny PAN, 2013