arXiv:1208.5003v2 [cs.LG] 27 Aug 2012 1 Identification of Probabilities of Languages Paul M.B. Vit´ anyi and Nick Chater Abstract We consider the problem of inferring the probability distribution associated with a language, given data consisting of an infinite sequence of elements of the languge. We do this under two assumptions on the algorithms concerned: (i) like a real-life algorothm it has round-off errors, and (ii) it has no round- off errors. Assuming (i) we (a) consider a probability mass function of the elements of the language if the data are drawn independent identically distributed (i.i.d.), provided the probability mass function is computable and has a finite expectation. We give an effective procedure to almost surely identify in the limit the target probability mass function using the Strong Law of Large Numbers. Second (b) we treat the case of possibly incomputable probabilistic mass functions in the above setting. In this case we can only pointswize converge to the target probability mass function almost surely. Third (c) we consider the case where the data are dependent assuming they are typical for at least one computable measure and the language is finite. There is an effective procedure to identify by infinite recurrence a nonempty subset of the computable measures according to which the data is typical. Here we use the theory of Kolmogorov complexity. Assuming (ii) we obtain the weaker result for (a) that the target distribution is identified by infinite recurrence almost surely; (b) stays the same as under assumption (i). We consider the associated predictions. I. I NTRODUCTION In cognition and science one learns by observation. The perceptual system of an individual person, or the data-gathering resources of a scientific community, incrementally gathers empirical data, and attempts to find the structure in that data. The question arises: under what conditions is it possible precisely to infer the structure underlying those observations? Or, relatedly, under what conditions could a machine learning algorithm potentially precisely recover this structure? We can model this problem as having Vit´ anyi is with the national research institute for mathematics and computer science in the Netherlands (CWI) and the University of Amsterdam. Address: CWI, Science Park 123, 1098 XG, Amsterdam, The Netherlands. Email: paulv@cwi.nl. Chater is with the Behavioural Science Group. Address: Warwick Business School, University of Warwick, Coventry, CV4 7AL, UK. Email: Nick.Chater@wbs.ac.uk. Chater was supported by ERC Advanced Grant “Cognitive and Social Foundations of Rationality.” DRAFT