A Model for Recursively Self Improving Programs Matt Mahoney Draft v3, Sept. 14, 2008, revised Dec. 17, 2010 mmahoney@cs.fit.edu, matmahoney@yahoo.com Abstract We formally define recursively self improving programs and show that such programs exist, but that their algorithmic complexity cannot grow faster than O(log n) given fixed goals. Background Vinge [1] predicted that if humans could produce artificial intelligence (AI) systems with greater than human intelligence, then so could those systems, only faster. The resulting positive feedback cycle would result in an intelligence explosion or singularity. The Singularity Institute for Artificial Intelligence [2] is investigating the threat of runaway AI, but the problem is poorly understood; lacking any precedent. Of central importance is an understanding of the limits of recursive self improvement (RSI). If RSI is possible, then it is critical that the initial goals of the first iteration of agents (seed AI) are friendly to humans and that the goals not drift through successive iterations, assuming we cannot control the goals of subsequent generations appearing in rapid succession. This is an extremely difficult problem in itself, because friendliness is not well defined, although attempts have been made [3]. On the other hand, if RSI is not possible, then we are possibly left with an evolutionary process in which self modifying and reproducing agents compete for computing resources with the most successful of these displacing the others. Any goal contrary to rapid reproduction would be unstable. The evolutionary process might be greatly accelerated because it might be based on software (such as self modifying internet worms) or self replicating nanotechnology (gray goo [4]) rather than on DNA. To talk about RSI, we must first define it. Generally, we wish to improve intelligence, which is itself hard to define. Turing proposed an imitation game [5] as a test for intelligence: the ability of a machine to fool a human into believing it was another human is sufficient (but not necessary) evidence of intelligence. However, the Turing test is not an appropriate test for RSI because it does not describe a test for superhuman intelligence. This seems to be a general problem. We can detect an IQ of 200 in children, but there is no equivalent test for adults. If Alice is smarter than Bob, then how is Bob to test Alice's intelligence? If Alice gives an answer that Bob believes is wrong, then who are we to believe? Legg and Hutter [6] surveyed a number of common definitions of intelligence and offered a formal definition of universal intelligence which does not depend on human characteristics. An agent's intelligence is defined as the expected reward in environments modeled by programs generated by coin flips in the AIXI model. In the AIXI model [7] an agent and environment are modeled as a pair of interacting Turing machines in which at each step the agent sends a symbol to the (unknown) environment and the environment sends a symbol plus a reward signal to the agent. The goal of the agent is to accumulate the greatest possible reward. A formal definition has the advantage of allowing mathematical analysis. For example, Hutter proved that the optimal strategy of the agent is to guess at each step that the environment is simulated by the shortest program consistent with all of the interaction observed so far. This would solve the general intelligence problem once and for all (for systems that fit the model), except for the fact that the strategy is not computable. Legg and Hutter's universal intelligence cannot be measured precisely because it would require testing over an infinite number of environments, although it could be approximated by random sampling.