version
history

HOW TO CITE
THIS ENTRY

Stanford Encyclopedia of Philosophy

A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

This document uses HTML 4/Unicode to display special characters. Older browsers and/or operating systems may not display these characters correctly.
last substantive
content change

MAY
6
2004

Computability and Complexity

A mathematical problem is computable if it can be solved in principle by a computing device. Some common synonyms for "computable" are "solvable", "decidable", and "recursive". Hilbert believed that all mathematical problems were solvable, but in the 1930's Gödel, Turing, and Church showed that this is not the case. There is an extensive study and classification of which mathematical problems are computable and which are not. In addition, there is an extensive classification of computable problems into computational complexity classes according to how much computation -- as a function of the size of the problem instance -- is needed to answer that instance. It is striking how clearly, elegantly, and precisely these classifications have been drawn.

What can be computed in principle? Introduction and History

In the 1930's, well before there were computers, various mathematicians from around the world invented precise, independent definitions of what it means to be computable. Alonzo Church defined the Lambda calculus, Kurt Gödel defined Recursive functions, Stephen Kleene defined Formal systems, Markov defined what became known as Markov algorithms, Emil Post and Alan Turing defined abstract machines now known as Post machines and Turing machines.

Surprisingly, all of these models are exactly equivalent: anything computable in the lambda calculus is computable by a Turing machine and similarly for any other pairs of the above computational systems. After this was proved, Church expressed the belief that the intuitive notion of "computable in principle" is identical to the above precise notions. This belief, now called the "Church-Turing Thesis", is uniformly accepted by mathematicians.

Part of the impetus for the drive to codify what is computable came from the mathematician David Hilbert. Hilbert believed that all of mathematics could be precisely axiomatized. He felt that once this was done, there would be an "effective procedure", i.e., an algorithm that would take as input any precise mathematical statement, and, after a finite number of steps, decide whether the statement was true or false. Hilbert was asking for what would now be called a decision procedure for all of mathematics.

As a special case of this decision problem, Hilbert considered the validity problem for first-order logic. First-order logic is a mathematical language in which most mathematical statements can be formulated. Every statement in first-order logic has a precise meaning in every appropriate logical structure, i.e., it is true or false in each such structure. Those statements that are true in every appropriate structure are called valid. Those statements that are true in some structure are called satisfiable. Notice that a formula, ϕ, is valid iff its negation, ¬ϕ, is not satisfiable.

Hilbert called the validity problem for first-order logic, the entscheidungsproblem. In a textbook, Principles of Mathematical Logic by Hilbert and Ackermann, the authors wrote, "The Entscheidungsproblem is solved when we know a procedure that allows for any given logical expression to decide by finitely many operations its validity or satisfiability. … The entscheidungsproblem must be considered the main problem of mathematical logic."

In his 1930 Ph.D. thesis, Gödel presented a complete axiomatization of first-order logic, based on the Principia Mathematica by Whitehead and Russell. Gödel proved his Completeness Theorem, namely that a formula is provable from the axioms if and only if it is valid. Gödel's Completeness theorem was a step towards the resolution of Hilbert's entscheidungsproblem.

In particular, since the axioms are easily recognizable, and rules of inference very simple, there is a mechanical procedure that can list out all proofs. Note that each line in a proof is either an axiom, or follows from previous lines by one of the simple rules. For any given string of characters, we can tell if it is a proof. Thus we can systematically list all strings of characters of length 1, 2, 3, and so on, and check whether each of these is a proof. If so, then we can add the proof's last line to our list of theorems. In this way, we can list out all theorems, i.e., exactly all the valid formulas of first-order logic, can be listed out by a simple mechanical procedure. More precisely, the set of valid formulas is the range of a computable function. In modern terminology we say that the set of valid formulas of first-order logic is recursively enumerable (r.e.). Gödel's Completeness theorem was not sufficient, however, to give a positive solution to the entscheidungsproblem. Given a formula, ϕ, if ϕ is valid then the above procedure would eventually list it out and thus could answer, "Yes, ϕ is valid." However, if ϕ were not valid then we might never find this fact out. What was missing was a procedure to list out all the non-valid formulas, or equivalently to list out all satisfiable formulas. A year later, in 1931, Gödel shocked the mathematical world by proving his Incompleteness Theorem: there is no complete and computable axiomatization of the first-order theory of the natural numbers. That is, there is no reasonable list of axioms from which we can prove exactly all true statements of number theory.

A few years later, Church and Turing independently proved that the entscheidungsproblem is unsolvable. Church did this by using the methods of Gödel's Incompleteness Theorem to show that the set of satisfiable formulas of first-order logic are not r.e., i.e., they cannot be systematically listed out by a function computable by the lambda calculus. Turing introduced his machines and proved many interesting theorems some of which we will discuss in the next section. In particular, he proved the unsolvability of the halting problem. He obtained the unsolvability of the entscheidungsproblem as a corollary.

Hilbert was very disappointed because his program towards a decision procedure for all of mathematics was proved impossible. However, as we will see in more detail in the rest of this article, a vast amount was learned about the fundamental nature of computation.

Turing Machines

In his 1936 paper, "On Computable Numbers, with an Application to the Entscheidungsproblem", Alan Turing introduced his machines and established their basic properties. He thought clearly and abstractly about what it would mean for a machine to perform a computational task. Turing defined his machines to consist of the following: The linear nature of its memory tape, as opposed to random access memory, is a limitation on computation speed but not power: a Turing machine can find any memory location, i.e., tape cell, but this may be time consuming because it has to move its head step by step along its tape. The beauty of Turing machines is that the model is extremely simple, yet nonetheless, extremely powerful. A Turing machine has potentially infinite work space so that it can process arbitrarily large inputs, e.g., multiply two huge numbers, but it can only read or write a bounded amount of information, i.e., one symbol, per step. Even before Turing machines and all the other mathematical models of computation were proved equivalent, and before any statement of the Church-Turing thesis, Turing argued convincingly that his machines were as powerful as any possible computing device.

Universal Machines

Each Turing machine can be uniquely described by its transition table: for each state, q, and each symbol, σ, δ(q) is the new state, the new symbol, and the head displacement. These transition tables, can be written as a finite string of symbols, giving the complete set of instructions of each Turing machine. Furthermore, these strings of symbols can be listed in lexicographic order as follows: M1, M2, M3, …, where Mi is the transition table, i.e., the complete set of instructions, for Turing machine number i. The transition table for Mi is the program for Turing machine i, or more simply, the ith program.

Turing showed that he could build a Turing machine, U, that was universal, in the sense that it could run the program of any other Turing machine. More explicitly, for any i, and any input w, U on inputs i and w would do exactly what Mi would do on input w, in symbols,

U(i,w)   =   Mi(w) .

Turing's construction of a universal machine gives the most fundamental insight into computation: one machine can run any program whatsoever. No matter what computational tasks we may need to perform in the future, a single machine can perform them all. This is the insight that makes it feasible to build and sell computers. One computer can run any program. We don't need to buy a new computer every time we have a new problem to solve. Of course, in the age of personal computers, this fact is such a basic assumption that it may be difficult to step back and appreciate it.

The Halting Problem

Because they were designed to embody all possible computations, Turing machines have an inescapable flaw: some Turing machines on certain inputs never halt. Some Turing machines do not halt for silly reasons, for example, we can mis-program a Turing machine so that it gets into a tight loop, for example, in state 17 looking at a 1 it might go to state 17, write a 1 and displace its head by 0. Slightly less silly, we can reach a blank symbol, having only blank symbols to the right, and yet keep staying in the same state, moving one step to the right, and looking for a "1". Both of those cases of non-halting could be easily detected and repaired by a decent compiler. However, consider the Turing machine MF, which on input "0", systematically searches for the first counter-example to Fermat's last theorem, and upon finding it outputs the counter-example and halts. Until Andrew Wiles relatively recently proved Fermat's Last Theorem, all the mathematicians in the world, working for over three centuries, were unable to decide whether or not MF on input "0" eventually halts. Now we know that it never does.

Computable Functions and Enumerability

Since a Turing machine might not halt on certain inputs, we have to be careful in how we define functions computable by Turing machines. Let the natural numbers, N, be the set {0,1,2,…} and let us consider Turing machines as partial functions from N to N.

Let M be a Turing machine and n a natural number. We say that M's tape contains the number n, if M's tape begins with a binary representation of the number n (with no unnecessary leading 0's) followed by just blank symbols from there on.

If we start the Turing machine M on a tape containing n and it eventually halts with its tape containing m, then we say that M on input n, computes m: M(n) = m. If, when we start M on input n, it either never halts, or when it halts, its tape does not contain a natural number, e.g., because it has leading 0's, or digits interspersed with blank symbols, then we say that M(n) is undefined, in symbols: M(n) = ↗. We can thus associate with each Turing machine, M, a partial function, M: N N {↗}. We say that the function M is total if for all nN, M(n) ∈ N, i.e., M(n) is always defined.

Now we can formally define what it means for a set to be recursively enumerable (r.e.) which we earlier described informally. Let S N. Then S is r.e. if and only if there is some Turing machine, M, such that S is the image of the function computed by M, in symbols,

S   =  {M(n) | nN; M(n) ≠ } .

Thus, S is r.e. just if it can be listed out by some Turing machine. Suppose that S is r.e. and its elements are enumerated by Turing machine M as above. We can then describe another Turing machine, P, which, on input n, runs M in a round-robin fashion on all its possible inputs until eventually M outputs n. If this happens then P halts and outputs "1", i.e., P(n)=1. If n S, then M will never output n, so P(n) will never halt, i.e., P(n)=↗. Let the notation P(n)↓ mean that Turing machine P on input n eventually halts. For a Turing machine, P, define L(P), the set accepted by P, to be those numbers n such that P on input n eventually halts,

L(P)   =   {n | P(n)↓} .

The above argument shows that if a set S is r.e. then it is accepted by some Turing machine, P, i.e., S = L(P). The converse of this statement holds as well. That is, S is r.e. if and only if it is accepted by some Turing machine, P.

We say that a set, S, is decidable if and only if there is a total Turing machine, M, that decides for all n N whether or not n S. Think of "1" as "yes" and "0" as "no". For all n N, if n S, then M(n) = 1, i.e., M on input n eventually halts and outputs "yes", whereas if n N, then M(n) = 0, i.e., M on input n eventually halts and outputs "no". Synonyms for decidable are: computable, solvable, and recursive.

For S N, the complement of S is N - S, i.e., the set of all natural numbers not in S. We say that the set S is co-r.e. if and only if its complement is r.e. If a set, S, is r.e. and co-r.e. then we can list out all of its elements in one column and we can list out all of its non-elements in a second column. In this way we can decide whether or not a given element, n, is in S: just scan the two columns and wait for n to show up. If it shows up in the first column then n S. Otherwise it will show up in the second column and n S. In fact, a set is recursive iff it is r.e. and co-r.e.

The Unsolvability of the Halting Problem

Turing asked whether every set of natural numbers is decidable. It is easy to see that the answer is, "no", by the following counting argument. There are uncountably many subsets of N, but since there are only countably many Turing machines, there can be only countably many decidable sets. Thus almost all sets are undecidable.

Turing actually constructed a non-decidable set. As we will see, he did this using a diagonal argument. The diagonal argument goes back to Georg Cantor who used it to show that the real numbers are uncountable. Gödel used a similar diagonal argument in his proof of the Incompleteness Theorem in which he constructed a sentence, J, in number theory whose meaning could be understood to be, "J is not a theorem."

Turing constructed a diagonal halting set, K, as follows:

K   =   {n | Mn(n)↓} .

That is, K consists of those Turing machines that eventually halt when input their own programs.

It is not hard to see that K is r.e. Suppose for the sake of a contradiction that K is also co-r.e., and let d be the number of a Turing machine that accepts the complement of K. That is, for any n,

n K   ⇔   Md(n)↓ .

But consider what happens when we substitute d for n in the above equation:

d K   ⇔   Md(d)↓ .

However, the definition of K tells us that:

d K   ⇔   Md(d)↓ .

Thus we have that

d K   ⇔   d K,

which is a contradiction. Thus our assumption that K is co-r.e. is false. Thus K is not recursive. It follows that it is not a computable problem to be given a Turing machine and its input and to decide whether or not the Turing machine will eventually halt on that input, i.e., the halting problem is unsolvable.

Primitive Recursive Functions

We next define the class of Primitive Recursive Functions. This is a very interesting class of functions defined by Gödel as part of his proof of the Incompleteness Theorem. We are interested in functions f from Nr to N, for r = 0, 1, 2, … . Here r is called the arity of the function f, i.e., the number of arguments that it takes. Gödel started with three very simple functions, the initial functions, and two natural closure operations, composition and primitive recursion, each of which take some already defined functions and use them to define a new one. We next explain his definitions in detail. This section is technical and can be safely skipped. The important idea is that the primitive recursive functions comprise a very large and powerful class of computable functions, all generated in an extremely simple way.

We begin with the three initial primitive recursive functions:

Now consider the following two operations: