The last data structure we'll look at is called a `heap'. One has to watch out for this term because it has at least two very different meanings even within computer science. We're talking here about the kind of heap that is a kind of binary tree (but not a binary search tree). It is well suited to keeping track of a `list' of prioritized items. As such, it can also form the basis of an efficient sorting algorithm. We'll look first at what a heap is, then how it can be used for sroting, and then we'll analyze the complexity of HeapSort.
A heap is a complete binary tre in which every node has the `heap property'. What is a complete binary tree? [wait] Yes, every level is full, except possibly for the lowest level, which is filled from left to right. So, let's look at a complete binary tree:
w
/ \
/ \
h b
/
g
Levels 0 and 1 are filled, and level 2, though not filled, is complete
from left to right. So, this is a complete binary tree. Notice that
the keys are not ordered as they would be for a binary search tree.
Let's look at a binary tree that is not complete.
w
/ \
/ \
h b
\
g
It is not complete because the lowest level has a gap in it.
[Draw a few other examples.]
Now let's talk about the `heap property'. You've gotten used to the idea of a property of a node - this is a new kind of property. THe property is that the key of a node must be greater than the keys of its immediate children. If this property is true at every node of a complete binary tree, then that tree is a heap. So, is the following a heap?
w
/ \
/ \
h b
/
g
[wait] Yes, for two reasons. First, it is a complete binary tree,
and second, every node has the heap property. Is the following
a heap?
w
/ \
/ \
h b
\
g
[wait] Right, it is not a heap because it is not a complete binary tree.
How about the following?
w
/ \
/ \
g b
/
h
[wait] Right, it is not a heap because one of the nodes (at `g') does
not have the heap property.
It turns out that there is a very nice way to represent a complete binary tree in an array. We need a method for mapping each element on the complete binary tree to a unique element of the array. It is because of our desire to use an array representation that we insist on the completeness of the binary tree. Let's look at a scheme. Let's draw a heap and a corresponding structure that shows the desired array index of each element, and the array.
w 0 0: w
/ \ / \ 1: h
/ \ / \ 2: b
h b 1 2 3: g
/ \ / \ 4: e
g e 3 4
Thus, one can represent a complete binary tree as an array, and
one can treat an array as a complete binary tree. We know how
to move from element to another in an array. What is the corresponding
idea for moving to a left child, or a right child? That is, given
the index of a node, how do we compute the index of its left child? its
right child? [wait] Yes, given index `i', `left' is at 2*i+1, and `right'
is at 2*i+2. Do you see a relationship between the indeces of two siblings?
[wait] Yep, a node that is a left child with index i would find its
right sibling at index i+1. Similarly, a right child with index k would
find its left sibling at index k-1. So, we have the ability to compute
various relative nodes in the tree from the array index of one.
One more, how does one compute the index of a node's parent? [wait]
Yes, (i-1)/2.
Now you know what a heap is. It is an excellent structure for locating the maximum of a list in one step. The largest key must be at the root. Where is the second largest key? [wait] Well, it's one level lower, but we don't know whether it is the left child or the right child.
Let's examine how the HeapSort algorithm makes use of a heap to sort the keys of a list. Here is HeapSort expressed at a high level.
1. Arrange the unordered keys as a heap. 2. While heap not empty a. Swap root with last key b. Reduce size of heap by one. c. Reheapify the remaining nodes in the heap.The idea here is to swap the root with the last key, putting the largest key at the end of the heap. This makes it a non-heap for a moment, but then the size of the heap is reduced by 1, which takes what was the largest key out of the heap. Now the smaller heap is reheapified (what we just put at the root may not be the largest key), and then we do it again. This is a really nice idea, and it is all done in one array.
Let's see how HeapSort would work on our example heap. It is already a heap, so we'll skip step 1 for the moment. Let's start with step 2.
w 0: w
/ \ 1: h
/ \ 2: b
h b 3: g
/ \ 4: e
g e
First, we swap root with last, giving:
e 0: e
/ \ 1: h
/ \ 2: b
h b 3: g
/ \ 4: w
g w
Now we reduce the size of the heap, effectively taking element 4
out of the heap:
e 0: e
/ \ 1: h
/ \ 2: b
h b 3: g
/ ----
g 4: w
Now we reheapify. How do we do that? [wait] Good, I see you've been
doing your reading. One starts at the root, and repeatedly swaps the
key with the larger of the child keys. So, this will first swap
`e' and `h', giving:
h 0: h
/ \ 1: e
/ \ 2: b
e b 3: g
/ ----
g 4: w
Then, it will swap `e' and `g', giving:
h 0: h
/ \ 1: g
/ \ 2: b
g b 3: e
/ ----
e 4: w
Now we do step 2 again. Tell me what to write [wait] Yes, we swap `h' and `e', producing
e 0: e
/ \ 1: g
/ \ 2: b
g b 3: h
/ ----
h 4: w
Now we reduce the size of the heap by 1, taking `h' out of the
heap.
e 0: e
/ \ 1: g
/ \ 2: b
g b ----
3: h
4: w
Again, we reheapify, in this case swapping `e' and `g'
g 0: g
/ \ 1: e
/ \ 2: b
e b ----
3: h
4: w
Now we do step 2 again, swapping what? [wait] Yes, we swap `g' and `b', producing
b 0: b
/ \ 1: e
/ \ 2: g
e g ----
3: h
4: w
Now we reduce the size of the heap by 1, taking `g' out of the
heap.
b 0: b
/ 1: e
/ ----
e 2: g
3: h
4: w
Again, we reheapify, in this case swapping `b' and `e'
e 0: e
/ 1: b
/ ----
b 2: g
3: h
4: w
You can see that the array holds a heap that is shrinking and
a sorted array that is growing. Both of these logical structures
are stored in the single array. Let's finish this off.
Now we do step 2 again, swapping `e' and `b', producing
b 0: b
/ 1: e
/ ----
e 2: g
3: h
4: w
Now we reduce the size of the heap by 1, taking `e' out of the
heap.
b 0: b
----
1: e
2: g
3: h
4: w
There is no need to do any more heap operations. One cannot operate
on a heap of size 1. The steps are still weel defined though. We
could think of it as swapping `b' with itself, then removing from heap.
Either way, the heap has dwindled to nothing, and the resulting list
in the array is sorted.
----
0: b
1: e
2: g
3: h
4: w
Now let's look at how to do step 1 of HeapSort. Specifically, how can we convert the initial array of keys into a heap? Let's imagine that we start with a terrible ordering with respect to a heap. For example, let's imagine that the array is initially already sorted. Let's draw our keys in heap showing this case.
b
/ \
/ \
e g
/ \
h w
One can see that the heap property does not hold at any node. How shall
we convert this to a heap? [wait] This is little tricky to see. Recall
that we swapped an element into the root position, it took just
one reheapify operation to restore the heap property at the root. This
was because the left and right subtrees were already heaps. Does this
give you an idea? [wait] Well, we can start from the leaves, and reheapify
each subtree as we go `up'. Let's try it.
We'll start at the last element, `w'. Well a leaf is already a heap, so there is nothing to do. Indeed, we can skip quickly to the `e' subtree, and reheapify it. What do we get? [wait] Yes, the `w' and `e' are exchanged, producing
b
/ \
/ \
w g
/ \
h e
Now we move back to the next subtree, in this case at `b', and reheapify. What happens? [wait] Yes, the `b' and `w' swap, giving
w
/ \
/ \
b g
/ \
h e
and then the `b' and `h' swap, giving
w
/ \
/ \
h g
/ \
b e
Is this a heap? Yes.
How could we code this step #1, assuming that we had already written a reheapify operation? [wait] Yes, since the heap is stored in an array, it is a simple matter to start at the end, and reheapify at each node (element) until the root is reached. Actually, since we can skip the leaves, where would be a better place (than the end) to start the loop? [wait] Yes, at the first non-leaf, which for a heap of n elements is at what indexc? [wait] Yes, it will be at element (n-2)/2.
Now that we know how to code the algorithm, let's analyze its worst case running time, in terms of key comparisons, as a function of n elements to sort. There are two steps. One is to establish the initial heap. The other is to step 2 repeatedly. I'm curious whether you have an intuition, before we do the analysis? [wait] Well, let's think first about a reheapify operation. It will approximately log2(n) swaps, and for each swap it does two key comparisons, so it will do O(logn) comparisons. We do at most n reheapifies, so one can guess that step 1 is O(nlogn). (We'll see in a moment that it is less complex than this). How about step #2? The swap does not require any key comparisons, nor does reducing the size of the heap. So, there are approximately n reheapifies, at a cost of approximately 2log2(n) comparisons each, so step 2 is also O(nlogn). Two O(nlogn) steps makes an overall cost of O(nlogn). Why? [wait]. So, we've got a rough estimate.
Now let's do some careful analysis and see how it compares to our rough version. Let's do the analysis in terms of number of swaps in which keys were compared to decide exchange. We have already noted that two comparisons are required for each swap. Since these differ by a constant factor, we can do our analyses in terms of number of swaps instead, which is somewhat more convenient. Let's compute the function in tabular form. How many swaps are needed to heapify three elements?
x
/ \
/ \
y z
[wait] Yes, one swap. So, we have a point for our function:
n swaps(n) ---- -------- 3 1
How about when n is 7? [wait] Well, we know that the two subtrees of three require one each, and then we know that two more will be required to reheapify from the root, giving
n swaps(n) ---- -------- 3 1 7 1+1+2=4
How about when n is 15? [wait] Good, it takes 4 each for the subtrees of 7, plus three more to reheapify from the root, giving
n swaps(n) ---- ----------- 3 1 7 1+1+2= 4 15 4+4+3= 11How about 31? [wait] Right,
n swaps(n) ---- ----------- 3 1 7 1+1+2= 4 15 4+4+3= 11 31 11+11+4= 26When n is 31? [wait]
n swaps(n) ---- ----------- 3 1 7 1+1+2= 4 15 4+4+3= 11 31 11+11+4= 26 63 26+26+5= 57So, what do you conclude about the cost of step 1? [wait] Yes, it is actually O(n).
Let's do a similar kind of analysis for step 2. How many swaps (during reheapify) for a heap of size 2? [wait] Yes, 0, because after we remove an element (step 2a), there is one element remaining. For 3? 1, for 4? 2, for 5? 4 for 6? 6 for 7? 8, for 8? 10. One can see that this is growing more quickly than linear, but more slowly than quadratic. So, the intution of nlogn is correct in this case. Overall, the O(n) of step 1 and the O(nlogn) of step 2 make O(nlogn) overall. This sort has execellent worst case properties.
That's it for the course material. Our remaining time will be devoted to review, answering questions, and course evaluation. In addition to the official evaluation, there is an unofficial one that provides additional useful information.