Download Induction, Loop Invariants, and List Sorting Algorithms and more Study notes Algorithms and Programming in PDF only on Docsity!
CS231 Algorithms Handout # 10’ Prof. Lyn Turbak October 11, 2001 Wellesley College
Induction, Loop Invariants, and List Sorting
This is a revised version of the October 5 handout #10 that fixes several bugs and fleshes out many missing details.
Induction
We want to be able to prove that an algorithm is correct – i.e., that it satisfies the specification of a solution for the problem being solved.
The divide/conquer/glue problem solving strategy naturally leads to recursive algorithms. How does one prove a recursive algorithm is correct?
Induction is a proof methodology for showing that recursive algorithms are correct. An inductive proof has the following structure:
- Show that the base case(s) of the recursive algorithm are correct.
- Assuming that the algorithm works correctly for all inputs of size < n (this is called the inductive hypothesis (IH)), show that the algorithm is correct for every input of size n.
The principle of induction says that (1) and (2) imply that the recursive algorithm is correct for all inputs.
Note that induction formally justifies the “wishful thinking” approach to writing recursive programs that you’ve seen in CS111/CS230.
Induction Example: Factorial
Here is the standard recursive definition of factorial, written in the language Haskell:
fact n = if n == 0 then 1 else n * fact(n-1)
Specification To prove that fact is correct, we must first have a formal specification for what it is supposed to compute. In the case of factorial, we want fact(n) to calculate:
n! =
∏^ n
k=
k = 1 · 2 · 3 ·... (n − 1) · n
Note that 0! =
k=1 k, which is defined to be 1.
Inductive Proof of the Correctness of fact
Base Case (n = 0): fact(0) returns 1 =
k=1 k^ = 0! Inductive Case (n > 0): n ∗ f act(n − 1) is returned by fact(n) = n ·
∏n− 1 k=1 by IH =
∏n k=1 =^ n!^ by algebra
List Notation
Here is box-and-pointer notation for a list:
Here is a tree (cons/nil) notation for the same list:
cons cons cons nil
Here is Haskell’s notation for the same list:
7 : (2 : (4 : []))
Here is Haskell’s abbreviated notation for the same list:
[7,2,4]
Textual List Function Notation
The same insert example from above can be written in Haskell using pattern matching syntax:
insert x [] = [x]
insert x (y:ys) = if x <= y then x:(y:ys) else y:(insert x ys)
Haskell also allows the second definition clause of insert to be written in guard notation as
insert x (y:ys) | x <= y = x:(y:ys) | x > y = y:(insert x ys)
or as
insert x (y:ys) | x <= y = x:(y:ys) | otherwise = y:(insert x ys)
Program Algebra
We can perform algebra on Haskell programs (e.g., substitute equals for equals).
insert 8 [3, 5, 9] ⇒ insert 8 (3 : (5 : (9 : []))) ⇒ 3 : (insert 8 (5 : (9 : []))) ⇒ 3 : (5 : (insert 8 (9 : []))) ⇒ 3 : (5 : (8 : (9 : []))) ⇒ [3, 5, 8 , 9]
This sort of reasoning doesn’t work in the presence of side effects (e.g. assigning to a variable, updating a data structure). Unlike Java, C, etc., Haskell does not support side effects.
Sortedness Specification
A list of numbers xs = [x 1 , x 2 ,... , xk] is sorted, written sorted (xs), if xi ≤ xi+1 for all i ∈ [1 .. k − 1]. Notes:
- The notation [x .. y] denotes the set of integers from x up to and including y. If y < x, then [x .. y] denotes the empty set. E.g.:
[3 .. 5] = { 3 , 4 , 5 } [3 .. 3] = { 3 } [3 .. 2] = {}
- The specification for sorted does need a special case for empty or singleton lists. Why not?
- Although we have defined sortedness for a list of numbers, everything we say here generalizes to any lists whose elements are “orderable”, but we want to keep things simple here.
Bags (Multisets)
A bag (a.k.a. multiset) is an unordered collection of elements that may contain multiple oc- currence of the same element. In contrast, a set is an unordered collection of elements without duplicates.
We shall use the notation {|... |} to denote a bag, where... is an enumeration of the elements in the bag. So {| 8 , 4 , 5 , 8 , 5 , 5 |} denotes a bag with one 4, three 5s, and two 8s. Since element order doesn’t matter, the same bag could be written {| 5 , 8 , 5 , 4 , 8 , 5 |} or {| 4 , 5 , 5 , 5 , 8 , 8 |}.
Bag Operations:
- bagEmpty : the empty bag
- x ∈ b : bag membership, true iff there is at least one occurrence of x in b.
- b 1 =bag b 2 : bag equality, true iff bags b 1 and b 2 have exactly the same elements (including occurrence information).
- b 1 ∪bag b 2 : the result of unioning all elements of bags b 1 and b 2.
- bagIns(x, b) : the result of inserting one occurrence of element x into bag b.
- bagDel (x, b) : the result of removing one occurrence of element x from bag b. (Does nothing if there is no occurence of x in b.)
- elts(xs) : the bag of elements from the list xs. Note that elts(xs) could be written in Haskell syntax as:
elts [] = emptyBag elts (x:xs) = bagIns(x, elts(xs))
Correctness Proof for insert: Inductive Case
There are two cases here, depending on whether x <= y or x > y.
Case 1: x <= y.
Here, a = x, as = (y:ys), and bs = x:(y:ys).
(insert1) Assuming that as = (y:ys) is sorted, we must show sorted (x:(y:ys)). Since y:ys is sorted, we only need show that x ≤ y. But this is true, by the precondition for Case 1. (insert2) We must show elts(x:(y:ys)) =bag bagIns(x, elts(y:ys)). It is easy to see this is true, since both sides denote bags containing x, y, and the elements of ys. It can also be shown formally by unwinding the definition of elts on x:(y:ys).
Case 2: x > y.
Here, a = x, as = (y:ys), and bs = y:(insert x ys).
(insert1) Assuming that as = (y:ys) is sorted, we must show sorted (y:(insert x ys)). We can show this by proving the following two claims:
- sorted (insert x ys): This follows by IH applied to (insert1), which is justified by the fact that ys is smaller than y:ys.
- y is at least as small as any element in insert x ys (formally: for all z in elts(insert x ys), y ≤ z): By IH applied to (insert2), elts(insert x ys) =bag bagIns(x, elts(ys)). By the precondition of Case 2, y < x, and by the assump- tion that y:ys is sorted, y is ≤ any element of ys. This is an example how the proof of one property – in this case (insert1) – can depend on the induction hypothesis applied to a different property – in this case, (insert2). (insert2) elts(y:(insert x ys)) =bag bagIns(x, elts(y:ys)). We can reason as fol- lows: elts(y:(insert x ys)) = bagIns(y, elts(insert x ys)) by the defn. of elts =bag bagIns(y, bagIns(x, elts(ys))) by IH (since ys is smaller than y:ys) and (insert2). =bag bagIns(x, bagIns(y, elts(ys))) order of insertion into bag doesn’t matter = bagIns(x, elts(y:ys)) by the defn. of elts
List Sorting Specification
Suppose that sort(ps) returns qs, where ps and qs are lists of integers. We say that sort is a sorting algorithm iff it satisfies the following two conditions:
(sort1) sorted (qs). (sort2) elts(qs) =bag elts(ps).
Insertion Sort
isort [] = [] isort (x:xs) = insert x (isort xs)
Here is an inductive proof that isort satisfies the sort specification:
Base Case: Here, ps = [] and qs = [].
(sort1) sorted ([]) is trivially true. (sort2) elts([]) =bag elts([]) is clearly true.
Inductive Case: Here, ps = x:xs and qs = insert x (isort xs).
(sort1) We show sorted (insert x (isort xs)) via the following steps:
- sorted (isort xs) by IH (sort1) (since xs is smaller than x:xs).
- sorted (isort xs) implies sorted (insert x (isort xs)) by (insert1). (sort2) We show elts(insert x (isort xs)) =bag elts(x:xs) as follows:
elts(insert x (isort xs)) =bag bagIns(x, elts(isort xs)) by (insert2) =bag bagIns(x, elts(xs)) by IH (isort2) (since xs is smaller than x:xs) =bag elts(x:xs) by defn. of elts
Correctness of Quick Sort
Using the specification of partition, we prove the correctness of qsort by induction.
Base Case: Exactly the same reasoning used in the base case of isort applies here.
Inductive Case: Here ps = x:xs and qs = (qsort ls) ++ [x] ++ (qsort gs).
(sort1) sorted ((qsort ls) ++ [x] ++ (qsort gs)) follows from these four facts:
- sorted (qsort ls) by IH (sort1)†.
- sorted (qsort gs) by IH (sort1)†.
- All elements of ls (in particular, the last one) are ≤ x by (part1).
- x is < all elements of gs (in particular, the first one) by (part2).
The applications of IH (sort1) marked †^ in steps (1) and (2) require justification. To invoke IH, we must argue that both ls and gs have a length that is strictly less that that of x:xs. This follows from (part3): the bag union of ls and gs is xs, so the length of either ls or gs can be at most the length of xs. But the length of xs is strictly less than the length of x:xs.
(sort2) We show elts((qsort ls) ++ [x] ++ (qsort gs)) =bag elts(x:xs) as follows:
elts((qsort ls) ++ [x] ++ (qsort gs)) =bag elts(qsort ls) ∪bag elts([x]) ∪bag elts(qsort gs) by distribution of elts over ++ =bag elts(ls) ∪bag elts([x]) ∪bag elts(gs) by IH (sort2)‡ =bag elts([x]) ∪bag elts(xs) by (part3) =bag bagIns(x, elts(xs)) by bag algebra =bag elts(x:xs) by defn. of elts
The applications of IH (sort2) marked ‡^ require justification. As in the proof of (sort1), we can argue that both ls and gs have a length that is strictly less that that of x:xs.
Partition: Non-Tail-Recursive Version
partition1 y [] = ([],[]) partition1 y (z:zs) | z <= y = (z:ls, gs) | z > y = (ls, z:gs) where (ls,gs) = partition1 y zs
Example:
(ls 1 ,gs 1 ) = partition 5 [7,2,4,6,3] ⇒ (ls 2 , 7:gs 2 ), where (ls 2 ,gs 2 ) = partition 5 [2,4,6,3] ⇒ (2:ls 3 , 7:gs 3 ), where (ls 3 ,gs 3 ) = partition 5 [4,6,3] ⇒ (2:(4:ls 4 ), 7:gs 4 ), where (ls 4 ,gs 4 ) = partition 5 [6,3] ⇒ (2:(4:ls 5 ), 7:(6:gs 5 )), where (ls 5 ,gs 5 ) = partition 5 [3] ⇒ (2:(4:(3:ls 6 )), 7:(6:gs 6 )), where (ls 6 ,gs 6 ) = partition 5 [] ⇒ (2:(4:(3:[])), 7:(6:[])) ⇒ ([2,4,3], [7,6])
Partition: Tail-Recursive (Iterative) Version
partition2 y ws = partTail y ws [] []
partTail y [] ls gs = (ls, gs) partTail y (z:zs’) ls gs | z <= y = partTail y zs’ (z:ls) gs | z > y = partTail y zs’ ls (z:gs)
Below is an iteration table for partTail in the example partition2 5 [7,2,4,6,3]. The column labelled zs is the list being partitioned.
zs ls gs [7,2,4,6,3] [] [] [2,4,6,3] [] [7] [4,6,3] [2] [7] [6,3] [4,2] [7] [3] [4,2] [6,7] [] [3,4,2] [6,7]
How do we prove partition2 correct? We can use induction, but must be careful – although some arguments decrease in size (zs), others increase in size (ls and gs). Furthermore, certain properties of ls and gs must be maintained at each recursive call; e.g., the elements of ls at each invocation of partTail must be ≤ y. Specializing induction to the case of loops leads to a proof technique called loop invariants.
Loop Invariants
The loop invariants proof technique is a specialization of proof-by-induction for iterations (a.k.a loops, a.k.a. tail-recursive functions). Given a loop with state variables s 1 ,... , sk, the technique involves the following steps. (We use the notation sj (i) to denote the value of state variable sj at the beginning of the ith iteration of the loop.)
- State one or more invariants that hold among the state variables of the loop. Formally, this is a k-ary relation LI(s 1 ,... , sk).
- Show that the loop invariants hold the first time the loop is entered. I.e., it is necessary to show that LI(s 1 (1),... , sk(1)) holds.
- Show that if the loop invariants are true at the beginning of iteration i, then they are true at the beginning iteration i + 1, i.e., after the body of the loop has executed, regardless of the path taken through the body. Formally, it must be shown that for all i, LI(s 1 (i),... , sk(i)) implies LI(s 1 (i + 1),... , sk(i + 1)).
- Show that that the loop terminates. This is usually done by definining a non-negative integer metric function M (s 1 (i),... , sk(i)). that characterizes the size of the problem at iteration i, and showing that the metric strictly decreases at every iteration.
- Show that the terminating state satisfies the desired correctness properties. The terminating state corresponds to the beginning of an iteration that is not executed. Formally, if the loop terminates just before the f inth iteration, it must be shown that LI(s 1 (f in),... , sk(f in)) implies the desired correctness property.
Iterative Factorial
Here is an iterative implementation of the factorial function:
fact2 n = factTail n 1
factTail 0 ans = ans factTail num ans = factTail (num - 1) (num * ans)
Below is an iteration table for factTail in the invocation fact2 5. We assume there is an implicit index variable i that counts the iterations of the loop (i.e., the number of calls to factTail). We refer to this implicit variable in later discussions.
i num ans 1 5 1 2 4 5 3 3 20 4 2 60 5 1 120 6 0 120
Proving partition2 Correct by Loop Invariants
State invariants There are three invariants that hold among the state variables zs, ls, and gs (each of which is assumed to be indexed by an implicit index variable i).
(partLI1) Every element of lsi is ≤ pivot y. (partLI2) Every element of gsi is > pivot y. (partLI3) elts(ws) =bag elts(zsi) ∪bag elts(lsi) ∪bag elts(gsi), where ws is the list parameter of partition2.
Note how the loop invariants are similar to the specifications (part1), (part2), and (part3). This is not coincidence; each (partLIx) will be used to show (partx) in the final step of the proof.
Show invariants hold on entry to loop In the initial invocation of partTail, zs 1 = ws, ls 1 = [], and ls 1 = [].
(partLI1) Each of the zero elements of ls 1 = [] is ≤ y. (partLI2) Each of the zero elements of gs 1 = [] is > pivot y. (partLI3) elts(ws) = elts(ws) ∪bag elts([]) ∪bag elts([]) by bag algebra = elts(zs 1 ) ∪bag elts(ls 1 ) ∪bag elts(gs 1 ) by defn. of partition
Show each loop iteration preserves invariants
(partLI1) Assume all elements of lsi are ≤ y. If zi ≤ y, then by the definition of partTail, lsi+1 = zi:lsi, and clearly all elements of lsi+1 are ≤ y. If zi > y, then by the definition of partTail, lsi+1 = lsi, and the claim holds as well. (partLI2) Reasoning similar to that used in (partLI1) shows this. (partLI3) Assume that elts(ws) =bag elts(zsi) ∪bag elts(lsi) ∪bag elts(gsi). The body of the ith invocation of partTail removes the first element zi of zsi to yield zsi+1, and either prepends zi to lsi to yield lsi+1, or prepends zi to gsi to yield gsi+1. In either case, the union of the three bags is unchanged from iteration i to iteration i + 1.
Show termination Define the metric function M (zsi, lsi, gsi) = length(zsi). M always returns a non-negative number, and the value strictly decreases with i since the length of zs decreases by 1 with each iteration. The iteration stops after a finite number of iterations f in when length(zsf in) = 0.
Show desired properties Assume that the loop terminates right before the f inth iteration, in which case partition2 returns the pair (lsf in,gsf in).
- (partLI1) implies that that every element of lsf in is ≤ y, satsifying (part1).
- (partLI2) implies that that every element of gsf in is > y, satsifying (part2).
- (partLI3) implies that that elts(ws) =bag elts(zsf in) ∪bag elts(lsf in) ∪bag elts(gsf in). By the base case of partTail, zsf in = []. Since elts([]) = the empty bag, bag algebra yields (part3): elts(ws) =bag elts(lsf in) ∪bag elts(gsf in).
Other Classical Sorting Algorithms
The following are other classical sorting algorithms. The proof of their correctness is left as an exercise.
Selection Sort:
ssort [] = [] ssort xs = l:(ssort (del l xs)) where l = least xs
least [] = error "empty list" least [y] = y least (y:ys) = min y (least ys)
del w [] = [] del w (z:zs) | w == z = zs | otherwise = z:(del w zs)
Merge Sort:
msort [] = [] msort [x] = [x] msort xs = merge (msort ys) (msort zs) where (ys,zs) = split xs
merge ps [] = ps merge [] qs = qs merge (p:ps) (q:qs) | p <= q = p:(merge ps (q:qs)) | otherwise = q:(merge (p:ps) qs)
split [] = ([], []) split (x:xs) = (x:zs, ys) where (ys, zs) = split xs
