Basic Algorithms Overview Algorithms Search algorithm Sort algorithm Induction proofs Complexity 21 December 2005 Ariel Shamir 1 Conceptual Hierarchy Algorithm/Model Program Code Today s lecture Compilers etc. Operating System Execution Cycle Hardware 21 December 2005 Ariel Shamir 2 1
Algorithms the How An algorithm is a procedure for executing a given task. An algorithm consist of a list of instructions. Inputs Algorithm Outputs 21 December 2005 Ariel Shamir 3 Algorithms in CS Is there an algorithm for every task? How can we know if an algorithm is correct? What is the time/space complexity? What is the best algorithm for a given task? What is the best way to express an algorithm? 21 December 2005 Ariel Shamir 4 2
Pseudo Code Computer algorithms are usually written in pseudo-code. Pseudo-code is code that is easy to understand and yet is easily transformed into programming language. 21 December 2005 Ariel Shamir 5 Search Algorithms Search algorithms are algorithms that search for a specific element in a given group of elements: Minimum Maximum Specific value We will use arrays as data structure to store elements. 21 December 2005 Ariel Shamir 6 3
Finding Minimum The minimum is the smallest element in a given group of elements. Groups can be represented in different data structures. The basic idea of the algorithm is the same for all data structures. We had in the first lecture an algorithm for finding minimum. Let s s revisit it 21 December 2005 Ariel Shamir 7 The Basic Idea Algorithm to find a minimum in a group of elements A: assign temporary_min the value of first element for each element a in A if a is smaller than temporary_min then assign temporary_min the value of a min(a) will be temporary_min 21 December 2005 Ariel Shamir 8 4
Java Code int min(int A[]) { int temporary_min = A[0]; for(int i=1; i<a.length; i++) if (A[i] < temporary_min) temporary_min =A[i]; return temporary_min; } 21 December 2005 Ariel Shamir 9 Finding Minimum 23 5 12 30 2 8 21 December 2005 Ariel Shamir 10 5
23 5 12 30 2 8 temporary _min=23 21 December 2005 Ariel Shamir 11 i=1 23 5 12 30 2 8 temporary _min=23 21 December 2005 Ariel Shamir 12 6
i=2 23 5 12 30 2 8 temporary _min=5 21 December 2005 Ariel Shamir 13 i=3 23 5 12 30 2 8 temporary _min=5 21 December 2005 Ariel Shamir 14 7
i=4 23 5 12 30 2 8 temporary _min=5 21 December 2005 Ariel Shamir 15 i=5 23 5 12 30 2 8 temporary _min=2 21 December 2005 Ariel Shamir 16 8
23 5 12 30 2 8 temporary _min=2 21 December 2005 Ariel Shamir 17 Finding Maximum The only difference is in the sign: int max(int A[]) { int temporary_max = A[0]; for(int i=1; i<a.length; i++) if (A[i] > temporary_max) temporary_max =A[i]; return temporary_max; } 21 December 2005 Ariel Shamir 18 9
How Long Does It Take? If we have N elements? Around N operations (must test all of them at least once!). Can we do better? On specific cases only with assumptions: the elements are within a given range 0..K. 21 December 2005 Ariel Shamir 19 Searching for Specific Value We can search for the specific value. The type of the result can be: Boolean found or not found. Integer place of the element in set (in our case index in array). The search is also done in linear way. We walk through our data and search for a given element. 21 December 2005 Ariel Shamir 20 10
Element Searching Implementation public class Sarray { public static boolean search(int a[], int value) { int i=0; while( i<a.length && a[i]!=value){ i++; } return (i < a.length); } } 21 December 2005 Ariel Shamir 21 Sort Algorithms Sorting a group of elements means ordering them according to some linear order. For numbers we can use < order. There are numerous sort algorithms. We will talk about several: Selection sort (( find minimum element). Insertion sort. Bubble sort & Merge sort. 21 December 2005 Ariel Shamir 22 11
Sort Algorithms Input and Output The input for the algorithm: Unordered group of n elements (possible representation is an array). The output: Ordered group of elements. 21 December 2005 Ariel Shamir 23 Selection Sort Outline Find the smallest element in a group of elements and exchange it with first one. Find the second smallest element and exchange it with the second one In the j-th iteration of the algorithm we find the j-th smallest element in a group of elements and exchange it with the j-th one. 21 December 2005 Ariel Shamir 24 12
Selection Sort 15 9 4 18 2 26 12 6 21 December 2005 Ariel Shamir 25 15 9 4 18 2 26 12 6 Find Minimum 21 December 2005 Ariel Shamir 26 13
15 9 4 18 2 26 12 6 Find Minimum 21 December 2005 Ariel Shamir 27 Replace 15 9 4 18 2 26 12 6 21 December 2005 Ariel Shamir 28 14
Replace 2 9 4 18 15 26 12 6 21 December 2005 Ariel Shamir 29 2 9 4 18 15 26 12 6 Find Minimum 21 December 2005 Ariel Shamir 30 15
2 9 4 18 15 26 12 6 Find Minimum 21 December 2005 Ariel Shamir 31 Replace 2 9 4 18 15 26 12 6 21 December 2005 Ariel Shamir 32 16
Replace 2 4 9 18 15 26 12 6 21 December 2005 Ariel Shamir 33 2 4 9 18 15 26 12 6 Find Minimum 21 December 2005 Ariel Shamir 34 17
2 4 9 18 15 26 12 6 Find Minimum 21 December 2005 Ariel Shamir 35 Replace 2 4 9 18 15 26 12 6 21 December 2005 Ariel Shamir 36 18
Replace 2 4 6 18 15 26 12 9 21 December 2005 Ariel Shamir 37 2 4 6 18 15 26 12 9 21 December 2005 Ariel Shamir 38 19
2 4 6 9 15 26 12 18 21 December 2005 Ariel Shamir 39 2 4 6 9 15 26 12 18 21 December 2005 Ariel Shamir 40 20
2 4 6 9 12 26 15 18 21 December 2005 Ariel Shamir 41 2 4 6 9 12 26 15 18 21 December 2005 Ariel Shamir 42 21
2 4 6 9 12 15 26 18 21 December 2005 Ariel Shamir 43 2 4 6 9 12 15 26 18 21 December 2005 Ariel Shamir 44 22
2 4 6 9 12 15 18 26 21 December 2005 Ariel Shamir 45 The End! 2 4 6 9 12 15 18 26 21 December 2005 Ariel Shamir 46 23
Selection Sort Code public static void selectionsort(int[] array) { int minindex; for(int j=0; j<array.length-1; j++) { // find j-th minimum element minindex = j; for(int k=j+1; k<array.length; k++) if(array[k]<array[minindex]) minindex=k; // exchange with j-th place element int temp = array[j]; array[j] = array[minindex]; array[minindex] = temp; } } 21 December 2005 Ariel Shamir 47 Find Minimum public static void selectionsort(int[] array) { int minindex; for(int j=0; j<array.length-1; j++) { // find j-th minimum element minindex = j; for(int k=j+1; k<array.length; k++) if(array[k]<array[minindex]) minindex=k; // exchange with j-th place element int temp = array[j]; array[j] = array[minindex]; array[minindex] = temp; } } 21 December 2005 Ariel Shamir 48 24
Replace public static void selectionsort(int[] array) { int minindex; for(int j=0; j<array.length-1; j++) { // find j-th minimum element minindex = j; for(int k=j+1; k<array.length; k++) if(array[k]<array[minindex]) minindex=k; // exchange with j-th place element int temp = array[j]; array[j] = array[minindex]; array[minindex] = temp; } } 21 December 2005 Ariel Shamir 49 Do This N times public static void selectionsort(int[] array) { int minindex; for(int j=0; j<array.length-1; j++) { // find j-th minimum element minindex = j; for(int k=j+1; k<array.length; k++) if(array[k]<array[minindex]) minindex=k; // exchange with j-th place element int temp = array[j]; array[j] = array[minindex]; array[minindex] = temp; } } 21 December 2005 Ariel Shamir 50 25
Induction Proof by induction on a set created by a creation rule {S 0, S k S k+1 }: Induction base - check that the claim is true in the basic situation, S0. Induction rule assume it is true in S k and check that the claim does not change under the update rule (or rules): S k S k+1. 21 December 2005 Ariel Shamir 51 Simple Induction Proof by induction on natural numbers: Induction base - check that the claim is true in the basic situation (for example for 0). Assumption assume the claim is true for k (or for all i <= k). Need to prove the assumption is true for k+1. Why it works? The telescope rule. 21 December 2005 Ariel Shamir 52 26
Example: Sum of n Numbers Proof 1+2+3+ +n +n = n(n+1)/2 Base: 1 = 1*2/2 true! Assumption: 1+2+ +k +k = k(k+1)/2 Need to prove: 1+2+ +(k+1) +(k+1) = (k+1)(k+2)/2 Proof: 1+2+ +k+(k+1) +k+(k+1) = k(k+1)/2 +(k+1) = (k(k+1)+2(k+1))/2 = (k+1)(k+2)/2! 21 December 2005 Ariel Shamir 53 Selection Sort Proof Lemma: after j passes, the j lowest elements occupy the first j places in the group and are stored in sorted order. Outcome: After n-1 n 1 passes the group is sorted! 21 December 2005 Ariel Shamir 54 27
Selection Sort Lemma Induction We have an iteration therefore the base case (S 0 ) is j=1. The update rule S k S k+1 is just another iteration. We assume the lemma is true for j=k (for all j <= k), and we prove it remains true after the k+1 iteration. 21 December 2005 Ariel Shamir 55 Proof by Induction (1) Lemma: after j passes, the j lowest elements occupy the first j places in the group and are stored in sorted order. Base (j=1): after 1 passes, the 1 lowest elements occupy the first 1 places in the group and are stored in sorted order. Assumption: assume after k passes, the k lowest elements occupy the first k places in the group and are stored in sorted order. 21 December 2005 Ariel Shamir 56 28
Proof by Induction (2) Need to prove: after k+1 passes, the k+1 lowest elements occupy the first k+1 places in the group and are stored in sorted order. Proof: in the k+1 iteration we find the smallest element in the remaining N-k N k group and put it in the k+1 place (How do we know its not smaller than the k first elements? assume otherwise). Therefore we get what we want after the K+1 iteration! 21 December 2005 Ariel Shamir 57 Insertion Sort The second sorting method will insert on the j-th pass the j-th element into its rightful place in a subgroup of j-1 j 1 elements that has already been sorted. After n-1 n 1 passes a group is sorted. 21 December 2005 Ariel Shamir 58 29
Inserting a Value to a Sorted List 1. Search for the correct place Linear search? Binary search? Key 2 4 9 15 18 26 12 21 December 2005 Ariel Shamir 59 Inserting the Key 2. Insert the key Key 12 2 4 9 15 18 26 2 4 9 12 15 18 26 21 December 2005 Ariel Shamir 60 30
Inserting to an Array Search can be done in binary search but the insertion to an array must move all elements forward. Therefore, we insert just by gradually moving the element to its place from the top. 21 December 2005 Ariel Shamir 61 Insertion to a Sorted Array 2 4 9 15 18 26 12 2 4 9 15 18 12 26 2 4 9 15 12 18 26 2 4 9 12 15 18 26 21 December 2005 Ariel Shamir 62 31
Insertion Sort 15 9 4 18 2 26 12 6 21 December 2005 Ariel Shamir 63 Key 15 9 4 18 2 26 12 6 21 December 2005 Ariel Shamir 64 32
Insert to sorted 9 15 4 18 2 26 12 6 21 December 2005 Ariel Shamir 65 Key 9 15 4 18 2 26 12 6 21 December 2005 Ariel Shamir 66 33
Insert to sorted 4 9 15 18 2 26 12 6 21 December 2005 Ariel Shamir 67 Key 4 9 15 18 2 26 12 6 21 December 2005 Ariel Shamir 68 34
Insert to sorted 4 9 15 18 2 26 12 6 21 December 2005 Ariel Shamir 69 Key 4 9 15 18 2 26 12 6 21 December 2005 Ariel Shamir 70 35
Insert to sorted 2 4 9 15 18 26 12 6 21 December 2005 Ariel Shamir 71 Key 2 4 9 15 18 26 12 6 21 December 2005 Ariel Shamir 72 36
Insert to sorted 2 4 9 15 18 26 12 6 21 December 2005 Ariel Shamir 73 Key 2 4 9 15 18 26 12 6 21 December 2005 Ariel Shamir 74 37
Insert to sorted 2 4 9 12 15 18 26 6 21 December 2005 Ariel Shamir 75 Key 2 4 9 12 15 18 26 6 21 December 2005 Ariel Shamir 76 38
Insert to sorted 2 4 6 9 12 15 18 26 21 December 2005 Ariel Shamir 77 Sorted! 2 4 6 9 12 15 18 26 21 December 2005 Ariel Shamir 78 39
Insertion Sort Code public static void insertionsort(int[] array) { for(int j=1; j<array.length; j++) { int key = array[j]; int k=j-1; // insert key to its right place while( k>=0 && array[k]>key) { array[k+1] = array[k]; k--; } array[k+1]=key; } } 21 December 2005 Ariel Shamir 79 Bubble Sort The idea is to bubble-up the largest values by swapping adjacent pairs. After j loops the j topmost values are the largest values and sorted! 21 December 2005 Ariel Shamir 80 40
Bubble Sort Pseudo- Code For all indices i from 0 to N-1 For all indices j from 1 to N-1-i If A[j-1] > A[j] Swap A[j] and A[j-1] 21 December 2005 Ariel Shamir 81 Bubble Sort 23 5 12 17 8 2 21 December 2005 Ariel Shamir 82 41
23 5 12 17 8 2 21 December 2005 Ariel Shamir 83 5 23 12 17 8 2 21 December 2005 Ariel Shamir 84 42
5 23 12 17 8 2 21 December 2005 Ariel Shamir 85 5 12 23 17 8 2 21 December 2005 Ariel Shamir 86 43
5 12 23 17 8 2 21 December 2005 Ariel Shamir 87 5 12 17 23 8 2 21 December 2005 Ariel Shamir 88 44
5 12 17 23 8 2 21 December 2005 Ariel Shamir 89 5 12 17 8 23 2 21 December 2005 Ariel Shamir 90 45
5 12 17 8 23 2 21 December 2005 Ariel Shamir 91 2 5 12 17 8 23 21 December 2005 Ariel Shamir 92 46
2 5 12 17 8 23 21 December 2005 Ariel Shamir 93 2 5 12 17 8 23 21 December 2005 Ariel Shamir 94 47
2 5 12 17 8 23 21 December 2005 Ariel Shamir 95 2 5 12 17 8 23 21 December 2005 Ariel Shamir 96 48
2 5 12 8 17 23 21 December 2005 Ariel Shamir 97 2 5 12 8 17 23 21 December 2005 Ariel Shamir 98 49
5 12 8 2 17 23 21 December 2005 Ariel Shamir 99 5 12 8 2 17 23 21 December 2005 Ariel Shamir 100 50
5 12 8 2 17 23 21 December 2005 Ariel Shamir 101 5 12 8 2 17 23 21 December 2005 Ariel Shamir 102 51
5 8 12 2 17 23 21 December 2005 Ariel Shamir 103 5 8 12 2 17 23 21 December 2005 Ariel Shamir 104 52
5 8 2 12 17 23 21 December 2005 Ariel Shamir 105 5 8 2 12 17 23 21 December 2005 Ariel Shamir 106 53
5 8 2 12 17 23 21 December 2005 Ariel Shamir 107 5 8 2 12 17 23 21 December 2005 Ariel Shamir 108 54
5 2 8 12 17 23 21 December 2005 Ariel Shamir 109 5 2 8 12 17 23 21 December 2005 Ariel Shamir 110 55
5 2 8 12 17 23 21 December 2005 Ariel Shamir 111 2 5 8 12 17 23 21 December 2005 Ariel Shamir 112 56
2 5 8 12 17 23 21 December 2005 Ariel Shamir 113 Complexity The efficiency of the algorithm is called it s s complexity. Complexity = Cost in time and space (memory). Usually complexity is computed as a function of the size of the input (why?). 21 December 2005 Ariel Shamir 114 57
Selection Sort Loops public static void selectionsort(int[] array) { int min,temp; for(int j=0; j<array.length-1; j++) { min=j; // find j-th minimum element for(int k=j+1; k<array.length; k++) if(array[k]<array[min]) min=k; // exchange with j-th place element temp=array[j]; array[j]=array[min]; array[min]=temp; } } 21 December 2005 Ariel Shamir 115 Counting Loops j 0 1 2 N-2 N-1 k 1..N 2..N 3..N (N-1)..N (N-1)..N # N-1 N-2 N-3 1 0 0+1+2+ (N-2)+(N-1) (N-2)+(N-1) = (N*(N-1))/2 =~ N 2 21 December 2005 Ariel Shamir 116 58
An Order of N 2 Check for Primes int numprimes = 0; for (int n = 2; n < number; n++) { isprime = true; for (int j = 2; j < n; j++) if (i % n == 0) isprime = false; } if No. (isprime) of Loops: 1+2+ (N (N-2)+(N-1) numprimes++; } 21 December 2005 Ariel Shamir 117 Counting Primes Running Times (ver. 1) Input size T N i /N 0 10K 2.5s 1 20K 10s 2 30K 22.5s 3 40K 40s 4 T i /T 0 1 4 9 16 F(n) = n 2 is Quadratic Growth 21 December 2005 Ariel Shamir 118 59
An Order of NN Check for Primes int numprimes = 0; for (int n = 2; n < number; n++) { isprime = true; int largestdivisor = (int)math.sqrt (n); for (int j = 2; j < largestdivisor; j++) if (i % n == 0) isprime = false; } if (isprime) numprimes++; } 21 December 2005 Ariel Shamir 119 Number of Loops 1+ 2+ 2+ (N-2)+ (N-1) < (N-1) + (N-1) + + + (N-1) = (N-1) 1)(N-1) < NNN 21 December 2005 Ariel Shamir 120 60
Counting Primes Running Times (ver. 2) Input size T N i /N 0 10K 1.1s 1 20K 3.1s 2 30K 5.6s 3 40K 8.6s 4 T i /T 0 1 2.8 5.1 7.8 F(n) = nn is Sub-Quadratic Growth 21 December 2005 Ariel Shamir 121 Insertion Sort Loops public static void insertionsort(int[] array) { for(int j=1; j<array.length; j++) { int key= array[j]; int k=j-1; // shift to right values larger then key while( k>0 and array[k]>key) { array[k+1] = array[k]; k--; } array[k+1]=key; } } 21 December 2005 Ariel Shamir 122 61
Sorting Complexity In our examples all algorithms have the same complexity. If the size of input is N elements all algorithms in worse case require around N 2 steps not the most efficient. There are N*log(N) algorithms which is also proven to be the best possible. 21 December 2005 Ariel Shamir 123 Merge Sort public static void mergesort (int[] a) { sort(a,0,a.length-1); } private static void sort( int a[], int low, int high) { if (low >= high ) return; int med = (low + high)/2; sort(a, low, med); sort(a, med+1, high); merge(a, low, med, high); } 21 December 2005 Ariel Shamir 124 62
Merge Method private static merge( int[] a, int low, int med, int high) { int[] temp = new int [high-low + 1]; i = 0 ; j = low ; k = med + 1; while (i < temp.length) { if (k > high a[j] < a[k]) temp[i++] = a[j++]; else temp[i++] = a[k++]; } for (int i = 0; i < temp.length ; i++) a[low + i] = temp[i]; } 21 December 2005 Ariel Shamir 125 63