Basic Algorithms Overview

Transcription

1 Basic Algorithms Overview Algorithms Search algorithm Sort algorithm Induction proofs Complexity 21 December 2005 Ariel Shamir 1 Conceptual Hierarchy Algorithm/Model Program Code Today s lecture Compilers etc. Operating System Execution Cycle Hardware 21 December 2005 Ariel Shamir 2 1

2 Algorithms the How An algorithm is a procedure for executing a given task. An algorithm consist of a list of instructions. Inputs Algorithm Outputs 21 December 2005 Ariel Shamir 3 Algorithms in CS Is there an algorithm for every task? How can we know if an algorithm is correct? What is the time/space complexity? What is the best algorithm for a given task? What is the best way to express an algorithm? 21 December 2005 Ariel Shamir 4 2

3 Pseudo Code Computer algorithms are usually written in pseudo-code. Pseudo-code is code that is easy to understand and yet is easily transformed into programming language. 21 December 2005 Ariel Shamir 5 Search Algorithms Search algorithms are algorithms that search for a specific element in a given group of elements: Minimum Maximum Specific value We will use arrays as data structure to store elements. 21 December 2005 Ariel Shamir 6 3

4 Finding Minimum The minimum is the smallest element in a given group of elements. Groups can be represented in different data structures. The basic idea of the algorithm is the same for all data structures. We had in the first lecture an algorithm for finding minimum. Let s s revisit it 21 December 2005 Ariel Shamir 7 The Basic Idea Algorithm to find a minimum in a group of elements A: assign temporary_min the value of first element for each element a in A if a is smaller than temporary_min then assign temporary_min the value of a min(a) will be temporary_min 21 December 2005 Ariel Shamir 8 4

5 Java Code int min(int A[]) { int temporary_min = A[0]; for(int i=1; i<a.length; i++) if (A[i] < temporary_min) temporary_min =A[i]; return temporary_min; } 21 December 2005 Ariel Shamir 9 Finding Minimum December 2005 Ariel Shamir 10 5

6 temporary _min=23 21 December 2005 Ariel Shamir 11 i= temporary _min=23 21 December 2005 Ariel Shamir 12 6

7 i= temporary _min=5 21 December 2005 Ariel Shamir 13 i= temporary _min=5 21 December 2005 Ariel Shamir 14 7

8 i= temporary _min=5 21 December 2005 Ariel Shamir 15 i= temporary _min=2 21 December 2005 Ariel Shamir 16 8

9 temporary _min=2 21 December 2005 Ariel Shamir 17 Finding Maximum The only difference is in the sign: int max(int A[]) { int temporary_max = A[0]; for(int i=1; i<a.length; i++) if (A[i] > temporary_max) temporary_max =A[i]; return temporary_max; } 21 December 2005 Ariel Shamir 18 9

10 How Long Does It Take? If we have N elements? Around N operations (must test all of them at least once!). Can we do better? On specific cases only with assumptions: the elements are within a given range 0..K. 21 December 2005 Ariel Shamir 19 Searching for Specific Value We can search for the specific value. The type of the result can be: Boolean found or not found. Integer place of the element in set (in our case index in array). The search is also done in linear way. We walk through our data and search for a given element. 21 December 2005 Ariel Shamir 20 10

11 Element Searching Implementation public class Sarray { public static boolean search(int a[], int value) { int i=0; while( i<a.length && a[i]!=value){ i++; } return (i < a.length); } } 21 December 2005 Ariel Shamir 21 Sort Algorithms Sorting a group of elements means ordering them according to some linear order. For numbers we can use < order. There are numerous sort algorithms. We will talk about several: Selection sort (( find minimum element). Insertion sort. Bubble sort & Merge sort. 21 December 2005 Ariel Shamir 22 11

12 Sort Algorithms Input and Output The input for the algorithm: Unordered group of n elements (possible representation is an array). The output: Ordered group of elements. 21 December 2005 Ariel Shamir 23 Selection Sort Outline Find the smallest element in a group of elements and exchange it with first one. Find the second smallest element and exchange it with the second one In the j-th iteration of the algorithm we find the j-th smallest element in a group of elements and exchange it with the j-th one. 21 December 2005 Ariel Shamir 24 12

13 Selection Sort December 2005 Ariel Shamir Find Minimum 21 December 2005 Ariel Shamir 26 13

14 Find Minimum 21 December 2005 Ariel Shamir 27 Replace December 2005 Ariel Shamir 28 14

15 Replace December 2005 Ariel Shamir Find Minimum 21 December 2005 Ariel Shamir 30 15

16 Find Minimum 21 December 2005 Ariel Shamir 31 Replace December 2005 Ariel Shamir 32 16

17 Replace December 2005 Ariel Shamir Find Minimum 21 December 2005 Ariel Shamir 34 17

18 Find Minimum 21 December 2005 Ariel Shamir 35 Replace December 2005 Ariel Shamir 36 18

19 Replace December 2005 Ariel Shamir December 2005 Ariel Shamir 38 19

20 December 2005 Ariel Shamir December 2005 Ariel Shamir 40 20

21 December 2005 Ariel Shamir December 2005 Ariel Shamir 42 21

22 December 2005 Ariel Shamir December 2005 Ariel Shamir 44 22

23 December 2005 Ariel Shamir 45 The End! December 2005 Ariel Shamir 46 23

24 Selection Sort Code public static void selectionsort(int[] array) { int minindex; for(int j=0; j<array.length-1; j++) { // find j-th minimum element minindex = j; for(int k=j+1; k<array.length; k++) if(array[k]<array[minindex]) minindex=k; // exchange with j-th place element int temp = array[j]; array[j] = array[minindex]; array[minindex] = temp; } } 21 December 2005 Ariel Shamir 47 Find Minimum public static void selectionsort(int[] array) { int minindex; for(int j=0; j<array.length-1; j++) { // find j-th minimum element minindex = j; for(int k=j+1; k<array.length; k++) if(array[k]<array[minindex]) minindex=k; // exchange with j-th place element int temp = array[j]; array[j] = array[minindex]; array[minindex] = temp; } } 21 December 2005 Ariel Shamir 48 24

25 Replace public static void selectionsort(int[] array) { int minindex; for(int j=0; j<array.length-1; j++) { // find j-th minimum element minindex = j; for(int k=j+1; k<array.length; k++) if(array[k]<array[minindex]) minindex=k; // exchange with j-th place element int temp = array[j]; array[j] = array[minindex]; array[minindex] = temp; } } 21 December 2005 Ariel Shamir 49 Do This N times public static void selectionsort(int[] array) { int minindex; for(int j=0; j<array.length-1; j++) { // find j-th minimum element minindex = j; for(int k=j+1; k<array.length; k++) if(array[k]<array[minindex]) minindex=k; // exchange with j-th place element int temp = array[j]; array[j] = array[minindex]; array[minindex] = temp; } } 21 December 2005 Ariel Shamir 50 25

26 Induction Proof by induction on a set created by a creation rule {S 0, S k S k+1 }: Induction base - check that the claim is true in the basic situation, S0. Induction rule assume it is true in S k and check that the claim does not change under the update rule (or rules): S k S k December 2005 Ariel Shamir 51 Simple Induction Proof by induction on natural numbers: Induction base - check that the claim is true in the basic situation (for example for 0). Assumption assume the claim is true for k (or for all i <= k). Need to prove the assumption is true for k+1. Why it works? The telescope rule. 21 December 2005 Ariel Shamir 52 26

27 Example: Sum of n Numbers Proof n +n = n(n+1)/2 Base: 1 = 1*2/2 true! Assumption: k +k = k(k+1)/2 Need to prove: (k+1) +(k+1) = (k+1)(k+2)/2 Proof: k+(k+1) +k+(k+1) = k(k+1)/2 +(k+1) = (k(k+1)+2(k+1))/2 = (k+1)(k+2)/2! 21 December 2005 Ariel Shamir 53 Selection Sort Proof Lemma: after j passes, the j lowest elements occupy the first j places in the group and are stored in sorted order. Outcome: After n-1 n 1 passes the group is sorted! 21 December 2005 Ariel Shamir 54 27

28 Selection Sort Lemma Induction We have an iteration therefore the base case (S 0 ) is j=1. The update rule S k S k+1 is just another iteration. We assume the lemma is true for j=k (for all j <= k), and we prove it remains true after the k+1 iteration. 21 December 2005 Ariel Shamir 55 Proof by Induction (1) Lemma: after j passes, the j lowest elements occupy the first j places in the group and are stored in sorted order. Base (j=1): after 1 passes, the 1 lowest elements occupy the first 1 places in the group and are stored in sorted order. Assumption: assume after k passes, the k lowest elements occupy the first k places in the group and are stored in sorted order. 21 December 2005 Ariel Shamir 56 28

29 Proof by Induction (2) Need to prove: after k+1 passes, the k+1 lowest elements occupy the first k+1 places in the group and are stored in sorted order. Proof: in the k+1 iteration we find the smallest element in the remaining N-k N k group and put it in the k+1 place (How do we know its not smaller than the k first elements? assume otherwise). Therefore we get what we want after the K+1 iteration! 21 December 2005 Ariel Shamir 57 Insertion Sort The second sorting method will insert on the j-th pass the j-th element into its rightful place in a subgroup of j-1 j 1 elements that has already been sorted. After n-1 n 1 passes a group is sorted. 21 December 2005 Ariel Shamir 58 29

30 Inserting a Value to a Sorted List 1. Search for the correct place Linear search? Binary search? Key December 2005 Ariel Shamir 59 Inserting the Key 2. Insert the key Key December 2005 Ariel Shamir 60 30

31 Inserting to an Array Search can be done in binary search but the insertion to an array must move all elements forward. Therefore, we insert just by gradually moving the element to its place from the top. 21 December 2005 Ariel Shamir 61 Insertion to a Sorted Array December 2005 Ariel Shamir 62 31

32 Insertion Sort December 2005 Ariel Shamir 63 Key December 2005 Ariel Shamir 64 32

33 Insert to sorted December 2005 Ariel Shamir 65 Key December 2005 Ariel Shamir 66 33

34 Insert to sorted December 2005 Ariel Shamir 67 Key December 2005 Ariel Shamir 68 34

35 Insert to sorted December 2005 Ariel Shamir 69 Key December 2005 Ariel Shamir 70 35

36 Insert to sorted December 2005 Ariel Shamir 71 Key December 2005 Ariel Shamir 72 36

37 Insert to sorted December 2005 Ariel Shamir 73 Key December 2005 Ariel Shamir 74 37

38 Insert to sorted December 2005 Ariel Shamir 75 Key December 2005 Ariel Shamir 76 38

39 Insert to sorted December 2005 Ariel Shamir 77 Sorted! December 2005 Ariel Shamir 78 39

40 Insertion Sort Code public static void insertionsort(int[] array) { for(int j=1; j<array.length; j++) { int key = array[j]; int k=j-1; // insert key to its right place while( k>=0 && array[k]>key) { array[k+1] = array[k]; k--; } array[k+1]=key; } } 21 December 2005 Ariel Shamir 79 Bubble Sort The idea is to bubble-up the largest values by swapping adjacent pairs. After j loops the j topmost values are the largest values and sorted! 21 December 2005 Ariel Shamir 80 40

41 Bubble Sort Pseudo- Code For all indices i from 0 to N-1 For all indices j from 1 to N-1-i If A[j-1] > A[j] Swap A[j] and A[j-1] 21 December 2005 Ariel Shamir 81 Bubble Sort December 2005 Ariel Shamir 82 41

42 December 2005 Ariel Shamir December 2005 Ariel Shamir 84 42

43 December 2005 Ariel Shamir December 2005 Ariel Shamir 86 43

44 December 2005 Ariel Shamir December 2005 Ariel Shamir 88 44

45 December 2005 Ariel Shamir December 2005 Ariel Shamir 90 45

46 December 2005 Ariel Shamir December 2005 Ariel Shamir 92 46

47 December 2005 Ariel Shamir December 2005 Ariel Shamir 94 47

48 December 2005 Ariel Shamir December 2005 Ariel Shamir 96 48

49 December 2005 Ariel Shamir December 2005 Ariel Shamir 98 49

50 December 2005 Ariel Shamir December 2005 Ariel Shamir

51 December 2005 Ariel Shamir December 2005 Ariel Shamir

52 December 2005 Ariel Shamir December 2005 Ariel Shamir

53 December 2005 Ariel Shamir December 2005 Ariel Shamir

54 December 2005 Ariel Shamir December 2005 Ariel Shamir

55 December 2005 Ariel Shamir December 2005 Ariel Shamir

56 December 2005 Ariel Shamir December 2005 Ariel Shamir

57 December 2005 Ariel Shamir 113 Complexity The efficiency of the algorithm is called it s s complexity. Complexity = Cost in time and space (memory). Usually complexity is computed as a function of the size of the input (why?). 21 December 2005 Ariel Shamir

58 Selection Sort Loops public static void selectionsort(int[] array) { int min,temp; for(int j=0; j<array.length-1; j++) { min=j; // find j-th minimum element for(int k=j+1; k<array.length; k++) if(array[k]<array[min]) min=k; // exchange with j-th place element temp=array[j]; array[j]=array[min]; array[min]=temp; } } 21 December 2005 Ariel Shamir 115 Counting Loops j N-2 N-1 k 1..N 2..N 3..N (N-1)..N (N-1)..N # N-1 N-2 N (N-2)+(N-1) (N-2)+(N-1) = (N*(N-1))/2 =~ N 2 21 December 2005 Ariel Shamir

59 An Order of N 2 Check for Primes int numprimes = 0; for (int n = 2; n < number; n++) { isprime = true; for (int j = 2; j < n; j++) if (i % n == 0) isprime = false; } if No. (isprime) of Loops: 1+2+ (N (N-2)+(N-1) numprimes++; } 21 December 2005 Ariel Shamir 117 Counting Primes Running Times (ver. 1) Input size T N i /N 0 10K 2.5s 1 20K 10s 2 30K 22.5s 3 40K 40s 4 T i /T F(n) = n 2 is Quadratic Growth 21 December 2005 Ariel Shamir

60 An Order of NN Check for Primes int numprimes = 0; for (int n = 2; n < number; n++) { isprime = true; int largestdivisor = (int)math.sqrt (n); for (int j = 2; j < largestdivisor; j++) if (i % n == 0) isprime = false; } if (isprime) numprimes++; } 21 December 2005 Ariel Shamir 119 Number of Loops (N-2)+ (N-1) < (N-1) + (N-1) (N-1) = (N-1) 1)(N-1) < NNN 21 December 2005 Ariel Shamir

61 Counting Primes Running Times (ver. 2) Input size T N i /N 0 10K 1.1s 1 20K 3.1s 2 30K 5.6s 3 40K 8.6s 4 T i /T F(n) = nn is Sub-Quadratic Growth 21 December 2005 Ariel Shamir 121 Insertion Sort Loops public static void insertionsort(int[] array) { for(int j=1; j<array.length; j++) { int key= array[j]; int k=j-1; // shift to right values larger then key while( k>0 and array[k]>key) { array[k+1] = array[k]; k--; } array[k+1]=key; } } 21 December 2005 Ariel Shamir

62 Sorting Complexity In our examples all algorithms have the same complexity. If the size of input is N elements all algorithms in worse case require around N 2 steps not the most efficient. There are N*log(N) algorithms which is also proven to be the best possible. 21 December 2005 Ariel Shamir 123 Merge Sort public static void mergesort (int[] a) { sort(a,0,a.length-1); } private static void sort( int a[], int low, int high) { if (low >= high ) return; int med = (low + high)/2; sort(a, low, med); sort(a, med+1, high); merge(a, low, med, high); } 21 December 2005 Ariel Shamir

63 Merge Method private static merge( int[] a, int low, int med, int high) { int[] temp = new int [high-low + 1]; i = 0 ; j = low ; k = med + 1; while (i < temp.length) { if (k > high a[j] < a[k]) temp[i++] = a[j++]; else temp[i++] = a[k++]; } for (int i = 0; i < temp.length ; i++) a[low + i] = temp[i]; } 21 December 2005 Ariel Shamir