Probability Foundations for Electrical Engineers Prof. Krishna Jagannathan Department of Electrical Engineering Indian Institute of Technology, Madras Lecture - 1 Introduction Welcome, this is Probability Foundations EE5110, Probability Foundation for Electrical Engineering. So, this is a post graduate level course on probability theory. So, this course will essentially, it is a somewhat triggers treatment of probability theory. I guess, you could say it is a more regulars treatment of probability theory. Then, engineers are normally use to. So, in this course we will take an axiomatic view and focus more on deriving fundamental theorems and proving results. Rather than, more under graduate like treatments, where you are probably the focuses is more on solving problems. And computing expectations and computing probabilities and so on. So, at that level, so you can think of these courses, a more conceptual course. They would be more emphasis on proves, more emphasis on rigorously deriving things, under greater degree of conceptual understanding. So, this course, so who should take this course. So, if you think about it in that way, is it right for you. So, these courses meant for people who essentially more mathematically minded. So, people who will benefit from eight or the once, who will probability to more mathematical research. So, if you are working on topics like ((Refer Time: 01:33)) computer networks or caustic control or machine learning, where you need a somewhat strong foundation, conceptual foundation in probability. This course will be very useful for you. So, this course EE5110 runs in two different avatars. It is offered in both semesters, both odd and even semesters. It has the same number, but the version that is offered in the even semester is more operational. It is more computational, operational and there is more emphasis on problem solving. So, if you are more practical person, who has greater use for problem solving and computing things and so on and not so much. If you are not so interested in really getting into the nuts and bolts of probability theory, may be the even semester course may be more appropriate for you. So, this is a call that
you may want to take. Whether you want to take it in this semester or during the even semester. It is call the same thing, but this version will be more theoretical, more conceptual. So, for example, this semester we with, this is the measure theoretic version of the course. This course will have measure theory in it. And the even semester version, although it is call the same thing, it does not have measure theory in it. It is a much more slightly more elementary treatment, conceptually. Although there will be more emphasis on problems and computational aspects of probability. So, you may want to decide which one you are, which one is for you in terms sets. You can always take this call in a few weeks, I guess. So, I am Krishna Jagannathan, Electrical Engineering Department. That is, where you contact me. So, the course has a website. This is, if you go to my web page, you can find the link to this web page. This link write here, has list the entire course contents. So, all the topics we will cover are in fact given here. So, I suggest you visit this today. (Refer Slide Time: 03:24) So, reference material. Let me write down here. So, there is no one book out there, which closely follows what we are going to do. So, there is a number of materials that I am going to suggest. So, you do not have to buy all these books. So, what I would I think, the book that you should probably consider seriously buying, is Grimmet and Stirzaker. This is probably, most well matched with the kind of a... This is a very good book, it is a very classic book.
So, this is something you may want to consider buying, Stirzaker 3rd addition, Probability and Random Processors, Oxford university press. It is an expensive book and it is about 2800 rupees and so. It is an excellent book though. So, if you ((Refer Time: 04:30)) this is be a standard reference. And there is another open source reference, which I will follow quite a bit. It is a MIT OCW Open Course Where. There is a link on the home page. So, if you go to the web page there is a link. So, this is also very useful source. It is a, we will follow certain lectures from there. It is a good source. So, these two are roughly at the level at which, we will be doing this course. And the other text books, which are either slightly at more elementary level or at a more advance level. So, I will list of couple of them also. So, that is a book by Bertsimas and Tsitsiklis. This is rightly more UG level. So, an excellent book Bertsimas and Tsitsiklis, Introduction to Probability. And there is David Williams, Probability with Martingales. This is a beautiful book, but it is more advanced than what we will need, what we will cover. But, there are certain topics here, which are just beautiful. So, this one and the one by Rosenthal, these two books are more advanced, David Williams and Rosenthal. They both more advance than what we will cover. But, they may be a few results and few thing, that I make refer to these books. But, many case both these books are beautifully written very, very good books and more advance probability theory, there is been. So, this is what, so roughly this is really what mostly you need focus on. These two references are enough, I think. And these are occasionally, I think it will be occasionally useful. There is also an effort going on by students, from previous years, to actually latex the notes from the previous year, so of this class. So, that I mean as I mentioned, there is no one book that covers all the material in one place. So, we difficult there it may be good idea to just put everything down on late take. So, students from previous years have actually collaborated and formed a group to latexed the notes from previous years. So, those notes are they, the first round of the, they editing is over. And so I will make them available to you.
(Refer Slide Time: 07:30) So, I will put them. So, class notes latxed. Let me write the spelling latexed class notes, to be uploaded periodically on modal. So, these notes will be uploaded from time to time. So, thing is these notes are I think they are all right. They not it fully polished. So, you have to realize that, they are just coming out of the press. So, they may be some minor errors and they may not be very polished. So, I give them to you. But, just remember the Kavya. That, they may be an occasional mistake or you know, they may be some errors or bugs or type version there. If you find any bugs or errors, please let us know. So, that the idea has to improve these notes as we go along. It is like a collaborate away effort to get it done. So, we will build up from basic. So, there is nothing that I am going to really assume other than some basic real analysis concepts. So, we will really be starting from the very basic stuffs. I will start, you know the real course material I will start on, from the next lecture. Today, I just want to get some sense from you, on what you think, why we study probability. And what it is that, probability theory does, why are you, why are we interested in it. So, can I get some of your feelers form you? Why you think? What is probability theory and why people study? Student: ((Refer Time: 09:00)) So, to study non deterministic events, this is that convincing answer for you?
Student: ((Refer Time: 09:13)) Mathematical tools is to study experiments with uncertain outcomes. So, it is basically what it is. So, if you want a very concise way of looking at a probability theory is a... It essentially is the science behind randomness. It is the science of randomness. So, there are in real life, we know that there are so many events that we encounter, which we do not seem to have a perfect control over or perfect ability to predict or a perfect knowledge of... Things, such as toss of a coin or toss of a die or what the weather, it is going to be tomorrow or whether a you know, whether a child is going to be male or female. These are things that, we do not seem to have control over or we do not seem to have complete understanding over. And so, these are... But, if think about these random events, so call this random or non determined events, is that. There is a larger pattern to it. That is a central point. Although apparently the one time, I toss a coin have no idea whether, it going to be heads or tails. Or I have no idea whether, I really can tell the temperature tomorrow. There is a larger pattern to these things. So, you know for example, let although you do not know the sex of the baby that is going to be born. On an average, half of them are male and half of them are female. So, roughly if you toss a coin million times, roughly half of them turn out to be heads. So, these are certain larger large scale pattern or you know long term certain patterns or what may be call. I guess, pattern is the right word. So, patterns that, one observes in this seemingly random events. And probability theory tries to quantify these aspects. It is a theory that helps us to mathematically capture the pattern behind the seemingly random events. That is really, what probability theory does. Now, this you know these games of chance have been played, even since in the ancient civilizations. So, it is been around for a long time. And actually, even for a last several centuries people have been computing odds and winning bets and so on. So, I guess for few centuries, people have been computing probability at some level or the other. Although, the mathematical theory of probability the more regulars foundation of it, where laid only about a century ago.
So, the probability theory as we know it today, is only about a hundred years old. It is primarily, the primary person the mathematician who primarily contributed to it, to the modern theory of probability. What do you know? Student: ((Refer Time: 11:59)) It is Laplace made contribution. In fact, Laplace was a yes. It did not make significant contributions about. That is about three centuries ago, I guess. But, more I said modern theory of probability is only 100 years old. So, very famous mathematician. Student: ((Refer Time: 12:18)) As a great Russian mathematician by name Andrey Kolmogorov. So, he is the father of modern probability theory. So, essentially he cast probability theory as a... He realize there is basically as a special case of, what is known as measure theory, which was developed by two French mathematicians primarily, Boral and Lay beg. And then of course, last 100 years there will be an explosive development of this probability theory. So, what we will do is, in fact this axiomatic modem probability, axiomatic approach. So, that question arises, why do we need it. So, even very before Kolmogorov, people have been computing probabilities. There may people have been computing there, chance of winning bets. So, chance of winning bets and so on. So, why do even bother, why do we need this more sophisticated theory or more rigorous mathematical theory. If you can compute probability, you can. That is all, we need. Practically, that is why we need. If I tell you that probability of a certain event is, you think that. So, let us say if the probability of an event is 1 by 3, why it occurs, roughly 1 over 3 times. And we know, this is all we really need in practice. See the reason that there is this. The people have to bother to make this rigorous mathematical theory, is because before Kolmogorov, people have been running into all such of paradoxes and contradictions. Now, because without have proper theory, you run into all such of problems. There are because, if you all only go by a certain intuitive understanding in convenient to difficulties. To illustrate this, those to illustrate the importance of an axiomatic theory, there is a very many paradoxes. Let you can come up with, if you not careful about doing
this probability theory in a rigorous way. So, there are many paradoxes that you can run into. So, what I will do is, I will just describe one such paradox to you, very famous paradox. And that will hopefully send the message across to you, on why need to be little more careful than just a normal intuitive understanding of probability. (Refer Slide Time: 14:51) Let so this paradox, it is a well known paradox. So, you take a circle. You take a circle and you inscribe a equilateral triangle in it. That see, this is the center. So, you have circle. And then inscribe equilateral triangle. So, let say this is radius. Let say the radius is just 1. It can be r or 1. And so, if this is 1 then this is also 1. And then this will be. So, this side will be square root of 3. Side of the triangle would be square root of 3, get it. So, the question is the following. So, take a circle and draw a chord uniformly at random. What is the probability that, this random chord is longer than square root of 3, which is the side of the inscribe equilateral triangle. So, the question is you draw. You basically close your eyes and draw a chord. And the question is. So, you draw chord. So, this is the length of the chord. The chord may like this or chord may like that. So, in this case this will be the length of the chord. So, if you pick a chord uniformly at random, what is the probability that the length of the chord in this case. This in this case, that is longer than the side of the inscribe equilateral triangle or rigorous mathematical
theory or. What is the probability? That, this is larger than square root of 3, which is the side of 3 equilateral triangle. So, this is a question. So, it turns out that. This is the reason. This is a paradox, is depending on how you look at it. You get different answers to this question. So, there is in fact, you will there are three perfectly reasonable sounding arguments, which give you three completely different answers. So, the first... So, let me see if I remember this correctly. So, the first way of seeing this. So, you want the chord to be longer than this side. So, what happened if you draw the in circle of this triangle. So, this guy is an equilateral triangle. Let us say, you draw the in circle of this triangle. And if it so turns out, that the chord you draw, let us say you draw the chord. If it is so turns out, that the midpoint of chord is inside the in circle. You can show that, it will be longer than square root of 3. Let so, in this case. In fact, the center of the chord is inside this in circle. And you can see that, this guy is longer than square root of 3, where as if is a chord like that and the midpoint is a outside the in circle, it will be shorter than square root of 3. So, you may argue here that. So, in this case, so this is a first possible construction. In this case, the probability that your random chord is longer than square root of 3, is simply the probability that, the center of the chord falls in side, this is a little circle, in circle. So, essentially you looking at... So, the question you really looking at is, where is the center of the chord. If it is inside the in circle, you are longer than square root of 3. If it is outside of in circle, it is shorter than square root of 3. So, essentially you looking at the probability that, the center of the chord is inside, that in circle. Now, you know that the radius of this in circle is how much. This is half, because this is the centroid. And so, this divides into 2 it to 2 to 1. So, this radius is half. So, the probability that your center of the chord falls into this circle, is how much. If it is uniformly at random then it should be area of this little circle divided by the total area. So, in this, so the midpoint of the chord, so if we take the r d m about the midpoint of chord. So, the answer you get is 1 by 4. See why? Because, you are looking at the point, the midpoint inside the radius of the circle of radius half whereas, the whole radius is 1.
So, area of this is of course, one fourth the area of the bigger circle. So, this is 1 answer. This is the perfectly reasonable argument. The trouble is that, I mean you probably say this is the answer and that is it. The trouble is, that there are also other perfectly reasonable sounding argument, which give you other answers, different answers. So, the second let say the second argument goes is follows. Let me erase this. (Refer Slide Time: 21:45) So, let us say that you fix. Actually, get out this triangle as well. Another way to draw a chord uniformly at random, it is to just fix one and half the chord, wherever you want. Let us say, you fix it here. And consider, so let say this is the tangent at that point. This is one and half the chord. And you can draw the chord like that or like that. Now, whether or not this chord is longer than square root of 3 or shorter than square root of 3 will depend on. So, if the here is where. So, if I draw an equilateral triangle from that point, it is look like that right. So, it depends on the angle that this guy makes with the tangent. So, if this chord is making an angle of. So, this equilateral triangle makes an angle of 60 degrees pi over 3. If this angle is less than pi on 3, I will be shorter. By equivalently, the angle is greater than 2 pi on 3. If it is like that will be shorter, say the chord be shorter, whereas if the angle made with this, a tangent is between 60 degrees and 120 degrees pi on 3 and 2 pi 3.
My chord will be longer than the side of that triangle, make sense. So, essentially what I am doing is fixing one end of the chord. And then looking at what angle it makes with the tangent. So, if from that point of view, it looks like. So, if this angle is uniform with a random chord. So, it does not prefer prepare any particular direction. So, if the angle is uniform then the probability that your longer than square root of 3, simply the probability that this theta, is laying between 60 and 120 pi on 3 and 2 pi on 3. Student: ((Refer Time: 23:38)) What is that probability equal to? 1 by 3, because well this angle is uniformly at random chosen, if at random. So, if you make this angle with tangent, this argument you get answer equal to 1 over 3. Already, you have two different answers to the, what seems like a same question. I will just argue that, I am not made any mistake. I am not cheating you with any some simple. This is not any mistake that is going on. I have completed it correctly and getting two different answers. Actually, there is one more way of getting different answer, all together. And that the argument is as follows. (Refer Slide Time: 24:35) So, there is a triangle. So, you take one side of the equivalent triangle. And you draw that perpendicular. So, what you doing is now. You are fixing the... So, you are fixing the direction of the chord. And you are just going to move the chord, up and down. So,
whichever angle you want, you fix the angle of the chord. So and you are just going to move it up and down. And you are going to draw the equivalent triangle parallel to it. The side of the equivalent triangle be parallel to it. So, now if you see, if you are only going to draw chords, which parallel to this guy. It does not have to be horizontal. It can be any other direction. I will just flip up the, I will just rotate the equivalent triangle. So, that the fact that, this is horizontal is not big deal. So, you can see that if you draw chord like that, it will be longer than square root of 3. And if you draw chord like that, it will be smaller than square root of 3. So, what we are seeing is. So, if you take that radius, the probability of the chord being longer than square root of 3 is simply the probability that is lying above this point. The center of the chord is lying above that point. So, if you are above this point, you are longer and you have below, your shorter. So, if you look at this radius and if you think, if you let say that the center of the chord is uniformly distributed on this radius. Then, you are looking at... So, this is midway. So, this is midway between this and this. So, you would conclude that, the probability the required probability is half, 1 by 2. Because, this is, in this length you are longer and this length you are shorter. So, in that case you will get. So, if you look at, what should I call this? So, distance center of circle and center of chord. So, that distance is uniformly distributed, because it is not. It does not prefer any particular radius. You will get the answer is half. So, there are... So, it is what was posed in English, as seemingly well post question. It has laid to three perfectly reasonable sounding answers. There is nothing. They have not made any mistakes here. So, there is no cheap error. It is actually a deeper problem going on. Now, the question is, what is happening? So, there are no paradoxes. There are all paradoxes, if at all you want to be consistent. You have to have a resolution. There should not be this kind of paradoxes, in there any theory you create. So, what is the resolution? Would you have any, you already know or do you have any guesses. So, this is a same question. So, I am taking the same English question. And it translating it to mathematical languages, I have derived it in three different ways with three different answers, seemingly correct all three of them. So, that question is a same question.
Student: ((Refer Time: 28:36)) So, I did quite understand what is said but I heard the word sample space. So, that is an important concept. Any other, guesses? Student: First method. Actually all the points are not equally likely even. Like the center point can have n codes, infinity number of calls going to that. Well all points can have infinity many points which is call going to that. Student: ((Refer Time: 29:11)) May not be infinite number of course going. Infinite number of course will be there, but as send a point as that point, it would not be there infinite. That is not true. Each point you will have infinitely many possible chords. That is not the answer reason. So, there is something... So, this is not some cheap error. This is not some little mistake that, I have made somewhere. So, it is actually in trigging question. Why this is happening? It is actually a slightly non trivial explanation. So, the one word I heard is this about sample phase, which is a very important concept that we will study. So, to state it in plain English, we are getting three different answers, because you are answering actually three different mathematical questions. So, what I post as a plain English question, which I here wrote as random chord, which in chords. So, we are getting three different answers, because we are actually they are answers to three different mathematical questions. So, the questions are different, the answers can be different. It seems like they are the same English question. But, mathematically actually they are answers to three different questions. In more formal terms, the sample phases of the probability phases involved are in fact, very different in the three cases. I will of course, define these terms more carefully as we go along. But, in this case the midpoint of the chord being within the circle, there the sample phase. We are looking at the center of the chord. And there is sample phase, we are looking at is the whole circle. And we are looking at uniform distribution of the center of the chord, within the circle. Similarly, in the second case we are looking at the uniform distribution
of the angle, that it makes. So, although it seems like there all uniformly drawn chord, mathematically we have sold three different mathematical problems. So, the question the mathematical questions under probability spaces, the sample spaces behind each of them are in fact are different. They are not the same it is not the same problem. We are getting three different answers, because we are answering three different questions, in mathematical terms. Is that? So, which is why we are getting three different answers? So, it is actually not what seems like the same English questions are different mathematical questions, corresponding to three different sample spaces. Therefore, you get three different answers. So, which is why now? So, this while I give this example is just to warn you or give you Kavya that, if you are generally lose about these things. So, if you generally say random chord without really, mentioning what the sample space or what the underlining probability spaces. You can get into all such of confusions. You can get any of these three answers. May be there are more answers, you can get if you. For some other problems, you will get different answers also. So, I hope this as kind of slightly open your eyes, at least to the possibility that. You have to be a bit more careful in talking about thinks, like a random point or a random chord or you know. These things you cannot be very lose about. You have to specify in a more precise way. Student: ((Refer Time: 32:20)) So, the three different questions are... So, we will do this, we can. Once we do all these sample phases, it is tough. It will be become more clear. So, in this case as I said, you are considering the distribution of a midpoint, your distribution of point inside the circle uniformly. And you are looking at the probability, the midpoint of the chord lying inside this circle. So, it is like, so the sample phase is the entire circle itself. And is uniform distribution you say the circle. And you looking at the center of the point being it is a smaller circle. Whereas for a second case for example. The sample space is this theta. Day time in the set of the thing, what you are varying is theta. It is not the center of the chord. So, it is finally. So, the sample phases between 0 1 2 pi. And you have uniform distribution in that.
So, there all different mathematical questions, not all the same questions. It is seems like a same question may English, but not in mathematical terms, in the second. So, in the final case for example, your sample spaces 0 1, in terms of 0 1, where you are putting the chord. So, they are actually different probabilities phases, different sample phases and therefore, different mathematical questions. So, you can learn. So, this called Bertrand's paradox. So, actually Wikipedia has a good article on this. And book by ((Refer Time: 33:55)) also has this, I think. You can there even simulated all three of this and shown on, what the chord is look like. So, this is pretty interesting article to the. And this is a very famous paradox in probability. I will point out few more paradoxes as we go along. There are some interesting paradoxes you can look up. There is another paradox in probability theory. On why you need message theoretic, measure theory view of probability, that also I will point out later. This is just saying that we have to be careful defining sample phases, now defining what you underlying probability spaces and so on.