ST2334 Probability and Statistics
A/P Ajay Jasra Office 06-18, S16 National University of Singapore E-Mail:
[email protected]
Department of Statistics and Applied Probability National University of Singapore 6 Science Drive 2, Singapore, 117546
Contents 1
Introd Introduct uction ion to Probab Probabilit ility y
1.1 Introd Introduct uction ion . . . . . . . . . . . . . . . 1.2 Probabilit Probability y Triples . . . . . . . . . . . 1.2.1 1.2 .1 Sample Sample Space Space . . . . . . . . . . 1.2.2 σ Fields . . . . . . . . . . . . 1.2.3 1.2 .3 Probab Probabilit ility y . . . . . . . . . . . 1.3 Conditional Conditional Probabilit Probability y . . . . . . . . 1.3.1 1.3 .1 Baye Bayess Theore Theorem m . . . . . . . . . 1.3.2 Theorem Theorem of Total Probabilit Probability y . 1.4 Indepen Independen dence ce . . . . . . . . . . . . . .
−
2
2
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
Random Variable Variables s and their Distribu Distributions tions
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
Introd Introduct uction ion to Stat Statist istics ics
3.1 Introd Introduct uction ion . . . . . . . . . . . . . . . . . 3.2 Maximum Maximum Likelihood Likelihood Estimation Estimation . . . . . 3.2.1 3.2 .1 Introd Introduct uction ion . . . . . . . . . . . . 3.2. 3.2.22 The The Me Meth thod od . . . . . . . . . . . . 3.2.3 3.2 .3 Examp Examples les of of Compu Computin ting g the the MLE 3.3 Hypothe Hypothesis sis Testing esting . . . . . . . . . . . . . 3.3.1 3.3 .1 Introd Introduct uction ion . . . . . . . . . . . . 3.3.2 Constructi Constructing ng Test Statistics Statistics . . . . 3.4 Bay Bayesian esian Inference Inference . . . . . . . . . . . . . 3.4.1 3.4 .1 Introd Introduct uction ion . . . . . . . . . . . . 3.4.2 3.4 .2 Baye Bayesia sian n Estima Estimatio tion n . . . . . . . . 3.4. 3.4.33 Exam Exampl ples es . . . . . . . . . . . . . .
2 2 2 3 4 5 6 6 7 8
2.1 Introd Introduct uction ion . . . . . . . . . . . . . . . . . . . . . . . 2.2 Random Random Variab Variables les and Distributi Distribution on Functi Functions ons . . . . 2.2.1 2.2 .1 Random Random Variabl ariables es . . . . . . . . . . . . . . . 2.2.2 Distributio Distribution n Functions unctions . . . . . . . . . . . . . 2.3 Discrete Discrete Random Random Variables ariables . . . . . . . . . . . . . . 2.3.1 Probabilit Probability y Mass Functions unctions . . . . . . . . . . 2.3.2 2.3 .2 Indepen Independen dence ce . . . . . . . . . . . . . . . . . . 2.3.3 2.3 .3 Expecta Expectatio tion n . . . . . . . . . . . . . . . . . . . 2.3. 2.3.44 Depen Depende denc ncee . . . . . . . . . . . . . . . . . . . 2.3.5 Conditional Conditional Distributio Distributions ns and Expectations Expectations . 2.4 Continuo Continuous us Random Random Variables ariables . . . . . . . . . . . . . 2.4.1 Probabilit Probability y Densit Density y Functions unctions . . . . . . . . . 2.4.2 2.4 .2 Indepen Independen dence ce . . . . . . . . . . . . . . . . . . 2.4.3 2.4 .3 Expecta Expectatio tion n . . . . . . . . . . . . . . . . . . . 2.4. 2.4.44 Depen Depende denc ncee . . . . . . . . . . . . . . . . . . . 2.4.5 Conditional Conditional Distributio Distributions ns and Expectations Expectations . 2.4.6 Functions unctions of Random Random Variables ariables . . . . . . . . 2.5 Convergen Convergence ce of Random Random Variab Variables les . . . . . . . . . . 2.5.1 Convergen Convergence ce in Distribution Distribution . . . . . . . . . . 2.5.2 Convergen Convergence ce in Probabilit Probability y . . . . . . . . . . 2.5.3 2.5 .3 Centra Centrall Limi Limitt Theo Theorem rem . . . . . . . . . . . . 3
. . . . . . . . .
8 8 8 8 9 9 10 11 13 15 17 18 18 19 21 24 26 31 31 32 33 34
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
34 34 34 34 35 37 37 38 39 39 39 40
4
Miscell Miscellane aneous ous Result Resultss
4.1 4.1 4.2 4.3 4.4 4.5 4.6
Set Set Theo Theory ry . . . . . . Summat Summation ion . . . . . . Exponential Exponential Function unction Taylor aylor Series Series . . . . . Integrat Integration ion Methods Methods . Distrib Distributi utions ons . . . . .
43
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
43 43 44 44 44 44
1
Course Information Lecture times: This course is held from January 2013-May 2013. Lecture times are at 0800-
1000 Wednesday and 1200-1400 Thursday at LT 27. The notes are available on my website: http://www.stat.nus.edu.sg/~staja/. I do not use IVLE. Office Hour: My office hour is at 1600 on Thursday in 06-18, Department of Statistics and Applied
Probability. I am not available at other times, unless there is a time-table clash. Assessed Coursework: During the course there will b e two assignments which make up 40% of
the final grade (equal weighting). The dates of when these assessments will be handed out will be provided at least two weeks beforehand and you are given two weeks to complete the assessment. The assessments are to be handed to me in person or, at the statistics office, S16, level 7. Due to the number of students on this course I do not accept assessments via e-mail (unless there are extreme circumstances, which would need to be verified by the department before-hand). There is NO mid-term examination. Exam:. There is a 2 hour final exam of 4 questions on 4th May 1-3pm. No calculators are allowed
and the examination is closed-book. You will be given a formula sheet to assist you in the examination (Tables 4.1 and 4.2). Problem Sheets: There are also 10 non-assessed problem sheets available on my website. Typed
solutions will be available on my website. In some weeks, we will discuss the solutions of the assessments after the deadline. Course Details: These notes are not sufficient to replace lectures . In particular, many examples
and clarifications are given during the class. This course investigates the following concepts: Basic concepts of probability, conditional probability, independence, random variables, joint and marginal distributions, mean and variance, some common probability distributions, sampling distributions, estimation and hypothesis testing based on a normal population. There are three chapters of course content, the first covering the foundations of probability. We then move on to random variables and their distributions which features the main content of the course and is a necessary prerequisite for further work in statistical modeling and randomized algorithms. The third chapter is a basic introduction to statistics and gives some basic notions which would be used in statistical modeling. The final chapter of the notes provides some mathematical background for the course, which you are strongly advised to read in the first week. You are expected to know everything that is in this chapter. In particular the notion of double summation and integration is used routinely in this course and you should spend some time recalling these ideas. References: The recommended reference for this course is [1] (Chapters 1-5). For a non-technical
introduction the book [2] provides an entertaining and intuitive look at probability.
Chapter 1
Introduction to Probability 1.1
Introduction
This Chapter provides a basic introduction to the notions that underlie the basics of probability. The level of notes is below that required for complete mathematical rigor, but does provide a technical development of these ideas. Essentially, the basic ideas of probability theory start with a probability triple of an ‘event-space’ (e.g. in flipping a coin a head or tail), sets of events (e.g.‘tail’ or ‘tail and head’) and a way to compare the ‘likelihood’ of events by a probability distribution (the chance of obtaining a tail). Moving on from these notions, we consider the probability of events, given other events are known to have occurred ( conditional probability ) (the probability a coin lands tails, given we know it has two ‘heads’). Some events have probabilities which decouple in a special way and are called independent (for example, flipping a fair coin twice, the outcome of the first flip, may not influence that of the second). The structure of this Chapter is as follows: In Section 1.2, we discuss the idea of a probability triple; in Section 1.3 conditional probability is introduced and in Section 1.4 we discuss independence.
1.2
Probability Triples
As mentioned in the introduction, probability begins with a triple:
• A sample space (possible outcomes) • A collection of sets on outcomes ( σ−field) • A way to compare the likelihood of events (probability measure) We will slowly introduce and analyze these concepts.
1.2.1
Sample Space
The basic notion we begin with is that of an experiment, with a collection of possible outcomes to which we cannot (usually) determine exactly what will happen. For example, if we flip a coin, or if we watch a football game and so on; in general we do not know for certain what will happen. The idea of a sample space is as follows. Definition 1.2.1. The set of all possible outcomes of an experiment is called the sample space
and is denoted by Ω. Example 1.2.2. Consider rolling a six sided fair die, once. There are 6 possible outcomes, and
thus: Ω = 1, 2, . . . , 6 . We may be interested (for example, for betting purposes) in the following events:
{
}
1. we roll a 1 2. we roll an even number 3. we roll an even number, which is less than 3
3 Notation Ω ω A Ac A B A B A B A∆B A B
∩ ∪ \ ⊆ ∅
Set Terminology Collection of objects Member of Ω Subset of Ω Complement of A Intersection Union Difference Symmetric difference Inclusion Empty set
Probability Terminology Sample Space Elementary event Event that an outcome in A occurs Event that no outcome in A occurs Both A and B A or B A but not B A or B but not both If A then B Impossible event
Table 1.1: Terminology of Set and Probability Theory
4. we roll an odd number One thing that we immediately realize is that each of the events in the above example are subsets of Ω, that is each event (1)-(4) correspond to:
{ } 2. { 2,4,6} 3. { 2} 4. { 1,3,5} 1. 1
As a result, we think of events as subsets of the sample space Ω. Such events can be constructed by unions, intersections and complements of other events (we will formally explain why, below). For example, letting A = 2, 4, 6 in the case (2) above, we immediately yield that the event (4) is Ac . Similarly letting A be the event in (2) and B be the event of rolling a number less than 4 we obtain that event (3) is A B. For those of you that have forgotten basic set notations and terminologies, see Table1 1.1. In particular, we will think of Ω as the certain event (we will roll a 1, 2, . . . , 6) and its complement Ω c = as the impossible event (we must roll something ).
{
}
∩
∅
1.2.2
σ
−Fields
Now we have a notion of an event, in particular, that they are subsets of of Ω. A particular question of interest is then: are all subsets of Ω events? The answer actually turns out to be no, but the technical reasons for this lie far out of the scope of this course. We will content ourselves to use a particular collection of sets F (a ‘set of sets’) of Ω which contains all the events that we can make probability statements about. This collection of sets is a σ field.
−
Definition 1.2.3. A collection of sets F of Ω is called a σ - field if it satisfies the following conditions:
1.
∅ ∈ F
∈ F then ∞i=1 Ai ∈ F 3. If A ∈ F then Ac ∈ F It can be shown that σ −fields are closed under countable intersections (i.e. if A 1 , . . . , ∈ F then ∞ A ∈ F ). Whilst it may seem quite abstract, it will turn out that, for this course, all the sets i=1 i we consider will lie in the σ −field F . 2. If A1 , . . . ,
Example 1.2.4. Consider flipping a single coin, letting H denote a head and T denoting a tail then:
Ω = H, T and F = Ω, , H , T .
{
}
{ ∅ { } { }}
−
Thus, to summarize, so far (Ω, F ) is sample space and a σ field. The former is the collection of all possible outcomes and the latter is collection of sets on Ω (the events) that follow Definition 1.2.3. Our next objective is define a way of assigning a likelihood to each event. 1
For those of you that have forgotten set theory, Section 4.1 has a refresher
4 Exercise 1.2.5. You are given De-Morgan’s Laws:
Ai
c
Aci
=
i
c
Ai
i
=
Aci .
i
Show that if A, B
∈ F , A ∩ B and A \ B are in F (just use the rules of σ-fields).
1.2.3
Probability
We now introduce a way to assign a likelihood to an event, via probability. One possible interpretation is the following: suppose my experiment is repeated many times, then the probability of any event is the limit of the ratio of times the event occurs over the number of events. We note that this is not the only interpretation of probability, but we do not diverge into a discussion of the philosophy of probability. Formally, we introduce the notion of a probability measure: Definition 1.2.6. A probability measure P on (Ω, F ) is a function P : F
→ [0, 1] which satisfies:
1.
P(Ω)
= 1 and P( ) = 0
∅
2. For A1 , A2 , . . . disjoint ( i = j , Ai
∩ Aj = ∅) members of F then
∀
P
∞
∞
Ai =
i=1
P(Ai ).
i=1
The triple (Ω, F , P), comprising a set Ω, a σ field F of subsets of Ω and a probability measure P, is called a probability space (or probability triple).
−
The idea is to associate the probability space with an experiment: Example 1.2.7. Consider flipping a coin as in Example 1.2.4, with Ω = H, T and F = Ω, , H , T
{
Then we can set:
{H }) = p
P(
{T }) = 1 − p
{ ∅ { } { }}.
}
∈
P(
p [0, 1].
If p = 1/2 we say that coin is fair.
{ }Ω (the power
Example 1.2.8. Consider rolling a 6-sided die, then Ω = 1, . . . , 6 and F = 0, 1
{
}
set of Ω the set of all subsets of Ω). Define a probability measure on (Ω, F ) as: P(A) =
pi
i A
∈
where i
∀ ∈ {1, . . . , 6}, 0 ≤ pi ≤ 1 and
6 i=1 pi =
P(A) =
∀A ∈ F
1. If p i = 1/6, i
∀ ∈ {1, . . . , 6} then:
Card (A) 6
where Card (A) is the cardinality of A (the number of elements in the set A). We now consider a sequence of results on probability spaces. Lemma 1.2.9. We have the following properties on probability space (Ω, F , P):
∈ F , P(Ac) = 1 − P(A). 2. For any A, B ∈ F , if A ⊆ B then, P(B) = P (A) + P(B \ A) ≥ P (A). 3. For any A, B ∈ F , P(A ∪ B) = P (A) + P(B) − P(A ∩ B). 4. (inclusion-exclusion formula) For any A 1 , . . . , An ∈ F then: 1. For any A
n
Ai =
P
i=1
∩ −
P(Ai )
i
where, for example,
P(Ai
Aj ) +
i
i
=
i
n j=1
j 1 i=1 etc.
−
n
P(Ai
n+1
∩ Aj ∩ Ak ) − · · · + (−1)
P
Ai .
i=1
5
Proof. For (1), A Ac = Ω, so 1 = P(Ω) = P(A Ac ) = P(A) + P(Ac ) (A Ac = ). Thus, one concludes by rearranging. For (2), as B = A (B A) and A (B A) = , (recall Exercise 1.2.5) P(B) = P(A) + P(B A). The inequality is clear from the definition of a probability. For (3), A B = A (B A), which is a disjoint union, and B A = (B (A B)) hence
∪
∪ ∪ \ ∪ \ P(A
Now (A
∪
∩ \
∩
∅
∅
\
\
\ ∩
∪ B) = P (A) + P(B \ A) = P (A) + P(B \ (A ∩ B)).
∩ B) ⊆ B so by (2) P(B
\ (A ∩ B)) = P (B) − P(A ∩ B)
and thus P(A
∪ B) = P (A) + P(B) − P(A ∩ B).
For (4), this can be proved by induction and is an exercise. To conclude the Section, we introduce two terms:
• An event A ∈ F is null if P(A) = 0. • An event A ∈ F occurs almost surely if P(A) = 1. Null events are not impossible. We will see this when considering random variables which take values in the real line.
1.3
Conditional Probability
We now move onto the notion of conditional probability. It takes the concept of a probability of an event, given another event is known to have occurred. For example, what is the probability that it rains today, given that it rained yesterday? We formalize the notion of conditional probability: Definition 1.3.1. Consider probability space (Ω, F , P) and let A, B
∈ F with P(B) > 0. Then the
conditional probability that A occurs given B occurs is defined to be:
∩ B) . |B) := P(AP(B)
P(A
Example 1.3.2. A family has two children of different ages. What is the probability that both
children are boys, given that at least one is a boy? The older and younger child may be either boys or girl so the sample space is: Ω = GG,BB,GB,BG
{
}
and we assume that all outcomes are equally likely P(GG) = P (BB) = P(GB) = P (BG) = 1/4 (the uniform distribution). We are interested in: P(BB
|GB ∪ BB ∪ BG)
= = =
∩ (GB ∪ BB ∪ BG)) P(GB ∪ BB ∪ BG)
P(BB
P(BB)
P(GB
∪ BB ∪ BG)
1/4 1 = . 3/4 3
One can also ask the question: what is the probability that in a family of two children, where the younger of two is a boy, that both are boys? We want: P(BB
|BG ∪ BB)
= = =
∩ (BG ∪ BB)) P(BG ∪ BB)
P(BB
P(BB) P(BG
∪ BB)
1/4 1 = . 1/2 2
6
1.3.1
Bayes Theorem
An important result in conditional probability is Bayes theorem . Theorem 1.3.3. Consider probability space (Ω, F , P) and let A, B
∈ F with P(A), P(B) > 0. Then
we have: P(B
P(B) |A) = P(AP|B) . (A)
Proof. By definition P(B
As P(B) > 0, P(A
∩ B) = P (A|B)P(B) and hence: P(B
1.3.2
∩ B) . |A) = P(AP(A)
P(B) . |A) = P(AP|B) (A)
Theorem of Total Probability
We will use Bayes Theorem after introducing the following result (the Theorem of Total Probability . We begin with a preliminary definition: Definition 1.3.4. A family of sets B 1 , . . . , Bn is called a partition of Ω if: n
∀i = j Bi ∩ Bj = ∅
and
Bi = Ω.
i=1
Lemma 1.3.5. Consider probability space (Ω, F , P) and let for any fixed n
a partition of Ω , with P(Bi ) > 0, i
∀ ∈ {1, . . . , n}. Then, for any A ∈ F :
≥ 2, B 1, . . . , Bn ∈ F be
n
P(A) =
i=1
|Bi )P(Bi ).
P(A
Proof. We give the proof for n = 2, the other cases being very similar. We have A = (A B1 ) (A B2 ) (recall B 1 B2 = Ω, and B 2 = B1c ), so
∩ ∪ ∩
∪
P(A)
= =
∩ B1) + P(A ∩ B2) P(A|B1 )P(B1 ) + P(A|B2 )P(B2 ). P(A
Example 1.3.6. Consider two factories which manufacture a product. If the product comes from factory I, it is defective with probability 1/5 and if it is from factory II, it is defective with probability 1/20. It is twice as likely that a product comes from factory I. What is the probability that a given product operates properly (i.e. is not defective)? Let A denote the event of ‘operates properly’ and B denote the event ‘the product is made in factory I’. Then: P(A)
|B)P(B) + P(A|Bc)P(Bc)
=
P(A
=
4 2 19 1 51 + = . 5 3 20 3 60
If a product is defective, what is the probability that it came from factory I? This is Bayes Theorem P(B
|Ac)
= =
P(Ac
|B)P(B)
P(Ac )
(1/5)(2/3) 8 = . 9/60 9
P(B
|Ac); by
7
1.4
Independence
The idea of independence is roughly that the probability of an event A is ‘unaffected’ by the occurrance of a (non-null) event B ; that is, P(A) = P (A B), when P(B) > 0. More formally:
|
Definition 1.4.1. Consider probability space (Ω, F , P) and let A, B
dent if
∈ F . A and B are indepen-
∩ B) = P (A)P(B). More generally, a family of F −sets A1 , . . . , An ( ∞ > n ≥ 2 ) are independent if P(A
n
P
n
Ai =
i=1
P(Ai ).
i=1
Two important concepts include pairwise independence : P(Ai
∩ Aj ) = P (Ai )P(Aj ) ∀i = j.
This does not necessarily mean that A 1 , . . . , An are independent events. Another important concept is conditional independence ; given an event C F , with P(C ) > 0 , A, B F are conditionally independent events if P(A B C ) = P (A C )P(B C ).
∈
∩ |
∈
|
|
This can be extended to families of sets A i given C . Example 1.4.2. Consider a pack of playing cards, and one chooses a card completely at random
(i.e. no card counting etc). One can assume that the probability of choosing a suit (e.g. spade) is independent of its rank. So, for example: P(spade
king ) = P (spade )P(king ) =
13 4 1 = . 52 52 52
Chapter 2
Random Variables and their Distributions 2.1
Introduction
Throughout the Chapter we assume that there is a probability space (Ω , F , P), but its presence is minimized from the discussion. This is the main Chapter of the course and we focus upon random variables and their distribution (Section 2.2). In particular, a random variable is (essentially) just a map from Ω to some subset of the real line. Once we are there we introduce notions of probability through distribution functions. Sometimes the random variables take values on a countable space (possibly countably infinite) and we call such random variables discrete (Section 2.3). The probabilities of such random variables are linked to probability mass functions and we use this concept to revisit independence and conditional probability. We also consider expectation (the ‘theoretical average’) and conditional expectation for discrete random variables. These ideas are revisited when the random variables are continuous (Section 2.4). The Chapter is concluded when we discuss the convergence of random variables (Section 2.5) and, in particular, the famous central limit theorem.
2.2
Random Variables and Distribution Functions
2.2.1
Random Variables
In general, we are often concerned by an experiment itself, but by an associated consequence of the outcome. For example, a gambler is often interested in his or hers profit or loss, rather that in the result that occurs. To deal with this issue, we introduce the notion of a random variable: Definition 2.2.1. A random variable is a function X : Ω
→ R such that for each x ∈
R,
{ω ∈ Ω : X (ω) ≤ x} ∈ F . Such a function is said to be F −measurable. For the purpose of this course, the technical constraint of F −measurability can be ignored; you
can content yourself to think of X (ω) as a mapping from the sample space to the real line and continue onwards. In general we omit the argument ω and just write X , with possible realized values (numbers) written in lower-case x. The distinction between a random variable X and its realized value is very important and you should maintain this convention. To move from the random variable, back to our probability measure P, we write events ω Ω : X (ω) x as X x , and hence we F write P( X x ) as P(X x) to mean P( ω Ω : X (ω) x ) (recall ω Ω : X (ω) x and so this makes sense). This leads us to the notion of a distribution function:
{ ≤ }
2.2.2
≤
{ ∈
{ ∈
≤ } { ≤ } ≤ } { ∈ ≤ } ∈
Distribution Functions
Definition 2.2.2. The distribution function of a random variable X is the function F : R
given by F (x) = P (X x), x
≤
∈ R .
Example 2.2.3. Consider flipping a fair coin twice; then Ω = Ω
→ [0, 1]
{ H H , T T , H T , T H } , with F =
and P(HH ) = P (T T ) = P (HT ) = P (T H ) = 1/4. Define a random variable X as the number of heads; so X (HH ) = 2 X (HT ) = X (T H ) = 1 X (T T ) = 0.
{0, 1}
9
The associated distribution function is:
F (x) =
0 1 4 3 4
1
if x < 0 if 0 x < 1 if 1 x < 2 if x 2
≤ ≤ ≥
A distribution function has the following properties, which we state without proof. See [1] for the proof. The lemma characterizes a distribution function: a function F is a distribution function for some random variable if and only if it satisfies the three properties in the following lemma. Lemma 2.2.4. A distribution function F has the following properties:
1. limx→−∞ F (x) = 0, limx→∞ F (x) = 1, 2. If x < y then F (x)
≤ F (y),
3. F is right continuous; limδ↓0 F (x + δ ) = F (x) for each x
∈ R .
Example 2.2.5. Consider flipping a coin as in Example 1.2.4, which lands heads with probability
p (0, 1). Let X : Ω
∈
→ R be given by X (H ) = 1
X (T ) = 0.
The associated distribution function is F (x) =
0 1 1
− p
if x < 0 if 0 x < 1 if x 1
≤ ≥
X is said to have a Bernoulli distribution.
We end the Section with another lemma, whose proof can be found in [1]. Lemma 2.2.6. A distribution function F of random variable X , has the following properties:
1.
− F (x) for any fixed x ∈ R , 2. P(x < X ≤ y) = F (y) − F (x) for any fixed x, y ∈ R with x < y, 3. P(X = x) = F (x) − limy↑x F (y) for any fixed x ∈ R . P(X
2.3 2.3.1
> x) = 1
Discrete Random Variables Probability Mass Functions
We now move onto an important class of random variable called discrete random variables. Definition 2.3.1. A random variable is said to be discrete if it takes values in some countable subset X =
{x1, x2, . . . } of R.
A discrete random variable takes values only at countable values and hence its distribution is a jump function, in that it shifts between different values when they change. An important concept is the probability mass function (PMF): Definition 2.3.2. The probability mass function of a discrete random variable X , is the function
f : X
→ [0, 1] defined by f (x) = P (X = x).
Remark 2.3.3. We generally use sans-serif notation X,
Z to
denote the supports of a random variable. That is the range of values for which its PMF (or PDF - as will be defined much later on) is (potentially) non-zero. Note however, the PMF may be defined on say Z or R but is taken as (potentially) non-zero only at those points in X it is always zero outside X. We call this the support of a random variable.
10 We remark that F (x) =
f (x) f (x) = F (x)
i:xi x
≤
F (y) − lim y↑x
which provides and association between the distribution function and the PMF. We have the following result, whose proof follows easily from the above definitions and results. Lemma 2.3.4. A PMF satisfies:
1. the set of x such that f (x) = 0 is countable, 2.
x
∈X f (x) = 1.
Example 2.3.5. A coin is flipped n times and the probability one obtains a head is p
(0, 1); Ω = H, T . Let X denote the number of heads, which takes values in the set X = 0, 1, . . . , n and is thus a discrete random variable. Consider x X , exactly nx points in Ω give us x heads and each point occurs with probability p x (1 p)n−x ; hence
{
}
n
∈
−
f (x) =
n x p (1 x
{
− p)n−x x ∈
∈
}
X .
The random variable X is said to have a Binomial distribution and this is denoted X Note that n n! = x (n x)!x! with x! = x
∼ B (n, p).
−
× (x − 1) × · · · × 1 and 0! = 1.
Example 2.3.6. Let λ > 0 be given. A random variable X that takes values on X = 0, 1, 2, . . . ,
{
is said have a Poisson distribution with parameter λ (denoted X
∼ P (λ)) if its PMF is:
f (x) =
2.3.2
λx e−λ x!
x
∈
}
X .
Independence
We now extend the notion of independence of events to the domain of random variables. Recall that events A and B are independent if the joint probability is equal to the product ( A does not affect B). Definition 2.3.7. Discrete random variables X and Y are independent if the events X = x
{ } and {Y = y } are independent for each (x, y) ∈ × . To understand this idea, let = {x1 , x2 , . . . }, = {y1 , y2 , . . . }, then X and Y are independent if and only if the events A i = {X = xi }, B j = {Y = y j } are independent for every possible pair i, j. Example 2.3.8. Consider flipping a coin once, which lands tail with probability p ∈ (0, 1). Let X X
Y
X
Y
be the number of heads seen and Y the number of tails. Then: P(X = Y =
1) = 0
and P(X =
1)P(Y = 1) = (1
so X and Y cannot be independent.
− p) p = 0
A useful result (which we do not prove) that is worth noting is the following. Theorem 2.3.9. If X and Y are independent random variables and g :
→ R, h : → R, then
X
the random variables g (X ) and h(Y ) are also independent.
Y
Xi ; In full generality1 , consider a sequence of discrete random variables X 1 , X 2 , . . . , Xn , X i they are said to be independent if the events X 1 = x 1 , . . . , X n = x n are independent for every possible (x1 , . . . , xn ) X 1 Xn . That is:
{
∈ × · · · ×
}
{
∈
}
n
P(X 1 = x 1 , . . . , Xn = x n ) =
i=1 1
P(X i = x i )
∀(x1, . . . , xn) ∈ 1 × · · · ×
this point will not make too much sense, until we see Section 2.3.4
X
Xn .
11
2.3.3
Expectation
Throughout your statistical training, you may have encountered the notion of an average or mean value of data. In this Section we consider the idea of the average value of a random variable, which is called the expected value . Definition 2.3.10. The expected value of a discrete random variable X on X, with PMF f , denoted E[X ] is
defined as E[X ] :=
x
xf (x)
∈X
whenever the sum on the R.H.S. is absolutely convergent. Example 2.3.11. Recall the Poisson random variable from Example 2.3.6 , X
∼ P (λ). The expected
value is: E[X ]
∞ λx e−λ
=
x
x!
x=0
∞
λx−1 (x 1)! x=1
= λe−λ
−
= λe−λ eλ = λ
where we have used the Taylor series expansion for the exponential function on the third line. We now look at the idea of an expectation of a function of a random variable: Lemma 2.3.12. Given a discrete random variable X on X, with PMF f and g : E[g(X )]
=
X
→ R :
g(x)f (x)
x
∈X
whenever the sum on the R.H.S. is absolutely convergent. Example 2.3.13. Returning to Example 3.2.1, we have 2
E[X
] =
∞
x2
x=0
= λe−λ
λx e−λ x!
∞
−
x=1
= λe−λ
x
λ(x−1) (x 1)!
∞ d
x=1
dλ
(λx )
1 (x
− 1)!
∞ − λ d = λe λ (λx−1 ) dλ
x=1
1 (x
d (λeλ ) dλ 2 = λ + λ
− 1)!
= λe−λ
where we have assumed that it is legitimate to swap differentiation and summation (it turns out that this is true here). An important concept is the moment generating function:. Definition 2.3.14. For a discrete random variable X the moment generating function (MGF)
is
M (t) = E [eXt ] =
where T is the set of t for which
X
x
∈X
ext f (x) <
∞.
ext f (x)
∈
t
T
12 Exercise 2.3.15. Show that E[X ]
2 E[X ]
= M (0)
= M (2) (0)
when the right-hand derivatives exist. The moment generating function is thus a simple way to obtain moments, if it is simple to differentiate M (t). Note also, that it can be proven that the MGF uniquely characterizes a distribution. Example 2.3.16. Let X
∼ P (λ) then: E[e
Xt
∞
] =
ext
x=0
λx −λ e x!
∞ (λet )x e−λ
=
x!
x=0
= exp λ(et
{
− 1)}.
Then M (0) = λ. An important special case of functions of random variables are: Definition 2.3.17. Given a discrete random variable X on X, with PMF f and k
moment of X is
∈ Z+, the kth
k E[X ]
and the kth central moment of X is
− E[X ])k ].
E[(X
Of particular importance, is the idea of the variance:
− E[X ])2].
Var[X ]
:= E [(X
Now, we have the following calculations:
− E[X ])2]
E[(X
=
x
=
x
=
x
= Hence we have shown that
∈X
(x
− E[X ])2f (x)
(x2
∈X ∈X
− 2xE[X ] + E[X ]2)f (x)
x2 f (x) 2
E[X
]
− 2E[X ] 2
− 2E[X ]
2 Var[X ] = E [X ]
xf (x) + E[X ]2
x
∈X
f (x)
x 2
2
+ E[X ] = E [X ]
− E[X ]2.
∈X
− E[X ]2.
(2.3.1)
≥
Note a very important point: for any absolutely convergent sum an , if an 0 for each n, then an 0. Now as Var[X ] := E [(X E[X ])2 ] = x∈X (x E[X ])2 f (x) and clearly
≥
−
− (x − E[X ])2 f (x) ≥ 0 ∀x ∈
X
it follows that Variances cannot be negative2 . Example 2.3.18. Returning to Examples 3.2.1 and 2.3.13 , when X
∼ P (λ), we have shown that
2 E[X ]
= λ 2 + λ
E[X ]
= λ.
Hence on using (2.3.1): Var[X ]
= λ.
That is, for a Poisson random variable, its mean and variance are equal. 2
This is very important and you will not be told again
13 Exercise 2.3.19. Compute the mean and variance for a Binomial distribution
B (n, p) random vari-
able. [Hint: consider the differentiation, w.r.t. x, of the equality
n
k=0
n k x = (1 + x)n . k
]. We now state a Theorem, whose proof we do not give. In general, it is simple to establish, once the concept of a joint distribution has been introduced; we have not done this, so we simply state the result. Theorem 2.3.20. The expectation operator has the following properties:
1. if X
≥ 0, E[X ] ≥ 0 2. if a, b ∈ R then E[aX + bY ] = a E[X ] + bE[Y ] 3. if X = c ∈ R always, then E[X ] = c . An important result that we will use later on and whose proof is omitted is as follows. Lemma 2.3.21. If X and Y are independent then E[XY ] = E [X ]E[Y ].
A notion of dependence (linear dependence) is correlation. This will be discussed in details, later on. Definition 2.3.22. X and Y are uncorrelated if E[XY ] = E [X ]E[Y ].
It is important to remark that random variables that are independent are uncorrelated. However, uncorrelated variables are not necessarily independent; we will explore this idea later on. We end the section with some useful properties of the variance operator. Theorem 2.3.23. For random variables X and Y
1. For a
∈ R , Var[aX ] = a2Var[X ],
2. For X, Y uncorrelated Var[X + Y ] = V ar[X ] + Var[Y ]. Proof. For 1. we have: = E[(aX )2 ) E[aX ]2 = a2 E[X 2 ] a2 E[X ]2 = a 2 Var[X ]
− −
Var[aX ]
where we have used Theorem 2.3.20 2. in the second line. Now for 2. we have: Var[(X + Y )]
= = = = =
− E[X + Y ])2)
E[(X + Y 2 2 E[X + Y
+ 2XY + E[X + Y ]2 2(X + Y )E[X + Y ]] 2 2 2 2 E[X + Y + 2XY + E[X ] + E[Y ] + 2E[XY ]] 2E[X + Y ]2 2 2 2 2 E[X ] + E[Y ] + 4 E[X ]E[Y ] + E[X ] + E[Y ] 2(E[X ]2 + E[Y ]2 + 2E[X ]E[Y ]) 2 2 2 2 E[X ] + E[Y ] E[X ] E[Y ] = V ar[X ] + Var[Y ]
−
−
−
−
−
where we have repeatedly used Theorem 2.3.20 2..
2.3.4
Dependence
As we saw in the previous Section, there as a need to define distributions on more than one random variable. We will do that in this Section. We start with the following definition (of which one can easily generalize to a collection of n 1 discrete random variables):
≥
Definition 2.3.24. The joint distribution function F : R2
discrete random variables, is given by
→ [0, 1] of X, Y where X and Y are
≤ ∩ Y ≤ y).
F (x, y) = P (X x
Their joint mass function f : R 2
→ [0, 1] is given by f (x, y) = P (X = x
∩ Y = y).
14 y =
−1
y = 0
y = 2
f (x)
3 18
2 18 3 18 3 18 8 18
6 18 5 18 7 18
1 18 2 18
x = 1 x = 2 x = 3 f (y)
0
4 18 7 18
0
3 18
Table 2.1: The Joint Mass Function in Example 2.3.26. The row totals are the marginal mass function on X (and sum to 1) and respectively the column totals are the marginal mass function on Y . In general, it may be that the random variables are defined on a space Z which may not be decomposable into a cartesian product X Y. In such scenarios we write the joint support as simply Z and omit X and Y , in this scenario, we will use the notation x or y to denote sums over the supports induced by Z. This concept will be clarified during a reading of the subsequent text. Of particular importance, are the marginal PMFs of X and Y :
×
f (x) =
f (x, y) f (y) =
y
Note that, clearly
f (x, y).
x
f (x) = 1 =
x
f (y).
y
Example 2.3.25. Consider Theorem 2.3.20 2. and for simplicity suppose Z = X E[aX + bY ]
=
Y.
Now, we have
(ax + by)f (x, y)
x
∈X y∈Y
=
ax
x
f (x, y) +
y
∈X
= a
= aE[X ] + bE[Y ].
}
E[XY ]
=
∈Y
f (x, y)
x
∈X
yf (y)
y
∈X
Example 2.3.26. Suppose X = 1, 2, 3 and Y = found in Table 2.3.26. From the table, we have:
by
y
∈Y
xf (x) + b
x
{
×
∈Y
{−1, 0, 2} then an example of a joint PMF can be
xyf (x, y) =
x
∈X y∈Y
29 18
(just sum the 9 values in the table, multiplying each time by the product of the associated x and y . Similarly 6 5 7 37 E[X ] = + 2 + 3 = 18 18 18 18 and 3 7 8 13 E[Y ] = 1 + 0 + 2 = . 18 18 18 18
−
We now formalize independence in a result, which provides us a more direct way to ascertain if two random variables X and Y are independent. Lemma 2.3.27. The discrete random variables X and Y are independent if an only if
f (x, y) = f (x)f (y)
∀x, y ∈ R .
More generally, X and Y are independent if and only if f (x, y) factorizes into the product g(x)h(y) with g a function only of x and h a function only of y . Example 2.3.28. Consider the joint PMF:
f (x, y) =
λx e−λ λy e−λ x! y!
X = Y =
{0, 1, . . . , }.
15
Now clearly
λ x e −λ f (x) = x x!
∈
λy e−λ f (y) = y y!
X
∈
Y .
It is also clear that via Lemma 2.3.27 the random variables X and Y are independent and identically distributed (and Poisson distributed). As in the case of a single variable, we are interested in the expectation of a functional of two random variables (strictly, in the proof of Theorem 2.3.23 we have already used the following result):
Lemma 2.3.29. E[g(X, Y )] =
(x,y)
∈X×Y g(x, y)f (x, y).
Recall that in the previous Section, we mentioned a notion of dependence called correlation. In order to formally define this concept, we introduce first the covariance and then correlation. Definition 2.3.30. The covariance between X and Y is Cov[X, Y ]
− E[X ])(Y − E[Y ])].
:= E [(X
The correlation between X and Y is ρ(X, Y ) = Exercise 2.3.31. Show that
Cov[X, Y ]
Var[X ]Var[Y ]
.
− E[X ])(Y − E[Y ])] = E [XY ] − E[X ]E[Y ].
E[(X
Thus, independent random variables have zero correlation. Example 2.3.32. Returning to Example 2.3.26, we have: Var[X ]
= 233/324
Var[Y ]
= 461/324
and Cov[X, Y ]
= 29/18
Thus
− (37/18)(13/18) = 41/324. √
ρ(X, Y ) = 41/ 107413.
Some useful facts about correlations that we do not prove:
• |ρ(X, Y )| ≤ 1. Random variables with correlation 1 or -1 are said to be perfectly positively or negatively correlated.
• The correlation coefficient is 1 iff Y increases linearly with X and -1 iff Y decreases linearly as X increases.
When we consider continuous random variables an example of uncorrelated random variables that are not independent, will be given.
2.3.5
Conditional Distributions and Expectations
In Section 1.3 we discussed the idea of conditional probabilities associated to events. This idea can then be extended to discrete-valued random variables: Definition 2.3.33. The conditional distribution function of Y given X , written F Y |x ( x), is defined
·|
by
|
≤ y |X = x)
F y|x (y x) = P (Y
for any x with P(X = x) > 0 . The conditional PMF of Y given X = x is defined by
|
|
f (y x) = P (Y = y X = x)
when x is such that P(X = x) > 0 .
16 We remark that, in particular, the conditional mass function of Y given X = x is:
|
f (y x) =
f (x, y) = f (x)
Example 2.3.34. Suppose that Y
f (x, y) . y f (x, y)
(λ) and X Y = y (y, p). Find the conditional probability mass function of Y X = x. Note that the random variables lie in the space Z = (x, y) : y 0, 1, . . . , x 0, 1, . . . , y . We have
{
} ∈{
|
∼ P
|
∼ B
{
}}
|
f (x, y) f (x y)f (y) = f (x) f (x)
|
f (y x) =
(x, y)
∈
Z
which is a version of Bayes Theorem for discrete random variables. Note that for (x, y) PMFs above are positive. Now for x 0, 1, . . .
∈{
f (x) =
}
∞
y x p (1 x
y=x
= (λp) =
Thus, we have for y
y
|
f (y x) =
Z all
the
−λ
−λ ∞ ((1 − p)λ)y−x
xe
x!
y x
∈
− p)y−x λ y!e (y
y=x
(λp)x −λp e . x!
∈ {x, x + 1, . . . }:
∈
− x)!
y −λ
− p)y−x λ y!e
px (1
(λp)x λp x! e
−
which after some algebra, becomes:
|
f (y x) =
(λ(1
− p))y−xe−λ(1− p) y ∈ {x, x + 1, . . . }. (y − x)!
Given the idea of a conditional distribution, we move onto the idea of a conditional expectation: Definition 2.3.35. The conditional expectation of a random variable Y , given X = x is E[Y
|X = x] =
|
yf (y x)
y
given that the conditional PMF is well-defined. We generally write E[Y X ] or E[Y x].
|
An important result associated to conditional expectations is as follows: Theorem 2.3.36. The conditional expectation satisfies: E[E[Y
|X ]] = E [Y ]
assuming the expectations all exist. Proof. We have E[Y ]
=
yf (y)
y
=
yf (x, y)
(x,y)
∈Z
=
|
yf (y x)f (x)
(x,y)
∈Z
=
[
x
=
y
E[E[Y
|
yf (y x)]f (x)
|X ]].
|
17 Example 2.3.37. Let us return to Example 2.3.34. Find E[X Y ], E[Y X ] and E [X ]. From Exercise
|
2.3.19 , you should have derived that if Z
|
∼ B (n, p) then E[Z ] = np. Then E[X |Y = y] = yp.
Thus E[X ]
|
= E [E[X Y = y]] = pE[Y ].
As Y
∼ P (λ)
E[X ]
= pλ.
From Example 2.3.34,
|
f (y x) =
(λ(1
− p))y−xe−λ(1− p) y ∈ {x, x + 1, . . . }. (y − x)!
Thus E[Y
Setting u = y
|X = x] =
∞ (λ(1 − p))y−x e−λ(1− p)
y
− x)!
(y
y=x
.
− x in the summation, one yields E[Y
|X = x]
∞
=
(u + x)
(λ(1
− p))ue−λ(1− p) u!
u=0
∞
=
u
(λ(1
− p))u e−λ(1− p) + x u!
u=0
= λ(1
− p) + x.
Here, we have used the fact that U
∼ P (λ(1 − p)).
We end the Section with a result which can be very useful in practice. Theorem 2.3.38. We have, for any g : R
→ R E[E[Y |X ]g(X )] = E [Y g(X )]
assuming the expectations all exist. Proof. We have E[Y g(X )]
=
yg(x)f (x, y)
(x,y)
=
∈Z
g(x)[
x
=
2.4
|
yg(x)f (y x)f (x)
(x,y)
=
∈Z
E[E[Y
|
yf (y x)]f (x)
y
|X ]g(X )].
Continuous Random Variables
In the previous Section, we considered random variables which produce real numbers within a countably finite or infinite set. However, there are of course a wide variety of applications where one sees numerical outcomes of an experiment that can lie potentially anywhere on the real-line. Thus we extend each of concepts that we saw for discrete random variables to continuous ones. In a rather informal (and incorrect) manner, one can simply think of the idea of replacing summation with integration; of course things will be more challenging than this, but one should keep this idea in the back of your mind.
18
2.4.1
Probability Density Functions
A random variable is said to be continuous if its distribution function F (x) = written as x F (x) = f (u)du
≤ x), can be
P(X
−∞
→ ∞), the R.H.S. is the usual Riemann integral and we will assume that the R.H.S. is
where f : R [0, differentiable.
Definition 2.4.1. The function f is called the probability density function (PDF) of the con-
tinuous random variable X . We note that, under our assumptions, f (x) = F (x). Example 2.4.2. One of the most commonly used PDFs is the Gaussian or normal distribution:
f (x) =
where µ
1 1 √ exp{− 2 (x − µ)2 } 2σ σ 2π
x
∈
X = R
∈ R , σ 2 > 0. We use the notation X ∼ N (µ, σ2).
One of the key points associated to PDFs is as follows. The numerical value f (x) does not represent the probability that X takes the value x. The technical explanation goes far beyond the mathematical level of this course, but perhaps an intuitive reason is simply that there are uncountably infinite points in X (so assigning a probability to each point is seemingly impossible). In general, one assigns probability to sets of ‘non-zero width’. For example let A = [a, b], < a < b < , then one might expect:
∈ A) =
P(X
−∞
∞
f (x)dx.
A
Indeed this holds true, but we are deliberately vague about this. We give the following result, which is not proved and should be taken as true. Lemma 2.4.3. If X has a PDF f , then
1.
X
f (x)dx = 1.
2.
P(X = x) =
3.
P(X
0 for each x
∈ [a, b]) =
∈ . b f (x)dx, −∞ ≤ a < b ≤ ∞. a X
Example 2.4.4. Returning to Example 2.4.2 , we have
∈
P(X
X )
=
∞
−∞
=
1 1 √ exp{− 2 (x − µ)2 }dx 2σ σ 2π
√
∞ 2 1 e−u /2 du = 1. 2π −∞
Here, we have used the substitution u = (x ∞ −u2 /2 du = 2π. −∞ e
2.4.2
− µ)/σ to go to the second line and the fact that
√
Independence
To define an idea of independence for continuous random variables, we cannot use the one for discrete random variables (recall Definition 2.3.7); the sets X = x and Y = y have zero probability and are hence trivially independent. Thus we use a new definition for independence:
{
}
{
}
Definition 2.4.5. The random variables X and Y are independent if
{X ≤ x} {Y ≤ y } are independent events for each x, y
∈ R .
As for PMFs one can consider independence through PDFs; however, as we are yet to define the notion of joint PDFs, we will leave this idea for now. We note that one can show that for any real valued functions h and g (at least within some technical conditions which are assumed in this course) that if X and Y are independent, so are the random variables h(X ) and g(Y ). We do not prove why.
19
2.4.3
Expectation
As for discrete random variables one can consider the idea of the average of a random variable. This is simply brought about by replacing summations with integrations: Definition 2.4.6. The expectation of a continuous random variable X with PDF f is given by E[X ]
=
xf (x)dx
X
whenever the integral exists. Example 2.4.7. Consider the exponential density:
f (x) = λe−λx
x
∈
X =
∼ E (λ). Then
[0,
∞)
λ > 0.
We use the notation X E[X ] =
∞
0
xλe−λx dx = [−xe−λx ]∞ + 0
∞
− λ1 e−λx]∞0 = λ1
e−λx dx = [
0
where we have used integration by parts. Example 2.4.8. Let us return to Example 2.4.2. Then E[X ]
= =
∞
−∞ ∞
x
1 1 √ exp{− 2 (x − µ)2 }dx 2σ σ 2π
√ 12π e−u /2du 2
(uσ + µ)
−∞ ∞
= σ
u
−∞
√ 12π e−u /2du + µ 2
2 σ = [ e−u /2 ]∞ −∞ + µ 2π = µ.
√ −
Here, we have used the substitution u = (x
− µ)/σ to go to the second line and the fact that ∞ 1 √ 2π e−u /2du = 1
2
−∞
to go to the third. Example 2.4.9. Consider the gamma density:
f (x) =
λα α−1 −λx x e Γ(α)
∈
x
∞
X =
[0,
∞)
λ,α > 0
where Γ(α) = 0 tα−1 e−t dt. We use the notation X X (λ). Now
∼ E
∼ G (α, λ) and note that if X ∼ G (1, λ) then
E[X ]
∞ λα xα e−λx dx Γ(α) 0 ∞ λα 1 uα e−u du Γ(α) λα+1 0 1 α Γ(α + 1) = . λΓ(α) λ
= = =
Where we have used the substitution u = λx to go the second line and Γ(α + 1) = αΓ(α) on the final line (from herein you may use that identity without proof). We now state a useful technical result Theorem 2.4.10. If X and g (x) ( g : R
→ R ) are continuous random variables
E[g(X )]
=
g(x)f (x)dx.
X
20 We remark that all the extensions to expectations discussed in Section 2.3.3 can be extended to the continuous case. In particular Definitions 2.3.17 and 2.3.22 can be imported to the continuous case. In addition the results: Theorems 2.3.20 and 2.3.23 and Lemma 2.3.21 all can be extended. It is assumed that this is the case from herein. Example 2.4.11. Let us return to Example 2.4.7 2
E[X
∞
] =
x2 λe−λx dx
0
∞
2 = [−x e−λx ]∞ 0 + 2
λ
0
2 . λ2
=
xλe−λx dx
Thus, for an exponential random variable: Var[X ]
=
1 . λ2
Example 2.4.12. Let us return to Example 2.4.2. Then 2
E[X
] = =
∞
−∞ ∞ −∞
= σ
2
x2
1 1 √ exp{− 2 (x − µ)2 }dx 2σ σ 2π
(uσ + µ)2
∞
2
u
−∞
= =
√ 12π e−u /2du 2
∞ 1 −u2 /2 1 −u2 /2 2 e du + µ + 2σµ u e du 2π −∞ 2π
√
√
∞ 2 2 σ2 [ ue−u /2 ]∞ + e−u /2 du + µ2 −∞ 2π −∞ 2 σ 2π + µ2 . 2π
√ { − √ √
∞
2
Here we have used −∞ u √ 12π e−u variable:
/2
}
∞
2
du = 0 and −∞ e−u
/2
du =
√ 2π.
Thus, for a normal random
= σ 2 .
Var[X ]
Example 2.4.13. Let us return to Example 2.4.9 Now 2
E[X
] = = =
∞ λα xα+1 e−λx dx Γ(α) 0 ∞ 1 λα uα+1 e−u du α+2 Γ(α) λ 0 1 α(α + 1) Γ(α + 2) = 2 λ Γ(α) λ2
where we have used Γ(α + 2) = (α + 1)αΓ(α). Thus, for a gamma random variable: Var[X ]
=
α . λ2
To end this section we consider an important concept; the MGF for continuous random variables. Definition 2.4.14. For a continuous random variable X the moment generating function
(MGF) is Xt
M (t) = E [e
where T is the set of t for which Exercise 2.4.15. Show that
]=
ext f (x)dx <
E[X ]
when the right-hand derivatives exist.
ext f (x)dx
X
X
= M (0)
∈
t
∞. 2 E[X ]
= M (2) (0)
T
21 As for discrete random variables, the moment generating function is a simple way to obtain moments, if it is simple to differentiate M (t). Note also, that it can be proven that the MGF uniquely characterizes a distribution. Example 2.4.16. Suppose X
∼ E (λ) then, supposing λ > t
−
∞
M (t) =
λe−x(λ−t) dx
0
=
λ
−
(λ t) λ . (λ t)
=
∞ e−x(λ−t) 0
−
Clearly M (t) =
λ (λ
and thus E[X ] = 1/λ.
− t)2
∼ N (µ, σ2) then
Example 2.4.17. Suppose X
M (t) = = = =
√ √
∞ 1 1 exp (x µ)2 + xt dx 2 2σ σ 2π −∞ ∞ 1 1 exp [(x (µ + tσ 2 ))2 (µ + tσ 2 )2 + µ2 ] dx 2 2σ σ 2π −∞ ∞ 2 2 1 σ t 1 exp µt + exp (x (µ + tσ 2 ))2 dx 2 2 2σ σ 2π −∞ 2 2 σ t exp µt + 2
√
{
{−
−
{−
−
−
}
{
{−
}
−
}
}
where we have used a change-of-variables u = (x
2.4.4
}
− (µ + tσ2))/σ to deal with the integral.
Dependence
Just as for discrete random variables, one can consider the idea of joint distributions for continuous random variables. Definition 2.4.18. The joint distribution function of X and Y is the function F : R2
given by
≤
→ [0, 1]
≤ y).
F (x, y) = P (X x, Y
Again, as for discrete random variables, one requires a PDF: Definition 2.4.19. The random variables are jointly continuous with joint PDF f : R 2
y
F (x, y) =
→ [0, ∞) if
x
f (u, v)dudv
−∞ −∞
for each (x, y)
∈ R 2.
In this course, we will assume (generally) that f (x, y) =
∂ 2 F (x, y). ∂x∂y
Note that, P(X = x, Y = y) = 0 and for sufficiently well defined sets A P((X, Y )
∈ A) =
⊆ ⊆ R 2 Z
f (x, y)dxdy
A
note that the R.H.S. is a double integral, but we will only write one integral sign in such contexts. As for discrete random variables, we can introduce the idea of marginal PDFs. Here, we take a little longer to consider these ideas:
22 Definition 2.4.20. The marginal distribution functions of X and Y are
F (x) = lim F (x, y) y
As
x
F (x) =
∞
F (y) = lim F (x, y). x
→∞
y
f (u, y)dy du
F (y) =
−∞ −∞
∞
f (x, u)dxdu
−∞ −∞
the marginal density functions of X and Y f (x) =
→∞
∞
f (x, y)dy
f (y) =
−∞
∞
f (x, y)dx.
−∞
We remark that expectation is much the same for joint distributions (as for Section 2.4.3) of continuous random variables: E[g(X, Y )]
=
g(x, y)f (x, y)dxdy =
∞
∞
g(x, y)f (x, y)dxdy.
−∞ −∞
Z
So as before if g (x, y) = ah(x) + bg(y) E[g(X, Y )]
= a E[h(X )] + bE[g(Y )].
We now turn to independence; we state the following result with no proof. If you are unconvinced, simply take the below as a definition. Theorem 2.4.21. The random variables X and Y are independent if and only if
F (x, y) = F (x)F (y)
or equivalently f (x, y) = f (x)f (y). Example 2.4.22. Let Z = (R+ )2 = X
×
Y
and
f (x, y) = λ 2 e−λ(x+y)
Then one has
f (x) = λe−λx x
∈
∈
(x, y)
Z , λ > 0.
f (y) = λe−λy y
∈
X
Y
In addition, what is the probability that X > Y ? P(X
> Y ) =
− − − 0
=
0
=
0
=
x
∞
f (x, y)dydx
0
∞
λe−λy dyλe−λx dx
0
∞ ∞
x
e−λy
(1
x 0
λe−λx dx
e−λx )λe−λx dx
0
=
= 1
−
∞ 1 e−λx + e−2λx 2 0 1 1 = . 2 2
Again, one has the idea of covariance and correlation for continuous valued random variables: Cov[X, Y ]
= =
− Z
ρ(X, Y ) =
− E[Y ])] = E [XY ] − E[X ]E[Y ] xyf (x, y)dxdy − xf (x)dx yf (y)dy
E[(X
E[X ])(Y
Cov[X, Y ]
Var[X ]Var[Y ]
23 Example 2.4.23. Let Z = R 2 and define
f (x, y) =
1
1
(x2 + y 2
− {− − − } {− − − − − {− − − } √ {− − − } √ {− − }
2π
ρ2
1
exp
ρ2 )
2(1
2xyρ)
∈
(x, y)
Z ρ
∈ [−1, 1].
Let us check if this is a PDF; clearly it is non-negative on Z. So
f (x, y)dxdy
∞
=
∞
1
−∞ −∞ 2π 1 ∞ 1
Z
=
ρ2
1
exp
2(1
1
exp
ρ2 )
ρ2 )
(y
(x2 + y 2
2
}
2xyρ) dxdy
∞
2 2
y ρ )
{− 2(1 −1 ρ2) (x − yρ)2}dxdy
exp
2(1 2π 1 −∞ −∞ ∞ 1 1 exp (y 2 y 2 ρ2 ) dy 2 2(1 ρ ) 2π −∞ ∞ 1 1 exp (1 ρ2 )y 2 dy = 1. 2) 2(1 ρ 2π −∞ ρ2
= =
−
Here we have completed the square in the integral in x and used the change of variable u = (x ρy)/(1 ρ 2 )1/2 . Thus f (x, y) defines a joint PDF. Moreover, it is called the standard bi-variate normal distribution. Now let us consider the marginal PDFs:
−
−
f (x) = = =
{− − − {− − ∞
1
−∞ 2π 1
ρ2
1
1
2 2 − ρ2) (x + y − 2xyρ)}dy x ∈ R ∞ 1 1 2 2 2 (x − x ρ )} exp{− (y − xρ)2 }dy 2 2(1 ρ ) 2(1 − ρ2 )
exp
exp 2π 1 ρ2 1 −x2 /2 e . 2π
2(1
−∞
√
That is X E[XY ]
∼ N (0, 1). By Similar arguments, Y ∼ N (0, 1). Now, let us consider: ∞ ∞ 1 1 = exp{− (x2 + y 2 − 2xyρ)}dxdy xy 2 2 2(1 − ρ ) 2π 1 − ρ
− − − − √
−∞ −∞
= = = = =
∞
1
2π 2π 2π
1 1
ρ2
1 1 1 1
ρ2 ρ2
−∞ ∞
1
2
2 2
{− 2(1 − ρ2) (y − y ρ )
y exp
{− 2(1 −1 ρ2) (y2 − y2ρ2)
} }
∞
−∞ ∞
{− 2(1 −1 ρ2) (x − yρ)2}dxdy
x exp
2
− ρ2)1/2 + ρy](1 − ρ2)1/2e−u /2dudy −∞ −∞ ∞ √ 2πE[X ] + ρy√ 2π}]dy 1 2 2 2 2 1/2 2 1/2 y exp{− (y y ρ ) [(1 ρ ) (1 ρ ) − } − { − 2(1 − ρ2 ) −∞ ∞ √ 2πdy 1 2 2 2 2 1/2 y exp{− (y y ρ ) (1 ρ ) ρy − } − 2(1 − ρ2 ) −∞ y exp
[u(1
2π 1 ρ2 ∞ 2 ρ y2 e−y /2 dy = ρ. 2π −∞
Here we have used the results in Examples 2.4.8 and 2.4.12 that the first and second (raw) moment of (µ, σ 2 ) are µ and µ 2 + σ 2 . Therefore
N
Cov[X, Y ]
= ρ
ρ(X, Y ) =
√ ρ1 = ρ.
Note also that if ρ = 0 then: f (x, y) =
so as f (x) =
1
2π
− 1
ρ2
√ 12π e−x /2 x ∈ R 2
{− 12 (x2 + y2)}
exp
f (y) =
∈
(x, y)
Z .
√ 12π e−y /2 y ∈ R 2
f (x, y) = f (x)f (y).
That is standard bi-variate normal random variables are independent if and only they are uncorrelated. Note that this does not always occur for other random variables.
24 We conclude this section with a mention of joint moment generating functions: Definition 2.4.24. For a continuous random variables X and Y the joint moment generating
function is Xt 1 +Y t2
M (t1 , t2 ) = E [e
where T is the set of t 1 , t2 for which Exercise 2.4.25. Show that
∂ E[X ] = M (t1 , t2 ) ∂t 1
]=
ext1 +yt2 f (x, y)dxdy
ext1 +yt2 f (x, y)dxdy <
∂ E[Y ] = M (t1 , t2 ) ∂t 2
,
t1 =t2 =0
when the derivatives exist.
2.4.5
∈
(t1 , t2 )
Z
Z
T
∞.
, t1 =t2 =0
∂ 2 E[XY ] = M (t1 , t2 ) ∂t 1 ∂t 2
t1 =t2 =0
Conditional Distributions and Expectations
The idea of conditional distributions and expectations can be further extended to continuous random variables. However, we recall that the event X = x has zero probability, so how can we condition on it? It turns out that we cannot have a rigorous answer to this question (i.e. at the level of mathematics in this course), so some faith must be extended towards the following definitions.
{
}
Definition 2.4.26. The conditional distributions function of Y given X = x is the function:
y
|
F (y x) =
f (x, v) dv −∞ f (x)
for any x such that f (x) > 0 . Definition 2.4.27. The conditional density function of F (y x), written f (y x) is given by:
|
|
f (y x) =
|
f (x, y) f (x)
for any x such that f (x) > 0 . We remark that clearly
|
f (y x) = Example 2.4.28. Suppose that Y X = x
|
so f (x, y) =
f (x, y)
. ∞ −∞ f (x, y)dy
∼ N (x, 1) and X ∼ N (0, 1), f (x, y) = f (y |x)f (x)
|
f (x, y) and f (x y). We have
1 exp 2π
{− 12 ((y − x)2 + x2)}
Now
f (y) = = = = = =
∞ 1 1 exp {− ((y − x)2 + x2 )}dx 2 −∞ 2π ∞ 1 1 exp
2π 1 exp 2π 1 exp 2π
2
{−y /2 {−y2/2 {−y2/4
=
R2 .
Z .
|
f (x, y) f (y x)f (x) = f (y) f (y) which is Bayes theorem for continuous random-variables. Now for y
|
f (x y) =
∈
(x, y)
Z
} } }
−∞ ∞ −∞ ∞
∈ R
{− 2 (2x2 − 2xy)}dx
exp
{−[(x − y/2)2 − y2/4]}dx
exp
2
e−u
−∞ √ 1 2π exp {−y 2 /4} √ 2π 2 1 √ 2√ exp{−y 2 /4}. 2π
/2
√ 12 du
Find the PDFs:
25
So Y
∼ N (0, 2). Then: 1 2π exp
{− 12 ((y − x)2 + x2)} x ∈ R . 1 √ 2√ exp {−y2 /4} 2π
|
f (x y) =
After some calculations (exercise) one obtains:
|
f (x y) =
that is X Y = y
|
2 exp 2π
{−(x − y2 )2}
x
∈ R
∼ N (y/2, 1/2).
Example 2.4.29. Let X and Y have joint PDF:
f (x, y) =
1 x
(x, y)
∈
Z =
Then for x [0, 1]
∈
{(x, y) : 0 ≤ y ≤ x ≤ 1}.
x
f (x) =
1 dy = 1. x
0
In addition, (x, y)
∈
Z:
1/x 1 = 1 x that is if x [0, 1], then conditional upon x Y [0,x] (the uniform distribution on [0, x]). Now, 2 2 suppose we are interested in P(X + Y 1 X = x). Let x 0 , and define
|
f (y x) =
∈
∼ U
≤ | ≥ A(x) := {y ∈ R : 0 ≤ y ≤ x, x2 + y 2 ≤ 1 } that is, A(x) = [0, x ∧ (1 − x2 )1/2 ], where a ∧ b = min{a, b}. Thus ( x ∈ [0, 1]) 2 2 P(X + Y ≤ 1 |X = x) = f (y|x)dy
A(x)
x (1 x2 )1/2
=
∧ −
0
=
x
1 dy x
∧ (1 − x2)1/2 . x
To compute P(X 2 + Y 2 1), we note, by setting A = (x, y) : (x, y) interested in P(A), which is:
≤
{
2
P(X
2
+ Y
≤ 1)
∧
=
=
x
0
hence
√
P(X
+ Y
=
1
√
1/ 2
(1
− x2)1/2 dx x
− x2)1/2 dx. x
1
dx +
0
Now we have, setting x = cos(θ)
(1
| 1/ 2
≤ 1)
|
√ ≤ 1/ 2
x
2
(1
√
− x2)1/2 dx. x
1/ 2
π/4
=
sin(θ) tan(θ)dθ
0
=
≤ 1}, we are
f (y x)dyf (x)dx
A(x)
1
2
+ y 2
f (x, y)dydx
0
≤ (1 − x2)1/2 if
2 Z, x
A 1
=
Now x 2
∈
= log(1 +
√ − √ 1 2) 2
π/4
| − sin(θ) 0
log( sec(θ) + tan(θ) )
(2.4.1)
26
where we have used integration tables to obtain the integra l 3 . Thus, returning to (2.4.1): 2 P(X
+ Y 2
≤ 1) = √ 12 + log(1 + √ 2) − √ 12 = log(1 + √ 2).
Given the previous discussion, the notion of a conditional expectation for continuous random variables: E[g(Y ) X = x] = g(y)f (y x)dy.
|
|
As for discrete-valued random variables, we have the following result, which is again not proved. Theorem 2.4.30. Consider jointly continuous random variables X and Y with g, h : R
h(X ), g(Y ) continuous. Then: E[h(X )g(Y )] = E [E[g(Y )
|X ]h(X )] =
whenever the expectations exist.
|
→ R with
g(y)f (y x)dy h(x)f (x)dx
Example 2.4.31. Let us return to Example 2.4.28. Then E[XY ]
|
= =
E[Y E[X Y ]] E[Y (Y /2)]
1 [2] = 1. 2
=
Thus as Var[X ] = 1 and Var[Y ] = 2
√ 12 . Example 2.4.32. Suppose Y |X = x ∼ N (x, σ12 ) and X ∼ N (ξ, σ22 ). To find the marginal distribuρ(X, Y ) =
tion of Y , one can use MGFs and conditional expectations. We have: Yt E[e ] Yt E[E[e
M (t) = =
|X ]]
Xt+(σ12 t2 )/2 E[e ]
=
2 2
= e(σ1 t
)/2
Xt E[e ]
2 2
2 2
= e(σ1 t )/2 eξt+(σ2 t )/2 = exp ξt + (σ12 + σ22 )t2 /2 .
{
}
Here we have used Example 2.4.17 for the MGF of a normal distribution. Thus we can conclude that
∼ N (ξ, σ12 + σ22).
Y
2.4.6
Functions of Random Variables
→
R a continuous and invertible function (continuity can Consider a random variable X and g : R be weakened, but let us use it for now). Suppose Y = g(X ); what is the distribution of Y ? We have:
F (y) = P (Y ≤ y) = P (g(X ) ≤ y) = P (X ∈ g −1 ((−∞, y)) =
g −1 (
f (x)dx.
−∞,y]
⊆ R , then g −1A = {x ∈ R : g(x) ∈ A}. Example 2.4.33. Let X ∼ N (0, 1) and set x √ 12π e−u /2du Φ(x) = Note that for A
2
−∞
to denote the standard normal CDF. Suppose Y = X 2 , then for y
≥ 0 (note √ √ y). 2 P(Y ≤ y) = P (X ≤ y) = P (− y ≤ X ≤
3
you will not be expected to perform such an integration in the examination
Y = R
+
)
27
Now
−√ y ≤ X ≤ √ y)
P(
Then f (y) =
√ ≤ √ − √ − √ −
−√ ≤ −√ y) − √ −
= P(X y) P(X = Φ( y) Φ( y) = Φ( y) [1 Φ( y)] = 2Φ( y) 1.
√ √
d 1 F (y) = Φ ( y) = dy y
1 √ 2πy e−y/2 y ∈
Y
that is Y (1/2, 1/2). This is also called the chi-squared distribution on one degree of freedom. We remark that Y and X are clearly not independent (if X changes so does Y and in a deterministic manner). Now 3 E[XY ] = E [X ] = 0.
∼G
In addition E[X ] = 0 and E[Y ] = 1 . So X and Y are uncorrelated, but they are not independent. We now move onto a more general change of variable formula. Suppose X 1 and X 2 have a joint density on Z and we set (Y 1 , Y 2 ) = T (X 1 , X 2 )
→
T where T is differentiable and invertible and where T : Z function of (Y 1 , Y 2 ) on T? As T is invertible, we set
T
⊆ R2.
What is the joint density
X 1 = T 1−1 (Y 1 , Y 2 ) X 2 = T 2−1 (Y 1 , Y 2 ) Then we define J (y1 , y2 ) :=
∂T 1−1 ∂y 1 ∂T 1−1 ∂y 2
∂T 2−1 ∂y 1 ∂T 2−1 ∂y 2
=
∂T 1−1 ∂T 2−1 ∂y 1 ∂y 2
−1 ∂T −1
− ∂∂yT 21
1
∂y 2
to be the Jacobian of transformation. Then we have the following result, which simply follows from the change of variable rule for integration: Theorem 2.4.34. If (X 1 , X 2 ) have joint density f (x, y) on Z, then for (Y 1 , Y 2 ) = T (X 1 , X 2 ), with
T as described above, the joint density of (Y 1 , Y 2 ), denoted g is: g(y1 , y2 ) = f (T 1−1 (y1 , y2 ), T 2−1 (y1 , y2 )) J (y1 , y2 )
|
|
∈
(y1 , y2 )
T .
Example 2.4.35. Suppose Z = (R+ )2 and
f (x1 , x2 ) = λ 2 e−λ(x1 +x2 )
∈
(x1 , x2 )
Z .
Let (Y 1 , Y 2 ) = (X 1 + X 2 , X 1 /X 2 ).
To find the joint density of (Y 1 , Y 2 ) we first note that T = Z and that (X 1 , X 2 ) =
Then the Jacobian is J (y1 , y2 ) :=
Thus:
g(y1 , y2 ) =
Y 1 Y 2 Y 1 , . 1 + Y 2 1 + Y 2
y2 1+y2 y1 (1+y2 )2
−
1 1+y2 y1 (1+y2 )2
y1 λ2 e−λy1 2 (1 + y2 )
=
y1 − (1 + y . 2 2)
(y1 , y2 )
∈
T.
One can check that indeed Y 1 and Y 2 are independent and, that the marginal densities are: g(y1 ) = λ 2 y1 e−λy1 y1
∈ R +
g(y2 ) =
1 y2 (1 + y2 )2
∈ R +.
28 Example 2.4.36. Suppose we have (X 1 , X 2 ) as in the case of Example 2.4.35 , except we have the
mapping: (Y 1 , Y 2 ) = (X 1 , X 1 + X 2 ).
Now, clearly Y 2
≥ Y 1, so this transformation induces the support for (Y 1, Y 2) as: = {(y1 , y2 ) : 0 ≤ y1 ≤ y 2 , y2 ∈ R + }. T
Then (X 1 , X 2 ) = (Y 1 , Y 2
− Y 1)
and clearly J (y1 , y2 ) = 1; thus g(y1 , y2 ) = λ 2 e−λy2
∈
(y1 , y2 )
T .
N (0, 1) random variables ( = R 2) and let 1 − ρ2 X 2 ) ρ ∈ [ −1, 1]
Example 2.4.37. Let X 1 and X 2 be independent
− − − √ √ −
(Y 1 , Y 2 ) = (X 1 , ρX 1 +
Then T = Z and
Z
(X 1 , X 2 ) = Y 1 ,
Then the Jacobian is
J (y1 , y2 ) :=
Thus g(y1 , y2 ) = =
1
2π 2π
− − 1 1
ρ2
1
ρ2
1 0
Y 2
ρY 1
ρ 1 ρ2 1 1 ρ2
− −
.
ρ2
1
1
=
1
ρ2
.
{− 12 (y12 + (y2 − ρy1)2/(1 − ρ2)}
(y1 , y2 )
∈
{− 2(1 −1 ρ2) (y12 + y22 − 2y1y2ρ)}
(y1 , y2 )
∈
exp exp
T
T.
On inspection of Example 2.4.23, (Y 1 , Y 2 ) follow a standard bivariate normal distribution. Distributions associated to the Normal Distribution
In the following Subsection we investigate some distributions that are related to the normal distribution. They appear frequently in hypothesis testing, which is something that we will investigate in the following Chapter. We note that, in a similar way to the way in which joint distributions are defined (Definition 2.4.19), one can easily extend to joint distributions of n 2 variables (so that there is a joint distribution F (x1 , . . . , xn ) and density f (x1 , . . . , xn )). We begin with the idea of the chi-square distribution on n > 0 degrees of freedom:
≥
f (x) =
1 xn/2−1 e−x/2 n/2 (2) Γ(n/2)
2 We write X n ; you will notice that also X Table 4.2, we have that
∼ X
x
∈
X = R
+
.
∼ G (n/2, 1/2), so that E[X ] = n.
M (t) = We have the following result:
(1
−
1 2t)n/2
Note that from
1 > t. 2
Proposition 2.4.38. Let X 1 , . . . , Xn be independent and identically distributed (i.i.d.) standard
normal random variables (i.e. X i
∼ X n2.
Then Z
i.i.d.
∼ N (0, 1)). Let Z = X 12 + · ·· + X n2 .
29
Proof. We will use moment generating functions. We have Zt E[e ] 2 t n i=1 Xi ] E[e n tX 2 E[e i ]
M (t) = =
=
i=1 X2t n E[e 1 ] .
=
Here we have used the independent property on the third line and the fact that the random variables are identically distributed on the last line. Now X12 t
E[e
√
∞ 1 1 1 exp (1 2t)x21 dx1 >t 2 2 2π −∞ ∞ 1 1 2 exp u du 2 2π(1 2t)1/2 −∞ 1 . (1 2t)1/2
] =
√
= =
{−
−
}
{−
}
−
Hence
M (t) =
(1
and we conclude the result.
−
−
−
−
1 2t)n/2
1 >t 2
The standard t distribution on n degrees of freedom is (n > 0): p(x) =
Γ([n + 1]/2) x 2 −(n+1)/2 1+ nπΓ(n/2) n
√
x
∈
X = R .
∼ T n. Then we have the following important result. Proposition 2.4.39. Let X ∼ N (0, 1) and independently Y ∼ X n2 , n > 0 . Let We write X
X
T =
Y n
Then T
∼ T n.
Proof. We have f (x, y) =
√ 12π e−
1 2
1
x2
.
1
2n/2 Γ(n/2)
y n/2−1 e− 2 y
Z = R
× R+ .
We will use Theorem 2.4.34, with the transformation defined by X T =
Y n
S = Y
and then marginalize out S . The Jacobian of transformation is: J (t, s) := Then f (t, s) = Then we have for t
s n
0 = 1
√ t 2 sn
s . n
1 s √ 2nπ2n/2 sn/2−1/2 exp{− [t2 /n + 1]} 2 Γ(n/2)
∈ R
f (t) = =
1
√ 2nπ2n/2Γ(n/2) 1 √ 2nπ2n/2 Γ(n/2)
∞
{− 2s [t2/n + 1]}ds
∞
1 t2 n+1 ( 12 + 2n ) 2
=
1 √ 2nπ2n/2 Γ(n/2)
=
Γ([n + 1]/2) t 2 − 1+ n nπΓ(n/2)
√
( 12 +
∈ R × R+.
sn/2−1/2 exp
0
0
(t, s)
1 t2 n+1 2 2n ) (n+1)/2
un/2−1/2 e−u du
Γ([n + 1]/2)
30 and we conclude.
−
The last distribution that we consider is the standard F distribution on d1 , d2 > 0 degrees of freedom: d +d 1 d1 d1 /2 d1 /2−1 d 1 − 1 2 2 f (x) = x 1+ x x X = R + d2 B( d21 , d22 ) d2
where
B(
∈
Γ( d21 )Γ( d22 ) d1 d 2 , ) = . 2 2 2 Γ( d1 +d ) 2
∼ F (d ,d ). We have the following result. Proposition 2.4.40. Let X ∼ X d2 and independently Y ∼ X d2 , d 1 , d2 > 0 . Let We write X
1
2
1
2
F =
Then F
X/d1 . Y /d2
∼ F (d ,d ) . 1
2
Proof. We have f (x, y) =
1
1
1
2d1 /2 Γ(d1 /2)
xd1 /2−1 e− 2 x
1
2d2 /2 Γ(d2 /2)
y d2 /2−1 e− 2 y
Z = R
+
× R+ .
We will use Theorem 2.4.34, with the transformation defined by X/d1 Y /d2 S = Y
T =
and then marginalize out S . The Jacobian of transformation is: J (t, s) := Then f (t, s) =
d1 s d2 d1 t d2
1 d1 s d1 st d /2+d /2 2 d2 2 1 Γ(d1 /2)Γ(d2 /2) d2
0 d1 s = . 1 d2
d1 /2 1
− − d st d /2−1 −s/2 e d s e 1 2 2
2
T =
To shorten the subsequent notations, let
1 d1 g = d /2+d /2 1 2 2 Γ(d1 /2)Γ(d2 /2) d2 Then, for t
∈ R + f (t) = gt
d1 /2 1
−
∞
= = = and we conclude.
d1 +d2 2
.
−1 exp{−s( 1 [1 + d 1 t ])}ds
2 d2 1 d 1 t −(d1 +d2 )/2 gt d1 /2−1 u(d1 +d2 )/2−1 e−u ( [1 + ]) du 2 d2 0 1 d 1 t −(d1 +d2 )/2 gt d1 /2−1 ( [1 + ]) Γ((d1 + d2 )/2) 2 d2 d +d d1 +d2 d 1 − 1 2 2 d1 /2−1 2 gΓ((d1 + d2 )/2)2 t 1+ t d2 d +d 1 d1 d1 /2 d1 /2−1 d 1 − 1 2 2 t 1+ t d2 B( d21 , d22 ) d2 0
=
s
d1 /2
∞
( R+ ) 2 .
31
2.5
Convergence of Random Variables
In the following Section we provide a brief introduction to convergence of random variable and in particular two modes of convergence:
• Convergence in Distribution • Convergence in Probability These properties are rather important in probability and mathematical statistics and provide a way to justify many statistical and numerical procedures. The properties are rather loosely associated to the notion of a sequence of random variables X 1 , X 2 , . . . , Xn (there will be infinitely many of them typically) and we will be concerned with the idea of what happens to the distribution or some functional as n . The second part of this Section will focus on perhaps the most important result in probability: the central limit theorem. This provides a characterization of a random variable:
→∞
n
√
1 S n = n
X i
i=1
with X 1 , . . . , Xn independent and identically distributed with zero mean and unit variance. This particular result is very useful in hypothesis testing, which we shall see later on. Throughout, we will focus on continuous random variables, but one can extend this notion.
2.5.1
Convergence in Distribution
Consider a sequence of random variables X 1 , X 2 , . . . , with associated distribution functions F 1 , F 2 , . . . , we then have the following definition: Definition 2.5.1. We say that the sequence of distribution functions F 1 , F 2 , . . . converges to a
distribution function F , if limn→∞ F n (x) = F (x) at each point where F is continuous. If X has d distribution function F , we say that X n converges in distribution to X and write X n X .
→
Example 2.5.2. Consider a sequence of random variables X n
F n (x) = 1
Then for any fixed x
∈ R +
− − x n
1
lim F n (x) = 1
n
→∞
Now if X
∼ E (1):
x
F (x) =
n
[0, n], n
≥ 1, with
X n .
e−u du = 1
− e−x.
→d X , X ∼ E (1).
Example 2.5.3. Consider a sequence of random variables X n
F n (x) = 1
. − sin(2nπx) 2nπ
Then for any fixed x [0, 1]
∈
lim F n (x) = x.
n
→∞
Now if X
∼ U [0,1] , x ∈ [0, 1]:
x
F (x) =
0
So X n
X n =
− e−x.
0
So X n
∈
x
∈
→d X , X ∼ U [0,1] .
du = x.
∈
X =
[0, 1], n
≥ 1, with
32
2.5.2
Convergence in Probability
We now consider an alternative notion of convergence for a sequence of random variables X 1 , X 2 , . . . Definition 2.5.4. We say that X n converges in probability to a constant c
for every > 0
P
∈ R (written X n → c) if
P →∞ (|X n − c| > ) = 0.
lim
n
We note that convergence in probability can be extended to convergence to a random variable X , but we do not do this, as it is beyond the level of this course. We have the following useful result, which we do not prove. Theorem 2.5.5. Consider a sequence of random variables X 1 , X 2 , . . . , with associated distribution
functions F 1 , F 2 , . . . . If for every x
∈
X n
lim F n (x) = c
n
then X n
→∞
P
→ c.
Example 2.5.6. Consider a sequence of random variables X n
F n (x) =
Then for any fixed x
∈
x 1 + x
n
∈
X = R
+
, n
≥ 1, with
.
X
lim F n (x) = 0.
n
→∞
Thus X n
P
→ 0.
We finish the Section with a rather important result, which we again do not prove. It is called the weak law of large numbers: Theorem 2.5.7. Let X 1 , X 2 , . . . be a sequence of independent and identically distributed random
variables with E[ X 1 ] <
| | ∞. Then
n
1 n
X i
i=1
P
→ E [X 1].
Note that this result extends to functions; i.e. if g : R 1 n
→ R with E[|g(X 1)|] < ∞ then
n
g(X i )
i=1
P
→ E [g(X 1)].
Example 2.5.8. Consider an integral
I =
g(x)dx
R
where we will suppose that g(x) = 0 . Suppose we cannot calculate I . Consider any non-zero pdf f (x) on R. Then g(x) I = f (x)dx R f (x)
and suppose that
R
g(x) f (x)dx < f (x)
∞.
Then by the weak law of large numbers, if X 1 , . . . , Xn are i.i.d. with pdf f then 1 n
n
i=1
g(X i ) f (X i )
P
→ I .
This provides a justification for a numerical method (called the Monte Carlo method) to approximate integrals.
33
2.5.3
Central Limit Theorem
We close the Section and Chapter with the central limit theorem (CLT) which we state without proof. Theorem 2.5.9. Let X 1 , X 2 , . . . be a sequence of independent and identically distributed random
variables with E[ X 1 ] <
| | ∞ and 0 < V ar[X 1] < ∞. Then √ n 1 n (X i − E[X 1]) →d Z n Var[X ]
1
i=1
where Z
∼ N (0, 1).
n
The CLT asserts that the distribution of the summation i=1 X i suitably normalized and centered, will converge to that of a normal distribution. An often used idea, via the CLT, is normal approximation. That is, for n ‘large’
− ∼ N
√ n where
1 n
n
(X i
Var[X 1 ]
i=1
·
(0, 1)
∼· means ‘approximately distributed as’, so we have 4, n
X i
i=1
∼· N (nE[X 1], nVar[X 1]).
Example 2.5.10. Let X 1 , . . . , Xn be i.i.d.
Z n =
E[X 1 ])
n i=1
X i is for b > t:
G (a/n,b) random variables.
n
Z n t
M (t) = E [e
]=
X t E[e i ]
= E [ex1 ]n =
i=1
Then the distribution of
− b
b
a/n n
t
where we have used the i.i.d. property and the MGF of a Gamma random variable (see Table 4.2 ). Thus Z n (a, b) for any n 1 . However, from the CLT one can reasonably approximate Z n , when n is large by a normal random variable with mean
∼ G
≥
n
and variance n
4
a a = bn b
a a = . nb2 b2
If Z ∼ N (0, 1) you can verify that if X = σZ + µ then X ∼ N (µ, σ2 )
Chapter 3
Introduction to Statistics 3.1
Introduction
In this final Chapter, we give a very brief introduction to statistical ideas. Here the notion is that one has observed data from some sampling distribution ‘the population distribution’ and one wishes to infer what the properties of the ‘population’ are on the basis of observed samples. In particular we will be interested in estimating the parameters of sampling distributions such as the maximum likelihood method (Section 3.2) as well as testing hypotheses about parameters ( 3.3). We end the Chapter with an introduction to Bayesian statistics (Section 3.4) which is another alternative way to estimate parameters, which is more complex, but much richer than the MLE method.
3.2 3.2.1
Maximum Likelihood Estimation Introduction
So far in this course, we have proceeded to discuss pmfs and pdfs such as (λ) or (µ, σ 2 ), but we have not discussed what the parameters λ or µ might be. We discuss in this Section, a particularly important way to estimate parameters called maximum likelihood estimation (MLE). The basic idea is this. Suppose one observes data x1 , x2 , . . . , xn and we hypothesize that they follow some joint distribution F θ (x1 , . . . , xn ), θ Θ (Θ is the parameter space e.g. Θ = R ). Then the idea of maximum likelihood estimation is to find the parameter which maximizes the joint pmf/pdf of the data, that is: θˆn = argmaxθ∈Θ f θ (x1 , . . . , xn ).
P
N
∈
3.2.2
The Method
Throughout this section, we assume that X 1 , . . . , Xn are mutually independent. So, we have X i F θ , and the joint pmf/pdf is:
i.i.d.
∼
n
f θ (x1 , . . . , xn ) = f θ (x1 )
× f θ (x2) × · · · × f θ (xn) =
f θ (xi ).
i=1
We call f θ (x1 , . . . , xn ) the likelihood of the data. As maximizing a function is equivalent to maximizing a monotonic increasing transformation of the function, we often work with the log-likelihood :
n
lθ (x1 , . . . , xn ) = log f θ (x1 , . . . , xn ) =
log f θ (xi ) .
i=1
If Θ is some continuous space (as it generally is for our examples) and θ = (θ1 , . . . , θd ), then we can compute the gradient vector:
∇lθ (x1, . . . , xn) = ∂lθ (x1∂θ, . 1. . , xn) , . . . , ∂lθ (x1∂θ, . d. . , xn) and we would like to solve, for θ ∈ Θ (below 0 is the d −dimensional vector of zeros) ∇lθ (x1, . . . , xn) = 0.
(3.2.1)
35 The solution of this equation (assuming it exists) is a maximum if the hessian matrix is negative definite: 2 2 2 H (θ) :=
∂ lθ (x1 ,...,xn ) ∂θ 12
∂ lθ (x1 ,...,x n ) ∂θ 1 ∂θ 2
.. .
.. .
2
.. .
2
∂ lθ (x1 ,...,xn ) ∂θ d ∂θ 1
∂ lθ (x1 ,...,x n ) ∂θ d ∂θ 2
| −
∂ lθ (x1 ,...,xn ) ∂θ 1 ∂θ d
· ··
.. .
2
∂ lθ (x1 ,...,xn ) 2 ∂θ d
· ··
|
×
.
If the d numbers λ1 , . . . , λd which solve λI d H (θ) = 0, with I d the d d identity matrix, are all negative, then θ is a local maximum of l θ (x1 , . . . , xn ). If d = 1 then this just boils down to checking whether the second derivative of the log-likelihood is negative at the solution of (3.2.1). Thus in summary, the approach we employ is as follows: 1. Compute the likelihood f θ (x1 , . . . , xn ). 2. Compute the log-likelihood l θ (x1 , . . . , xn ) and its gradient vector
∇
∈ Θ, call this solution θn (we are assuming there
3. Solve lθ (x1 , . . . , xn ) = 0, with respect to θ is only one θn ).
∇lθ (x1, . . . , xn).
4. If H (θn ) is negative definite, then θˆn = θn .
In general, point 3. may not be possible analytically (so for example, one can use Newton’s method). However, you will not be asked to solve lθ (x1 , . . . , xn ) = 0, unless there is an analytic solution.
3.2.3
∇
Examples of Computing the MLE
Example 3.2.1. Let X 1 , . . . , Xn be i.i.d.
(λ) random variables. Let us compute the MLE of λ = θ , given observations x1 , . . . , xn . First, we have
P
n
f λ (x1 , . . . , xn ) =
λxi −λ e x! i=1 i n
= e−nλλ
i=1 xi
Second the log-likelihood is:
1
. n i=1 xi !
n
lλ (x1 , . . . , xn ) = log(f λ (x1 , . . . , xn )) =
−nλ +
xi log(λ)
i=1
n
− log(
xi !).
i=1
The gradient vector is a derivative: dlλ (x1 , . . . , xn ) = dλ
1 n + λ
−
Thirdly
xi .
i=1
n
xi = 0
i=1
1 λn = n
Fourthly,
n
− − 1 n + λ
so,
d2 lλ (x1 , . . . , xn ) = dλ2
n
xi .
i=1
1 λ2
n
xi < 0
i=1
for any λ > 0 (assuming there is at least one i such that xi > 0 ). Thus assuming there is at least one i such that xi > 0 n 1 ˆ λn = xi . n i=1
36 ˆ n as in Example 3.2.1, we note that Remark 3.2.2. As a theoretical justification of λ 1 n
n
xi
i=1
will converge in probability (see Theorem 2.5.7 ) to E[X 1 ] = λ if our assumptions hold true. That is, we recover the true parameter value; such a property is called consistency - we do not address this issue further. Example 3.2.3. Let X 1 , . . . , Xn be i.i.d.
(λ) random variables. Let us compute the MLE of λ = θ , given observations x1 , . . . , xn . First, we have
E
n
f λ (x1 , . . . , xn ) =
λe−λxi
i=1 n
= λn exp
{−λ
}
xi .
i=1
Second the log-likelihood is: n
lλ (x1 , . . . , xn ) = log(f λ (x1 , . . . , xn )) = n log(λ)
−λ
xi .
i=1
The gradient vector is a derivative: n
−
dlλ (x1 , . . . , xn ) n = dλ λ
Thirdly
n
− n λ
so
xi = 0
i=1
1 n
λn = (
Fourthly,
xi .
i=1
n
xi )−1 .
i=1
d2 lλ (x1 , . . . , xn ) = dλ2
− nλ2 < 0.
Thus ˆ n = ( 1 λ n Example 3.2.4. Let X 1 , . . . , Xn be i.i.d. 2
n
xi )−1 .
i=1
N (µ, σ2) random variables. Let us compute the MLE of
(µ, σ ) = θ , given observations x1 , . . . , xn . First, we have n
f θ (x1 , . . . , xn ) =
√ √ i=1
=
1
2πσ 2
1
{− 2σ1 2 (xi − µ)2}
exp
{− 2σ1 2
n
exp
2πσ 2
n
(xi
i=1
− µ)2}.
Second the log-likelihood is: lθ (x1 , . . . , xn ) = log(f θ (x1 , . . . , xn )) =
−
n log(2πσ 2 ) 2
−
1 2σ 2
n
(xi
i=1
− µ)2.
The gradient vector is
∂l θ (x1 , . . . , xn ) ∂lθ (x1 , . . . , xn ) , = ∂µ ∂σ 2
1 σ2
n
(xi
i=1
− µ), −
n 1 + 2 2σ 2(σ2 )2
n
(xi
i=1
2
− µ)
.
37
Thirdly, we must solve the equations n
− − − − 1 σ2
− 2σn2 + 2(σ12)2
(xi
µ) = 0
(3.2.2)
i=1 n
µ)2
(xi
= 0
(3.2.3)
i=1
simultaneously for µ and σ 2 . Since (3.2.2) can be solved independently of (3.2.3), we have: 1 µn = n
n
xi .
i=1
Thus substituting into (3.2.3), we have
n
1 2(σ 2 )2
that is:
i=1
σn2 =
Fourthly the hessian matrix is:
− − H (θ) =
n σ2 n i=1 (xi
1 (σ 2 )2
µn )2 =
(xi
1 n
n 2σ 2
n
µn )2 .
(xi
i=1
− (σ1) ni=1n(xi − µ) n 1 2 i=1 (xi − µ) 2(σ ) − (σ ) 2 2
− µ)
2 2
2 3
.
|
Note that when θn = (µn , σn2 ) the off-diagonal elements are exactly 0, so when solving λI 2 H (θn ) = 0, we simply need to show that the diagonal elements are negative. Clearly for the first diagonal element n < 0 σn2 if (xi µn )2 > 0 for at least one i (we do not allow the case σn2 , in that we assume this doesn’t occur). In addition
−
Thus
3.3.1
n
− − − −
n 2(σn2 )2
− (σ12 )3 n
µn )2 =
(xi
i=1
θˆn =
3.3
−
| −
1 n
n
1 xi , n i=1
n 2(σn2 )2
n
xi
i=1
n = (σn2 )2
1 n
n
xj
2
n < 0. 2(σn2 )2
.
j=1
Hypothesis Testing Introduction
The theory of hypothesis testing is concerned with the problem of determining whether or not a statistical hypothesis, that is, a statement about the probability distribution of the data, is consistent with the available sample evidence. The particular hypothesis to be tested is called the null hypothesis and is denoted by H 0 . The ultimate goal is to accept or reject H 0 . In addition to the null hypothesis H 0 , one may also be interested in a particular set of deviations from H 0 , called the alternative hypothesis and denoted by H 1 . Usually, the null and the alternative hypotheses are not on an equal footing: H 0 is clearly specified and of intrinsic interest, whereas H 1 serves only to indicate what types of departure from H 0 are of interest. A statistical test of a null hypothesis H 0 is typically based on three elements: 1. a statistic T , called a test statistic 2. a partition of the possible values of T into two distinct regions: the set K of values of T that are regarded as inconsistent with H 0 , called the critical or rejection region of the test, and its complement K c , called the non-rejection region 3. a decision rule that rejects the null hypothesis H 0 as inconsistent with the data if the observed value of T falls in the rejection region K , and does not reject H 0 if the observed value of T belongs instead to K c .
38
3.3.2
Constructing Test Statistics
In order to construct test-statistics, we start with an important result, which we do not prove. Note that all of our results are associated to normally distributed data. Theorem 3.3.1. Let X 1 , . . . , Xn be i.i.d.
¯n X
N (µ, σ2) random variables. Define 1 n
:=
s2n
n
n
n
− 1
:=
X i
i=1
1
(X i
i=1
− X ¯n)2
Then ¯n 1. X
− µ ∼ N (0, σ2/n). 2. (n − 1)s2n /σ2 ∼ X n2−1 . ¯ n − µ and (n − 1)s2 /σ 2 are independent random variables. 3. X n We note then, that by Proposition 2.4.39, it follows that T (µ) :=
¯ n (X
¯ n − µ) − µ)/(σ/√ n) = (X ∼ T n−1.
s2n /σ2
s2n /n
Note that as one might imagine Proposition 2.4.38 is used to prove Theorem 3.3.1. Note that one can show that the n−1 distribution has a symmetric pdf, around 0. Now consider testing H 0 : µ = µ 0 against the two-sided alternative H 1 : µ = µ 0 (that is, we are testing whether the population mean, the ‘true’ mean of the data, is a particular value). Now we know for any µ R that T (µ) n−1 , thus if the null hypothesis is true T (µ0 ) should be a random variable that is consistent with a n−1 random variable. To construct the rejection region of the test, we must choose a confidence level, typically 95%. Then we want T (µ0 ) to lie in 95% of the probability. However, this still does not tell us what the rejection region is; this is informed by the alternative hypothesis, that H 1 : µ = µ 0 ; which indicates that a value which is inconsistent with the n−1 distribution lies in each ‘tail’ of the distribution. Thus the procedure is:
T −
∈
∼ T T
T −
1. Compute T (µ0 ). 2. Decide upon your confidence level (1
−
− α)%. This defines the rejection region.
−
3. Compute the t values ( t, t). These are the numbers in R such that the probability (under a t is α/2. n−1 random variable) of exceeding t is α/2 and the probability of being less than
T − − 4. If − t < T (µ0 ) < t then we do not reject the null hypothesis, otherwise we reject the null hypothesis.
A number of remarks are in order. First, a test can only disprove a null hypothesis. The fact that we do not reject the null on the basis of the sample evidence does not mean that the hypothesis is true. Second, it is useful to distinguish between two types of errors that can be made:
• Type I error: Reject H 0 when it is true. • Type II error: Do not reject H 0 when it is false. In statistics, Type I error is usually regarded as the most serious. The analogy is with the judiciary system, where condemning an innocent is typically considered a much more serious problem than letting a guilty person free. A second test statistic we consider is for testing whether two samples have the same variance. Let i.i.d. 2 X 1 , . . . , Xn and Y 1 , . . . , Ym be independent sequences of random variables where X i (µX , σX ) and Y i
i.i.d.
2 (µY , σY ).
∼ N
Now we know from Theorem 3.3.1 that (n
− 1)s2X,n /σX2 ∼ X n2−1
∼ N
39 and independently:
− 1)s2Y,m /σY 2 ∼ X m2 −1.
(m
2 2 2 2 Now suppose that we want to test H 0 : σX = σY against H 1 : σX = σY . Now if H 0 is true, by Theorem 3.3.1 and Proposition 2.4.40.
2 F (σX )
:= (n
−
2 1)s2X,n /σX /(n
− 1)
(m
− 1)s2Y,m /σX2 /(m − 1) = s2X,n /s2Y,m ∼ F n−1,m−1
Thus, we perform the following procedure: 2 1. Compute F (σX ).
2. Decide upon your confidence level (1
− α)%. This defines the rejection region.
3. Compute the F values (f , f ). These are the numbers in R+ such that the probability (under a n−1,m−1 random variable) of exceeding f is α/2 and the probability of being less than f is α/2.
F
−
−
2 4. If f < F (σX ) < f then we do not reject the null hypothesis, otherwise we reject the null hypothesis.
Remark 3.3.2. All of our results concern Normal samples. However one can use the CLT (Theorem
2.5.9 ) to extend these tests to non-normal data. We do not follow this idea in this course.
3.4 3.4.1
Bayesian Inference Introduction
So far we have considered point estimation methods for the unknown parameter. In the following Section, we consider the idea of Bayesian inference for unknown parameters. Recall, from Examples 2.3.34 and 2.4.28, that there is a Bayes theorem for discrete and continuous random variables. Indeed, there is version of Bayes theorem that can mix continuous and discrete random variables. Let X and Y be two jointly defined random variables such that X or Y are either continuous or discrete. Then Bayes theorem is: f (y x)f (x) f (x y) = f (y)
|
|
as long as the associated pmfs/pdfs are well defined. Throughout the section, we assume that the pmfs/pdfs are always positive on their supports.
3.4.2
Bayesian Estimation
Throughout, we will assume that we will have a random sample X 1 , X 2 , . . . , Xn which are conditionally independent, given a parameter θ Θ. As we will see, in Bayesian statistics, the parameter i.i.d. is a random variable. So, we assume X i θ F ( θ) where F ( θ) is assumed to be a distribution function for any θ Θ. Thus we have that the joint pmf/pdf of X 1 , . . . , Xn θ is:
∈
∈
| ∼
·|
·|
|
n
|
f (x1 , . . . , xn θ) =
|
f (xi θ).
i=1
The main key behind Bayesian statistics is the choice of a prior probability distribution for the parameter θ Θ. That is, Bayesian statisticians specify a probability distribution on the parameter θ before the data are observed. This probability distribution is supposed to reflect the information one might have before seeing the observations. To make this idea concrete, consider the following example:
∈
Example 3.4.1. Suppose that we will observe data X i θ
i.i.d.
| ∼ E (λ). Then one has to construct a + probability distribution on λ ∈ R . A possible candidate is λ ∼ G (a, b). If one has some prior beliefs about the mean and variance of λ (say E[λ] = 10, are.
Var[λ]
= 10) then one determine what a and b
40 Throughout, we will write the prior pmf/pdf as π(θ). Now the way in which Bayesian inference works is to update the prior beliefs on θ via the posterior pmf/pdf. That is, ‘in the light of the data’ the distributional properties of the prior are updated. This is achieved by Bayes theorem; the posterior pmf/pdf is:
|
f (x1 , . . . , xn θ)π(θ) f (x1 , . . . , xn )
|
π(θ x1 , . . . , xn ) = where f (x1 , . . . , xn ) =
|
f (x1 , . . . , xn θ)π(θ)dθ
Θ
if θ is continuous and, if θ is discrete: f (x1 , . . . , xn ) =
|
f (x1 , . . . , xn θ)π(θ).
θ Θ
∈
For a Bayesian statistician, the posterior is the ‘final answer’, in that all statistical inference should be associated to the posterior. For example, if one is interested in estimating θ then one can use the posterior mean:
|x1, . . . , xn] =
E[Θ
∈
|
θπ(θ x1 , . . . , xn )dθ.
Θ
R (or indeed any univariate component of θ), then we can compute a In addition, consider θ confidence interval, which is called an credible interval in Bayesian statistics. The highest 95%posterior-credible (HPC) interval is the shortest region [ θ, θ], such that
θ
|
π(θ x1 , . . . , xn )dθ = 0.95.
θ
| − |
By shortest, we mean the [θ, θ], such that θ θ is smallest. It should be clear by now, that the posterior distribution is much ‘richer’ than the MLE, in the sense that one now has a whole distribution which reflects the parameter, instead of a point estimate. What also might be apparent now, is the fact that the posterior is perhaps difficult to calculate. For example the posterior mean will require you to compute two integrations over Θ (in general) and this may not be easy to calculate in practice. In addition, it could b e difficult, analytically to calculate a HPC. As a result, most Bayesian inference is done via numerical methods, based on Monte Carlo (see Example 2.5.8); we do not mention this further.
3.4.3
Examples
Despite the fact that Bayesian inference is very challenging, there are still many examples where one can do analytical calculations. Example 3.4.2. Let us consider Example 3.4.1. Here we have that
f (x1 , . . . , xn ) =
∞
n
} {−
n
λ exp
0
a
=
b Γ(a) a
= =
So, as:
b Γ(a) [ ba Γ(a) [
xi
i=1
ba a−1 −bλ λ e dλ Γ(a)
λn+a−1 exp
{−λ[
1 n i=1 xi + b] 1 n i=1 xi + b]
we have:
|
π(λ x1 , . . . , xn ) =
λn+a−1 exp [
n 1 i=1 x i +b]
}
xi + b] dλ
i=1
∞
n+a
un+a−1 e−u du
0
n+a
ba n+a−1 f (x1 , . . . , xn λ)π(λ) = λ exp Γ(a)
|
n
{− {−
∞
0
λ
Γ(n + a). n
λ[
i=1
n i=1 n+a
λ[
}
xi + b]
xi + b]
Γ(n + a)
}
41
i.e.
n
|
∼ G (n + a, b +
λ x1 , . . . , xn
xi ).
i=1
Thus, the posterior distribution on λ is in the same family as the prior, except, with updated parameters, reflecting the data. So, for example:
|x1, . . . , xn] = b +n + a . n xi
E[Λ
i=1
In comparison to the MLE in Example 3.2.3 , we see that the posterior mean and MLE correspond as a, b 0 .
→
Example 3.4.3. Let X i λ
i.i.d.
| ∼ P (λ), i ∈ {1, . . . , n}. Suppose the prior on λ is G (a, b). Then ∞ 1 ba a−1 −bλ x f (x1 , . . . , xn ) = λ exp{−nλ} n λ e dλ xi ! Γ(a)
n i=1
0
=
{− {− {− } ∼ G ∞
a
b
Γ(a)
n i=1 xi ! a
b
=
i
Γ(a)
i=1
n
i=1 xi +a
λ
0
n
1 n i=1 xi ! [n + b]
=
Γ(a)
So as:
|
f (x1 , . . . , xn λ)π(λ) =
we have:
|
π(λ x1 , . . . , xn ) =
λ
i=1 xi +a
−1 e−u du
n
Γ(
xi + a).
i=1
n
i=1 xi +a
λ n i=1 xi !
n
−1 exp
i=1 xi +a
1 [n+b]
i.e.
i=1 xi +a
ba
Γ(a)
n
u
0
n
1 n i=1 xi ! [n + b]
}
λ[n + b] dλ
∞
i=1 xi +a
a
b
−1 exp
−1 exp
n
i=1 x i +a
}
λ[n + b]
λ[n + b] n i=1
Γ(
xi + a)
n
|
λ x1 , . . . , xn
xi + a, n + b .
i=1
Thus, the posterior distribution on λ is in the same family as the prior, except, with updated parameters, reflecting the data. So, for example:
|x1, . . . , xn] =
E[Λ
n i=1 xi + a
n + b
.
In comparison to the MLE in Example 3.2.1, we see that the posterior mean and MLE correspond as a, b 0 .
→
Example 3.4.4. Let X i µ
i.i.d.
| ∼ N (µ, 1), i ∈ {1, . . . , n}. Suppose the prior on µ is N (ξ, κ). Then
f (x1 , . . . , xn ) =
√ {− − √ √ {− ∞
1 2π
−∞
=
1 2π
n
1 2
exp
n
i=1
∞
1 exp 2πκ −∞
n
µ)2
(xi 1 2
1 1 exp{− (µ − ξ )2 }dµ } √ 2πκ 2κ
n
(xi
i=1
− µ)2 − 2κ1 (µ − ξ )2}dµ.
To compute the integral, let us first manipulate the exponent inside the integral: n
− 1 2
(xi
i=1
2
− µ) −
Let c(ξ,κ,x1 , . . . , xn ) = n
− 1 2
(xi
i=1
2
− µ) −
1 (µ 2κ 2
− 12 ξκ +
1 (µ 2κ
2
− ξ )
− ξ )
= =
2
n i=1
− −
=
−
1 2 1 [µ (n + ) 2 κ
−
ξ 2µ( + κ
n
n
ξ 2 xi ) + + x2i ] κ i=1 i=1
x2i , then 1 2 1 [µ (n + ) 2 κ
−
−
1 1 (n + ) 2 κ
µ
n
ξ 2µ( + κ
xi )] + c(ξ,κ,x1 , . . . , xn )
i=1 n i=1 xi ) 2 ) n + κ1
( κξ +
n
−
( ξ + i=1 xi ) ( κ n + κ1
2
+ c(ξ,κ,x1 , . . . , xn ).
42
Let c (ξ,κ,x1 , . . . , xn ) = c(ξ,κ,x1 , . . . , xn ) + 21 (n + κ1 )( f (x1 , . . . , xn ) = =
√ √ √ √ 1 2π
n
1 2π
n
ξ (κ + n i=1 xi ) 1 n+ κ
1 exp c (ξ,κ,x1 , . . . , xn ) 2πκ
{
}
∞
2
, then we have
{−
exp
−∞
√
−
1 1 (n + ) µ 2 κ
1 2π exp c (ξ,κ,x1 , . . . , xn ) . (n + κ1 )1/2 2πκ
{
}
( κξ + ni=1 xi ) 2 ) dµ n + κ1
}
So as:
|
f (x1 , . . . , xn µ)π(µ) =
√ √ 1 2π
n
1 exp c (ξ,κ,x1 , . . . , xn ) exp 2πκ
{
} {−
we have
{−
exp
|
π(µ x1 , . . . , xn ) =
1 1 2 (n + κ )
− µ
√ 2π
−
( κξ + ni=1 xi ) 2 1 1 (n+ ) µ ) 2 κ n + κ1
ξ (κ + n i=1 xi ) 2 ) 1 n+ κ
}
}
1 1/2 (n+ κ )
i.e.
|
µ x1 , . . . , xn
∼ N
n
( κξ + i=1 xi ) 1 , (n + )−1 1 κ n + κ
Thus, the posterior distribution on µ is in the same family as the prior, except, with updated parameters, reflecting the data. So, for example:
( κξ + ni=1 xi ) E[Ξ x1 , . . . , xn ] = . n + κ1
|
In comparison to the MLE in Example 3.2.4, we see that the posterior mean and MLE correspond as κ , for any fixed ξ R .
→ ∞
∈
We note that in all of our examples the posterior is in the same family as the prior. This is not by chance; for every member of the exponential family, with parameter θ it is often possible to find a prior which obeys this former property. Such priors are called conjugate priors.
Chapter 4
Miscellaneous Results In the following Chapter we quote some results of use in the course. We do not cover this material in lectures and is there for your convenience and revision. The following Sections give general facts which should be taken as true. You can easily find the theory behind each of these results in any undergraduate text in maths or statistics.
4.1
Set Theory
∪ {
{ }
{ } }
Recall that A B means all the elements in A or all the elements of B . E.g. A = 1, 2 , B = 3, 4 then A B = 1, 2, 3, 4 . In addition that A B means all the elements in A and B . E.g. A = 1, 2, 3 , B = 3, 4 then A B = 3 . Finally recall Ac is the complement of the set A, that is everything in Ω that is not in A, e.g. Ω = 1, 2, 3 and A = 1, 2 , then A c = 3 . Let A, B , C be sets in some state-space Ω. Then the following holds true:
∪
{ }
}
∩
{ }
∩
{
}
∪ B = B ∪ A A ∪ (B ∪ C ) = (A ∪ B) ∪ (A ∪ C ) A
{ }
A A
{}
∩ B = B ∩ A ASSOCIATIVITY ∩ (B ∩ C ) = (A ∩ B) ∩ (A ∩ C ) DISTRIBUTIVITY.
Note that the second rule holds when mixing intersection and union, e.g.: A
∪ (B ∩ C ) = (A ∪ B) ∩ (A ∪ C ).
We also have De-Morgan’s laws: (A
∪ B)c = Ac ∩ Bc
A neat trick for probability: A = (A
4.2
(A
∩ B)c = Ac ∪ B c.
∩ B) ∪ (A ∩ Bc).
Summation 1 1
−z e
z
=
∞
|| ∈ ∈ ∈ || − | | zk
k=0
=
k=0 n
(1 + z)n
=
k=0 n
(1 + z)−n
=
k=0
− log(1 − z)
z < 1 GEOMETRIC
∞ zk
=
log(1 + z) =
z
k!
n k z k
∞
Z
n + k k z k
n
+
BINOMIAL Z
+
||
, z < 1 NEGATIVE BINOMIAL
z < 1 LOGARITHMIC
k
k+1 z
( 1)
k=1
EXPONENTIAL
n
∞ zk
k=1
R
k
k
{
z < 1 LOGARITHMIC
44
| |
Note that, suppose z < 1, then we have n
− zn−j+1 , 1−z
1 z k = z j
k=j
4.3 For x
Exponential Function
∈ R +: lim
n
→∞
lim
n
→∞
4.4
− x n x n
1+ 1
n
= ex
n
= e−x .
Taylor Series
Under assumptions that are in effect in this course, for infinitely differentiable f : R f (x) =
∞ (x − c)k
k!
k=0
4.5
f (k) (c) c
→ R :
∈ R .
Integration Methods
Some ideas for integration:
d [f (g(x))] = g (x)f (g(x)) dx
g (x)f (g(x))dx = f (g(x)) + C.
The methods of ‘integration by parts’ and ‘integration by substitution’; you should have covered these prior to your university education. Consider the following integration ‘trick’. Suppose you want to compute
g(x)dx
where g is a positive function. Suppose that there is a PDF f (x) such that f (x) = cg(x) where you know c. Then
g(x)dx =
Recall that
∞
I =
1 1 f (x)dx = . c c
2
e−u
/2
du =
√
2π.
−∞
This can be established by the fact that I 2 =
∞
∞
2
e−u
/2 v 2 /2
−
dudv
−∞ −∞
and then changing to polar co-ordinates.
4.6
Distributions
A Table of discrete distributions can be found in Table 4.1 and continuous Distributions in Table 4.2. Recall that ∞ Γ(a) =
0
ta−1 e−t dt a > 0.
45
∈
Note that one can show Γ(a + 1) = aΓ(a) and for a Z+ , Γ(a) = (a a,b > 0 Γ(a)Γ(b) B(a, b) = . Γ(a + b) One can show also that
1
B(a, b) =
0
ua−1 (1
− u)b−1du.
− 1)!.
In addition that for
46
n ) } t ) n 1 e e . p p − ) p p + t ) + p − e − ) ) − e 1 p ( 1 ( 1 p p λ e ( e F − { p − = G − 1 p e 1 − 1 ( q M ( ( x e 1 t a h 2 t p 2 ) e p / t p ) ] ) o ) p p − / p − N X − 1 . [ s r 1 ( − ( 1 n a ( p 1 o i V p n λ ( n t u ] b i r p p X p [ / / t s i E p n λ 1 n D e x t q e F r c − D s i C 1 D f n o p e x n l b − − x n a ) x ) p p T − p 1 : 1 ) − − 1 − . p 1 x 4 ) 1 ( ( − p l e F 1 xp − ! 1 1 b − e ( x − − a M x x n n x 1 T P p λ ( t
t
t
t
t
λ
x
+
Z
∈
+
Z
∈
n n , , ) ) ) ) 1 1 1 1 , , + , , 0 ( 0 R ( 0 ( 0 ( . r ∈ ∈ ∈ ∈ ∈ a P p p λ p p
} }
. n , . . } . t , . . . r 2 . o } . , , , p 1 1 1 2 p , , , , u 0 0 0 1 S { { { { X
) ) p p ) , , ) p λ n 1 ( ( ( ( e
} .
. . , 1 + n , n
{
) p , n ( e
B B P G N
47
. ] ) b ( Γ ) a ( Γ [ / ) b
2
t a
)
/
t ) b F e a − t b t + − G b − t ( λ − b a M e t λ ( e a
2
t b
) 1 +
+ a ( Γ =
2 1 b / + b a ] 2 1 ) a ( − a X ) ) [ b 2 b r − 2 b λ , + a b / / a a V ( 1 a b ( ( B t ) 2 b a / h ) t + b ] e a t b ( X + λ / [ o / / a E ( 1 a a a N . s x n λ o i − t F a a e u b D − − − i r C x b 1 t s i 1 D − s b u ) o x u n − i t 1 ( n ) 1 o x a − b C − x a − ( x f e o 1 1 − e − − ) l x a e b b x , b a F a λ ) π a T a 1 2 ( D 1 − − e b ( : P b λ Γ √ B 2 . 4 e l ∞ b + a < R 2 T ) b × + < R a + R ( 2
2
b
1 2
a
+
R ∈ ∈
< R ∈ ) ) . b b r , , a ∞ ∈ b , a a P − λ a ( ( X
t r o ] p ] 1 p b , , + + u a 0 S [ R R R [ ) ) b ) b , b , ] , a ( a b ) , a λ ( a ( ( e [
U E G N B
