Question abou the derivation of mean and variance of the sample average

[dadalee1102]

Hi, David,
I got a question about the derivation of mean and variance of the sample average. In order to derive them, we need to obtain two I.I.D random variables, such as X1 and X2 from a population. However, why the mean and variance of X1 and X2 are the same ? The text said because they are I.I.D, but I still don't understand why.

Thank you for your help.

 

[David Harper CFA FRM]

Hi hsintalee,

X1 and X2, or sometimes X(1) and X(2), typically would refer to the OUTCOMES (aka, sample points) ... i think of them as "draws" or instances of the random variable. So, consider a single six-sided die. The random variable might be "D" and it is characterized by a discrete uniform distribution: p[D(i) = 1] = 1/6, p[D(i) = 2] = 1/6, etc ...

D(1) or D1 is just the outcome (sample point) of rolling the dice once, maybe D1 = 3
D(2) or D2 is the next roll, maybe D2 = 5

The definition of a random sample is that the draws (sample points) are i.i.d.. That has two parts, independent and identical. It means that, if we roll two die together, like a craps game, this is indeed a random sample. Each roll is drawn from the IDENTICAL uniform distribution and the rolls are INDEPENDENT (one die has no impact on the next).

The "identical" in i.i.d. refers to fact that these outcomes (sample points) each (or all, if we roll many) draw from the same exact distribution. As the distribution is always the same, so must its first two moments (mean and variance) be the same.

In this way, we might illustrate two types of samples that, technically, are NOT random samples b/c they violate i.i.d.:

1. if the die somehow were not independent or correlated with each other (maybe magnets between them!), or
2. if we rolled a six-sided plus a ten-sided die (as in Dungeons and Dragons!): those are not identical distributions

I hope that helps, David

 

[dadalee1102]

David, thanks for the reply.
But now I got another question. Since what you said is only true if we draw from discrete uniform distribution. If the population we draw from is not uniform, then p[D(i) = 1] = 1/6, and p[D(i) = 2] will not necessary be 1/6, right?

Then under this case, mean the variance of the two sample will not be the same? and then how do we derive the mean and variance of sample average?

Thank you.

 

[David Harper CFA FRM]

Hi hsintalee,

There may be some semantic circularity here. The definition of "random sample" is, while un-intuitive in my opinion, is narrow: it is to repeatedly draw samples from the exact same distribution (same random variable). This is the identical in i.i.d., such that if each sample point is drawn from an "identical" distribution, each sample point is represented by the same distribution (i.e., the mean and variance must be preserved as they are merely characteristics, the first two moments, of the same distribution). So, it's pretty much a function of our restrictive meaning of "random sample."

The discrete uniform distribution, a single sided die, is just my one example; it could be any distribution. If we change the distribution, then it's mean and variance will be different, but still: a random sample from that distribution must draw each sample point from an (identical) distribution with the same mean and variance (as the last sample point, as the next sample point).

The central limit theorem is striking because, amazingly, it does not care what is the underyling distribution:

Variance of the sample average = variance [ sample average] = variance [(X1 + X2 + ... Xn)/n]:
= variance [1/n*X1 + 1/n*X2 + .... + 1/n*Xn]
= variance [1/n*(X1 + X2 + .... + Xn)]
= (1/n)^2*variance(X1 + X2 + .... + Xn)
= 1/n^2*[variance(X1) + variance(X2) + ... variance(Xn)]
= 1/n^2*n*variance(X)
= variance(X)/n; in this way, the standard error of the sample average = SQRT[standard devation(x)/SQRT(n)]

Note that this reduction is only possible under i.i.d.:
* independence allows us to forgo the covariance terms in summing variances, and
* identical allows us to realize that, by definition of a random sample (!), the variance(X1) sample point = variance (X2) sample point

I hope this helps, David

 

[dadalee1102]

thxs for your detailed explanation!