Power and Significance

Significance level

Given a fixed [decision rule] for a hypothesis test, the significance level is the probability that a random sample will lead the researcher to reject the null hypothesis when the null hypothesis is true. The significance level of a test must be specified before the test is performed and is usually denoted by the Greek letter alpha (α). The significance level also defines the rejection region for the test:

If the [p-value] is less than α, then the decision is to reject the null hypothesis. If the p-value is great than α, the decision is to fail to reject the null hypothesis
If the test statistic exceeds a critical value determined by the significance level and the sampling distribution of the test statistic, then the null hypothesis is rejected. Otherwise, we fail to reject the null hypothesis.

Power

The power of a test is the probability that a particular sample will lead the researcher to reject the null hypothesis when the null hypothesis is false. The power of an hypothesis test can only be calculated for a particular value of the population parameter.

Example

We are interested in whether a particular coin is fair (that there is an equal probability of heads and tails). Let p represent the probability that a coin flip will results in heads. We will test the coin by flipping it 25 times and calculating the proportion of heads that we observe. So the random sample will be the outcomes of the 25 flips of the coin, and the statistic is the proportion of those flips that result in heads. Our hypotheses are:

$H_0: p = 0.5 \\ H_a: p \ne 0.5$

We choose a significance level of 10%. Roughly speaking, if we repeated our experiment 100 times with a fair coin, we would expect to reject the hypothesis that the coin is fair in ten of those experiments even though the coin is actually fair. We would make a Type I error in 10% of our trials.

The test statistic for this test is given by:

$\frac{\hat{p} - p}{\sqrt{p(1-p)/N}}$

This statistic has a normal distribution with mean of zero and standard deviation of one. Given this distribution, we will have a test with a significance level of 10% if we will reject the null hypothesis when the absolute value of the test statistic exceeds 1.645:

$\lvert \frac{\hat{p} - p}{\sqrt{p(1-p)/N}}\rvert > 1.645$

Using some algebra, another way to state the rejection criterion is to say that we reject the null hypothesis when the absolute value of the proportion of heads we observe exceeds :

$\lvert \frac{\hat{p} - p}{\sqrt{p(1-p)/N}}\rvert > 1.645$

$\implies \frac{\hat{p} - p}{\sqrt{p(1-p)/N}} >1.645 \text{ or } \frac{\hat{p} - p}{\sqrt{p(1-p)/N}} < -1.645 \\ \frac{\hat{p} - 0.5}{\sqrt{0.5(1-0.5)/25}} >1.645 \text{ or } \frac{\hat{p} - 0.5}{\sqrt{0.5(1-0.5)/25}} < -1.645 \\$

or, $\hat{p} > 0.6645 \text { or } \hat{p} < 0.3355$ .

The power of the test can only be calculated for a specific value of p. Let's assume that the coin is such that p = 0.6. We need to calculate the probability of observing a sample proportion whose absolute value is either greater than 0.6645 or less than 0.3355. when the probabiity of heads is 0.6. That is, we calculate the probability that $\hat{p}$ > 0.6645 or $\hat{p}$ < 0.3355 when p = 0.6. This probability is 0.259.

Power and Significance

Significance level

Power

Example

Labels