Power and Significance

Added by Marlynne Ingram, last edited by Marlynne Ingram on Feb 03, 2008  (view change)

Definitions

Given a fixed [decision rule] for a hypothesis test, the significance level is the probability that a particular sample will lead the researcher to reject the null hypothesis when the null hypothesis is true. The calculation is performed as if the researcher had the ability to sample repeatedly from the population. The significance level of a test must be specified before the test is performed and is usually denoted by the Greek letter alpha (α). The significance level also defines the rejection region for the test. If the [p-value] is less than α, then the decision is to reject the null hypothesis. If the p-value is great than α, the decision is to fail to reject the null hypothesis

The power of a test is the probability that a particular sample will lead the researcher to reject the null hypothesis when the null hypothesis is false. The power of an hypothesis test can only be calculated for a particular value of the parameter of interest.

Example

We are interested in whether a particular coin is fair. Let p represent the probability that a coin flip will results in heads. We will test the coin by flipping it 25 times and calculating the proportion of heads that we observe. Our hypotheses are:

H 0: p = 0.5
H a: p ≠ 0.5

We choose a significance level of 10%. Roughly speaking, if we repeated our experiment 100 times with a fair coin, we would expect to reject the hypothesis that the coin is fair in ten of those experiments even though the coin is actually fair. We would make a Type I error in 10% of our trials.

The test statistic for this test is given by:

This statistic has a normal distribution with mean of zero and standard deviation of one. Given this distribution, we will have a test with a significance level of 10% if we will reject the null hypothesis when the absolute value of the test statistic exceeds 1.86:

Using some algebra, another way to state the rejection criterion is to say that we reject the null hypothesis when the absolute value of the proportion of heads we observe exceeds 0.1645:

The power of the test can only be calculated for a specific value of p. Let's assume that the coin is such that p = 0.6. We need to calculate the probability of observing a sample proportion whose absolute value exceeds 0.1645 when the probabiity of heads is 0.6. That is, we calculate the probability that > 0.1645 or < -0.1645 when p = 0.6. This value is 0.259.

Minitab commands

The power of a test can be calculated using the command Stat>Power and Sample Size. The following is from the Minitab help manual:

  1. Choose Stat > Power and Sample Size > 1-Sample Z, 1-Sample t, or 2-Sample t.
  2. Do one of the following:
    1. Solve for power
      1. In Sample sizes, enter one or more numbers. For a two-sample test, the number you enter is considered the sample size for each group. For example, if you want to determine power for an analysis with 10 observations in each group for a total of 20, you would enter 10.
      2. In Differences, enter one or more numbers.
    2. Solve for sample size
      1. In Differences, enter one or more numbers.
      2. In Power values, enter one or more numbers.
    3. Solve for the minimum difference
      1. In Sample sizes, enter one or more numbers. For a two-sample test, the number you enter is considered the sample size for each group.
      2. In Power values, enter one or more numbers.

Minitab will solve for all combinations of the specified values. For example, if you enter 3 values in Sample sizes and 2 values in Differences, Minitab will compute the power for all 6 combinations of sample sizes and differences.

  1. In Standard deviation, enter an estimate of the population standard deviation (s) for your data. See Estimating standard deviation.
  2. If you like, use one or more of the available dialog box options, then click OK.

Labels

 
(None)