The following example from Lock et al, Statistics: Unlocking the Power of Data describes an experiment to learn about the effectiveness of two drugs in the treatment of cocaine addiction.

Cocaine addiction is very hard to break. Even among addicts trying hard to break the addiction, relapse is common. (A relapse is when a person trying to break out of the addiction fails and uses cocaine again.) One experiment investigates the effectiveness of the two drugs desipramine and lithium in the treatment of cocaine addiction. The subjects in the study were cocaine addicts seeking treatment, and the study lasted six weeks. The 72 subjects were randomly assigned to one of three groups (desipramine, lithium, or a placebo, with 24 subjects in each group) and the study was double blind.

The results of the study are summarized in the following table.

Treatment for cocaine addiction
Relapse No Relapse

Desipramine 10 14

Lithium 18 6

Placebo 20 4

We focus on the question: “Is taking lithium more effective at preventing relapse than taking a placebo?”

The Model

Here a model describes the relationship between drug use and the prevention of relapse from cocaine addiction. One possible model is that the use of the placebo or lithium has no effect on the prevention of relapse. If one considers one population of cocaine patients who take the placebo and another population of cocaine patients who take lithium, then model implies that the proportions of relapse patients in the two hypothetical populations would be equal. Another possible model would say that the proportion of relapse patients in the lithium population would be smaller than the proportion of relapse patients in the placebo group. We have described two models "choice of drug has no effect on prevention of relapse" and "the lithium drug does help in preventing a relapse" and we want to use the data to help make a decision about the appropriate model.

A Statistic

Here a statistic is something we compute from the observed data that indicates that lithium may be more effective than the placebo in preventing a relapse from cocaine addition. Here we observe that 6 out of 24 out of lithium group had no relapse compared to 4 out of 24 for the placebo group. A reasonable choice of statistic is

$X = \hat p_L - \hat p_P$ ,

where $\hat p_L$ is the proportion of lithium patients who have no relapse and $\hat p_P$ is the proportion of placebo patients with no relapse. Here we observe X = 6/24 – 4/24 = 0.083

A Randomization Test

This is a special case of a designed experiment and the subjects were assigned to the lithium and placebo groups at random. Of the 48 patients used, the number of ways of assigning the subjects to the two groups is 48 C 24.

We can represent the data by two columns – the group and a response indicator if the patient had no relapse (1) or relapse (0).

Group Response

Placebo 1

Placebo 0

…

Placebo 1

Lithium 0

Lithium 1

…

Lithium 1

If the choice of treatment has no effect on preventing a relapse is true, that is, the “no effect” model is true,G then the group labels of Placebo and Lithium are not informative. In particular, if the “no effect” model is true, the distribution of the statistic X would not change by randomly mixing up the group labels.

We use the following simulation experiment to compute a randomization distribution for X.

Step 1: Randomly mix up the 24 Placebo and 24 Lithium group labels.

Step 2: Using the randomly chosen labels, compute the statistic $X = \hat p_L - \hat p_P$

Repeat this process a large number (say, 1000) times, collecting the values of X. The 1000 values of X are graphed using a histogram. The observed value of X is displayed using a red vertical line.

This histogram represents the plausible values for the difference in proportions X if the "no effect" model is true, that is, there is really no differences in the proportions of the populations of lithium and placebo who would experience no relapse from cocaine addiction. As one might expect, the values of X are centered about the value 0, but it is very possible that X would be as small as -0.2 or as large as 0.2. How does the observed value X = 0.083 compare to the values of X that we predict under the no-effect model? Actually, the observed value of X is in the middle part of this distribution. To say it a different way, the X = 0.083 is not surprising if really there were no differences in the lithium and placebo population proportions. Since X = -0.83 is not an unlikely value, we say that there is insignificant evidence to conclude that the lithium drug is better than the placebo in preventing relapse from cocaine addiction.

Using StatCrunch

StatCrunch has applets to perform simulations for different randomization tests such as this one. In StatCrunch, select the menu item Applets -> Resampling -> Randomization test for two proportions.

For this example, we have summary data. For the lithium group, 6 of the 24 patients did not suffer a relapse from addiction, and for the placebo group, 4 of the 24 did not suffer a relapse. In the box, select "From summary" and enter the number of "successes" (6 and 4) and the "number of observations" (24 and 24)

When we select Compute, the following window will appear. I press the 1000 times (doing the simulation 1000 iterations) button, and we see the following output.

The histogram looks a bit different from the earlier histogram due to the way the values of X were placed in bins. Remember the observed value of X = 6/24 - 4/24 = 0.083. StatCrunch tells us that the probability a randomized difference is at least as large as 0.083 is 0.367. Since this probability is large, there is little evidence that the population proportion of the lithium group is larger than the population proportion of the placebo group.