Signal or chance?
Shuffle group labels to discover how surprising a difference really is.
Shuffle two groups
1 question · No time limit
Example
Example
Under no treatment effect, what fraction of valid assignments produce a mean difference at least 7/2 away from zero, in either direction?
8 participants were randomly assigned: 4 to condition A and 4 to B. Each number is an observed score. Assume neither condition changes any participant’s score. Shuffle labels while keeping both groups the same size. The observed mean difference A − B is −7/2.
1. Predict
| Condition A | Condition B |
|---|---|
| 43 | 49 |
| 31 | 52 |
| 46 | 35 |
| 40 | 38 |
Scores stay fixed. Only the A and B labels move, with equal group sizes.
Mean difference A − B: −7/2
Make a prediction first. It is not scored.
2. Test your idea
Shuffle the labels to build a distribution of differences.
Random shuffles estimate the fraction. Count every assignment to see the exact value.
Try the controls, then check your solution.
Why?
Choose which 4 of the 8 participants receive A: C(8, 4) = 70 equally likely assignments. A label assignment is counted separately even when some scores match.
The rule counts a mean difference at least 7/2 away from zero, in either direction. Of the 70 assignments, 36 meet it, including ties. The exact fraction is 36/70 = 18/35.
This fraction describes how often an equally extreme difference would occur under the no-effect assumption and this random assignment scheme. It is not the probability that the assumption is true. A large fraction does not prove the conditions are equivalent.
How the assignments are counted
For each assignment, add the scores labeled A and subtract the scores labeled B. Divide by 4 to obtain the difference between the means.
Compare each difference with the stated boundary. Equality counts too.
Every valid assignment has the same probability, so add one to the count for each assignment that meets the rule.
Revisit your prediction
The exact count is 36 of 70 assignments (51.4%). Compare this with your first impression of the observed gap.
Keep exploring the experiment.
3. Change the rules
A linked challenge. This answer is not scored.
Continue your unfinished round
Your answers are saved in this tab. Continue from where you left off.
Continuing uses no additional practice start.
Discard the unfinished round?
Your answers from this round will be discarded. You can choose your settings before starting a new round.
Starting the new round uses one practice start.
- Answered
- 0 / 1
- Active time
- 0:00
Round complete.
Optional and off by default. Share starts, completions, replays and error categories. For visits from ChatGPT, starts and completions can also include the first QuickenBrain page and broad country. No answers, drawings or names.
How this data is usedHelp
Try the controls, then check your solution.
Your first checked solution is recorded. You can keep experimenting afterwards.
About this practice
Learn a method
Shuffle to test a differenceShuffle group labels to discover how surprising a difference really is.
For a broader mix of practices, explore the statistics practice test. Membership is required to start a test.
Report a problem
Optional and off by default. Share starts, completions, replays and error categories. For visits from ChatGPT, starts and completions can also include the first QuickenBrain page and broad country. No answers, drawings or names.
How this data is used