Permutation test

So, let's say you and your friend Lucy are playing a game of rock-paper-scissors. You both decide to play the game five times to see who wins more times.

Now, let's say Lucy wins three times, and you win two times. But you start wondering, "Hmm, is five games enough to really tell who's better at this game? Maybe we need to play more games to be sure."

But playing more games might take too long, so instead, you decide to do something called a permutation test.

Basically, a permutation test shuffles, or mixes up, the data (in this case, the results of each game) many times to see how often the observed outcome (Lucy winning three times and you winning twice) could have happened just by chance.

Let's say we use a computer to randomly shuffle the results of the five games over and over again. Each time, we count the number of times Lucy wins three or more times out of the five games. We do this maybe 1,000 or 10,000 times to get a good estimate.

If we find that, out of all these shuffles, only 10% of the time does Lucy win three or more times out of five, we might conclude that her three wins out of five games is pretty significant and not just due to chance.

In essence, a permutation test helps us figure out if our observations are just a coincidence or if there is actually a real difference between the things we are comparing (like Lucy and you and your rock-paper-scissors skills).