Null hypothesis, alternative hypothesis and hypothesis testing

A fact is a simple statement that everyone believes. It is innocent, unless found guilty. A hypothesis is a novel suggestion that no one wants to believe. It is guilty, until found effective.

• Edward Teller

Often we can't get to unambiguous conclusions using machine learning models and we end up saying: "we can't reject the null hypothesis". But, what does that mean? This article tries to explain what does null hypothesis mean, according to Statistics.

What's a hypothesis?

Sometimes, we feel that something is off about what we are told as truth. That's what happened to Charles Darwin when he observed different species of mockingbirds in each island of the Galapagos archipelago. He knew that there were just one species of mockingbird in South America, but found three more in those islands. So he formulated a hypothesis: "What if those new species are evolved separately, by natural selection, from the continent mockingbird which, somehow, arrived into the islands flying?"

After that, Darwin knew he had to prove this hypothesis to transform it into a well-established theory, but this is another story.

More formally: a hypothesis is an educated guess made to explain some event.

Ok. Then, why is the Null Hypothesis called "Null"?

What Darwin was proposing is what is called Alternative Hypothesis: a premise that challenges the established beliefs. So the Null Hypothesis is the one that doesn't change the status quo. It's called "null" because it brings no difference to the knowledge we have (null difference)

Oh, I see. And how do we define this "Alternative" Hypothesis?

The Alternative Hypothesis has to bring some change compared with the Null Hypothesis. This change can go from just modifying some value to the total opposition of the Null Hypothesis. For instance, you could propose a variation in the value of Earth's gravity (say, that is greater than $9.8ms^{-2}$ ) or affirm that there is no gravity at all!

The Alternative Hypothesis is proposed, what now? How do we test it?

• First, we should measure the value we want to evaluate. For example, we measure the Earth's gravity a hundred times and take the mean of all the records. Or, say that we want to analyse the mean age of the population of Barcelona: to do that, we take a sample of the entire population and average their ages (being careful that our sample is representative of Barcelona's population!).
• Then we propose a statistical model for the null hypothesis. We are going to use this model (for example, a Normal distribution) to check if the value we measured is compatible with it.
• We compute then the probability of our measured value occurring under the null hypothesis. This probability is called the p-value.
• And if this p-value is below 1%-5% (depending on the author), we reject the Null Hypothesis, because the measured value has a probability too low to be compatible with it. This 1%-5% (or 0.01-0.05) is called the significance level and is usually written as $\alpha$ .

If the probability is greater than $\alpha$ , we cannot reject the Null Hypothesis, because there is a higher probability that the measured value belongs to its modelled distribution. Even so, this doesn't mean that the Null Hypothesis is definitely confirmed! There is no absolute certainty in science.

TL;DR

• A hypothesis is an educated guess made to explain some event.
• The null hypothesis is the one that doesn't change our current knowledge.
• The alternative hypothesis proposes a change in some measurable feature.
• We accept the alternative hypothesis if its probability of occurrence under the null hypothesis (p-value) is below 1%-5%.

More:

• Stephanie Glen, Null Hypothesis Definition and Examples, How to State from StatisticsHowTo.com:
• Exclusion of the null hypothesis from Wikipedia
• Statistical significance from Wikipedia
• Vinita Silaparasetty , What is a Hypothesis in Machine Learning? from Medium