Test Your Thinking

In analysis, data alone doesn’t tell the full story. We might observe patterns or differences in numbers; maybe one group appears to perform better than another, or two variables seem to move together. But how do we know whether these patterns are real or just due to random chance?

This is where statistical testing becomes essential. It allows us to evaluate data objectively, and make evidence-based decisions about the relationships and differences we observe.

Statistical tests

A statistical test is a mathematical procedure used in hypothesis testing to evaluate whether a relationship exists between variables, or whether differences between groups are meaningful. These tests help us assess whether the evidence in our sample data is strong enough to draw conclusions about the broader population.

In essence, statistical tests allow us to test a claim (hypothesis) by quantifying the likelihood that the observed results are due to chance.

What is hypothesis testing?

Hypothesis testing is a structured approach used to evaluate whether the data provides sufficient evidence to support a claim about a population. It helps determine whether an observed effect or relationship in the sample data is likely to exist in the population, or if it could have occurred by random chance.

Hypothesis testing follows a sequence of steps, outlined below.

Step 1: State the hypothesis

We begin by clearly defining two competing statements, mentioned below.

• Null hypothesis (H₀): The default assumption is that there is no effect, no difference, or no relationship between groups or variables.
  Example: A new drug has no impact on blood pressure compared to the current standard treatment.

• Alternative hypothesis (H₁): The research hypothesis proposes that an effect, difference, or relationship exists.
  Example: The new drug significantly reduces blood pressure compared to the standard treatment.

Step 2: Choose an appropriate statistical test

The choice of statistical test depends on:

• The type of data (categorical vs. numerical).

• The number of groups or variables involved.

• Whether data meet assumptions for parametric testsParametric tests are statistical tests that assume the data follows a specific distribution, usually a normal distribution. These tests also assume that the data meets certain conditions, such as equal variances across groups. Examples of parametric tests include t-tests, ANOVA, and linear regression. (e.g., normality, equal variances).

Common tests include t-tests, ANOVA, regression tests, etc., and we’ll explore some of these in more detail in the upcoming sections.

Step 3: Set a significance level ($α$)

Before analyzing data, set the significance level ($α$). This is the threshold for deciding if a result is statistically significant. The standard value is $α = 0.05$.

If the probability of observing the result under the null hypothesis is less than $α$, we consider the finding statistically significant.

Step 4: Calculate the test statistic

Using the selected test, compute a test statistic, which is a numerical summary that measures how far the observed data deviates from what the null hypothesis predicts.

Examples of test statistics include:

• t-statistic for t-tests.

• F-statistic for ANOVA.

• χ² statistic for chi-square tests.

• z-statistic for z-tests.

• r statistic for correlation tests.

A larger test statistic generally indicates stronger evidence against the null hypothesis.

Step 5: Determine the p-value

The p-value quantifies the probability of obtaining a test statistic as extreme as (or more extreme than) the observed one, assuming the null hypothesis is true.

• Low p-value (≤ α): Strong evidence against H₀, so we reject the null hypothesis.

• High p-value (> α): Insufficient evidence to reject H₀.

Calculation of p-values involves:

• Identifying the relevant probability distribution for the test statistic. For example:

  • Use the t-distribution for t-tests.

  • Use the F-distribution for ANOVA.

  • Use the chi-square distribution for chi-square tests.

  • Use the normal (z) distribution for z-tests.

• Finding the p-value by locating the probability associated with the calculated test statistic through:

  • Statistical tables: Traditional printed or online tables provide critical values and cumulative probabilities for various distributions (e.g., t-tables, F-tables).

  • Statistical software/tools: Programs such as Google Sheets, Excel, Python (SciPy library), R, SPSS, or online calculators can directly compute p-values given the test statistic and degrees of freedom. This reduces manual errors and effort.

Step 6: Make a decision

Compare the p-value to $α$ to decide:

• If $p-value ≤ α$ ...