Choosing Statistical Tests — A Guide

Correct selection of a statistical test is one of the most important elements of data analysis in scientific work. An improperly chosen test can lead to erroneous conclusions and undermine the credibility of the entire study — whether it's a thesis or a scientific article. The challenge is that dozens of statistical tests exist, and their selection depends on many factors: variable types, data distribution, sample size, dependencies between observations, and the nature of the research hypothesis. In this comprehensive guide, we present a systematic approach to selecting statistical tests, explain the logic behind each test, and offer advice on how to avoid the most common mistakes.

Determining Variable Types — The Foundation of Test Selection

The first and absolutely crucial step is correctly determining the type of variables. Variables can be quantitative (continuous — e.g., weight, height, temperature; or discrete — e.g., number of children, number of hospitalizations) or qualitative (nominal — e.g., sex, blood type, eye color; or ordinal — e.g., satisfaction level on a Likert scale, disease stage). The type of variable determines which statistical methods will be appropriate.

Parametric tests (t-test, ANOVA, Pearson correlation, linear regression) require quantitative variables with approximately normal distribution and homogeneous variances. Non-parametric tests (Mann-Whitney, Kruskal-Wallis, Spearman, Wilcoxon) are more versatile — they can be applied to ordinal data, skewed distributions, or small samples. Important: a Likert scale (1-5 or 1-7) is formally an ordinal scale, although in research practice it is often treated as continuous — this decision should be justified in the methodology.

Correctly distinguishing between dependent and independent variables is equally important. The dependent variable (outcome) is the one you want to explain or predict. The independent variable (predictor) is the one that potentially influences the dependent variable. In an experiment, the independent variable is manipulated by the researcher (e.g., type of treatment), while the dependent variable is measured (e.g., health improvement).

Checking Assumptions — The Step Many Skip

Before applying any test, check whether your data meet its assumptions. This is the step that students and beginning researchers most often skip, yet it is critical for analysis correctness. For parametric tests, the main assumptions are: normality of distribution, homogeneity of variance (homoscedasticity), and independence of observations.

Normality of distribution is checked using the Shapiro-Wilk test (preferred for samples n < 50) or the Kolmogorov-Smirnov test (for larger samples). It's also worth visually assessing the distribution using a histogram and Q-Q plot. Note: with large samples (n > 30-50), normality tests may reject the null hypothesis even with slight deviations from normality — in such cases, visual assessment and the central limit theorem, which ensures that means from large samples have approximately normal distribution, are more important.

Homogeneity of variance (homoscedasticity) is checked using Levene's test. If variances in compared groups differ significantly, we can apply the Welch correction variant of the t-test (default in most statistical software) or a non-parametric test. Independence of observations — the hardest assumption to formally test — means that one observation's result does not influence another's. Repeated measurements on the same person violate this assumption and require using tests for dependent samples.

Comparing Two Groups

For comparing two independent groups with quantitative data of normal distribution, we use Student's t-test for independent samples. The t-test compares means in two groups and answers the question: is the observed difference between means large enough to be considered statistically significant? Example: comparing mean exam scores in experimental and control groups.

If data don't meet the normality assumption or we're dealing with an ordinal variable, we use the Mann-Whitney U test (also known as the Wilcoxon rank-sum test for independent samples). This non-parametric test compares ranks (not means), making it robust against outliers and skewed distributions. It is the most common substitute for the t-test in social and medical research.

For comparing two measurements on the same group (e.g., before and after intervention), we use the paired t-test or its non-parametric counterpart — the Wilcoxon signed-rank test. The key distinction: independent samples are two different groups of people; dependent samples are the same people measured twice.

Comparing Three or More Groups

For comparing more than two groups, we use analysis of variance (ANOVA) or its non-parametric counterpart — the Kruskal-Wallis test. A common mistake is conducting multiple t-tests between all pairs of groups instead of ANOVA — this approach inflationary increases the risk of Type I error (false positive).

One-way ANOVA compares means across several groups and answers the question: do at least two groups differ significantly? If ANOVA yields a significant result, the next step is post-hoc tests that identify which specific pairs of groups differ. The most popular post-hoc tests are: Tukey's HSD — the most conservative, recommended for comparable group sizes; Bonferroni — simple to interpret but conservative with many comparisons; Scheffé — the most liberal, allowing complex comparisons.

Two-way ANOVA allows analyzing the effects of two independent variables simultaneously and their interaction. For example: the effect of teaching method (3 methods) and sex (2 levels) on test scores. An interaction means that the effect of one variable depends on the level of the other — e.g., method A is better for women but method B for men.

Repeated measures ANOVA is used when the same individuals are measured multiple times — e.g., before intervention, immediately after, and after 3 months. The non-parametric counterpart is the Friedman test.

Tests for Qualitative Data

For qualitative (nominal) data, the most commonly used tests are the chi-square test (χ²) and Fisher's exact test. The chi-square test checks the independence of two categorical variables — e.g., whether there is an association between sex and voting preferences. It requires that expected frequencies in each cell of the cross-tabulation be at least 5. If this condition is not met (with small sample sizes), we use Fisher's exact test, which calculates exact probability instead of relying on the chi-square distribution.

For 2×2 tables (two binary variables), we also calculate the odds ratio (OR) or relative risk (RR) as effect size measures. OR = 1 means no association, OR > 1 means increased odds of the event occurring, OR < 1 means decreased odds. A confidence interval for OR that does not include 1 indicates statistical significance of the association.

McNemar's test is used when we have qualitative data in a dependent pairs design — e.g., 'yes/no' responses from the same individuals before and after intervention. It is the counterpart of the paired t-test but for binary variables.

Correlations and Regression

Correlation measures the strength and direction of the linear association between two variables. Pearson's correlation coefficient (r) is used for quantitative data with normal distribution and a linear relationship. The value of r ranges from -1 (perfect negative correlation) to +1 (perfect positive correlation), and r = 0 indicates no linear correlation. Cohen's convention: |r| < 0.3 — weak correlation, 0.3-0.5 — moderate, > 0.5 — strong.

Spearman's rank correlation coefficient (ρ) is used when data don't meet the normality assumption, the relationship is monotonic (but not necessarily linear), or we have ordinal data. Kendall's tau is an alternative to Spearman, preferred with small samples and many tied ranks.

Linear regression allows modeling the relationship between a dependent variable (quantitative) and one (simple regression) or multiple (multiple regression) independent variables. Unlike correlation, regression allows prediction and assessment of individual predictors' effects. Logistic regression is used when the dependent variable is binary (e.g., sick/healthy, passed/failed). The results are odds ratios (OR) for each predictor.

Survival Analysis and Specialized Tests

In medical and clinical research, survival analysis is frequently used. Kaplan-Meier curves present survival probability over time, and the log-rank test compares survival curves between groups. Cox regression (proportional hazards model) allows analyzing the effect of multiple variables on the risk of an event occurring over time.

Multivariate analysis of variance (MANOVA) extends ANOVA to situations with more than one dependent variable. Analysis of covariance (ANCOVA) allows controlling the influence of confounding variables. Mixed-effects models combine fixed and random effects and are particularly useful for hierarchical data (e.g., students nested in classes) or unbalanced repeated measures.

Effect Size — Why the P-value Is Not Everything

The p-value only tells us about the probability of obtaining the observed (or more extreme) result if the null hypothesis were true. It says nothing about the practical significance of the result or the strength of the effect. Therefore, reporting an effect size measure alongside the p-value is essential. The most commonly used measures are: Cohen's d (difference between means in standard deviation units: small effect ≈ 0.2, medium ≈ 0.5, large ≈ 0.8), eta-squared (η²) for ANOVA (small ≈ 0.01, medium ≈ 0.06, large ≈ 0.14), correlation coefficient r as an effect measure, and odds ratio (OR) for categorical data.

Confidence intervals (CI) provide more information than the p-value alone because they show the range of plausible parameter values. A 95% CI that does not include the null value (0 for differences, 1 for OR) indicates statistical significance at the α = 0.05 level, but additionally informs about estimation precision.

Corrections for Multiple Comparisons

Remember about corrections for multiple comparisons. When performing multiple tests simultaneously, the risk of Type I error (false discovery) increases. With 20 independent tests at α = 0.05, the probability of at least one false positive is 1 - (0.95)^20 = 64%. Popular correction methods include: Bonferroni correction — simple (divide α by the number of tests) but very conservative; Holm-Bonferroni procedure — less conservative, preserves power with many tests; false discovery rate (FDR) control using the Benjamini-Hochberg method — preferred in exploratory studies with many comparisons (e.g., genomics, proteomics).

Common Mistakes in Statistical Test Selection

Based on years of working with researchers and students, the most common mistakes are: using a t-test instead of ANOVA when comparing more than two groups (and multiple t-tests between pairs), using parametric tests without checking normality and variance homogeneity assumptions, confusing correlation with causation ('students who drink more coffee have higher grades' does not mean coffee improves grades), ignoring multiple comparisons, reporting only p-values without effect size measures, and 'p-hacking' — manipulating analyses to achieve p < 0.05.

If statistical analysis is not your strong suit, don't hesitate to consult a specialist. Incorrectly selected tests and improperly interpreted results can undermine even the best-designed study. Statistical consultation at the research planning stage (not after data collection) is an investment that pays off in the quality of the entire scientific work.

Explore our professional statistical analysis services — we help with test selection, computations, and results interpretation. If you're writing a thesis, also read our practical guide to statistical analysis in a thesis. Have questions? Get in touch.