Bayesian vs. Frequentist Estimation Under Small-Sample and Non-Normal Conditions: A Full-Factorial Monte Carlo Evaluation

Abstract

The replication crisis in social and behavioral sciences has systematically exposed the fundamental limitations of Null Hypothesis Significance Testing (NHST). In these disciplines, small samples and non-normal data distributions are the operational norm, frequently leading to underpowered studies and fragile empirical findings. Addressing this methodological gap, we conducted a comprehensive Monte Carlo simulation (144,000 iterations) to evaluate alternative inferential frameworks under realistic research constraints. We systematically compared frequentist approaches (Welch’s t-test and Wilcoxon-Mann-Whitney) with Bayesian estimation across varied sample sizes (n = 20 to 200), effect sizes, data distributions, and prior specifications. Estimation accuracy was rigorously assessed using Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and coverage rates, incorporating an explicit null condition to precisely capture Type I error dynamics. Simulation results demonstrate that under small-sample conditions typical of social and health research, Bayesian estimation utilizing an empirically derived informative prior substantially outperforms traditional frequentist models. For instance, at n = 20, the Bayesian approach reduced MAE by approximately 65% (0.124 versus 0.358) while meaningfully increasing statistical power. Critically, prior sensitivity analyses reveal that the choice of prior dictates inferential performance far more than the basic Bayesian-frequentist dichotomy. This advantage introduces a fundamental trade-off: while informative priors optimize power and minimize error in limited samples, they incrementally inflate Type I error rates. These empirically grounded findings provide behavioral researchers with actionable, condition-specific guidance for navigating inferential decisions under the realistic constraints of their discipline.