Automated Auditing Framework for Detecting Overfitting and Data Leakage in Machine Learning Models

Abstract

Machine learning now informs decisions in hospitals, banks, and hiring pipelines, where a model's integrity matters as much as its accuracy. Overfitting, data leakage, and feature bias still slip past routine evaluation and surface only after deployment. In this paper, we present an automated auditing framework of six modules that catches these problems beforehand. Overfitting is read from the train-test accuracy gap, stabilized over repeated stratified splits rather than a single partition. Leakage is screened by feature-target correlation and an index-overlap check; feature bias by Mean Decrease Impurity, permutation importance, a model-agnostic SHAP step, and a Gini index. Statistical validation uses stratified cross-validation and paired t-tests corrected for multiple comparisons, and a fairness module reports demographic parity and equalized-odds gaps when a protected attribute is available. In the ablation study the full framework scores 0.677, ahead of every reduced version. We tested it on four datasets with six classifiers. On Breast Cancer Wisconsin it flagged Gradient Boosting and Decision Tree as overfitting; a 30-split analysis confirmed both and placed Random Forest below the 0.05 gap (mean 0.037), where the conventional single-split test would have mislabeled it in about a quarter of splits. Injected leakage was caught at correlation 0.971.