No resampling wins on PR-AUC in fraud tutorial

The tutorial builds on an earlier walkthrough that used bank telemarketing data, moving to credit card fraud, where almost every transaction is legitimate. The dataset behind it comes from Kaggle's Credit Card Fraud Detection set, collected and analyzed in joint research by the payments company Worldline and the Machine Learning Group at ULB in Brussels, and the publisher itself recommends the Precision-Recall curve area for such skewed data.

The most striking finding is that the simplest option came out on top. With the classifier held fixed as a decision tree, the run without any resampling posted the best PR-AUC and F1, while the sampling methods lifted recall at the cost of precision — meaning more normal transactions get stopped. Choosing between them, the tutorial argues, means weighing the loss from one missed fraud against the loss from one blocked legitimate payment.

Model choice moved the numbers further. Under the same RandomOverSampler condition, random forest led on PR-AUC and F1, logistic regression caught the most fraud but with low precision, and GBDT had the highest ROC-AUC. The final model's confusion matrix — 81 of 98 frauds caught, 17 missed, 8 normal transactions wrongly flagged — shows why the piece closes on operational concerns: time-based splits, duplicate rows of unknown origin, probability calibration after oversampling, drift and the lag before fraud labels are confirmed.