XGBoost and Gradient Boosting tie at 0.880 ROC-AUC in tutorial test

The tutorial builds its comparison in two stages. First, it tracks a regression example by hand: starting from the mean prediction, it adds small depth-2 decision trees fitted to the residuals, one at a time with a learning rate of 0.45. Training MSE falls from 0.18915 with no trees to 0.01294 after four, illustrating how each tree corrects what the previous ensemble got wrong rather than rebuilding it.

The second exercise moves to binary classification and fixes the same core settings (100 estimators, learning rate 0.1, depth 3) across four models. The near-tie between Gradient Boosting and XGBoost is presented as a lesson in experiment discipline: the tutorial stresses that equal n_estimators does not mean equal model complexity, and that test data was never used for tuning.

The material deliberately stops short of XGBoost's distinctive machinery. Regularization and second-order derivatives are deferred to the next lesson, along with early stopping, which the author notes is not yet automated here. The framing suggests the series treats XGBoost as an extension of ideas the reader has already built by hand, not as a black box.