xgboost
XGBoost is a gradient-boosted decision tree library: a C++ core with Python bindings that trains ensembles of trees for classification, regression and ranking. On tabular data (spreadsheets, database rows, features you engineered) it is still among the strongest models available and routinely beats neural networks. It offers two Python APIs, scikit-learn style estimators (XGBClassifier, XGBRegressor) and a lower-level native API around DMatrix, plus GPU training via device="cuda" and distributed training on Dask and Spark.
Still the reference model for tabular ML, actively maintained with serious corporate backing. Pick it when accuracy on structured data pays the bills; pick LightGBM for raw CPU speed or sklearn's built-in boosting when you want fewer moving parts.
Use it if
- Your data is tabular and you want top-tier accuracy: boosted trees remain the default winner on structured data and most ML competitions reflect that
- You want scikit-learn compatibility: XGBClassifier drops into sklearn pipelines, grid search and cross-validation unchanged
- You have a GPU and large datasets; device="cuda" typically speeds training up by an order of magnitude with a one-line change
- You need learning-to-rank, quantile regression, survival analysis or custom objectives, which XGBoost supports natively
- Your data is images, audio or free text: use deep learning; gradient boosting needs engineered features and does not learn representations
- You have a small dataset and want something simple to explain and maintain: sklearn's HistGradientBoosting or even logistic regression gets close with far less dependency weight
- Install footprint matters: current wheels require Python 3.12+ and the Linux wheel is large because GPU support is bundled; macOS additionally needs the libomp runtime from Homebrew
- Training speed on very large CPU-only datasets is the bottleneck: LightGBM is usually faster to train at comparable accuracy
Setup reality
pip install xgboost pulls a prebuilt wheel, no compiler needed, but it is a hefty download on Linux since CUDA support is baked in; there is a smaller CPU-only variant if you look for it. On macOS the wheel imports only after brew install libomp, an error message that greets most Mac users on first run. The 3.x line also raised the floor to Python 3.12. API-wise the recurring trap is version drift: early_stopping_rounds moved from fit() to the constructor in 2.x, and old gpu_hist / gpu_id parameters were replaced by device="cuda", so pre-2.0 tutorials actively mislead. Categorical support works but requires pandas category dtype plus enable_categorical=True.
Patterns
Train a classifier with the sklearn APItrain-classifier
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = XGBClassifier(n_estimators=500, learning_rate=0.05, max_depth=6)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))Labels must be 0-based integers for multiclass; use sklearn's LabelEncoder if yours are strings.
Stop when the validation score stallsearly-stopping
model = XGBClassifier(
n_estimators=2000,
learning_rate=0.05,
early_stopping_rounds=50, # constructor, not fit(), since 2.x
eval_metric="logloss",
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
print(model.best_iteration)Passing early_stopping_rounds to fit() was removed in 2.x; set a generous n_estimators and let early stopping pick the real count.
Native API with DMatrixnative-api
import xgboost as xgb
dtrain = xgb.DMatrix(X_train, label=y_train)
dval = xgb.DMatrix(X_val, label=y_val)
booster = xgb.train(
{"objective": "binary:logistic", "eta": 0.05, "max_depth": 6},
dtrain,
num_boost_round=2000,
evals=[(dval, "val")],
early_stopping_rounds=50,
)
preds = booster.predict(xgb.DMatrix(X_test))The native API exposes everything (custom objectives, callbacks) but returns raw probabilities, not classes; threshold them yourself.
Train on GPUgpu-training
model = XGBClassifier(device="cuda", tree_method="hist", n_estimators=1000)
model.fit(X_train, y_train)device="cuda" replaced tree_method="gpu_hist" and gpu_id in 2.0; if your data lives on CPU it is copied to GPU each fit, so consider cupy/cuDF inputs for big jobs.
Use categorical columns without one-hot encodingcategorical-features
import pandas as pd
X = df.copy()
for col in ["city", "plan"]:
X[col] = X[col].astype("category")
model = XGBClassifier(enable_categorical=True, tree_method="hist")
model.fit(X, y)Both pieces are required: pandas category dtype AND enable_categorical=True; plain object/string columns raise an error.
Save and load a model safelysave-load-model
model.save_model("model.ubj") # or model.json
from xgboost import XGBClassifier
loaded = XGBClassifier()
loaded.load_model("model.ubj")Use save_model, not pickle: the ubj/json formats survive library upgrades, while pickled models break across versions.
Which features matterfeature-importance
import pandas as pd
imp = pd.Series(model.feature_importances_, index=X_train.columns)
print(imp.sort_values(ascending=False).head(10))
# more detail from the booster:
model.get_booster().get_score(importance_type="gain")Default importance is gain-based and biased toward high-cardinality features; use SHAP values when the ranking actually drives decisions.
Handle class imbalanceimbalanced-classes
neg, pos = (y_train == 0).sum(), (y_train == 1).sum()
model = XGBClassifier(
scale_pos_weight=neg / pos,
eval_metric="aucpr",
)
model.fit(X_train, y_train)scale_pos_weight rebalances the gradient for binary tasks; it skews predicted probabilities, so recalibrate if you need real probabilities rather than ranking.
Cross-validate boosting roundscross-validation
import xgboost as xgb
dtrain = xgb.DMatrix(X, label=y)
cv = xgb.cv(
{"objective": "binary:logistic", "eta": 0.05, "max_depth": 6},
dtrain,
num_boost_round=1000,
nfold=5,
early_stopping_rounds=50,
metrics="auc",
)
print(len(cv), cv["test-auc-mean"].iloc[-1])xgb.cv returns a DataFrame with one row per surviving round; its length is the round count to reuse for the final fit.
Tune with sklearn searchhyperparameter-tuning
from sklearn.model_selection import RandomizedSearchCV
search = RandomizedSearchCV(
XGBClassifier(tree_method="hist", n_estimators=300),
param_distributions={
"max_depth": [3, 4, 5, 6, 8],
"learning_rate": [0.01, 0.05, 0.1],
"subsample": [0.7, 0.85, 1.0],
"colsample_bytree": [0.7, 0.85, 1.0],
"min_child_weight": [1, 5, 10],
},
n_iter=30, cv=3, scoring="roc_auc", n_jobs=-1,
)
search.fit(X_train, y_train)max_depth, learning_rate and subsample carry most of the signal; for anything serious use Optuna, which the project integrates with officially.
Cut memory with QuantileDMatrixmemory-efficient-training
import xgboost as xgb
qtrain = xgb.QuantileDMatrix(X_train, label=y_train)
qval = xgb.QuantileDMatrix(X_val, label=y_val, ref=qtrain)
booster = xgb.train({"tree_method": "hist"}, qtrain,
num_boost_round=500, evals=[(qval, "val")])QuantileDMatrix pre-bins features and can halve memory versus DMatrix with hist; validation sets must pass ref=qtrain to share the bin edges.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| lightgbm | PyPI | Faster CPU training on large datasets with comparable accuracy; leaf-wise growth and lighter install |
| catboost | PyPI | Lots of categorical features you do not want to encode yourself; strong defaults with minimal tuning |
| scikit-learn | PyPI | HistGradientBoostingClassifier covers most everyday cases with zero extra dependencies |