Your model isn't underfitting. Your features are lazy.

Here's the scene I've watched play out on a dozen teams. Accuracy plateaus. Someone rips out the logistic regression, drops in XGBoost, and waits for the jump. It doesn't come — or it comes with two points you can't explain to anyone. So the week disappears into hyperparameter tuning, and you end up with a slower, heavier, less interpretable model that's barely better than where you started. The model was almost never the bottleneck. The features were. This post is the long, practical version of that argument. We'll define the two camps in plain language, run real code, look at when boosting genuinely wins, and then walk through the failure mode nobody warns you about — the one where the fancy model is "winning" because it's quietly cheating. A note before we start: keep your examples generic. We'll predict a numeric target — think demand, a quantity, a score on a tabular dataset. The principles are the same everywhere, and you should validate them on your own data. The two camps, in plain terms Linear / logistic regression fits a straight-line relationship: each feature gets a weight (a coefficient), and the prediction is a weighted sum. Logistic regression is the same idea bent for classification — it outputs a probability. from sklearn.linear_model import LogisticRegression model = LogisticRegression ( max_iter = 1000 ) model . fit ( X_train , y_train ) # the whole model, readable in one line per feature: for name , weight in zip ( feature_names , model . coef_ [ 0 ]): print ( f " { name : < 20 } { weight : + . 3 f } " ) That loop is the entire model. A positive weight means "more of this pushes the prediction up," and you can hand that table to a stakeholder and defend every number. The cost: it assumes the relationship is roughly linear and that features act independently. Real data often isn't that polite. Gradient boosting (XGBoost, LightGBM, sklearn's GradientBoostingClassifier ) builds hundreds of small decision trees, each one correcting the mistakes of th