AdaBoost from Scratch: How a Pile of Dumb Rules Becomes a Smart Classifier

Here is a question that sounds like a trick: can you build an accurate classifier out of models that are barely better than flipping a coin? Surprisingly, yes. That is the whole idea behind boosting, and AdaBoost is the algorithm that made it famous. I built it from scratch and dropped it into an interactive demo — here's how it actually works, real math, no hand-waving. Play with the live version: https://dev48v.infy.uk/ml/day21-adaboost.html The weak learner: a decision stump AdaBoost's building block is the simplest classifier you can imagine: a decision stump . It is a decision tree with exactly one split. Look at one feature, compare it to one threshold, and call everything on one side "+1" and everything on the other side "−1". That's it. One line, one cut. def stump_predict ( X , dim , thresh , polarity ): pred = np . ones ( len ( X )) if polarity == 1 : pred [ X [:, dim ] <= thresh ] = - 1 else : pred [ X [:, dim ] > thresh ] = - 1 return pred On anything that isn't trivially separable, a single stump is hopeless — on a checkerboard layout it barely passes 55-60%. That is exactly why it's a "weak learner": a model that only beats random guessing by a hair. The magic is in how we combine hundreds of them. Sample weights: a moving spotlight The engine of AdaBoost is a weight on every training point that says "how much does getting this one right matter?" Everything starts equal: n = len ( X ) w = np . full ( n , 1.0 / n ) # uniform: every point weighs 1/n These weights are a probability distribution — they sum to 1. After each round they change: points we got right get lighter, points we missed get heavier. Since we always pick the next stump to minimise weighted error, the heavy points end up dominating the search. The next stump is effectively forced to stare at whatever the committee keeps blowing. Weighted error, not a plain count When we hunt for the best stump each round, we don't count mistakes — we add up the weight of the mistakes: def weighted_error