The Fix Was Not a Cleverer Model

I spent four months tuning a custom weather ensemble. It was worse than guessing. The fix was not a better ensemble. It was admitting someone already built the right thing and giving it away for free. What I built and why it failed The original weather bot counted forecast members. It pulled raw output from four systems: GFS, AIGEFS, ECMWF IFS, and AIFS. Up to 164 individual simulations per contract. The logic was simple. If at least three of four systems agreed on direction, the bot traded. If they disagreed, it sat out. That sounds reasonable. It was not. I ran 112 settled trades through the system and scored the model with a Brier score. The model scored 0.2858. Predicting the historical base rate, with no model at all, scores 0.2439. Lower is better. My model was worse than making no prediction. The problem was not direction. Direction was right about 60 percent of the time. The problem was confidence. The model spread its probabilities 2.1 to 4.0 times too narrow. It was certain when it should have been uncertain. In prediction markets, confidence sizes your bets. A model that is too confident trades too big on the wrong calls. The confident wrong calls cost more than the confident right ones made. There was also a systematic temperature bias at the gridpoint level, peaking around seven degrees Fahrenheit. The model leaned warm in a way that was not in the data. It was in the model. What I should have done first Before building anything, I should have checked whether the thing I was building already existed in better form. NOAA publishes the National Blend of Models. It blends dozens of forecast systems and applies statistical post-processing no individual model can match. It produces calibrated, bias-corrected, station-level probabilistic temperature guidance. For exactly the stations Kalshi settles on. For free. The NBM already does what I was trying to do by hand. It corrects the biases I was measuring. It produces uncertainty ranges I was approximating with