99% token accuracy, zero learning. Field notes from fine-tuning vision models with RL.

Over the past year I have been fine-tuning open vision-language models - 9B dense up to a 35B mixture-of-experts - with supervised fine-tuning and GRPO-style reinforcement learning on verifiable rewards. Most of what I learned was not about algorithms. It was about the ways a training run can look healthy while doing nothing, or crash for reasons that have nothing to do with your code. Three failures, in increasing order of how long they fooled me. Failure 1: the metric that measured the wrong thing (18 hours) I ran an 18-hour supervised fine-tune that reported token accuracy climbing steadily to 99%. Looked like a textbook run. The real evaluation metric - accuracy on multiple-choice questions - never moved. The cause was a mismatch between what I supervised and what I evaluated. The training loss was over free-text reasoning traces; the evaluation scored a single extracted answer letter. The model got extremely good at reproducing the shape of the training text - hence 99% token accuracy - without that transferring to the decision I actually cared about. Token accuracy is a proxy, and proxies drift from the target exactly when you stop checking. The fix was structural, not a hyperparameter: supervise the thing you evaluate. If the deliverable is a constrained answer, the training signal has to reach that answer, not just the prose around it. The general rule I took: any training metric that is not your evaluation metric is a hypothesis about correlation, and you should check that correlation before you spend GPU-days on it. Failure 2: the crash that was two libraries disagreeing about position ids The GRPO trainer for the 9B vision model crashed in the forward pass, deep inside rotary position embedding code. Nothing in my training code had changed. The diagnosis took a while because the bug lived at the boundary between components: the text sequence length was derived from token-type ids, while the vision sequence length came from the image grid - and image-pad t