When to Ship an AI Feature Behind a Flag

Every team already knows how to put a feature behind a flag. What is different here is that the thing most likely to need changing at three in the morning is not whether the feature is on — it is which model it calls, which prompt it uses, and how much it is allowed to do without asking. Why the usual flag is not enough A conventional feature flag answers one question with a boolean, and it is the right shape because a conventional feature has one failure mode: it is broken. An AI feature has several, and they want different responses. The provider is degraded — you want a different model, not the feature off. A prompt change regressed quality — you want the previous prompt, which is not a code deploy. The feature is fine but a specific customer’s data is producing bad output — you want it off for them and on for everyone else. Spend is running above forecast — you want the cheap model or the degraded path, not an outage. A single boolean answers none of these, so the response to each becomes a deploy, and a deploy is the slowest tool available at the moment you most need speed. There is a second reason, specific to this dependency. The behaviour you are flagging can change without you deploying anything, because the model is somebody else’s and it can be updated underneath you. Flags are usually a mechanism for controlling your own changes; here they are also the mechanism for reacting to changes you did not make, which is why detecting a provider-side behaviour change and having a flag to respond with are two halves of one control. Four things to flag separately Axis Description Feature on/off The ordinary flag. Per-tenant and per-segment, because the common case is a problem confined to one customer's data rather than a global outage. Model selection Which model each call site uses, as configuration. This is what lets you switch providers during an incident, run a canary on a new model, or drop to a cheaper one under budget pressure — without shipping code. Promp