I built on-device workout rep counting in Flutter — here's what actually worked

I'm building TrainWiz , a Flutter app that turns real exercise into a pet-raising game: you do squats or push-ups, your phone counts the reps, and a little creature levels up and evolves. The core technical problem sounds trivial and absolutely is not: count reps from the camera, on-device, without uploading a single frame. Here's what broke along the way, and what finally worked. Why on-device Two reasons: privacy and latency. A fitness camera that streams your body to a server is a non-starter for most people, and rep feedback has to feel instant or the whole "game" loop dies. So everything runs locally with tflite_flutter + an on-device pose model — no footage ever leaves the phone. Naive attempt #1: joint-angle thresholds The obvious approach: track the knee angle, count a rep when it dips below X° and comes back up. // looks fine in a demo, dies in the real world final kneeAngle = angleBetween ( hip , knee , ankle ); if ( ! _down && kneeAngle < 100 ) _down = true ; if ( _down && kneeAngle > 160 ) { reps ++ ; _down = false ; } It demos beautifully. Then real users prop the phone on the floor, stand at an angle, and it falls apart. The trap: a phone camera gives you 2D pose. A "120° knee angle" flattens completely depending on where the camera sits — the same squat reads as 90° or 150° purely from perspective. Lifting to 3D via the model's z doesn't save you either; monocular z is noisy enough that the angle jitters across your threshold and double-counts. Naive attempt #2: a "body-line" gate Next idea: figure out which exercise you're doing so I can pick the right signal. Standing (squat) vs. horizontal (push-up) should be easy — just check if shoulder, hip and heel form a straight line, right? Wrong, again for the 2D reason. In a real push-up shot from the front-corner, shoulder–hip–heel are not collinear on the image plane — perspective bends them. I gated push-up counting on "body is a straight line" and it would just... stop counting mid-set. Nothing is more