Build a caption QA harness in Python: WER, missed entities, timing and reading rate

TL;DR We're building a caption evaluation harness that scores a WebVTT file on four axes instead of one: word error rate under a fixed normalizer, missed entity rate on domain terms, median cue timing offset, and reading rate in characters per second. Python 3.12, jiwer , whisper_normalizer , webvtt-py . Run it on every model or vendor change. A caption file can score 96% accurate and still be unusable. WER counts substitutions, insertions and deletions and weighs each one the same, so "fifteen milligrams" becoming "fifty milligrams" costs exactly as much as "the" becoming "a". It also throws away every timestamp before it starts, which means synchronization and readability are invisible to it. Let's measure the other three things. 0. Setup 🛠️ python3 -m venv .venv && source .venv/bin/activate pip install jiwer whisper_normalizer webvtt-py $ pip list | grep -Ei 'jiwer|whisper|webvtt' jiwer <your version> webvtt-py <your version> whisper-normalizer <your version> Pin whatever you install, and pin it in CI. The APIs below move between majors, which is exactly why the next tip exists. 💡 Tip: jiwer.compute_measures() is gone in recent versions. It is jiwer.process_words() now, and it returns a WordOutput dataclass. Most blog posts you will find still use the old name. 1. Parse the VTT into text plus timings # captions.py from dataclasses import dataclass import webvtt @dataclass class Cue : start : float end : float text : str @property def duration ( self ) -> float : return self . end - self . start @property def lines ( self ) -> list [ str ]: return self . text . split ( " \n " ) @property def flat ( self ) -> str : return " " . join ( l . strip () for l in self . lines ) @property def chars_per_second ( self ) -> float : return len ( self . flat ) / self . duration if self . duration > 0 else float ( " inf " ) def _to_seconds ( ts : str ) -> float : h , m , s = ts . split ( " : " ) return int ( h ) * 3600 + int ( m ) * 60 + float ( s ) def load_vtt ( path : str ) -