This is how I added an in-browser auto captions feature to my YouTube Shorts converter web application using Whisper AI and ffmpeg.wasm

A few weeks ago I launched Convert to Shorts — a free browser-based tool that converts horizontal videos to YouTube Shorts format (9:16) without uploading anything to a server. I wrote about the ffmpeg.wasm + Vite setup in a previous article. The most requested feature after launch was auto captions. Captions significantly boost Shorts engagement since most people watch without sound, and manually typing captions is tedious. The challenge: how do you add free auto captions to a privacy-first tool that never uploads your video to a server? The answer: run Whisper AI in the browser. The stack - Transformers.js ( @xenova/transformers ) — Hugging Face's JavaScript port of the Transformers library, runs ONNX models in the browser via WebAssembly Whisper tiny — OpenAI's speech recognition model, 75MB, surprisingly accurate for clear speech Web Audio API — for extracting and resampling audio from the video file ffmpeg.wasm — for burning captions into the video ASS subtitles — the subtitle format libass (inside ffmpeg.wasm) understands. Step 1: Audio extraction Whisper expects mono 16kHz audio as a Float32Array. The Web Audio API handles this cleanly: async function extractAudio ( file : File , trimStart : number , trimEnd : number ): Promise < Float32Array > { const arrayBuffer = await file . arrayBuffer (); const audioContext = new AudioContext ({ sampleRate : 16000 }); const audioBuffer = await audioContext . decodeAudioData ( arrayBuffer ); const sampleRate = audioContext . sampleRate ; const startSample = Math . floor ( trimStart * sampleRate ); const endSample = Math . floor ( trimEnd * sampleRate ); // Mix down to mono, slice to trim range const channelData = audioBuffer . getChannelData ( 0 ); const trimmed = channelData . slice ( startSample , endSample ); await audioContext . close (); return trimmed ; } Creating the AudioContext at 16kHz means the browser automatically resamples from whatever the source rate is (usually 44.1kHz or 48kHz). No manual resampling nee