Building a real-time desktop AI copilot for calls: the hard parts
Half a year ago I asked a simple question: during an online call, could a short, to-the-point hint appear on my screen in a second or two — while the other person is still talking? Not an after-the-fact transcript, but help in the moment. The result is a desktop assistant (macOS + Windows). Below is an honest breakdown of what turned out to be hard, and which solutions worked. Engineering only, no marketing. Architecture in one paragraph On the device there are only two things: audio capture and a thin UI overlay. All the "brains" (provider keys, prompts, model selection) live on the server. The client gets a short-lived per-session token and streams audio; the server returns the transcript and the generated answer. I picked this split not for "security theater" but because otherwise keys and prompts would have to be baked into the binary — and both leak instantly. Hard part #1: system audio, not the microphone The mic only captures you. You need the other party's audio — i.e. the system output. And that's where the platform pain starts: macOS. For a long time there was no native "give me system audio" API; the classic path was a virtual audio device (BlackHole/Soundflower-style) or, in recent versions, ScreenCaptureKit, which can hand you a process's audio. ScreenCaptureKit turned out to be the best option: no kernel extensions for the user to install. Windows. WASAPI loopback saves you — you can grab whatever is going to the output device, without virtual cables. Takeaway: "system audio capture" is not one feature but two different subsystems for two OSes, and most of the early bugs were about permissions and device selection, not about audio itself. Hard part #2: latency is everything A hint that arrives 6 seconds late is useless — the conversation has already moved on. The latency budget has three parts: STT (speech → text). Streaming only. Batch "recognize after the phrase ends" immediately adds 1–2 seconds. The key metrics weren't "overall accuracy on a benchm