标签：#voice

There's a school of voice-agent development that treats turn detection as something you buy, not something you build. Pick a streaming STT provider, let its end-of-turn logic decide when the user is done, and move on. For a lot of teams that's the right move — and if you're weighing the options, our breakdown of turn detection vs forced endpoints is the place to start. But some teams have already solved turn detection. They've tuned their own voice-activity detection over thousands of calls, they know their audio, and they trust their endpointing more than any default. For those teams, a streaming model's built-in turn logic isn't a feature — it's something to work around. What they want is narrower and faster: hand over a finished chunk of speech, get accurate text back, get out of the way. That's the case for bringing your own turn detection and pairing it with fast ASR over HTTP. Turn detection is an architectural decision, not a default Here's the framing that matters. In a streaming setup, the STT model is a participant in the conversation — it's watching the audio and deciding, continuously, whether the user has finished. That's genuinely useful when you want the provider to own that judgment. But it means the model is inserting its own decision between "user stopped talking" and "you get the transcript." If you already know the turn is over — because your VAD just fired — you don't want the model deliberating. You want it transcribing. Every millisecond the STT layer spends re-deciding a question you've already answered is latency you're adding for no benefit. So the decision isn't "which provider has the best turn detection." For these teams it's "who owns the turn boundary?" If the answer is you, then the ideal STT layer is one that does exactly one thing: turn a finished clip into accurate text, fast. Built-in vs. bring-your-own Built-in (streaming). The model reads tonality, pacing, and rhythm to detect end-of-turn — with Universal-3.5 Pro Realtime, aroun

Rime is handling over 100 million calls each month across multiple companies

OpenAI says its new voice mode can speak and listen at the same time, a key ability for live translation.

The Promise: A Private, Voice-Activated Shell The dream of a voice-activated command line is compelling: speak a command, see it executed. But for many developers, piping terminal input through a cloud-based API is a non-starter. This is the promise of a project like karolswdev/HoldSpeak , a cross-platform tool for local voice typing. Could it be the core of a truly local-first, push-to-talk shell assistant? I paired it with Ollama and a local llama3.2 model to find out. The goal was simple: hold a key, speak a command like "list files by size," release the key, and have the correct shell command appear, gated by a final confirmation prompt. This project turned out to be a tale of two stacks: one for voice that was surprisingly clean, and one for language that revealed the sharp edges of the local-first promise. Building the Demo To test this idea, I built a small Python script to tie these components together. You can find the complete code for this experiment, including the prompt engineering, in my demo project on GitHub: voice-activated-shell-demo . Setup Instructions Recreating this local-first voice assistant involves a few distinct steps: Install HoldSpeak from Source : Since we need to use it as a library, clone the repository and install it in editable mode. git clone https://github.com/karolswdev/HoldSpeak.git cd HoldSpeak pip3 install -e . Install and Run Ollama : Use Homebrew (on macOS) to install the Ollama CLI, then start the server. brew install ollama ollama serve Pull a Local LLM : In a separate terminal, pull a small, capable model. I used llama3.2 . ollama pull llama3.2 Grant Permissions (macOS) : To allow the hotkey listener to work, your terminal application (e.g., iTerm, Terminal.app) must be given Accessibility permissions in System Settings > Privacy & Security > Accessibility . Run the Demo Script : With the setup complete, you can run the final Python script that integrates all these components. Finding the Seams in HoldSpeak HoldSpeak pres

The boat-agent stack here runs on a prime directive: if there's something usable out there, improve it; build our own only as a last resort. So when we needed a SignalK MCP server, the honest first move wasn't to write one — it was to evaluate the one that already exists. VesselSense/signalk-mcp-server (TypeScript, MIT) is good work. It exposes SignalK to an agent through a single execute_code tool: the model writes JavaScript, the server runs it in a sandboxed V8 isolate ( isolated-vm ), and only the result comes back. Its README claims a 90–96% token reduction versus traditional named MCP tools — 2,000 tokens down to 120 for a vessel-state query, 13,000 down to 300 for a multi-call workflow. Those numbers are plausible, and they line up with the broader industry result that code execution beats tool-calling on token efficiency for complex multi-step work. We read it, ran the numbers against our own agent, and kept our discrete-named-tool signalk-mcp anyway — then harvested three of VesselSense's ideas into our roadmap. This post is that evaluation: the two philosophies, why the obvious-sounding win doesn't bind for a voice-first agent, and a decision framework you can reuse before you adopt-or-build your own MCP server. This is a design-reasoning post, not a debugging saga, but it maps to the same arc: a question, the dead-end that looks like an obvious yes, and the call that actually held. The question Two SignalK MCP servers, two genuinely different designs: VesselSense/signalk-mcp-server sailingnaturali/signalk-mcp ───────────────────────────── ─────────────────────────── one tool: execute_code discrete named tools: → agent writes JavaScript read_sensor(path) → runs in a V8 isolate battery_state(bank) → queries SignalK, returns depth_state() only the result get_route() get_local_time() TypeScript / Node + isolated-vm list_paths(prefix) claims 90–96% fewer tokens get_active_alarms() Python, end-to-end The adopt-vs-keep question: does the token-efficiency win bin

Sure, anyone can use OpenAI’s chatbot. But with smart engineering, you can get way more interesting results.

📝 Originally published in Japanese on Zenn. This is the English version. Canonical: https://zenn.dev/uya0526_design/articles/satellite3_metrics-rationale 📚 This is satellite article #3 in my "Read-Aloud Speed Meter dev log" series. For the whole picture, see the main article . Where This Sits The read-aloud speed meter converts speaking speed into an evaluation label like "slightly fast," and stagnation rate into one like "few." Those labels ultimately become the foundation for Claude Haiku's feedback. So — on what basis did I draw the thresholds (the dividing lines)? This article digs into that "basis." The short answer from my research: I couldn't find a paper that defines an academic threshold for "N characters/min = fast/slow." This is a record of how I drew the lines honestly once I'd learned there was no firm basis. More than the metric numbers themselves, I believe being transparent about why I chose those numbers is what makes an evaluation app trustworthy. 💡 I'm an ex-Java engineer learning TypeScript in public. This one is mostly about design decisions. Why Obsess Over the "Basis"? An evaluation app passes judgment on the user: "your reading is slightly fast." Once you're passing judgment, if you can't explain "why we can say that," it's just guesswork. This app in particular passes the labels straight to Claude Haiku to generate coaching. If the foundational label has an unclear basis, the feedback built on top of it is a castle on sand. So I decided to nail down the basis for the thresholds first. Two things to research: The judgment basis for speaking speed (characters/min) The judgment basis for stagnation rate (the proportion of silence) As it turned out, these two had completely different kinds of basis. Speed Thresholds: No Academic Threshold → Draw From General Rules of Thumb What I found For speaking speed, I first looked for academic backing. Here's what I found: Speaking speed has traditionally been measured against mora count, but prior researc

Arriving six years after Google’s last smart speaker, the new HomePod-style device was redesigned to play host to Gemini’s chatbot.

My voice agent took 1.2 seconds. Users hated it. So I made it lie. A while back I shipped a voice agent that took roughly 1,200ms to respond. Not catastrophic on paper. Pretty bad in practice. Users would ask a question, get a beat of silence, and start over. Some thought the mic had cut out. One tester told me, with a straight face, that my agent was "thinking too hard." I tried everything legitimate first. Smaller LLM. Streaming TTS. Region-pinned endpoints. I shaved off about 200ms and felt clever for a week. Then I measured again and realized I was still on the wrong side of every latency threshold that matters. So I gave up on being faster and started working on being a better liar. This is the playbook I wish I had when I started: five perception tricks that reduce felt latency without touching the actual numbers. They're the voice-AI equivalent of a magician's misdirection. Your right hand waves at the audience. Your left hand swaps the card. The cliff you can't engineer your way out of In a previous article I broke down the three latency cliffs for voice AI. The short version: Around 200ms : the brain starts to register the pause as "slow." This is the conversational baseline humans use with each other. Around 500ms : the conversation breaks. The user starts to wonder if they need to repeat themselves. Around 800ms : they've quietly given up. Even if your answer arrives, the trust is gone. If your stack is doing STT plus LLM plus TTS plus network, hitting 200ms end-to-end is, frankly, a fantasy for most teams. You can chase it. You can throw money at it. You can cache and prefetch and stream. At some point you bottom out. That's where perception work begins. The user can't measure your p99 latency. They can only measure how the agent feels . Those are two different problems and they have two different solutions. 5 tricks I now use to mask latency 1. Acknowledgment tokens ("Got it", "On it", "Let me check") What it is: A short, instant utterance played the mo

Last sprint, a team I talked to demoed a voice agent that handled support calls impressively — right up until a caller asked "can you email me those instructions?" and the room went quiet. The agent could talk about the docs. It had no address to send them from. The workaround on the whiteboard afterwards was grim: relay through a shared noreply@ , lose the replies, reconcile threads manually in the ticketing system. Voice agents hit this wall constantly, because phone calls generate follow-up artifacts — reset instructions, documents, meeting recaps — and email is how callers expect to receive them. The clean fix is the same one that works for text agents: the voice agent gets its own mailbox. The identity half A Nylas Agent Account is a hosted mailbox you create through the API — Agent Accounts are in beta — and the voice use case from the product docs is exactly the scenario above: a voice agent taking support calls sends documents, reset instructions, or meeting recaps from its own voice-agent@yourcompany.com address the moment the caller asks. The part that makes it more than a send pipe: when the caller replies, the reply returns through the same account, so the full conversation is one thread in one mailbox. The phone call and its written follow-ups stop living in separate systems. Each account is a real grant with a grant_id that works against the existing Messages, Threads, and Webhooks endpoints, ships with six system folders, and sends up to 200 messages per account per day on the free plan. The plumbing half The voice agents recipe covers how the runtime actually calls email tools. The flow is the same regardless of vendor: speech → STT → LLM (function-calling) → subprocess(nylas …) → JSON → LLM → TTS → speech The LLM decides on a tool, the runtime spawns a Nylas CLI subprocess with --json , the result comes back, and the model composes a spoken response. On LiveKit, a tool is just a decorated function: from livekit.agents import function_tool import sub

** How I instrument ASR, LLM, TTS, and the client with OpenTelemetry, and which number in each layer I actually look at ** TL;DR. A voice agent is four moving parts stuck together: speech to text, the model that writes the reply, text to speech, and the client that plays the audio back. End to end latency hides which of those four is slow on any given turn, so I stopped tracking it as one number and started tracing each stage as its own OTel span with a shared session id. The number I watch hardest is barge-in: when the user starts talking over the agent, how many milliseconds until the agent actually stops sending audio. In our setup we want that under 200ms, and when p95 barge-in creeps past that, the agent feels like it is talking at you instead of with you. Everything below is how I wire the spans, what attributes go on each one, and the p95 I page on per layer. The thing I keep saying, and the thing that keeps being true: voice agents fail in production not because of raw latency but because nobody simulated the audio and LLM pipeline together. You can have a fast ASR, a fast model, a fast TTS, and a voice agent that still feels broken, because the failure lives in the seams between them and in the parts (barge-in, jitter) that no single-stage benchmark touches. Tracing is how I get the seams to show up. A note before the layers. This is just the setup we run, the spans we emit, and the mistakes that made us add each attribute. Some of it is probably specific to our stack and will not transfer. I will flag that where I can. The shape of a turn, and why one span is not enough One turn is: user says a thing, agent says a thing back. Underneath that is roughly: audio frames come in, ASR turns them into text (streaming partials as it goes); the text plus history goes to the LLM, which streams tokens back; as text comes out, TTS turns it into audio, also streaming; the client receives audio frames and plays them, with some buffering to smooth out jitter. If you wrap

Over the past few months I’ve been building VoiceIntego, an AI voice agent that answers calls and books appointments for service businesses (dental clinics, HVAC, plumbing). Here are some of the technical lessons that surprised me along the way. Latency is the whole game With text chatbots, a 2-second delay is fine. On a phone call, anything over ~800ms feels broken — people start talking over the AI. The hard part isn’t the LLM response; it’s the round trip: speech-to-text → LLM → text-to-speech, all streaming. You have to stream every stage and start TTS before the full response is generated. Interruptions break naive pipelines Real callers interrupt. “Actually, can we do Tuesday instead—” mid-sentence. A simple request/response loop can’t handle this. You need barge-in detection: monitor the incoming audio stream and cancel the current TTS playback the moment the caller starts speaking again. Booking logic needs guardrails, not vibes Letting the LLM “decide” availability is a recipe for double-bookings. The reliable pattern: the LLM extracts intent (date, time, service), then deterministic code checks the actual calendar API and confirms. The model handles language; your code handles truth. Confirmation loops matter more than you’d think Always read the booking back: “So that’s a cleaning on Tuesday the 9th at 2pm — correct?” Phone audio is noisy and names/times get misheard constantly. One extra confirmation turn cuts errors dramatically. Phone numbers and edge cases everywhere Voicemail detection, callers who mumble, background noise, people who say “yeah” to mean no. The happy path is maybe 20% of the work. If you’re building something in this space, happy to compare notes. You can see what I’m working on at VoiceIntego .