标签：#voiceai

Every AI video pipeline eventually has to answer an unglamorous question: what did we actually pay for that clip? On the main video-generation service, the answer for months had been "a hardcoded constant." That's fine until the vendor changes its own pricing, or a code path pays for the same synthesis twice, or a voice engine mints a clone, bills for it, and never sends it downstream. Over a ten-PR run I audited and rebuilt the voice and lip-sync pipeline from the billing layer up, then used the vendor's own SKU tiers to cut cost 7x without touching output quality. A cost model built from hardcoded constants isn't a cost model. It's a guess that happens to compile. Billing what the vendor actually charges PR #224 was workstream one of three from a sibling-tool audit: port the cost-accounting fixes that Presenter Generation and Variant Multiplier had already found, verifying each one against this repo's own code rather than assuming the same defect existed in the same place. Anthropic returns exact token counts on every response. Nothing in the pipeline read them — every charge was a hardcoded per-call constant, so the ledger and the vendor invoice diverged the moment usage drifted from whatever number had been typed in at launch. The same PR closed a second gap: two editor-facing routes could spend money — kicking off a generation, retrying a step — outside any run . A run is the unit everything else (budgets, audit trail, the cost ledger) is keyed to. A spend with no run attached is a spend the ledger can't even see, which is worse than a wrong number. Paying twice for a take the model returns unchanged PR #225 found the sibling bug's twin: some vendor calls return the exact same asset on a retry — no new synthesis happened — and the pipeline billed a second time anyway because "call succeeded" and "call did new work" were treated as the same fact. The fix is the boring, correct kind: hash the output, and only charge when the hash changes from the take you already

Building FinSaathi: A Voice-First AI Financial Assistant Financial information can be difficult to understand. Banking terms, loans, credit scores, payments, and other financial decisions can quickly become overwhelming when users have to navigate everything through forms and complicated interfaces. So I wanted to explore a simpler interaction: What if financial guidance could start with a conversation? That idea became FinSaathi , a voice-first AI financial assistant. What I Built The first goal was simple: get a real-time voice assistant working end-to-end and deploy it. The current architecture is: Next.js Frontend → LiveKit → Python AI Agent → Voice/AI Services The frontend is deployed on Vercel, while the LiveKit agent is deployed on Railway. Users can open the application, start a conversation, and interact with the FinSaathi agent through voice. The Tech Stack Frontend Next.js React TypeScript LiveKit Components Tailwind CSS Vercel Backend Python LiveKit Agents UV Docker Railway Voice / AI LiveKit Murf AI/LLM services Data SQLite for application memory and call-related data The Part That Took More Time Than Expected Getting the agent to work locally was relatively straightforward. Getting the same system to actually run in production was a different problem. The Railway deployment initially failed with: python: can't open file '//src/agent.py': [Errno 2] No such file or directory The problem turned out to be related to how the application path and startup command were being handled inside the Docker deployment. After fixing the container and Railway startup configuration, the deployment moved further — and exposed another issue. Because the container runs the application as a non-root user, UV initially could not create its cache directory: Permission denied: '/app/.cache/uv' Fixing the permissions allowed the actual LiveKit AgentServer to start successfully. The production logs then showed the agent listening for connections and registering its worker with L

It was during the peak monsoon season when I read a distress report from a family stranded on their rooftop. Power was flickering, rain was hammering against the walls, and cellular data was down to a crawling 2G edge. They had a phone with 14% battery, but opening an emergency app or downloading heavy government disaster PDFs was impossible. All they could do was place a direct phone call. That moment stayed with me. When panic sets in and water is rising inside your living room, you don't navigate drop-down menus or type search queries into a browser. You need to speak, and you need a voice that answers immediately with verified life-saving relief info. That became the driving mission behind Sentinel — an autonomous, real-time Voice AI emergency dispatcher that I built over 10 days during the #VoiceForBharat challenge. The Problem I Wanted to Solve In emergency response across India, victims and disaster managers face three immediate hurdles: Information Fragmentation: Emergency guidelines, live rainfall alerts, and shelter capacities exist across different departments. A caller in panic needs instant answers (e.g., "Is there a shelter in Guwahati with medical support and space for pets?" ). The Friction of Touch UIs: Wet screens, low digital literacy, and high adrenaline make text interfaces fail. Voice is the most natural, accessible lifeline. Context Collapse: When standard chatbots escalate a user to another team or system, they drop the context and force the distressed victim to repeat their story from scratch. How Sentinel Works Under the Hood To make Sentinel feel like a natural human dispatcher, every millisecond of latency had to be eliminated. The system streams voice bidirectionally through a unified WebRTC pipeline: Speech-to-Text (STT): Deepgram Nova-3 transcribes incoming audio streams in real time with multilingual code-mixing support (English & Hindi). Brain & Reasoning (LLM): Google Gemini handles real-time disaster triage, safety guardrails, and

Over the last 10 days, I built Anisha, a voice agent for Learning & Literacy that can talk with users, remember them, provide learning exercises, make outbound calls, escalate to humans, track outcomes, and hand conversations to specialist agents. 👋 Meet Anisha Anisha is built using: Python LiveKit for real-time voice Murf Falcon for text-to-speech LLM for reasoning Custom tools and memory Human escalation Call analytics Specialist handoff The core flow is: User → STT → LLM → Tools / Memory / Specialist → Murf Falcon → User She can also handle Hindi and natural Hinglish, making the experience more suitable for Indian learners. 🧠 What Did I Build? Instead of trying to build everything at once, I added capabilities step by step. Memory — Anisha can remember relevant information about returning users. Learning Tools — She can provide exercises and track successful completion. Outbound Calls — I connected the agent to an outbound calling workflow. Human Escalation — Conversations that need human help can be escalated and tracked. Call Analytics — I added outcome tracking to understand whether conversations led to successful learning interactions. Math Specialist — Anisha can hand mathematics-related conversations to a dedicated specialist agent. This turned a basic voice assistant into a small multi-capability voice AI system. 😅 The Bug That Taught Me the Most The specialist handoff looked simple: Anisha → Math Specialist But there was a problem. The Math Specialist was being triggered correctly, yet its introduction was being spoken using Anisha's voice. The issue wasn't the text. It was the interaction between the active agent, session, and TTS configuration. I changed the handoff flow so Anisha doesn't speak the specialist's introduction. The specialist takes over and uses its own configured Murf voice. That taught me: In real-time voice AI, changing the agent also means managing the audio pipeline and agent state correctly. 🔧 What I Learned The biggest lesson from t

Building Samar: My 10-Day Voice AI Agent Journey with Murf Falcon Over the past 10 days, I built Samar , a multilingual AI voice agent for a Bharat Digital Bank use case as part of the 10 Days of Voice Agents – VoiceForBharat Edition challenge. The project started as a simple voice assistant and gradually evolved into a more complete Voice AI system capable of remembering users, using real-time tools, making outbound calls, escalating sensitive situations to humans, analyzing calls, and handing specialized conversations to another AI agent. 🎯 The Problem Banking can sometimes be difficult to navigate, especially when users need quick information or assistance without going through multiple screens and menus. I wanted to build a voice-first banking assistant that could provide natural conversations while also maintaining security and knowing when it should involve a human. That's where Samar comes in. 🤖 What is Samar? Samar is a multilingual banking voice agent designed to help users with general banking-related queries. It can: Answer general banking questions Provide financial information Remember returning users with consent Fetch real-time information using tools Find nearby branches Provide exchange-rate information Make outbound reminder calls Escalate sensitive issues to human support Track call analytics Hand specialized conversations to a specialist agent The voice experience is powered by Murf Falcon , the fastest TTS API used in this challenge. 🏗️ How the System Works At a high level, the voice interaction follows this flow: User Speech ↓ Speech-to-Text ↓ LLM / Agent Logic ↓ Memory or Tool Calling ↓ Text-to-Speech ↓ User hears the response The system uses real-time voice communication through LiveKit, an LLM for reasoning and conversation, speech recognition for understanding the user, and Murf Falcon for natural voice generation. 🚀 Important Features 1. Voice AI with Guardrails Samar has a clear banking role and follows safety rules. It does not ask users

Building Roshni: An Ultra-Low Latency, Multi-Agent Financial Voice Assistant for Bharat 🇮🇳 How I built an end-to-end, multilingual financial voice AI using Murf Falcon, LiveKit Agents, Deepgram Nova-3, Google Gemini, and Next.js during the 10 Days of AI Voice Agents Challenge. 🌟 1. The Problem & Why Voice Matters for Bharat In India, financial inclusion has accelerated rapidly with UPI, digital banking, and government-backed credit initiatives. However, navigating complex interest rates, eligibility criteria for government schemes (like PM Mudra or Sukanya Samriddhi Yojana), and understanding formal banking terms remains intimidating for millions of citizens—especially in regional and tier-2/3 heartlands where digital interfaces can be overwhelming. Text-first interfaces fail where voice thrives. When rural entrepreneurs or first-time bank customers have questions, they don't want to navigate complex web forms or read dense PDFs. They want to ask a direct question in their language and get an immediate, clear, spoken answer. To solve this, I built Roshni AI (and her specialist counterpart, Vikram ) — an ultra-low latency, conversational financial assistant engineered for natural voice interactions in English, Hindi (Devanagari script), and Hinglish. 🏗️ 2. High-Level Architecture & Tech Stack Building a real-time conversational agent requires synchronizing four core pipelines with sub-second latency: [ 👤 User Microphone ] │ (WebRTC Audio Stream) ▼ ┌─────────────────────────────┐ │ LiveKit Agents Worker │ └──────────────┬──────────────┘ │ ┌───────────────────────┼───────────────────────┐ ▼ ▼ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │ Deepgram │ ────► │Google Gemini│ ────► │ Murf Falcon │ │ Nova-3 │ │ (LLM) │ │ Fast TTS │ │ (Fast STT) │ │ │ │ (Anisha / Samar)│ └─────────────┘ └──────┬──────┘ └────────┬────────┘ │ (Tool / Handoff) │ ▼ ▼ ┌───────────────┐ [ 🔊 Audio Output ] │ SQLite Memory │ │ & Analytics │ └───────────────┘ The Stack: TTS (Text-to-Speech):

My first voice agent took 1,200ms to answer a spoken sentence. Then I rewrote three seams in the pipeline and it dropped to 340ms. No new hardware, no new models, no smaller LLM. The words the user says, the words the agent says back, the same. What changed was the shape of the wait. If you have ever built a voice agent that felt polite but slow, this is the part of the pipeline where the seconds hide. The 1,200ms baseline was polite and wrong Here is what my first version did, in the order it did it: Record until the user stops talking (~200ms of tail silence). Send the whole clip to Whisper. Wait for the transcript. Send the transcript to the LLM. Wait for the full response. Send the full response to Piper. Wait for the WAV. Play the WAV. Each stage was fine on its own. The pipeline was a one-lane road. Whisper could not start until recording finished. The LLM could not start until Whisper finished. Piper could not start until the LLM was done. The user waited for the sum. The car metaphor gets old fast, so I will use a real one. This is what the timeline looked like on my machine: [record]--[200ms silence]--[whisper 380ms]--[LLM 480ms]--[piper 340ms]--[playback] ^ 1,200ms Every one of those bars was blocking the next. I had built a relay race where each runner waited for the previous runner to sit down. Trick 1: Frame-based STT so Whisper starts before the user stops The first fix is to stop treating the user's speech as a single file. Feed the audio to Whisper in 20-30ms frames as it is captured. By the time the user hits the tail silence, most of the transcription is already done. You only wait for the last few frames plus a short flush. Pipecat is the reference implementation. Its whole model is frame-based: every stage processes 20-30ms chunks and hands them forward as soon as they are ready. There is no batch, no full-clip handoff, no "wait for this stage to complete." Its own docs quote sub-500ms voice-to-voice when all models are hosted on the same GPU clu

A HIPAA-compliant AI voice agent for healthcare typically costs $40,000-$150,000 to build, depending on call complexity and EHR integration, plus $2,000-$15,000/month to operate. The build cost isn't dominated by the speech model — it's dominated by the compliance and data-retention layer wrapped around it. Most cost estimates for "AI voice agents" quietly assume a sales or support use case, where a wrong transcription costs you an annoyed customer. In healthcare, a wrong transcription in a medication name or a dropped consent statement is a liability. That difference reshapes the budget. Where the money actually goes 1. Speech recognition (10-20% of build cost) This is the smallest line item, despite being the part founders worry about most. You have three options: Managed API with a BAA (e.g., enterprise-tier Deepgram, Azure Speech, Google Healthcare API) — fastest to ship, but you're paying per-minute and locked into the vendor's accuracy on medical terminology. Fine-tuned open-weight model — better accuracy on clinical vocabulary and accents, but adds MLOps overhead. Self-hosted model — highest control over data residency, needed if your contracts or state law prohibit sending PHI to a third party. If your patient population speaks Gulf Arabic or another dialect underserved by mainstream ASR, budget separately for this — see our breakdown on Arabic speech recognition costs for how accent and dialect coverage move accuracy and price independently of the base model choice. 2. Compliance infrastructure (30-40% of build cost) This is where healthcare voice AI diverges hardest from a generic voice bot: Business Associate Agreements with every vendor in the call path (ASR, LLM, telephony, storage) Encryption at rest and in transit, with key management you can audit Role-based access control on transcripts and recordings Immutable audit logs of who accessed what patient data and when The U.S. Department of Health and Human Services publishes the actual HIPAA Security R

My voice agent took 1.2 seconds. Users hated it. So I made it lie. A while back I shipped a voice agent that took roughly 1,200ms to respond. Not catastrophic on paper. Pretty bad in practice. Users would ask a question, get a beat of silence, and start over. Some thought the mic had cut out. One tester told me, with a straight face, that my agent was "thinking too hard." I tried everything legitimate first. Smaller LLM. Streaming TTS. Region-pinned endpoints. I shaved off about 200ms and felt clever for a week. Then I measured again and realized I was still on the wrong side of every latency threshold that matters. So I gave up on being faster and started working on being a better liar. This is the playbook I wish I had when I started: five perception tricks that reduce felt latency without touching the actual numbers. They're the voice-AI equivalent of a magician's misdirection. Your right hand waves at the audience. Your left hand swaps the card. The cliff you can't engineer your way out of In a previous article I broke down the three latency cliffs for voice AI. The short version: Around 200ms : the brain starts to register the pause as "slow." This is the conversational baseline humans use with each other. Around 500ms : the conversation breaks. The user starts to wonder if they need to repeat themselves. Around 800ms : they've quietly given up. Even if your answer arrives, the trust is gone. If your stack is doing STT plus LLM plus TTS plus network, hitting 200ms end-to-end is, frankly, a fantasy for most teams. You can chase it. You can throw money at it. You can cache and prefetch and stream. At some point you bottom out. That's where perception work begins. The user can't measure your p99 latency. They can only measure how the agent feels . Those are two different problems and they have two different solutions. 5 tricks I now use to mask latency 1. Acknowledgment tokens ("Got it", "On it", "Let me check") What it is: A short, instant utterance played the mo