标签：#murf

Introduction For the past 10 days, I took part in the 10 Days of AI Voice Agents — #VoiceForBharat Edition challenge. During this challenge, I built an AI voice agent named Sadie. My goal was not just to make an AI that could talk. I wanted to build a voice agent that could understand users, remember conversations, use tools, make phone calls, connect users to humans, and hand conversations to specialist agents. This journey helped me understand that building a voice agent is much more than connecting an LLM with a text-to-speech API. The Problem Many people find it easier to speak than type. This can be especially useful for people who want to: Ask questions using their voice Learn through conversation Get quick information Speak in Hindi or English Use Hindi and English together Get help without using complicated interfaces I wanted to build a voice assistant that could make learning and getting information feel more natural. Instead of typing a question, users can simply speak to Sadie. What I Built Sadie is an AI voice agent that can: Have real-time voice conversations Understand Hindi-English code-mixed conversations Follow personality and safety rules Remember information with user permission Use external tools Make outbound phone calls Escalate conversations to humans Track call information Hand conversations to specialist agents The specialist agents I built are: Grammar Specialist Maths Specialist Full Stack Development Specialist How the System Works The basic architecture of my project is: User | | Voice ↓ LiveKit | ↓ Deepgram STT | ↓ Google Gemini / | \ / | \ Memory Tools Specialists \ | / \ | / ↓ Murf Falcon | ↓ User Voice Main Components Deepgram handles Speech-to-Text. Google Gemini acts as the brain of the agent and understands the user's request. Murf Falcon converts the AI's response into natural speech. LiveKit handles real-time audio communication. I also added memory, external tools, telephony, human escalation, analytics, and specialist agents

Over the last 10 days, I built Anisha, a voice agent for Learning & Literacy that can talk with users, remember them, provide learning exercises, make outbound calls, escalate to humans, track outcomes, and hand conversations to specialist agents. 👋 Meet Anisha Anisha is built using: Python LiveKit for real-time voice Murf Falcon for text-to-speech LLM for reasoning Custom tools and memory Human escalation Call analytics Specialist handoff The core flow is: User → STT → LLM → Tools / Memory / Specialist → Murf Falcon → User She can also handle Hindi and natural Hinglish, making the experience more suitable for Indian learners. 🧠 What Did I Build? Instead of trying to build everything at once, I added capabilities step by step. Memory — Anisha can remember relevant information about returning users. Learning Tools — She can provide exercises and track successful completion. Outbound Calls — I connected the agent to an outbound calling workflow. Human Escalation — Conversations that need human help can be escalated and tracked. Call Analytics — I added outcome tracking to understand whether conversations led to successful learning interactions. Math Specialist — Anisha can hand mathematics-related conversations to a dedicated specialist agent. This turned a basic voice assistant into a small multi-capability voice AI system. 😅 The Bug That Taught Me the Most The specialist handoff looked simple: Anisha → Math Specialist But there was a problem. The Math Specialist was being triggered correctly, yet its introduction was being spoken using Anisha's voice. The issue wasn't the text. It was the interaction between the active agent, session, and TTS configuration. I changed the handoff flow so Anisha doesn't speak the specialist's introduction. The specialist takes over and uses its own configured Murf voice. That taught me: In real-time voice AI, changing the agent also means managing the audio pipeline and agent state correctly. 🔧 What I Learned The biggest lesson from t