Building Anisha: My 10-Day Journey to a Voice Agent for Learning & Literacy
Over the last 10 days, I built Anisha, a voice agent for Learning & Literacy that can talk with users, remember them, provide learning exercises, make outbound calls, escalate to humans, track outcomes, and hand conversations to specialist agents. 👋 Meet Anisha Anisha is built using: Python LiveKit for real-time voice Murf Falcon for text-to-speech LLM for reasoning Custom tools and memory Human escalation Call analytics Specialist handoff The core flow is: User → STT → LLM → Tools / Memory / Specialist → Murf Falcon → User She can also handle Hindi and natural Hinglish, making the experience more suitable for Indian learners. 🧠 What Did I Build? Instead of trying to build everything at once, I added capabilities step by step. Memory — Anisha can remember relevant information about returning users. Learning Tools — She can provide exercises and track successful completion. Outbound Calls — I connected the agent to an outbound calling workflow. Human Escalation — Conversations that need human help can be escalated and tracked. Call Analytics — I added outcome tracking to understand whether conversations led to successful learning interactions. Math Specialist — Anisha can hand mathematics-related conversations to a dedicated specialist agent. This turned a basic voice assistant into a small multi-capability voice AI system. 😅 The Bug That Taught Me the Most The specialist handoff looked simple: Anisha → Math Specialist But there was a problem. The Math Specialist was being triggered correctly, yet its introduction was being spoken using Anisha's voice. The issue wasn't the text. It was the interaction between the active agent, session, and TTS configuration. I changed the handoff flow so Anisha doesn't speak the specialist's introduction. The specialist takes over and uses its own configured Murf voice. That taught me: In real-time voice AI, changing the agent also means managing the audio pipeline and agent state correctly. 🔧 What I Learned The biggest lesson from t