How I Built a Prompt-to-Music AI Agent & Browser-Based Karaoke Separator with React & ONNX
Tags: react , webdev , onnx , audio Introduction Music generation, vocal separation, and intelligent arrangement have traditionally been server-side tasks requiring complex pipelines and expensive GPU clusters. But what if we could bring the entire interactive music-creation experience—both real-time preview , offline export , prompt-based AI music generation , and local Karaoke processing —directly into the browser? In this post, I'll share how I built AI Groove Pad , a client-side React and Tone.js application featuring: A Prompt-to-Music AI Agent: Enter any prompt (e.g., "Create an energetic Tamil Kuthu beat with a driving bassline and a Nadaswaram melody" ), and the agent composes and adds the tracks directly to the arrangement. A Client-Side Karaoke Separator: Runs a local neural network with 84% accuracy using ONNX Runtime Web to separate vocals and accompaniment locally. 3. High-Performance Audio Engine: Tone.js scheduling, synth fallbacks, and real-time playback. The Tech Stack Frontend UI: React + TypeScript + Tailwind CSS for a premium, glassmorphic dark-mode interface. Audio Engine: Tone.js v15 (built on top of the Web Audio API) for sample playback, precise timing scheduling, and synthesis. Client-Side AI: ONNX Runtime Web ( onnxruntime-web ) executing a local neural network with 84% accuracy for vocal/accompaniment separation (Karaoke mode). AI Music Agent: A natural language agent interface that takes user prompts to compose midi sequences, beats, harmony, and arrangements in real-time. * Offline Rendering: OfflineAudioContext for high-speed, non-realtime rendering of arrangements straight to .wav files. 🤖 The Prompt-to-Music AI Agent With AI Groove Pad , users don't need to be music theory experts. They simply write what they want to hear. The AI Agent interprets the prompt and generates a multi-track composition containing: Groove & Beats: Automatically maps drum samples and rhythmic patterns (e.g. Parai drum, Pambai hits for Kuthu). Melody & Harmony