Building a Local-First Voice Copilot for the Shell with HoldSpeak and Ollama

The Promise: A Private, Voice-Activated Shell The dream of a voice-activated command line is compelling: speak a command, see it executed. But for many developers, piping terminal input through a cloud-based API is a non-starter. This is the promise of a project like karolswdev/HoldSpeak , a cross-platform tool for local voice typing. Could it be the core of a truly local-first, push-to-talk shell assistant? I paired it with Ollama and a local llama3.2 model to find out. The goal was simple: hold a key, speak a command like "list files by size," release the key, and have the correct shell command appear, gated by a final confirmation prompt. This project turned out to be a tale of two stacks: one for voice that was surprisingly clean, and one for language that revealed the sharp edges of the local-first promise. Building the Demo To test this idea, I built a small Python script to tie these components together. You can find the complete code for this experiment, including the prompt engineering, in my demo project on GitHub: voice-activated-shell-demo . Setup Instructions Recreating this local-first voice assistant involves a few distinct steps: Install HoldSpeak from Source : Since we need to use it as a library, clone the repository and install it in editable mode. git clone https://github.com/karolswdev/HoldSpeak.git cd HoldSpeak pip3 install -e . Install and Run Ollama : Use Homebrew (on macOS) to install the Ollama CLI, then start the server. brew install ollama ollama serve Pull a Local LLM : In a separate terminal, pull a small, capable model. I used llama3.2 . ollama pull llama3.2 Grant Permissions (macOS) : To allow the hotkey listener to work, your terminal application (e.g., iTerm, Terminal.app) must be given Accessibility permissions in System Settings > Privacy & Security > Accessibility . Run the Demo Script : With the setup complete, you can run the final Python script that integrates all these components. Finding the Seams in HoldSpeak HoldSpeak pres