How I Built a Self-Hosted Family AI Health Steward (Your Health Data, on Your Shelf)

TL;DR — I built and open-sourced AI Health Steward , a self-hosted, private AI health manager for families. It reads photos of lab reports with multimodal LLMs, builds a structured per-person health profile, shows trends on a dashboard, and answers health questions grounded in your actual data — all running on your own server. Privacy isn't a feature; it's the whole point. Star it on GitHub . The problem: your health data is a product Every family has a shoebox — or a folder — of medical reports: blood tests, blood-pressure logs, prescriptions, scan findings. And every "convenient" health app wants to hold those records for you. But hold them where ? On someone else's cloud, to be monetized, analyzed, or lost when the startup pivots. Health records are the most sensitive data you own. They shouldn't be a product. They should live on your shelf. So I built the opposite: a self-hosted AI health steward where the data never leaves your server. What it does 📄 Take a photo of a lab report → structured data. A multimodal LLM extracts key metrics (BP, glucose, lipids, CBC…) with your confirmation before anything is filed. 🧬 A person-level health profile as the single source of truth — basics, metrics, diagnoses, medications, allergies, lifestyle, family history, and data provenance (where each value came from). 📈 Trend visualization with anomaly markers and clinical critical-value alerts (e.g. BP ≥ 180/110 triggers a "see a doctor" banner). 💬 AI consultation grounded in real data — not a generic chatbot. Intent routing + function calling means answers reflect your profile, not Wikipedia. 🗓️ Personalized checkup plans via a 1+X+Y framework, with budget tiers and safety/contraindication screening. 📋 Periodic health summaries (weekly/monthly/yearly), risk scales (PHQ-9, GAD-7, diabetes, ASCVD), and follow-up/medication reminders . 🧠 RAG over your own history — archived reports are vectorized so you can ask "what did my A1C trend look like over 3 years?" The architecture ┌────