标签：#selfhosted

Six months ago I moved my RAG pipeline from Pinecone to self-hosted Qdrant. My vector search bill went from $210/month to $6.50/month. Same latency. Same recall. Here's exactly how. The Setup My app does document Q&A for legal contracts. The numbers: 5.2 million vectors (1536-dim, OpenAI embeddings) ~800K queries/month P99 latency requirement: < 50ms On Pinecone Serverless, this cost me roughly $210/month — storage plus read units plus write units for daily ingestion of new documents. What I Moved To A single Hetzner CX32 server: 4 vCPU, 8 GB RAM, 80 GB SSD €8.50/month (about $9.20) Qdrant running in Docker Automated daily backups to S3-compatible storage ($0.50/month) Total: ~$10/month. That's a 95% cost reduction. The Migration Was Easier Than Expected bash# Export from Pinecone (I used their scroll API) python export_pinecone.py --index legal-docs --output vectors.jsonl Start Qdrant docker run -d -p 6333:6333 -v ./storage:/qdrant/storage qdrant/qdrant Import python import_qdrant.py --input vectors.jsonl --collection legal-docs The whole migration took an afternoon. The Qdrant Python client is straightforward, and the API is surprisingly similar to Pinecone's. Performance Comparison I ran the same 10,000 test queries against both setups: MetricPinecone ServerlessQdrant Self-HostedP50 latency23ms4msP99 latency89ms12msRecall@100.970.97Monthly cost$210$10 The self-hosted Qdrant is actually faster because the data sits in memory on the same machine. Pinecone Serverless loads data from object storage on demand, which adds cold-start latency. When Self-Hosting Is a Bad Idea I want to be honest about the trade-offs: Don't self-host if: You have zero DevOps experience and no one on the team does You need 99.99% uptime SLA for enterprise customers Your vector count is growing unpredictably (10M one month, 100M the next) You're a team of 1-2 and every hour on infra is an hour not building product Do self-host if: Your scale is predictable (you know roughly how many vectors

Last week I couldn't shake an idea: what if I had an AI that knew everything I know ? Not ChatGPT — something on my hardware, holding my knowledge, answering to no one's API bill. Yesterday I built it. Here's the honest breakdown. What it does NEXUS runs on a regular Windows laptop — aging i7, 16GB RAM, no GPU. It: Remembers everything. Drop any file in a folder; 60 seconds later it's searchable memory. Answers from MY knowledge. "Which of my projects were formally closed and why?" — it answers from my actual records. Watches the live web. Every 2 hours it pulls Hacker News and news feeds, learns what's trending, pings my Telegram. Reports to my phone. 7 AM daily briefing: what it learned, what's running, what needs me. The stack — all free, all open source Ollama runs the models (Llama 3.2, Mistral 7B). Open WebUI is my private ChatGPT. Qdrant stores memory. n8n automates. SearXNG searches privately. PostgreSQL, Redis, and MinIO handle data. Commercial equivalent: $300–500/month . My cost: electricity. The memory trick nobody explains simply Parse — extract text from any file Chunk — split into ~300-word pieces Embed — each chunk becomes 768 numbers representing its meaning Store — a database that searches by similarity Your question becomes 768 numbers too, and the database finds memories with similar meaning — not matching keywords. I asked "how do I get clients cheaper" and it found my notes on "reducing customer acquisition cost." Different words. Same meaning. That's the magic. What surprised me A 2GB model is genuinely useful. Llama 3.2 3B answers from my knowledge in seconds, on CPU. The automation matters more than the AI. The watched folder + Telegram bot turned a cool demo into a system I actually use. Windows is fine. Docker Desktop + WSL2 ran all nine services without drama. The bill, honestly Hardware: $0 (laptop I own) Software: $0 (open source) APIs: $0 (all local) Time: one focused day The only future cost is a cloud GPU server (~$65/mo) when I outg

I have been building Lunarr , a self-hosted web media server for people who want to scan, organize, and watch their own movie and TV libraries in the browser. It is still early, but the core idea is simple: Add local or SFTP media libraries, scan them, match metadata, and play them through a clean web UI. Lunarr is not trying to replace Plex or Jellyfin overnight. Those projects are mature and cover a huge surface area. Lunarr is currently focused on being small, direct, and practical for self-hosted setups where media may live on the same machine or on remote SFTP storage. What Lunarr does today Lunarr currently supports: Local movie and TV libraries SFTP movie and TV libraries TMDb metadata matching Movie, show, season, and episode organization Browser playback Direct streaming when the browser can play the file Temporary HLS remux/transcode when needed Seekable request-driven HLS playback Sidecar .vtt subtitle detection Admin/user accounts Library sharing controls Manual scans, scheduled scans, and local file watching Docker deployment The SFTP support is one of the important parts for me. A lot of self-hosted setups do not keep media on the same machine as the web app. Lunarr can scan remote folders and, when possible, play seekable remote media without first copying the whole file locally. Playback model Lunarr tries direct playback first when the browser can handle the file. When direct playback is not suitable, Lunarr uses temporary HLS playback. Instead of transcoding an entire movie up front, it generates segments around what the player is actually requesting. For example, if you jump from 55 minutes to 13 minutes and then to 80 minutes, Lunarr does not transcode everything between those points. It repositions FFmpeg, generates the requested segment, and prepares a small lookahead window so playback can continue. That keeps CPU and disk usage more proportional to what the viewer is actually watching. Quick start with Docker docker run -d \ --name lunarr \ -

Overnight agents do good work, then dump it in a log file or a noisy Slack channel. Here's a pattern for delivering their output to a private, end-to-end encrypted inbox you read with your coffee. You point an agent at a nightly job — audit the dependencies, summarize yesterday's support tickets, check the infra, scan the repo for regressions. It runs at 3 a.m. and does good work. Then the work goes... where? Usually one of three bad places: A log file you'll never open. A Slack channel that's already 200 messages deep by the time you wake up. A plaintext file on a server , which is fine until the report contains a leaked key, a customer name, or a security finding — and now it's sitting in cleartext on a box you don't fully trust. And the fix you'd reach for first — "just email the report to me" — is the one that bites hardest. You can do it cleanly: a locked-down, send-only API key sends mail and nothing else. But the path of least resistance is "connect your email account," and that grant is far wider than the job needs — now the agent can read and send your mail, not just hand you a file. I learned this the hard way. I once connected an agent to my email so it could send me updates — and it took that as license to start replying to my incoming messages on its own, without my ever asking. Mail went out under my name that I never wrote. The job was "send me a file." The access I'd handed over was "run my inbox." The work is good. The delivery is the broken part. Here's a pattern that fixes it: your overnight agent delivers its report to a private, end-to-end encrypted inbox, and you read it with your coffee — decrypted in your browser, with a passkey. What we're building cron, 3 a.m. ↓ agent does the work ↓ encrypted delivery ↓ your inbox (read at 8 a.m.) The agent produces a report (Markdown, PDF, a CSV, whatever), hands it to the Agent Relay CLI, and the CLI encrypts it locally before it ever leaves the machine. The server stores only ciphertext. When you open t

LLM KV Cache Optimization, Open Model Evaluation, & Agent Engineering Skills for Local Deployment Today's Highlights This week, a groundbreaking KV cache layer promises to supercharge local LLM inference, alongside a new workbench for evaluating open language models. Additionally, a trending repository provides production-grade engineering skills for building robust AI agents, crucial for self-hosted deployments. LMCache: Supercharge Your LLM with the Fastest KV Cache Layer (GitHub Trending) Source: https://github.com/LMCache/LMCache LMCache introduces a novel KV cache optimization layer designed to significantly accelerate Large Language Model (LLM) inference. The KV cache (Key-Value cache) is a critical component in LLM decoding, storing previously computed keys and values for attention layers to avoid redundant calculations. Optimizing this cache is paramount for achieving high throughput and low latency, especially when running large models on consumer-grade hardware or self-hosted servers. This project aims to provide the fastest KV cache solution, directly addressing a key bottleneck in local LLM deployment and performance. By improving KV cache efficiency, LMCache enables developers and researchers to run more complex models or serve more users with existing hardware, making advanced LLMs more accessible for local inference scenarios. Details on its architecture and comparative benchmarks against existing solutions will be critical for understanding its impact on various open-weight models and frameworks like vLLM or llama.cpp. Comment: Faster KV cache is a game-changer for anyone running LLMs locally. This project could unlock new performance levels for open models on consumer GPUs. olmo-eval: An evaluation workbench for the model development loop (Hugging Face Blog) Source: https://huggingface.co/blog/allenai/olmo-eval The olmo-eval workbench from AllenAI provides a comprehensive system for evaluating language models throughout their development lifecycle.

The box that ran everything started dying in April. Not dramatically. Machines almost never die dramatically. It started with instability... the kind you explain away once, side-eye twice, and start losing sleep over the third time production goes down while you're in the middle of something else. The host under my entire stack... public site, analytics, security tooling, the AI crew's memory layer... was getting flaky. And flaky hardware only trends one direction. Here's the thing about a homelab that lives in a 40ft fifth wheel: there is no second team. No vendor escalation. No change advisory board. No maintenance window negotiated three weeks out. There's me, a crew of governed AI instances, and a reclaimed Dell T3600 about to get the biggest promotion of its second life. So we didn't try to heal the sick box. We built a new hull alongside it and started moving the ship... plank by plank... while it was still sailing. One ground rule, set day one: the old host stays untouched and keeps serving production until the new hull is proven. Not "mostly proven." Proven. Hold that thought, it matters at the end. Moving containers is the easy part. Docker made that boring years ago, and boring is a compliment in infrastructure. What's never boring is the inventory of everything you assumed and never wrote down. A migration doesn't test your stack. It tests your assumptions. Here's what mine were hiding. 01: The umbilical nobody documented Security stack went first. Suricata, Zeek, Wazuh, CrowdSec, Falco, the whole alphabet, up clean on the new hull. Then the MCP server, the piece that gives the AI crew its hands, refused to come up right. It was hard-wired over HTTP to the crew's memory backend. A live dependency, in production for months, documented exactly nowhere. The crew that documents every f*cking thing had never documented its own umbilical cord. Fix was trivial once we could see it: deploy the brain before the hands. Reorder, redeploy, done. But the lesson isn't

Six months ago I got tired of switching between apps to talk to AI. ChatGPT in the browser. Claude in another tab. Local models in a terminal. It was like having five friends who all live in different cities and refuse to visit each other. So I did what any developer with too many GPUs and too little patience would do: I built my own assistant and put it where I already spend my day — Telegram. It's not a chatbot for customers. It's not a business automation tool. It's just... my assistant. It lives in a private chat on my phone and handles the stuff I used to do manually. Here's what six months of actually using it has looked like. What I Actually Built (And Why Telegram) I already had three machines running Ollama at home — a Mac Mini M4, a Windows PC with an RTX 3060, and an Ubuntu box. Three endpoints, eight models, and me constantly forgetting which model was good for what. Telegram was the obvious choice because: I'm already there all day (friends, family, a few dev groups) It works on my phone, my Mac, and my watch The Bot API is dead simple I can send voice messages, photos, documents — and the bot can handle all of them The setup: a Python bot running on the Mac Mini, connected to all three Ollama endpoints. When I message it, the bot classifies what I want, routes to the right model on the right machine, and replies in the same chat thread. Sounds simple. Took three evenings to get right. Took six months to make actually useful. The Things I Actually Use It For Here's the honest list. Not the marketing pitch — the real daily usage: 1. Quick questions without context switching "Summarize this article" (I paste a link). "Explain this error" (I paste a stack trace). "Rewrite this email less formally." These used to mean opening a browser tab, logging in, maybe hitting a rate limit. Now I just... send a message. The reply comes back in 2-8 seconds depending on which model handles it. The routing is simple but effective: quick chat → small model on the Mac. Cod

I Tried PewDiePie's Open-Source AI Workspace. It's Actually Good. Yes, that PewDiePie. Felix Kjellberg (110M YouTube subscribers) spent late 2025 building a home AI lab — 8 modified RTX 4090s, 256GB of VRAM, running on Arch Linux. He called it "The Swarm." He crashed it running 64 models in parallel. The web frontend he built for it? He open-sourced it. Called it Odysseus . It hit 59,000 GitHub stars fast. I dug into the code expecting a glorified Ollama wrapper. It's not. What it actually is Odysseus isn't just another chat UI. It bundles things no other self-hosted tool does in one place: Chat — local or cloud models (Ollama, vLLM, llama.cpp, OpenAI, OpenRouter, GitHub Copilot) Agent mode — shell, files, web, MCP tools, per-tool toggles Cookbook — scans your GPU, recommends models that actually fit, downloads and serves them in one click Deep Research — multi-step web research that writes you a cited report Email — IMAP/SMTP with AI triage, auto-tagging, draft replies Calendar — CalDAV sync with Radicale, Nextcloud, Apple, Fastmail Memory — persistent, evolving across all your conversations No cloud account. No telemetry. MIT license. Everything lives in your data/ folder. The Cookbook is the standout feature Every other self-hosted UI assumes you already know what model to run. Odysseus doesn't. It scans your hardware, scores 270+ models against your actual VRAM, and gives you a one-click download-and-serve. It understands GGUF vs FP8 vs AWQ. It picks the right backend (vLLM, llama.cpp, Metal on Apple Silicon). Downloaded models persist in a volume — no re-downloading after container restarts. For someone who wants local AI but finds the ecosystem confusing, this is the most accessible on-ramp that currently exists. The code is better than the meme suggests The README has a little ASCII bear face. Don't let it fool you. The entry point app.py is 1,092 lines of real production thinking. A few things that stood out: The .env loader handles Windows BOM silently: loa

I recently rebuilt my homelab from scratch. The goal was simple: one machine, everything containerised, zero exposed ports, GPU-accelerated local AI, and a fully automated backup setup. No cloud subscriptions for the tools I use every day. This is the full technical breakdown — what I'm running, how it's wired together, and the hard-won fixes that cost me hours so you don't have to repeat them. What I'm Running Eight services, 26 containers, one machine: Service Purpose Portainer Docker management UI Uptime Kuma Service monitoring (7 monitors) NocoDB Self-hosted Airtable — CRM & leads n8n Workflow automation Open WebUI Local AI chat interface Ollama Local LLM inference (GPU) AFF!NE Collaborative docs & whiteboards Plane Project management (roadmaps, sprints) Duplicati Encrypted daily backups Cloudflare Tunnel Zero Trust secure access — no open router ports All external-facing services sit behind Cloudflare Zero Trust with email OTP. No passwords to manage, no VPN clients — Cloudflare handles authentication at the edge. Architecture ┌──────────────────────────────────┐ │ Cloudflare Edge (Zero Trust) │ │ *.yourdomain.com — email OTP │ └──────────────┬───────────────────┘ │ HTTPS ┌──────────────▼───────────────────┐ │ Ubuntu Machine │ │ │ │ cloudflared (outbound tunnel) │ │ │ │ │ ┌─────▼────────────────────┐ │ │ │ homelab-net (bridge) │ │ │ │ │ │ │ │ portainer uptime-kuma │ │ │ │ nocodb n8n │ │ │ │ open-webui affine │ │ │ │ plane-* duplicati │ │ │ │ ollama (GPU passthrough) │ │ │ └───────────────────────────┘ │ └───────────────────────────────────┘ Everything runs on a shared Docker bridge network ( homelab-net ). The cloudflared container maintains an outbound-only encrypted tunnel — no inbound ports open on the router at all. Ollama runs in Docker with NVIDIA GPU passthrough. The AI model inference happens on the GPU, leaving CPU headroom for all other services. Prerequisites Ubuntu 24.04 LTS Docker Engine + Compose v2 NVIDIA GPU with driver 535+ NVIDIA Container Too

NousResearch Agent, Open-Source Notebook LM, & Local Multimodal OCR for Consumer GPUs Today's Highlights Today's highlights feature new open-source tools empowering local AI inference and deployment, including an adaptive agent from NousResearch, a self-hostable AI-powered notebook, and a lightweight multimodal OCR solution. These practical GitHub trending projects enable developers to build and run advanced AI applications directly on consumer hardware. NousResearch Unveils Hermes Agent for Adaptive Local AI (GitHub Trending) Source: https://github.com/NousResearch/hermes-agent NousResearch, a prominent contributor to the open-weight LLM ecosystem with models like the Hermes series, has unveiled hermes-agent , a new GitHub trending project described as "The agent that grows with you." This initiative represents a significant step towards practical, adaptive AI agents designed for local execution. While specific architectural details are awaiting a deeper dive into the repository, the "grows with you" philosophy strongly implies advanced capabilities for personalized learning, continuous adaptation, and long-term memory integration—features crucial for self-hosted AI applications. Such an agent is highly relevant for developers focused on local inference, as it provides an open-source framework to build sophisticated agentic workflows, potentially integrating seamlessly with local LLM runtimes such as llama.cpp or vLLM . This allows users to leverage powerful open-weight models directly on their consumer GPUs, enhancing privacy and reducing reliance on cloud services. The project's emergence from NousResearch solidifies its potential as a robust foundation for next-generation local AI applications. Comment: A NousResearch agent is exciting; it implies strong open-source model compatibility and local deployment. I'm keen to see its learning mechanisms and integration potential with local LLM runtimes. PaddlePaddle's Lightweight OCR Toolkit Bridges Images to Local LLM

This article was originally published on aifoss.dev --- title: 'Chroma vs Qdrant vs Weaviate 2026: RAG Database Compared' description: 'Compare Chroma, Qdrant, and Weaviate for local RAG in 2026: version snapshots, filtering tradeoffs, hybrid search, quantization, and a clear pick by use case.' pubDate: 'May 27 2026' tags: ["vectordb", "ai", "rag", "python", "opensource"] The three most commonly recommended open-source vector databases for RAG — Chroma, Qdrant, and Weaviate — are not interchangeable. Chroma is a prototyping tool that grew into a real product. Qdrant is a production workhorse written in Rust with the best filtering performance of the three. Weaviate is an enterprise-grade platform with hybrid search and the most built-in integrations. Using Weaviate when you need Chroma adds unnecessary ops overhead. Using Chroma when you need Qdrant means migrating under pressure when your collection outgrows it. Versions covered: ChromaDB v1.5.9 (May 2026), Qdrant v1.17.1 (March 2026), Weaviate v1.37 (May 2026). The quick answer Situation Best choice Local prototyping, notebooks, under 100K vectors Chroma Embedded in a Python process — no separate service Chroma Production RAG with filtering-heavy queries Qdrant Multi-user deployment, concurrent queries Qdrant Memory-constrained deployment at millions of vectors Qdrant Hybrid search (BM25 + vector in one query) Weaviate Multi-modal retrieval (text + images + audio) Weaviate Built-in re-ranking or generative AI modules Weaviate Kubernetes, team-operated, agentic MCP workflows Weaviate Getting from zero to working RAG in 10 minutes Chroma What each tool actually is ChromaDB (Apache 2.0, chroma-core/chroma ) started as a pure-Python embedded database and was rebuilt in Rust for the v1.0 release. The Rust core eliminates Python's GIL bottlenecks and delivers roughly 4× faster writes and queries compared to the pre-1.0 implementation — write throughput went from ~10K to ~40K+ vectors/second in server mode. Chroma's des

A homelab box that never sleeps runs 8,760 hours a year. So the spec that decides what it costs you is not the one on the box. It is the one nobody prints: how many watts it pulls sitting at the login prompt doing nothing. I kept hitting this while shopping for a Proxmox node, so I put the measured numbers in one place. More on that at the end. First, why the spec sheet lies to you. TDP is a thermal budget, not a power reading TDP is the heat the cooler has to handle at full tilt. It is a design target for the heatsink, not a measurement of what the chip draws, and it says almost nothing about idle. Your homelab box spends 95%+ of its life idle, so the number that runs up the meter is idle wall power, and that number is never on the product page. The arithmetic is unforgiving. One watt running continuously is 8.76 kWh a year. So the gap between a 7 W box and a 35 W box is not 28 watts, it is about 245 kWh a year, every year, for as long as the box is on. Plug in your own rate to get the dollars; the point is the gap compounds. Where TDP actively misleads you A few measured results from the dataset I'll link below, all from third-party wall-meter readings, not vendor claims: The new N100 wave is genuinely low. A Minisforum UN100C measures 5 to 7 W at idle. Beelink, GMKtec and Trigkey N100 boxes land in the 6 to 10 W range. For a Pi-hole, a few containers and some light VMs, this tier is hard to beat on running cost. AMD mini PCs idle far higher than their marketing suggests. A Minisforum UM790 Pro measures 25 to 45 W at idle. A Beelink SER6 Pro lands at 20 to 35 W. These are fast little machines, but if you picked one expecting "small box, small draw," the meter disagrees, and over a year that delta is real money. Newer and higher-TDP is not lower-idle. A Dell OptiPlex 7060 Micro idles just over 18 W on its 65 W-TDP desktop chip. The older 7070 with a six-core part sits around 13 W, and the low-power "T" SKUs lower still. The CPU's TDP class predicted idle better tha

Did you know that a 35-billion-parameter model can generate tokens at the same compute cost as a 4B model? That single fact made me abandon a multi-model agent architecture I'd spent a weekend building. But I had to run the benchmarks first to understand why. Here's the full breakdown, with commands, numbers, and the architectural reason it all falls apart on shared-memory hardware. The Discovery That Changed Everything I'd been running qwen3.6:35b on my Minisforum UM790Pro for weeks as my daily coding assistant. 17.8 tokens/second -- genuinely usable for interactive work. But I kept wondering: could I run a lightweight sidecar model alongside it for quick classification and tool-calling in an agent pipeline? Before I even started benchmarking, I dug into what qwen3.6:35b actually is under the hood. It's a Mixture of Experts model: 256 total experts with only 8 activated per token. The architecture also incorporates SSM (State Space Model) components alongside traditional attention -- Mamba-style layers that handle certain sequence patterns more efficiently than pure transformers. The math hit me: 8 out of 256 experts means each token only touches roughly 4-5B parameters worth of compute. The model carries 36 billion parameters of knowledge , but its per-token cost is comparable to a small dense model. I was planning to run a separate 4B model for "fast tasks" next to a model that already operates at 4B-class speed. But I had to prove it with numbers. Hardware and Ollama Setup The UM790Pro specs that matter for this experiment: CPU: AMD Ryzen 9 7940HS (Zen 4, 8C/16T) iGPU: AMD Radeon 780M (12 RDNA 3 compute units) RAM: 96 GB DDR5-5600 (~80 GB/s bandwidth) GPU memory pool: 2 GB dedicated VRAM + 46 GB GTT = 48 GB GPU-accessible That 48 GB GPU pool sounds enormous until you realize it's carved from the same DDR5 that the CPU also uses. There is no separate GDDR6 bus. Everything -- CPU inference, GPU inference, KV caches, OS operations -- flows through one 80 GB/s pipe.