I Benchmarked 3 Local LLMs on My Laptop — Here's What the Numbers Actually Show

The Problem With Choosing a Local Model Everyone has an opinion on which local LLM is best. "Use Llama — it's the most popular." "Mistral 7B has the best quality." "Phi-3 Mini is small and efficient." None of these claims come with numbers. Specifically: your numbers, on your hardware, for your workload. I built a benchmarking system to change that. Three models, 30 prompts, full latency distribution, memory profiling per inference call, and a JSON validation layer to measure structured output reliability. Here's what I found — and why the results matter for anyone deploying local models in production. The Setup Three models tested: llama3.2:3b — 3B parameters, Q4_K_M quantization, 2 GB download phi3:mini — 3.8B parameters, Q4_K_M, 2.3 GB download mistral:7b — 7B parameters, Q4_K_M, 4.1 GB download Hardware: CPU only, no GPU acceleration. This is the worst-case baseline — the scenario that exposes real latency and memory numbers. 30 test prompts across 5 categories: Short factual (10): "What is the capital of France?" Reasoning (8): "Explain why the sky appears blue." Code generation (5): "Write a Python function to reverse a string." Structured output (5): "List 3 frameworks in JSON format with name and use_case." Multi-step (2): Complex chained reasoning tasks. Architecture POST /query → Pydantic validation → Ollama HTTP API → JSON Validator → QueryResponse POST /benchmark → Load test_prompts.json → For each prompt: psutil memory before → Ollama → psutil memory after → NumPy: P50/P95/P99 latency, avg TPS, peak/avg memory → BenchmarkResult JSON The benchmark runs prompts sequentially, not in parallel. Parallel would contaminate the per-prompt memory measurements. Results Llama 3.2 3B (Q4_K_M) avg_tokens_per_second : 42.3 p50_latency_ms : 1203 p95_latency_ms : 3847 p99_latency_ms : 5120 peak_memory_mb : 6953 avg_memory_mb : 6842 total_test_duration_s : 87.4 Interpretation: P50 at 1.2 seconds is excellent. P95 at 3.8 seconds misses a 3-second SLA — the outliers are m