标签：#vllm

TL;DR Short version: no. I dropped a much older GPU ( Quadro P2000, 5GB, Pascal, 2016 ) next to an RTX 3090 (24GB, Ampere) on the same box, ran the same context-length ladder (8K→128K) through Ollama and vLLM on qwen3-coder:30B-A3B , and got zero extra usable context in either engine — and a 74% decode-speed hit for the trouble. Ollama hits the identical Chunk too big wall at ctx=65536 whether the P2000 is there or not. vLLM refuses tensor-parallel across the two cards entirely — not a VRAM problem, a flat compute-capability rejection ( Minimum capability: 75. Current capability: 61. ) that fails in 40 seconds, before any memory profiling. And the one real, measured effect of adding the P2000 to Ollama: decode speed goes from 76 → 19.5 tok/s at ctx=49152 once the P2000 gets pulled in as an actual compute device. Full narrative version — the two-stage collapse, the prompt-cache validation bug caught mid-sweep, the CUDA13-silently-drops-Pascal finding — is on Medium .## The setup ardi (dual Xeon E5-2680 v4, 128GB RAM, openSUSE Leap) has a Quadro P2000 sitting in a second slot next to the RTX 3090 this whole series has run on so far. Same model as phase 1 ( qwen3-coder:30B-A3B ), same box, four legs: {Ollama, vLLM} × {3090 only, 3090+P2000 tandem}, priced through HomeLab Monitor against real GPU power draw. Ollama: same wall, extra tax ctx 3090 only decode tok/s tandem decode tok/s P2000 VRAM (tandem) 8,192 124.3 122.0 6 MB / 0% 24,576 108.2 70.0 62 MB / 0% 32,768 99.4 61.0 62 MB / 0% 49,152 75.7 19.5 3,580 MB / 55% 65,536 fatal: Chunk too big fatal: identical Chunk too big — Two separate costs, not one: decode already falls behind at ctx=24576 while the P2000 is still basically idle (62MB, 0% util) — some scheduling overhead just from having a second visible device. Then the real collapse hits at ctx=49152, when the P2000 actually gets pulled into the compute path (3.58GB, 55% util) and decode craters to 19.5 tok/s . Same context ceiling either way, worse speed the wh

If you want to reproduce my current local Hermes Agent + Qwen3.6-27B setup, this is the shape I would start from. Target One local coding agent. One 24GB GPU. Long context. Tools enabled. Thinking enabled. No child agents fighting the main request. The goal is not peak tok/s on a short prompt. The goal is: can the same agent session keep working after hours of tool calls without losing prefix locality, timing out during prefill, or getting wrecked by auxiliary requests? Model This setup is intentionally text-only. I am not serving the multimodal GGUF variant here. The working configuration uses groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit through vLLM with --language-model-only . That choice matters. On a 24GB RTX 3090, the text-only GPTQ-Marlin path gave the best balance I found between long context, prefix caching, stable agent behavior and usable decode speed. Vision should be handled by a separate service/model if needed. vLLM The useful shape: CUDA_VISIBLE_DEVICES = 0 vllm serve groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit \ --served-model-name qwen3.6-27b-gptq-pro-4bit \ --dtype float16 \ --quantization gptq_marlin \ --tensor-parallel-size 1 \ --max-model-len 131072 \ --max-num-seqs 1 \ --kv-cache-dtype fp8_e5m2 \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --gpu-memory-utilization 0.95 \ --max-cudagraph-capture-size 32 \ --language-model-only I used a recent vLLM nightly, not an old stable image ( 0.20.1rc1.dev16+g7a1eb8ac2 ). The two flags people will want to argue about: --max-num-seqs 1 --max-model-len 131072 I use max_num_seqs=1 deliberately. With an agent, parallelism is not free. Title generation, context compression, retries, browser checks, tool calls and side jobs can all steal KV/cache locality from the main request. On one 24GB GPU I prefer one useful request over two requests sabotaging each other. 131k context is tight, but workable here. If your service OOMs, reduce context before adding MTP or enf