Qwen3.6-27B + vLLM + Hermes on 24GB VRAM: May 2026 Recipe
If you want to reproduce my current local Hermes Agent + Qwen3.6-27B setup, this is the shape I would start from. Target One local coding agent. One 24GB GPU. Long context. Tools enabled. Thinking enabled. No child agents fighting the main request. The goal is not peak tok/s on a short prompt. The goal is: can the same agent session keep working after hours of tool calls without losing prefix locality, timing out during prefill, or getting wrecked by auxiliary requests? Model This setup is intentionally text-only. I am not serving the multimodal GGUF variant here. The working configuration uses groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit through vLLM with --language-model-only . That choice matters. On a 24GB RTX 3090, the text-only GPTQ-Marlin path gave the best balance I found between long context, prefix caching, stable agent behavior and usable decode speed. Vision should be handled by a separate service/model if needed. vLLM The useful shape: CUDA_VISIBLE_DEVICES = 0 vllm serve groxaxo/Qwen3.6-27B-GPTQ-Pro-4Bit \ --served-model-name qwen3.6-27b-gptq-pro-4bit \ --dtype float16 \ --quantization gptq_marlin \ --tensor-parallel-size 1 \ --max-model-len 131072 \ --max-num-seqs 1 \ --kv-cache-dtype fp8_e5m2 \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --gpu-memory-utilization 0.95 \ --max-cudagraph-capture-size 32 \ --language-model-only I used a recent vLLM nightly, not an old stable image ( 0.20.1rc1.dev16+g7a1eb8ac2 ). The two flags people will want to argue about: --max-num-seqs 1 --max-model-len 131072 I use max_num_seqs=1 deliberately. With an agent, parallelism is not free. Title generation, context compression, retries, browser checks, tool calls and side jobs can all steal KV/cache locality from the main request. On one 24GB GPU I prefer one useful request over two requests sabotaging each other. 131k context is tight, but workable here. If your service OOMs, reduce context before adding MTP or enf