I Ran 10 AI Coding Models Through 5 Tasks: A Data Scientist's Take
I Ran 10 AI Coding Models Through 5 Tasks: A Data Scientist's Take I'll be honest — I went into this expecting a clear winner. I came out with a scatter plot, three regressions, and a deeper appreciation for why "best" is the most dangerous word in machine learning. Over the past three weeks I've been grinding through prompts with ten different LLMs, all routed through the same endpoint, scoring every output on a 1–10 rubric that I tried very hard not to bias. The pricing data is pulled directly from the provider pages. The scores are mine. If you disagree with a score, you're probably right — n=1 per task per model is a laughably small sample size, and I say that as someone who publishes papers with bigger samples. But trends still emerged. Let me walk you through what I found. The Lineup Before I touch a single benchmark, here's the cast. I've grouped them by family so you can see the obvious concentration in the open-source Chinese ecosystem, which personally I find fascinating — three of the top five are DeepSeek or Qwen variants. # Model Provider Output $/M Category 1 DeepSeek V4 Flash DeepSeek $0.25 General (strong code) 2 DeepSeek Coder DeepSeek $0.25 Code-specialized 3 Qwen3-Coder-30B Qwen $0.35 Code-specialized 4 DeepSeek V4 Pro DeepSeek $0.78 Premium general 5 DeepSeek-R1 DeepSeek $2.50 Reasoning (code thinking) 6 Kimi K2.5 Moonshot $3.00 Premium general 7 GLM-5 Zhipu $1.92 Premium general 8 Qwen3-32B Qwen $0.28 General purpose 9 Hunyuan-Turbo Tencent $0.57 General purpose 10 Ga-Standard GA Routing $0.20 Smart routing One quick note on Ga-Standard — it's a routing layer that picks a backend model per request. So the score fluctuates. I averaged across runs. How I Tested Five prompts. Each one designed to probe a different cognitive layer: Function implementation — flatten a nested list recursively in Python Bug fix — chase down an async/await race condition in JavaScript Algorithm — Dijkstra's shortest path in TypeScript with proper types Code review — sec