标签：#evaluation

Every model launch comes with a chart. Bars, usually, or a spider diagram, showing the new model edging past its rivals on a row of benchmarks with acronyms most people cannot expand. The bar is taller. The press writes it up as a leap. And within a week, users report that the new state-of-the-art model is, for their actual work, about the same as the last one or occasionally worse. The benchmark said one thing. Reality said another. This happens so reliably that it is worth understanding the mechanics of the gap. The test is public, which ruins the test The most fundamental problem is contamination. Many popular benchmarks are published, discussed, and sitting on the open web — which is exactly where models get their training data. When the questions and answers to your exam are in the study material, a high score measures memorisation as much as ability. Nobody needs to cheat deliberately; the leak is structural. A model can score brilliantly on a benchmark it has effectively already seen and then flounder on a genuinely novel version of the same task. A benchmark stops measuring intelligence the moment it becomes famous enough to end up in the training data. Fame is the thing that breaks it. The number becomes the marketing, and the marketing corrupts the number There is a commercial feedback loop that makes benchmark figures even less trustworthy than their technical limitations alone would suggest. A high score is not just an engineering result; it is a marketing asset worth an enormous amount in attention, funding and credibility. That raises the stakes on every fractional improvement, and where the stakes are high, the temptation to select, frame and present the numbers favourably is irresistible. Vendors choose which benchmarks to headline, which comparisons to draw, and which unflattering results to leave in an appendix or omit entirely. The chart on the launch slide is not a neutral readout; it is a curated argument. This is not necessarily fraud — it rare

The eval that killed the temporal knowledge graph asserted one thing: at time T, the agent should report the state that was true at T. It failed 41% of the time. The graph had the right facts. It just handed the agent the wrong one. That number is what saved us from shipping. Every static retrieval metric looked fine. The graph answered "what is the status of Node A" with a confident, well-formed response. Trouble is, "what is the status" is a temporal question wearing a static question's clothes, and nothing in our test suite had noticed the difference until we wrote a test that actually asked about time. What I expected The pitch for a temporal knowledge graph (TKG) is genuinely good. You store facts as quadruples instead of triples: (subject, predicate, object, timestamp) or, better, (subject, predicate, object, valid_from, valid_to) . Now your agent memory isn't a flat pile of embeddings, it's a structured record of what was true and when. This is the natural next step past pure vector recall, and it slots neatly into the decay-based thinking I've written about before in Eviction Without Deletion . Instead of letting old facts fade by activation weight, you make validity windows explicit. My hope was that the graph would fix the exact failure mode that plagues flat vector memory: the agent confidently recalling a stale fact because it's semantically close to the query. With valid_from and valid_to on every edge, staleness becomes a filter, not a guess. Ask for the state at time T, filter edges where T falls inside the window, done. On paper it's cleaner than a decay curve because there's no fuzziness. A fact is either valid at T or it isn't. Schema-wise, it was simple enough. In a property graph it looks like this: // A temporal fact: Node A was in maintenance for a fixed window MATCH ( n: Server { name: 'node-a' }) CREATE ( n ) - [ :HAS_STATE { status: 'maintenance' , valid_from: datetime ( '2026-07-20T02:00:00Z' ), valid_to: datetime ( '2026-07-20T04:30:00Z' )

I benchmarked a vision-language model and scored it at 0.31. The real number was 0.70. Same model, same weights, same hardware, same 100 questions. The only thing that changed was how I read its output. I had already written up the 0.31 as a capability finding and concluded the model was unsuitable. That conclusion was wrong, and the failure was entirely in my harness. Here is the mistake, because I doubt I am the only one making it. The setup I was evaluating a batch of open-weight and frontier models on a multiple-choice benchmark: multi-view driving scenes, four options per question, one correct answer. Standard stuff. The prompt asked for reasoning followed by a final line, Answer: X . My scoring code did the obvious thing: m = re . search ( r " Answer:\s*([A-D]) " , output ) pred = m . group ( 1 ) if m else None # None scores as wrong That last comment is the bug. What actually happened The model I was testing is a "thinking" model. It emits a long internal reasoning trace before it commits to an answer. I had a generation budget of 1024 tokens. On easy questions it reasoned briefly, emitted Answer: B , and scored fine. On hard questions it reasoned at length, hit the token cap mid-thought, and never emitted the answer line at all. So the harness scored every one of those as wrong. 64 of 100 questions returned no parseable answer. Zero of those were image-loading errors or crashes. They were all truncation. And the truncation was not random: Uncertainty 0/8 answered Counterfactual 0/3 answered Safety-critical Planning 1/11 answered Safety-critical Prediction 3/12 answered Look at that distribution. The questions the model failed to answer were precisely the questions that required the most reasoning. My harness was systematically discarding the model's performance on exactly the hard subset I was trying to measure, and reporting the result as a capability ceiling. Of the 36 it did answer, it got 86% right. The model was fine. My measurement was garbage. The fix

I Built an Agent Evaluation Harness for Local AI — Here's What Most People Get Wrong DOYR | Not financial/legal/tax advice. For educational purposes only. Three months ago, I started building AI agents for my trading business. First agent: Fetches Nifty option chain data. Second agent: Analyzes PCR, OI, max pain. Third agent: Predicts direction using XGBoost. Fourth agent: Sends Telegram alerts. I had 4 agents doing 5 jobs. And I had no idea if they were any good . Sure, my trading results were +₹96,000 over 6 months. But was that because my agents were smart, or because I was overriding their bad decisions? I couldn't answer that question. So I built something to find out. An Agent Evaluation Harness. What Is an Agent Evaluation Harness? An Agent Evaluation Harness is a systematic framework for testing AI agents. It answers one question: "How good is this agent, actually?" Most people skip evaluation. They build an agent, test it once or twice manually, and call it "done." Then they wonder why it fails in production. An evaluation harness forces you to: Define success metrics — what does "good" mean? Create test suites — what scenarios will you test? Run evaluations — how does the agent perform across all scenarios? Measure regressions — did a change make the agent worse? Track improvements — is version 2 better than version 1? This is not optional. This is engineering 101 . Why Most Agent Evaluations Are Wrong I reviewed 50+ "agent evaluation" frameworks online. Here's what I found: Mistake 1: Single-Task Testing What they do: Test the agent on one task. "Can it book a flight?" → Yes/No. What's wrong: Real agents face thousands of variations of the same task. "Book a flight from Delhi to Mumbai on Friday" vs "Book a flight from Delhi to Mumbai next Friday" vs "Book a flight from Delhi to Mumbai on August 15th." A good harness tests variations , not just one example. Mistake 2: No Edge Cases What they do: Test happy paths only. "Book a flight when everything works.

How EvalPort's Grader System Works When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own way of scoring LLM outputs — DeepEval uses metric classes, Promptfoo uses assertion objects, Inspect AI uses solver functions. We needed a system expressive enough to cover 90%+ of real-world eval needs, but simple enough that any framework could implement it. The result: 11 grader types that carry their own semantics. A grader isn't just a name — it specifies its parameters, its model, its threshold. An eval suite is self-describing. The 11 Grader Types exact_match — Compare output to expected output, optionally ignoring case. contains — Check if the output contains a substring. regex — Match against a regular expression. semantic_similarity — Embed output and expected output, compare cosine similarity against a threshold. llm_judge — Use an LLM to evaluate the output against a prompt template. The most powerful grader. json_schema — Validate that the output is valid JSON matching a JSON Schema. json_path — Extract a value from JSON output using a JSONPath expression, then compare it. code — Run a function to evaluate the output. human — Defer to human review. model_graded — Compare the output to a reference answer using a model. custom — Escape hatch for graders not covered by built-in types. How Graders Connect to Test Cases A test case references graders by ID. Multiple graders can evaluate the same test case. The ResultSet records each grader's score separately. Why This Design Works Self-describing: An eval suite carries everything a framework needs to execute it. Framework-agnostic: Any framework can implement any subset of grader types. Extensible: The custom type lets frameworks bring their own graders. Comparable: Results from different frameworks use the same grader IDs. Try It pip install evalport-sdk npm install evalport-sdk Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md Repo: https://gith

Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo, Inspect AI, and lm-evaluation-harness all solve the same core problem (checking whether a model's output is correct) but none of them can read each other's eval datasets. That means every time a team wants to compare frameworks, or move from a notebook prototype to a production eval pipeline, they end up hand-rewriting the same test cases over and over. OpenEval is an attempt to fix that by defining a small, portable JSON Schema for eval test cases, graders, and results, plus tooling to move data between frameworks instead of retyping it. What's in the repo: A versioned JSON Schema spec for test cases, grader configs, and result records. TypeScript and Python SDKs for reading and writing OpenEval-formatted datasets. A CLI with validate, convert, init, and summarize commands. Converters for popular frameworks so existing datasets can be brought in or exported out without a manual rewrite. The project just published v1.0.0 to npm and PyPI, and issues are open on 17+ framework integrations if anyone wants to help wire up a converter for a framework not yet covered. Repo: https://github.com/adhabnr-ux/openeval Would love feedback from anyone who has hit this same portability problem while switching between eval tools.

Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation How we replaced fragile prompt chains with typed schemas, validation gates, and evaluation at every step — 94% task success vs 60% baseline The Prompt Chain Trap January 2024. We built a "research agent" — 12 prompts chained together: Decompose question → 2. Search planning → 3. Execute searches → 4. Extract facts → 5. Synthesize → 6. Fact-check → 7. Format → ... It worked 60% of the time. The other 40%: Step 3 returned malformed JSON → Step 4 crashed Step 5 hallucinated citations → Step 6 missed it Step 7 output wrong format → Downstream consumer failed No visibility into which step failed Debugging meant reading 12 LLM calls' worth of logs. Adding a step broke three others. The Shift: Agents as Typed Workflows We moved from prompt chains to structured workflows with: Pydantic schemas for every step input/output Guardrails that validate and auto-retry Explicit state machine (not implicit chaining) Evaluation harness per step (not just end-to-end) ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Decompose │──▶│ Search │──▶│ Extract │──▶│ Synthesize │ │ Question │ │ Planning │ │ Facts │ │ Answer │ │ │ │ │ │ │ │ │ │ In: Query │ │ In: Plan │ │ In: Results │ │ In: Facts │ │ Out: SubQ[] │ │ Out: Steps │ │ Out: Fact[] │ │ Out: Answer │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ ▼ ▼ ▼ ▼ [Schema] [Schema] [Schema] [Schema] [Guardrail] [Guardrail] [Guardrail] [Guardrail] [Eval: 0.9] [Eval: 0.85] [Eval: 0.9] [Eval: 0.95] Core Abstractions # agent_eval/schemas.py from pydantic import BaseModel , Field from typing import Literal , Any class DecomposeInput ( BaseModel ): user_query : str context : dict = Field ( default_factory = dict ) class DecomposeOutput ( BaseModel ): sub_questions : list [ str ] = Field ( min_length = 1 , max_length = 5 ) requires_tools : bool reasoning : str class PlanInput ( BaseModel ): sub_questions : list [

Building AI Agents That Don't Hallucinate: Structured Workflows, Guardrails, and Per-Step Evaluation How we replaced fragile prompt chains with typed schemas, validation gates, and evaluation at every step — 94% task success vs 60% baseline The Prompt Chain Trap January 2024. We built a "research agent" — 12 prompts chained together: Decompose question → 2. Search planning → 3. Execute searches → 4. Extract facts → 5. Synthesize → 6. Fact-check → 7. Format → ... It worked 60% of the time. The other 40%: Step 3 returned malformed JSON → Step 4 crashed Step 5 hallucinated citations → Step 6 missed it Step 7 output wrong format → Downstream consumer failed No visibility into which step failed Debugging meant reading 12 LLM calls' worth of logs. Adding a step broke three others. The Shift: Agents as Typed Workflows We moved from prompt chains to structured workflows with: Pydantic schemas for every step input/output Guardrails that validate and auto-retry Explicit state machine (not implicit chaining) Evaluation harness per step (not just end-to-end) ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Decompose │──▶│ Search │──▶│ Extract │──▶│ Synthesize │ │ Question │ │ Planning │ │ Facts │ │ Answer │ │ │ │ │ │ │ │ │ │ In: Query │ │ In: Plan │ │ In: Results │ │ In: Facts │ │ Out: SubQ[] │ │ Out: Steps │ │ Out: Fact[] │ │ Out: Answer │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ ▼ ▼ ▼ ▼ [Schema] [Schema] [Schema] [Schema] [Guardrail] [Guardrail] [Guardrail] [Guardrail] [Eval: 0.9] [Eval: 0.85] [Eval: 0.9] [Eval: 0.95] Core Abstractions # agent_eval/schemas.py from pydantic import BaseModel , Field from typing import Literal , Any class DecomposeInput ( BaseModel ): user_query : str context : dict = Field ( default_factory = dict ) class DecomposeOutput ( BaseModel ): sub_questions : list [ str ] = Field ( min_length = 1 , max_length = 5 ) requires_tools : bool reasoning : str class PlanInput ( BaseModel ): sub_questions : list [

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",

Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40% How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 The RAG Reality Check Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: Legal contracts: 512 tokens splits clauses mid-sentence API docs: 1000-token chunks drown signal in noise Customer tickets: Conversational context needs overlap, not fixed windows Latency: 500ms embedding + 200ms vector search + 300ms LLM = 1s+ per query We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. Chunking: One Size Fits None # rag/chunking.py from abc import ABC , abstractmethod from dataclasses import dataclass @dataclass class Chunk : text : str metadata : dict token_count : int chunk_id : str class ChunkingStrategy ( ABC ): @abstractmethod def chunk ( self , document : str , metadata : dict ) -> list [ Chunk ]: ... class FixedTokenChunker ( ChunkingStrategy ): """ Baseline. Good for homogeneous content. """ def __init__ ( self , chunk_size = 512 , overlap = 50 ): self . chunk_size = chunk_size self . overlap = overlap class RecursiveChunker ( ChunkingStrategy ): """ Respects structure: markdown headers, code blocks, paragraphs. """ def __init__ ( self , separators = [ " \n ## " , " \n ### " , " \n\n " , " \n " , " " ], chunk_size = 512 ): self . separators = separators self . chunk_size = chunk_size class SemanticChunker ( ChunkingStrategy ): """ Uses embedding similarity to find natural boundaries. """ def __init__ ( self , model = " text-embedding-3-small " , threshold = 0.7 ): self . model = model self . threshold = threshold class AgenticChunker ( ChunkingStrategy ): """ LLM decides boundaries. Expensive but highest quality for complex docs. """ def __init__ ( self , model = " gpt-4o-mini " ): self . model = model Our production config by

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment The Problem: Why "Vibe Checks" Fail in Production Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." What Production Evaluation Actually Needs Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs: Domain-specific judges — Your criteria, not generic "helpfulness" Speed — Evaluation must run in CI/CD, not overnight Regression detection — Know immediately when a prompt change breaks things CI/CD integration — Block merges that degrade quality Golden dataset management — Versioned, stratified, growing test cases Architecture: The Evaluation Pipeline ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ Core Abstractions # eval/base.py @dataclass ( frozen = True ) class TestCase : id : str input : dict [ str , Any ] expected : dict [ str , Any ] | None = None tags : list [ str ] = field ( default_factory = list ) # ["edge-case",