标签：#llm

We stopped measuring retrieval quality by how many tokens we could fit into the prompt. When...

Quick answer: The PromptEval Prompt Quality Report scored over 1,000 real prompts across 12 use cases. The average was 52 out of 100, and only 8% reached "good" (75+). The strongest single predictor of a good prompt is whether it defines its output format, worth 27 points on average. In 9 of 10 prompts, the weakest dimension was robustness. This is the PromptEval Prompt Quality Report . Over 1,000 real prompts have been scored on PromptEval , submitted by real users across use cases from customer support to healthcare to code. Each was scored from 0 to 100 on four structural dimensions: clarity, specificity, structure, and robustness. Every figure below comes from that set. No prompt text is stored; the analysis is anonymous and aggregate. Only 8% of the 1,000+ scored prompts reached "good" (75 or higher). Fewer than 1% reached "excellent." Source: PromptEval Prompt Quality Report, 2026 How the scores break down Here is how the scores spread across the set. The bar for "good" is 75, the point where a prompt is clear, specified, and holds up under variation. Score range Share of prompts 0 to 40 (failing) 25% 41 to 60 (below par) 31% 61 to 74 (functional but mediocre) 36% 75 to 84 (good) 8% 85 to 100 (excellent) under 1% Roughly 92% of prompts never reach "good," and almost none reach "excellent." This includes prompts from people who clearly know the tools. The gap is not talent. It is a few missing pieces that repeat. What separates a good prompt from a bad one For each structural element, we compared the average score of prompts that had it against those that did not. These are averages across the set, not a controlled experiment, so read them as correlation. But the gaps were large and consistent. The prompt... Avg with Avg without Point gap Defines the output format 58 31 +27 Has explicit constraints (what not to do) 63 41 +22 Assigns a role or persona 57 42 +15 Includes at least one example 64 51 +13 Prompts that define their output format score 27 points higher

Disclosure: I maintain Lynkr , an open-source router whose design decisions this post explains. The failure modes described are patterns widely reported across router issue trackers and local-LLM forums — the examples are representative reconstructions, not captured transcripts. The problem is real either way; ask anyone who's routed a coding agent to a 7B model. Everyone who gets their first LLM router working does the same thing within the hour: point the expensive coding agent at a free local model and watch the bill drop to zero. Then the agent tries to edit a file. The graveyard of downgraded sessions If you browse the issue tracker of any Claude Code router — or r/LocalLLaMA on any given week — you'll find the same story in a hundred variations. The routing works perfectly. The session dies anyway. The killers, in rough order of frequency: 1. Malformed tool arguments. The agent decides to call Edit , and the model produces arguments that are almost JSON: { "file_path" : "src/auth.js" , "old_string" : "if (token) {" , "new_string" : "if (token && !expired) {" One missing brace. The harness rejects the call, the model retries, produces a different malformation, and you're three turns deep into fixing nothing. Frontier models emit structurally valid tool calls with boring reliability; sub-10B models do it most of the time — and "most of the time," at 30 tool calls per session, means every session breaks. 2. Stale string matching. Edit -style tools require the old_string to match the file exactly. Small models paraphrase from memory instead of quoting — they'll "remember" the line as if (token) { when the file says if (accessToken) { . The edit fails, the model re-reads the file, burns 2,000 tokens, tries again with a different paraphrase. This is the single most reported failure, because it looks like the router's fault and is actually a capability cliff. 3. Hallucinated context. Ask a small model to run tests and it may confidently call Bash with npm test -- --g

The Log Is the Agent AI Engineer World's Fair Coverage Ishaan Sehgal Ishaan Sehgal Ishaan Sehgal Follow for Daily Context Jun 30 The Log Is the Agent # aie # agents # ai 48 reactions 89 comments 5 min read

Most of the "AI for observability" work I see right now hands the language model the judgment. I think that's backwards. Feed it the alert, feed it some metrics, ask it what's wrong, what should be done, and let it make the judgement call. Based on my experience working with language models, I decided that inverting the process provides better results. The short version: in my alerting pipeline, the set of allowable classifications is fixed in deterministic Python, and the model has to pick from it. The LLM's only job is to turn a structured verdict into an easily digestible sentence. It never decides whether something is bad, how bad it is, or what category of problem it is. It narrates within a decision space the code has already locked down. TL;DR: Instead of letting an LLM decide what's wrong with an alert, I let deterministic Python make every operational decision and restrict the model to explaining the result in plain English. The code classifies, validates, and aggregates; the LLM only narrates. That keeps the data consistent, prevents hallucinated classifications, and ensures the monitoring pipeline continues working even if the model fails. The problem I run a small managed monitoring service. Alertmanager fires, a webhook lands, and historically that webhook produced a line like HighMemoryUsage on host web-vm, severity warning, which is accurate, but not terribly helpful. The person reading it still has to know what HighMemoryUsage implies, whether this host always runs hot, and whether to care. I wanted plain-English context attached to the alerts without altering the alert delivery process. The obvious move was to throw the whole alert at an LLM and ask it to explain. I tried that in the first iteration of this experiment, expecting it to be somewhat accurate, but not entirely reliable, and it did not disappoint, the model was confidently inconsistent. The same alert, fired three times, produced three different "root cause" categories. One run called a

Most AI incidents don't happen because the model gave a bad answer. They happen because nobody was...

👋 Hey there, Tech Enthusiasts! I'm Sarvar, a Cloud Architect who loves turning complex tech problems...

A month ago we added a card_token column to the users table so a background job could retry failed Pro charges. It lasted about two days. Storing card data in your own database drops you into PCI-DSS (the compliance standard that kicks in the moment card data touches your systems), so we pulled it and moved to Stripe-managed payment methods. Last week the charges started failing again. New Claude Code session, no memory of any of that. Its plan? Add a card_token column to users and retry. I don't really blame the agent. It had the context the first time and it was right. The problem is that context died when the session closed. That's the part I never see mentioned about building with agents: the code sticks around, the reasoning doesn't. People leave a trail without trying. A commit message, a PR comment, the Slack thread before it. Agents don't, and the prompt that explained everything is gone by morning. So I built Selvedge to hold onto the reasoning. What happened the second time Selvedge is a local MCP server the agent calls as it works. There's a four-line block in our CLAUDE.md that says, roughly: before you touch an entity, check if we've been here before. $ selvedge prior-attempts users.card_token users.card_token Prior attempt 28 days ago ( reverted after 2 days ) Reasoning Added to store card tokens for one-click retries. Outcome REVERTED — kept card data out of our own DB to stay clear of PCI-DSS scope ; moved to Stripe-managed methods. So it didn't add the column. It charged off_session against the saved Stripe PaymentMethod instead. Charge retried, no card data in our database, done. We paid for that lesson once. How it works The agent writes down why live, in the moment, from the same context that made the change. That's the whole trick. A lot of the "git blame for AI" tools take your diff afterward and ask a second model to explain it. That's a guess. It reads well, but you can't really build on it. Selvedge stores what the agent actually meant, in i

Before adopting DSPy, prove the LM program has a contract DSPy is easy to undersell. If you describe it as "a nicer way to write prompts", you will probably test the wrong thing. The better first test is this: can your language-model workflow be expressed as a small program with declared inputs, declared outputs, measurable examples, and an optimizer that is allowed to change prompts without changing the business boundary? That is the point where DSPy starts to make sense. The upstream README positions DSPy as a framework for programming, rather than prompting, foundation models. The current Doramagic project pack for stanfordnlp/dspy also points at the same starting command: pip install dspy . The package metadata I checked today declares name="dspy" , version 3.3.0b1 , and Python >=3.10,<3.15 . The GitHub API snapshot showed the repository was still active, with a push on 2026-07-05 and recent pull requests around empty evaluation sets, document formatting, and GEPA trace attribution. That activity is useful, but it is not the adoption proof. The proof is whether your own task can survive three separations. 1. Separate the task contract from the prompt In DSPy terms, the first artifact worth reading is not a clever prompt. It is the Signature. A Signature says what goes in and what must come out. A Module wraps that contract into something callable. An Adapter turns the contract into the provider-facing prompt format and parses the model response back into fields. That gives you a practical review question: Can the team name the output fields before anyone starts tuning wording? If the answer is no, DSPy will not rescue the workflow. You are still negotiating the task itself. The first pass should be a tiny contract: input text, retrieved context if needed, answer, citations, confidence, or whatever fields the product actually needs. Do not start with agents, tools, or multi-stage pipelines. 2. Separate compilation from runtime correctness DSPy optimizers can impr

If someone compromised one of your project's dependencies today, would they be able to steal your...

Introduction Every post in this series has quietly touched a piece of the same problem. Building Agentic Workflows in Java said toolUse.input() is untrusted and must be validated before it reaches your code. Building Reliable LLM Applications in Java said the model will confidently invent facts, so ground it and get typed output instead of parsing prose. Neither post named the thing underneath both statements: anything that crosses from outside your code into the model, or from the model back into your code, is untrusted input — a request body from the network, not a trusted internal value. This post names that boundary directly and gathers the defenses in one place — prompt injection (direct and indirect), input validation, output validation, and PII redaction — with the SAFE pattern shown beside every unsafe one it replaces, since this is the security-forward capstone of the series. The Trust Boundary: Three Kinds of Untrusted Input An LLM application has three places where untrusted text enters: User input — anything a person types, uploads, or submits through an API. Retrieved content — Making RAG Accurate in Java built a pipeline that ranks and returns chunks from a document store; those chunks were written by whoever authored the source document, not by you, and a malicious or compromised document can carry text aimed at the model reading it, not at a human reader. Model output — untrusted the moment it's about to be used rather than displayed : passed to a tool, interpolated into a query, or fed into another LLM call as context. A model that just read attacker-controlled retrieved text can be manipulated into producing attacker-controlled output. The single rule under all three: text is data until your code has explicitly decided it's safe to use for anything more than display. Nothing below is executed against a live API — every snippet is illustrative, and none of it uses a real key or a real record. Direct Prompt Injection: Defending the System Prompt A di

Introduction Every post in this series has quietly touched a piece of the same problem. Building Agentic Workflows in Python said a tool's input is untrusted and must be validated before it reaches your code. Building Reliable LLM Applications in Python said the model will confidently invent facts, so ground it and get typed output instead of parsing prose. Neither post named the thing underneath both statements: anything that crosses from outside your code into the model, or from the model back into your code, is untrusted input — a request body from the network, not a trusted internal value. This post names that boundary directly and gathers the defenses in one place — prompt injection (direct and indirect), input validation, output validation, and PII redaction — with the SAFE pattern shown beside every unsafe one it replaces, since this is the security-forward capstone of the series. The Trust Boundary: Three Kinds of Untrusted Input An LLM application has three places where untrusted text enters: User input — anything a person types, uploads, or submits through an API. Retrieved content — Making RAG Accurate in Python built a pipeline that ranks and returns chunks from a document store; those chunks were written by whoever authored the source document, not by you, and a malicious or compromised document can carry text aimed at the model reading it, not at a human reader. Model output — untrusted the moment it's about to be used rather than displayed : passed to a tool, interpolated into a query, or fed into another LLM call as context. A model that just read attacker-controlled retrieved text can be manipulated into producing attacker-controlled output. The single rule under all three: text is data until your code has explicitly decided it's safe to use for anything more than display. Nothing below is executed against a live API — every snippet is illustrative, and none of it uses a real key or a real record. Direct Prompt Injection: Defending the System Prompt

Introduction We already covered picking the right model tier for the task and caching a large shared prefix in https://pg-blogs.netlify.app/posts/11-building-reliable-llm-apps-in-java/ . Those two lines were the tip of a bigger discipline: LLM cost is not a fixed line item, it's an engineering variable — one you can measure and shrink with the same rigor you'd apply to database query time or container memory. This post goes deeper: how input/output pricing actually works, the exact cache_control shape and how to prove a cache hit rather than assume one, the Batches API for work that isn't latency-sensitive, and model routing — using a cheap model to triage, escalating only the hard cases to a stronger one. The honest framing throughout: measure before you optimize. Every technique here has a cost of its own; applied to the wrong workload, "optimization" makes things slower or more expensive. Token Economics: Why the Prefix Is the Bill Anthropic (like every hosted LLM provider) prices input and output tokens separately, and output is always pricier — the model has to generate output autoregressively, one token informed by all the ones before it, while input can be processed in parallel. Representative pricing from the current model catalog: Model Input Output Claude Opus 4.8 $5.00 / MTok $25.00 / MTok Claude Sonnet 5 $3.00 / MTok $15.00 / MTok Claude Haiku 4.5 $1.00 / MTok $5.00 / MTok Two consequences follow directly: Long system prompts, tool definitions, and RAG context are read on every request , not written once. A 20K-token system prompt sent on every one of 10,000 requests is 200M input tokens — at Opus 4.8 rates, $1,000 before a single output token is generated. The shared prefix , not the user's question, is usually where the money goes. A verbose model wastes money twice — once on the extra output tokens themselves, and again because the next turn's messages history now carries that verbosity forward as input on every subsequent call. Trimming max_tokens an

Introduction https://pg-blogs.netlify.app/posts/10-building-reliable-llm-apps-in-python/ closed with a section on picking the right model per task and caching a shared prefix. That was the entry point into a bigger discipline: LLM spend is an engineering variable, not a fixed bill — one you can measure and reduce with the same rigor you'd apply to query latency or memory footprint. This post goes deeper on four levers: how input/output pricing actually works and why the prefix is usually where the money goes, the exact cache_control shape and how to prove a cache hit instead of assuming one, the Batches API for work that isn't latency-sensitive, and model routing — a cheap model triaging requests and escalating only the hard ones. The throughline is honest: measure before you optimize. Every lever here has its own cost; misapplied, it makes things slower or pricier, not cheaper. Token Economics: Why the Prefix Is the Bill LLM providers price input and output tokens separately, and output always costs more — generation is autoregressive (each token depends on every one before it), while input can be processed in parallel. Representative pricing from the current model catalog: Model Input Output Claude Opus 4.8 $5.00 / MTok $25.00 / MTok Claude Sonnet 5 $3.00 / MTok $15.00 / MTok Claude Haiku 4.5 $1.00 / MTok $5.00 / MTok Two things follow: A long system prompt, tool list, or RAG context is billed as input on every request , not written once. Send a 20K-token system prompt on 10,000 requests and that's 200M input tokens — at Opus 4.8 rates, $1,000 before the model has generated a single output token. The shared prefix , not the user's actual question, is usually the dominant cost. Verbose output costs twice — once directly (more output tokens billed at the higher rate), and again because the next turn's history carries that verbosity forward as input. Asking for concise output and setting a sane max_tokens is a cost control, not just a style choice. This is why the tw

Introduction Building Reliable LLM Applications in Python put it plainly: treat model output as a hypothesis to verify, not a fact to trust. Testing Best Practices in Python put the same discipline in pytest terms: a suite only earns trust by asserting the right things at the right level, unhappy paths included. This post is where those two ideas meet — a pytest assertion either passes or fails against a fixed expected value; an LLM's output is a paragraph of prose that might be right in spirit while differing token-for-token from anything you wrote down in advance. Evaluating it takes a harness, not an assert . That harness has three parts: a golden dataset of representative cases with known-good expected behavior, scoring that turns each case into a pass/fail or a number, and regression testing that runs the harness on every change and fails the build when the score drops. Making RAG Accurate in Python already gave you half of this story — recall@k, precision@k, MRR, nDCG measure whether retrieval found the right chunks. This post measures the other half: whether the generated answer built from those chunks is actually good, which is a genuinely different question a retrieval metric can't answer on its own. Everything below is illustrative, non-executed Python, grounded in the same Anthropic SDK shapes as posts 10/11. The Golden Dataset: Curating Cases, Not Just Inputs A golden dataset is a small, hand-curated set of (input, expected behavior) pairs that represents the ways your application is actually used — not a random sample, and not just the cases that already work. Each case needs enough structure to be scored automatically later: from dataclasses import dataclass , field @dataclass class EvalCase : id : str category : str # "extraction", "qa", "summarization", ... input : str # the prompt/question sent to the system under test expected_exact : str | None = None # non-None only for cases scorable by exact match must_contain : list [ str ] = field ( default_f

Introduction Building Reliable LLM Applications in Java put it plainly: treat model output as a hypothesis to verify, not a fact to trust. Testing Best Practices in Java put the same discipline in JUnit terms: a suite only earns trust by asserting the right things at the right level, unhappy paths included. This post is where those two ideas meet — a JUnit test either passes or fails against a fixed expected value; an LLM's output is a paragraph of prose that might be right in spirit while differing token-for-token from anything you wrote down in advance. Evaluating it takes a harness, not an assertEquals . That harness has three parts: a golden dataset of representative cases with known-good expected behavior, scoring that turns each case into a pass/fail or a number, and regression testing that runs the harness on every change and fails the build when the score drops. Making RAG Accurate in Java already gave you half of this story — recall@k, precision@k, MRR, nDCG measure whether retrieval found the right chunks. This post measures the other half: whether the generated answer built from those chunks is actually good, which is a genuinely different question a retrieval metric can't answer on its own. Everything below is illustrative, non-executed Java, grounded in the same Anthropic Java SDK shapes as posts 10/11. The Golden Dataset: Curating Cases, Not Just Inputs A golden dataset is a small, hand-curated set of (input, expected behavior) pairs that represents the ways your application is actually used — not a random sample, and not just the cases that already work. Each case needs enough structure to be scored automatically later: public record EvalCase ( String id , String category , // "extraction", "qa", "summarization", ... String input , // the prompt/question sent to the system under test String expectedExact , // non-null only for cases scorable by exact/programmatic match List < String > mustContain , // key facts a correct answer must mention (programma

Introduction Every agent needs tools, and every tool needs a way to reach the model. Building Agentic Workflows in Python built that connection by hand — a hand-written JSON schema, a loop that dispatches on block.name . LLM Frameworks vs. the Raw SDK in Python showed LangChain's @tool turning a plain function into that same schema via bind_tools . Both are still bespoke : the tool lives inside one process, wired to one agent, in one language. The Model Context Protocol (MCP) solves a different problem: it standardizes the wire format between an AI application and a tool server, so the server doesn't have to be rewritten per agent, per framework, or per language. This post covers what that buys you, builds a minimal MCP server and a client that consumes it — both on the official Python SDK — and gives an honest answer to when reaching for a protocol is worth it over a direct tool call. The Problem MCP Solves Without a shared protocol, every pairing of agent framework and tool needs its own glue code: a LangChain @tool wrapper, a hand-rolled schema for the raw SDK, a different wrapper again for whatever framework a teammate picks next — an integration per framework, per tool. That's an M×N problem. MCP flattens it to M+N. A server exposes tools, resources, and prompts once, over a standard JSON-RPC protocol. Any host application — Claude Code, Claude Desktop, VS Code, or your own agent — creates an MCP client that speaks that same protocol, regardless of which framework built the host. Write the server once; every MCP-aware host can use it without new integration code. The protocol itself is intentionally boring: JSON-RPC 2.0 messages for lifecycle negotiation, tool discovery, and tool execution. Discovery ( tools/list ) and execution ( tools/call ) are the two calls that matter for this post: // tools/list response (abbreviated) { "jsonrpc" : "2.0" , "id" : 2 , "result" : { "tools" : [ { "name" : "get_account_balance" , "description" : "Look up the balance for an ac

Introduction Every agent needs tools, and every tool needs a way to reach the model. Building Agentic Workflows in Java built that connection by hand — a hand-written Tool schema, a loop that dispatches on toolUse.name() . LLM Frameworks vs. the Raw SDK in Java showed LangChain4j and Spring AI turning an annotated Java method into that same schema via reflection. Both are still bespoke : the tool lives inside one process, wired to one agent, in one language. The Model Context Protocol (MCP) solves a different problem: it standardizes the wire format between an AI application and a tool server, so the server doesn't have to be rewritten per agent, per framework, or per language. This post covers what that buys you, builds a minimal MCP server and a client that consumes it — both on the official Java SDK — and gives an honest answer to when reaching for a protocol is worth it over a direct tool call. The Problem MCP Solves Without a shared protocol, every pairing of agent framework and tool needs its own glue code: a LangChain4j tool wrapper, a Spring AI @Tool method, a hand-rolled schema for the raw SDK — three integrations for one capability, repeated for every tool and every framework you add. That's an M×N integration problem. MCP flattens it to M+N. A server exposes tools, resources, and prompts once, over a standard JSON-RPC protocol. Any host application — Claude Code, Claude Desktop, VS Code, or your own agent — creates an MCP client that speaks that same protocol, regardless of which framework built the host. Write the server once; every MCP-aware host can use it without new integration code. The protocol itself is intentionally boring: JSON-RPC 2.0 messages for lifecycle negotiation, tool discovery, and tool execution. Discovery ( tools/list ) and execution ( tools/call ) are the two calls that matter for this post: // tools/list response (abbreviated) { "jsonrpc" : "2.0" , "id" : 2 , "result" : { "tools" : [ { "name" : "get_account_balance" , "description"

I'm probably not the only one who checks every few months whether a GPU alternative has finally shipped, mostly so I can cancel a few subscriptions. Nobody doubts it's physically possible or that people have tried. The real question is why it hasn't actually happened, and the answer is economic and structural, not technical. GPUs are not uniquely ideal. They're uniquely general LLM workloads are dense matmul, high parallelism, memory-bandwidth-bound compute. GPUs handle this well but weren't built for it specifically. An ASIC purpose-built for transformer inference should beat a GPU on perf-per-watt and perf-per-dollar, and in narrow slices, it already does: Groq's LPU beats GPUs on single-stream inference throughput for models that fit its architecture Cerebras' WSE cuts interconnect overhead by putting the whole model on one wafer Google TPUs have run production workloads for years and are now sold externally via GCP So specialized hardware can win, sometimes even in production. The real question isn't whether something can beat a GPU, it's why none of these have dented Nvidia's share. 1. The capital barrier Custom silicon needs hundreds of millions in NRE cost, access to TSMC's leading-edge nodes with multi-year allocation queues, and several iterations before a design is commercially viable. That caps the field to hyperscaler balance sheets or venture funding measured in billions. The barrier isn't just the chip either. CUDA, the surrounding tooling, and production pipelines took a decade of capital and engineering to mature, and matching that means rebuilding all of it, not swapping a part. That's a second capital sink on top of the silicon itself. There's also a timing risk specific to fixed-function silicon: if the underlying model architecture shifts significantly, an ASIC taped out for today's transformer variant can become dead weight, while a GPU just needs a software update to run whatever comes next reasonably well. That risk hasn't actually played out,

In-process cron ( node-cron , @nestjs/schedule , OS crontab) runs inside one Node process. That is fine for a single instance, but it does not survive restarts gracefully, deduplicate across replicas, or share infrastructure with your other background jobs. BullMQ stores queues and schedulers in Redis . Job Schedulers (BullMQ 5.16+) are the recommended way to enqueue recurring work on a cron pattern or fixed interval. The same workers that process one-off jobs also process scheduled ones, with retries, backoff, and concurrency you already get from BullMQ. This post covers Job Schedulers in plain Node.js, operations and pitfalls, a NestJS setup with @nestjs/bullmq , and a runnable demo with a fast cron heartbeat and a daily cleanup cron. Prerequisites Node.js version 26 Redis at redis://localhost:6379 (included in the demo docker-compose.yml , or use Postgres and Redis containers with Docker Compose ) npm i bullmq For the NestJS section: npm i @nestjs/bullmq bullmq BullMQ 2.0+ does not require a separate QueueScheduler instance. Use the Job Scheduler API ( upsertJobScheduler ), not the deprecated repeat option on queue.add() . Mental model Piece Role Queue Holds jobs waiting to run Worker Executes jobs Job Scheduler Factory that enqueues jobs on a schedule Scheduled job A job instance produced by a scheduler A scheduler id is stable across deploys. Calling upsertJobScheduler with the same id updates the schedule in place instead of creating duplicates. Queue and worker Share one Redis connection config between the queue and the worker: import { Queue , Worker } from ' bullmq ' ; const connection = { host : ' localhost ' , port : 6379 }; const queue = new Queue ( ' reports ' , { connection }); const worker = new Worker ( ' reports ' , async ( job ) => { console . log ( `[ ${ job . name } ]` , new Date (). toISOString (), job . data ); }, { connection }, ); worker . on ( ' failed ' , ( job , error ) => { console . error ( job ?. name , error . message ); }); Start the