标签：#LLM

At the beginning of June I started a “ Claude withdrawal ” challenge. The plan was to run MiniMax 3 for a month, to see if I can get the same level of quality, but at 5x less the price. Until then, Claude Code was my main driver, with MiniMax on the backup, for when I was running out of quota, or sometimes for code review. The monthly bill for Claude was $100 on the Max plan, whereas for MiniMax I would pay $20 for the Token plan. All in all, it seemed like an interesting experiment. Then, half way through the challenge, Grok came into the picture. I got a very interesting offer at $35 for 3 months, then $35/month. But Grok has something neither Claude, nor MiniMax can give me out of the shelf: video and image generations. The only unknown was if switching from Claude Code to Grok will still maintain the same coding power. So I instantly took the offer, and did whatever I had to do to understand if this was the right path. And here comes the “whatever I had to do”, in plain technical terms. Switching from Claude Code to Grok – the Actual Steps The switch itself was interesting because I didn’t want to lose the Claude Code interface. I like the harness. The way it works with my codebase, the commands, the flow. So I used a helper called cliproxyapi . It’s a small proxy that sits between the Claude Code client and whatever model you point it at. You run it locally, tell it to forward requests to Grok’s API instead of Anthropic’s. Then you launch Claude Code the same way you always do, but it talks to Grok under the hood. Here’s how it goes in practice. Step 1: Install the proxy. I used brew to install it, I’m on a Mac, and also because I wanted to have it started as a service. Step 2: Set two environment variables. One is the target API base URL, for Grok that’s something like https://api.x.ai . The other is your API key. "env" : { "ANTHROPIC_BASE_URL" : "http://localhost:8317" , "ANTHROPIC_API_KEY" : "cliproxy-local-key" } , Notice how we use “cliproxy-local-key”, be

Structured output broke on us three times. The third time taught us what "operator-ready" means. Last quarter we shipped a contract-extraction agent to an enterprise legal team. Schema validation passing at 97%. Human reviewers satisfied with the output quality in testing. Rollout went smoothly. Then it broke. Three times. In three completely different ways. The first two failures we fixed with better prompts and stricter schemas. The third one taught us something the first two hadn't: that "operator-ready" is not a technical checklist. It's a claim about your agent's behavior under conditions you didn't design it for. Failure one: the validation paradox Week two. A lease agreement came through with a renewal clause formatted as a table instead of prose. Our extractor looked for renewal terms in a specific JSON path. The table format populated the schema differently. Validation passed. The extracted renewal date was off by two years. The fix was obvious in retrospect: add a canonical-format normalization step before extraction. But the lesson was sharper than that. Schema validation tells you the shape of the output, not whether the content is correct. A JSON object with the right keys and the right types can still contain wrong values. Our 97% validation success rate was measuring the wrong thing. It was measuring structure conformance, not content accuracy. After this failure, we separated validation into two signals: schema validity (does the object have the required fields) and field confidence (do we have evidence the content is correct). We started logging both. An output is trusted only when both signals are above threshold. Failure two: the retry loop that lies Month one. A particular clause type appeared in a contract format we hadn't trained our test set on. The extractor failed schema validation on the first attempt. Our retry logic kicked in, filled missing fields with model-inferred defaults, and passed validation on the third try. The output looked rig

You built a system to surface information so a person could decide. Somewhere it started deciding for them — the output stopped saying "here's what the documents show" and started saying "you should do X." Nobody designed that drift. An LLM, when asked a question, produces an answer-shaped thing, and an answer easily becomes a verdict. What everyone tries A prompt instruction: "Don't make recommendations." "Only state what's in the documents." People add the line and assume the boundary is enforced. Why it doesn't work A prompt instruction is a request, not a guardrail. The model follows it most of the time, then on the input that matters produces a confident recommendation anyway, because nothing structurally prevents it. "Please don't make recommendations" is to a guardrail what a sticky note saying "please don't enter" is to a locked door. And the stakes are higher than they look. When output drifts from evidence to verdict, accountability moves. As long as the system returns evidence and a human decides, the human owns the decision. The moment the system returns a verdict and the human defers, the system is deciding things it was never validated to decide — and when one is wrong, accountability is a blank. High-stakes fields separate evidence extraction from judgment on purpose; most RAG systems erase that line by default. The one shift Decide what the output is and enforce it structurally. An output should declare itself: answer, evidence, missing facts, or out-of-scope. "Return decision material, not a decision" has to live in the output contract and in gates — not in a polite request to the model. The system supplies frames; the human supplies verdicts. This is the output boundary — one of three places production RAG dies. Read the full version on my blog , where this connects to the RAG Failure Diagnosis Kit for teams debugging production RAG.

Most backend engineers already know how to handle HTTP errors. 400 means the request is bad. 401 means auth failed. 429 means rate limited. 500 means something broke upstream. Retry a few times, add exponential backoff, log the response body, move on. That works fine for many APIs. It works badly for LLM APIs. LLM providers may use normal HTTP status codes, but the operational meaning behind those errors is different enough that treating them like ordinary REST failures can make your app slower, more expensive, and harder to debug. The mistake I kept making Early on, I handled LLM failures the same way I handled every other external API: if ( response . status === 429 || response . status >= 500 ) { retryWithBackoff (); } Simple. Familiar. Dangerous. That logic misses the actual question your app needs to answer: What kind of LLM failure happened, and what should the product do next? Because an LLM API failure is rarely just "one HTTP request failed." It can break: a user-facing chat response a background agent run a document generation job a tool-calling workflow a batch evaluation pipeline a structured JSON generation step And each one needs different handling. Not all 429s mean the same thing For a normal API, 429 Too Many Requests usually means: Slow down and retry later. With LLM APIs, 429 can mean several different things. It might be a temporary rate limit: { "error" : { "message" : "Rate limit reached" , "type" : "rate_limit_error" } } Retrying with backoff may help here. But it might also mean quota exhaustion: { "error" : { "message" : "You exceeded your current quota" , "type" : "insufficient_quota" } } Retrying this does not help. It just adds latency, noisy logs, and a worse user experience. It could also be model-specific pressure. One model may be overloaded while another model from the same provider, or a different provider, would work fine. So your handler should distinguish between: temporary rate limit hard quota exhaustion model-level capacity is

Yapay zeka mühendisi (AI Engineer) olmak, yalnızca ChatGPT’ye veya Claude'a akıllıca promptlar yazmaktan ibaret değildir. Yapay zeka modellerini kullanarak gerçek dünyadaki karmaşık problemleri çözen, sürdürülebilir, güvenli ve ölçeklenebilir yazılımlar inşa etmek ciddi bir mühendislik disiplini gerektirir. Eğer temel basamakları sağlam kurmadan doğrudan en üstteki otonom ajan (agentic) yapılarına atlamaya çalışırsanız, üretim ortamına çıktığınızda beklenmedik hatalarla (hallucination), kontrol edilemeyen maliyetlerle ve aşırı yüksek yanıt gecikmeleriyle (latency) karşılaşırsınız. Bu rehberde, sıfırdan başlayıp üretim seviyesine (Production-ready) kadar uzanan 17 katmanlı AI Engineer Yol Haritası'nı adım adım inceleyeceğiz. Phase 1: Foundation (Temel Katman) Binalar gibi, yapay zeka uygulamaları da güçlü temeller üzerine kurulur. Bu aşamada amacımız yazılım geliştirme ve temel model etkileşim mekanizmalarını kavramaktır. Python & Data Yolculuğun ilk ve en kritik basamağı Python programlama dili ve veri yönetimidir. Yapay zeka dünyasında veri okuma, temizleme, dönüştürme ve analiz etme süreçleri günlük işlerin büyük kısmını oluşturur. Python'ın liste, sözlük gibi yerleşik veri yapılarını, nesne tabanlı programlama (OOP) mantığını ve fonksiyonel yapısını kavramak şarttır. Ayrıca veri manipülasyonu için Pandas ve sayısal işlemler için NumPy gibi kütüphanelerde yetkinlik kazanılmalıdır. Detaylı bilgi için Python Resmi Dokümantasyonu incelenebilir. Software Engineering & APIs Kodunuzun lokal bilgisayarınızda sadece "çalışması" yeterli değildir. Üretim ortamında çalışacak kodun temiz, okunabilir, sürdürülebilir ve test edilebilir (Unit/Integration Tests) olması gerekir. Ayrıca modelleri uygulamalara entegre ederken REST API tasarımları, authentication (kimlik doğrulama) mekanizmaları, asenkron programlama, hata yönetimi (Error Handling) ve servis mimarileri hayati önem taşır. Git gibi versiyon kontrol sistemleri ve temel Docker bilgisi bu katmanın ayrılmaz parçasıdır. Pro

Visibility Collapse in the Post-LLM Engineering Stack Artificial intelligence is still improving. But something important has changed in how that improvement is perceived. For developers and engineers working closely with frontier models, the experience is no longer one of explosive capability jumps. Instead, it feels like: incremental improvement under increasing structural constraints This shift is not about stagnation. It is about uncertainty in how AI capability is exposed, deployed, and interpreted. Capability vs Visibility: the new separation Recent frontier model systems (such as Fable 5, as described in industry discussions) highlight an important architectural pattern: Certain capabilities are no longer fully exposed in production environments: advanced coding assistance deep debugging autonomy bioinformatics reasoning cybersecurity-related reasoning This does not necessarily imply reduced model capability. Instead, it reflects a system-level separation: model capability ≠ deployed capability System interpretation: Modern AI stacks are becoming layered systems: Raw Model → Safety Layer → Policy Filter → Deployment Interface → User Access This means developers are no longer interacting with models directly. They are interacting with constrained capability surfaces. Perceived slowdown in LLM progress Despite continued benchmark improvements: reasoning scores increase gradually multimodal capabilities expand tool-use frameworks improve The perceived acceleration of AI has weakened. Compared to 2022–2023, there are fewer qualitative jumps. From an engineering perspective, this suggests a transition: from capability discontinuity → capability smoothing In other words: AI is still improving, but improvements are less visible at the system interaction level. Economic mismatch: scaling vs returns The AI ecosystem is currently defined by a structural tension: Inputs: massive GPU infrastructure investment multi-billion-dollar training runs hyperscaler-scale capital a

A large-scale audit of AI-as-judge evaluation — covering over half a million individual judgments — finds that AI judges are consistently reliable but not valid, meaning they give the same answer repeatedly without that answer being correct. Published work and popular benchmarks like Chatbot Arena have treated consistency as proof of trustworthiness, and the audit shows that assumption is unfounded. Key facts What: Using one AI to grade another is now common — but the biggest audit yet shows these graders are consistent without being correct. A judge that always picks "answer A" scores perfectly on consistency. When: 2026-06-19 Primary source: read the source (arXiv 2606.19544) The distinction matters: a judge is reliable if it's consistent (same question, same answer), and valid if those answers are actually correct. The audit's central finding is that AI judges are reliable without being valid, and the field has been treating the first as evidence of the second. Because consistency is easy to measure and looks reassuring, it has stood in for actual trustworthiness across a lot of published work. A new audit makes the problem stark: a judge that ignores both answers and always picks the one labeled "A" would be perfectly consistent — flawless reliability, identical verdict every time — and completely worthless, because it never read anything. Consistency is trivially easy to fake and says almost nothing about whether the judging is sound. Yet "the judge agrees with itself" has done significant reassurance work in papers and benchmarks, and the always-pick-A example shows exactly how empty that reassurance is. When the researchers corrected for the agreement you'd get by chance — as any fair test should — confident-looking scores deflated noticeably. Gaps between models that seemed meaningful shrank or blurred. Accepted folk wisdom also took a hit: the long-standing worry that AI judges are suckers for longer, wordier answers turned out to be far weaker than assumed

Key Takeaways You can't unit-test a coach agent the way you test a pure function — the output is non-deterministic and "good" is a judgment call, not an assertion. An LLM-as-judge harness lets you grade a whole test set automatically against a rubric, which is the only way solo-scale eval stays sustainable. But the judge is itself a fallible model. If you don't design around its known biases — position, verbosity, self-preference, and quiet drift when the judge model updates — you build a green dashboard that means nothing. The mitigations that actually work are mechanical, not prompt-magic: shuffle order on every pairwise call, pin the judge version, keep a small human-labelled anchor set, and re-check the judge against it. The problem I actually had FamNest's coach agent generates responses to parents — check-ins, encouragement, the occasional gentle redirect. I have a growing pile of these interactions, and every time I change a prompt, swap a model, or adjust the pipeline, I need to know one thing: did I just make it better or worse? For normal code, that's what tests are for. I change something, the suite runs, red or green, done. But there's no assertEqual for "was this an empathetic, useful response to a tired parent." The output changes every run even at temperature zero-ish, and the quality bar is a human judgment, not a fixed string. Two responses can be worded completely differently and both be good. One can match my "expected output" word for word and still be worse than a version that didn't. So the honest options were: read every response by hand every time I change something (does not scale past about week two), or build a harness where a model grades the outputs against a rubric. I built the harness. Then I spent an uncomfortable amount of time learning all the ways a harness like that can lie to you. What the harness actually is At its simplest, it's a loop: def evaluate ( test_cases , coach_agent , judge ): results = [] for case in test_cases : res

A ChatGPT subscription starts at $20 a month and is one of the cheapest ways to run inference. OpenAI has also been fairly relaxed lately about third-party agents using them , which makes the deal even better for a lot of us. But a subscription can't be used as freely as pay-per-token access , and the providers police the difference. Anthropic recently narrowed its subscriptions to first-party apps; OpenAI has its own limits. Here's what will get you banned from an OpenAI subscription. Sharing your subscription A ChatGPT subscription is strictly personal. One subscription, one user. Sharing yours breaks OpenAI's terms of service. That also covers account pooling and account rotation, where several people share the same credentials to dodge rate limits. Running it in automation Automation (CI, runners, schedulers) should run on per-token pricing, not a subscription . Once a system calls the OpenAI API with your token while you're not in the loop, the usage stops being personal. No unattended production system should run on a ChatGPT subscription. Serving other users For now, you can point an autonomous agent like OpenClaw or Hermes at your ChatGPT subscription, as long as it only talks to you. The moment that agent starts chatting with other people, or serving them in any way, it turns into a team use case , and that inference should be paid per usage. Putting it in a commercial product Same logic here. Making an LLM call authenticated with an individual ChatGPT subscription inside a product you ship breaks OpenAI's terms. That access is subsidized, and reselling it in any form isn't what it's meant for. If you've built something just for yourself and you're the only user, you're probably fine. The bottom line A ChatGPT subscription is personal . Anything that stretches past personal use can get you restricted or banned. If you're not sure your usage counts, move it to pay-as-you-go. If you want to keep the subscription for your own work and fall back to per-token pr

Tree of Thoughts was a genuine leap. Instead of reasoning in one straight line, it branches into several lines, scores them, prunes the dead ends, and searches for the best path — so a puzzle that would sink a single chain of thought becomes solvable. But a tree has one restriction baked right into its shape, and once you see it you can't unsee it: every node has exactly one parent. A branch can be extended or abandoned. It can never be combined with another branch. That matters more than it sounds. Real problems decompose, and when they do, different branches each get part of the answer right. Branch A nails the first half; branch B nails the second half; neither is fully correct on its own. A tree is forced to pick one and throw the other's good half away. Graph of Thoughts (GoT) removes exactly that restriction. 🕸️ Interactive demo (a real merge-sort that branches, merges, and refines — with live-verified scores): https://dev48v.infy.uk/prompt/day21-graph-of-thoughts.html The core idea: thoughts are nodes in a graph GoT reframes reasoning as building a graph . Each vertex is a thought — a partial solution or intermediate result. Each edge is an operation that produced one thought from one or more others. Because it's a general graph and not a tree, a thought is allowed to have several parents, and edges can even loop back on themselves. That single change in the data structure is the entire conceptual leap. Everything else is just the operations you're now free to run on that network. The four operations generate (branch) — the familiar move, straight from Tree of Thoughts. From one thought, produce several different next thoughts. This can also be a split : break the input into independent sub-problems solved on separate branches. Diversity matters here — near-duplicates waste budget. score / rank — turn each thought into a number so the controller can compare them. Objective scorers win: a validator, a test, a metric. In the demo, the scorer is deliberately con

Contorium is a local-first system that introduces persistent project cognition into AI-assisted development workflows. Instead of treating AI as a tool that operates on code, Contorium treats the project itself as a structured, evolving system. ⸻ 🧠 Problem Modern AI coding workflows suffer from a structural limitation: Even with tools like: Cursor Claude Code MCP-based agents IDE copilots context is still: fragmented session-based non-persistent weakly structured This leads to: repeated explanations, lost reasoning, and architectural drift ⸻ 🧩 Solution: Project Cognitive Runtime (PCR) Contorium introduces a runtime model where project understanding is persistent and structured. ⸻ Core Components ⸻ PIK — Project Intent Kernel PIK defines the system-level intent of a project: primary goal constraints non-goals priority weighting It acts as a stable semantic anchor. ⸻ CIL — Cognitive Interaction Layer CIL captures reasoning: why decisions were made what alternatives were considered how context influenced outcomes It makes reasoning persistent instead of ephemeral. ⸻ Timeline Layer All system changes are recorded as events: code changes AI outputs tool interactions architectural decisions This enables replay and evolution tracking. ⸻ Drift Detection Layer A continuous alignment system compares: current behavior vs PIK intent It detects: intent drift structural drift behavioral drift And produces measurable deviation signals. ⸻ 🔁 System Loop Contorium forms a continuous loop: PIK defines intent Execution produces behavior Timeline records evolution Drift system evaluates alignment Suggestions guide correction This creates a self-regulating project system. ⸻ 🧠 Key Insight Contorium is not an AI coding tool. It is a: Project Cognitive Runtime (PCR) A system where software projects maintain structured intelligence over time. ⸻ 🚀 Why it matters The bottleneck in AI development is no longer capability. It is continuity of understanding across: time tools agents sessions Conto

If you run a RAG system in production, you eventually hit a dilemma that has nothing to do with your model and everything to do with your cache. Cache the answers to save tokens and latency, and one day a source document changes — but your cache keeps cheerfully serving the answer it built from the old document. Nobody gets an error. The number is just quietly wrong. Cache nothing , and every single call re-retrieves the same chunks, re-reads them, and re-pays the full context bill to rebuild an understanding you already built five minutes ago for a nearly identical question. Stale or expensive. Most teams pick "expensive" because at least it's correct, then bolt on a TTL and hope. This post is about why the TTL doesn't save you, and about two specific, mechanical fixes that let you cache RAG context and stay fresh. I maintain an open-source library called Coalent that implements both, so I'll use it for the runnable examples — but the two ideas are portable and worth stealing even if you never pip install anything. Failure mode 1: the stale RAG cache (and why a TTL won't save you) Here's the standard "answer cache" sitting in front of retrieval: answer = cache . get ( query ) if answer is None : chunks = retriever . retrieve ( query ) answer = llm . synthesize ( query , chunks ) cache . set ( query , answer , ttl = 3600 ) return answer This works until billing.md changes. The refund window goes from 30 days to 14. Your cache has an answer keyed on "what is our refund policy?" that says 30, and it will keep saying 30 for up to an hour — or forever, if the same question keeps refreshing a TTL that never expires under load. The reason this is hard is that the cache key (the query) has no relationship to the thing that changed (the source). You cached an answer; you threw away the fact that this particular answer was derived from billing.md . So when billing.md changes, you have no way to find the answers that depended on it. The TTL is a confession that you can't answ

An AI feature can feel impressive and still be a bad product decision. The demo is fast. The answer sounds useful. The team is excited. Then usage grows and nobody can answer the basic questions: Is it accurate enough? Is it saving time? Which customers trust it? Why did costs spike? Should we scale it, fix it, or kill it? That is the trap an AI metrics baseline prevents. A baseline is not a dashboard full of vanity charts. It is a small set of before-and-after measurements that tells you whether an AI workflow is getting better, getting worse, or merely getting more expensive. Why AI features fail without a baseline Most software teams already track uptime, errors, and conversion. AI features need those too, but they also need new signals because model behavior is probabilistic. A normal API either returns the expected response or throws an error. An AI workflow can return: a fluent answer that is wrong a correct answer with missing evidence a useful answer that costs too much a slow answer that users abandon a safe answer that refuses too often a cheap answer that hurts trust a high-rated answer that does not improve the business workflow Without a baseline, every production discussion becomes opinion-driven: "The model seems better." "Users like it." "The new prompt reduced hallucinations." "The expensive model is worth it." Maybe. Maybe not. The baseline turns those claims into measurable comparisons. What an AI metrics baseline is An AI metrics baseline is the starting measurement for the workflow before you optimize or scale it. It answers five questions: What does the workflow cost today? How good are the outputs today? How fast and reliable is the experience today? Do users adopt and reuse it? Does it improve the real task it claims to improve? You do not need 80 metrics on day one. You need a small set of metrics that match the feature's risk and purpose. For example: Feature Useful baseline Support answer bot resolution rate, citation quality, escalation r

The Problem LLM evaluation tools like Patronus AI excel at hallucination detection, toxicity checks, and semantic relevance. But they don't catch the structural failures: A JSON response missing required fields A function call with malformed parameters Output that violates schema constraints Latency budget overruns silently degrading UX Cost explosions from runaway token usage These aren't hallucinations. They're verification failures. The Solution correctover-patronus is an adapter that runs Correctover's 87 deterministic verification rules as native Patronus evaluators. Every verdict comes with a recomputable proof hash — meaning you can verify the verifier. pip install correctover-patronus The 6 Dimensions Dimension What It Checks Example Structure Output format validity JSON parses correctly Schema Field presence & types Required fields exist Identity Semantic relevance to input Response addresses the question Integrity Forbidden pattern absence No Tracebacks or error messages Latency Response time budget Under 30s threshold Cost Token usage budget Under 10k token limit Usage Full 6-Dimension Verification from correctover_patronus import CorrectoverEvaluator , CorrectoverConfig config = CorrectoverConfig ( min_confidence = 0.7 , latency_rules = { " max_ms " : 5000 }, cost_rules = { " max_tokens " : 4000 } ) evaluator = CorrectoverEvaluator ( config = config ) result = evaluator . evaluate ( task_input = " Summarize this article... " , task_output = " The article discusses... " , task_context = { " source " : " article " , " word_count " : 1500 } ) print ( f " Overall: { result . score : . 2 f } ( { ' PASS ' if result . pass_ else ' FAIL ' } ) " ) print ( f " Proof hash: { result . metadata [ ' proof_hash ' ] } " ) for dim , info in result . metadata [ ' dimensions ' ]. items (): print ( f " { dim } : { info [ ' status ' ] } (score= { info [ ' score ' ] : . 2 f } ) " ) Individual Dimensions from correctover_patronus import correctover_structure , correctover_inte

Of all the things involved in turning a woman's life into a 60-second reel, I assumed that the writing would be the easy part. Surely telling a good story in 150 words is exactly what a large language model should be good at; yet it proved surprisingly difficult. Draft after draft suffered from common AI weaknesses: a tendency to use hyperbole and inspirational language, to generalize, follow generic founder arcs ("built in a basement"), and focus on morbid details. (For this effort, I was using Sonnet 4.6, which I found to be better — more grounded, more true to facts, less inventive — than Opus 4.7 at creating longer bios.) Getting to something publishable took a significant amount of work. That said, the human in this story also found writing good short story arcs surprisingly challenging, so the effort spent on getting the system to do it decently was well worth it. Here's some context, then what I did. I've been publishing short biographies of notable women for a while now, on a website called Tycoona . Why notable women? Because in each field there are so many women who contributed so much and are little known. I've never understood why Corita Kent , the pop-artist nun, isn't as famous as Andy Warhol, why Hetty Green , the Gilded Age value investor called both the "Witch of Wall Street" and the "Queen of Wall Street," disappears into history behind Benjamin Graham and his protege Warren Buffett. The problem I faced is that no one was seeing my bios, so I decided to make short videos or reels in hopes of increasing my reach. Creating the technical infrastructure for the reels on top of my existing system was fun and relatively straightforward. Each reel consists of a series of "beats," and Remotion turns those beats into a vertical reel, complete with on-screen text and royalty-free images & attributions pulled from Wikimedia, Flickr, or Library of Congress. For content, the beat generator relies on the knowledge base of validated facts that my system creates f

You've got a folder of a few hundred screenshots and you want the text out of each one. Or you want to generate a batch of images for a side project. Or you just want to drop a single "summarize this" call into a script you're writing on a Sunday afternoon. So you open the pricing page for the official API, do the math on per-token billing plus setting up keys and a payment method, and it's hard to justify, because the exact same model will do the exact same thing for free in a browser tab. There are really two ways to get a model like ChatGPT or Gemini to do work for you. The web UI is free, or already covered by a subscription you're paying for anyway, but you drive it by hand. The API is scriptable, but you pay by the token. Most of the time that trade-off is fine. But for a whole category of work like hobby projects, throwaway scripts, research, or anything that doesn't need production-grade reliability, you're stuck picking between "free but manual" and "automated but paid." Which raises the obvious question: why not automate the free web UI? It's just a webpage. You open it, type in the box, click send. It turns out that hides a few fiddly problems, which I ran into enough times that I eventually built a small library for them. In this article we'll work through what it takes to automate these UIs, and at the end I'll show how little code it comes down to. 1. What it takes to drive a chat UI A single round trip with ChatGPT or Gemini breaks down into four jobs: Get your text into the input box Optionally attach a file Wait for the model to finish answering And read the answer back out. Every one of these is harder than it sounds, because the page is a modern single-page app that was never built to be driven by a script. We'll use Selenium with undetected-chromedriver, and for now assume the browser is already open (we'll get to launching it in the next section). To keep the code readable I'll show whichever of the two platforms makes each problem clearest, and

I had a bug last month that took most of a Saturday to find. A support bot we shipped started promising refund timelines that didn't match policy. Customer complaints, frantic Slack messages, the usual. The prompt had changed three weeks earlier. Nobody could remember why. Git blame pointed to a one-line edit inside a 200-line SYSTEM_PROMPT constant. No PR description, no diff worth reading. That's when I knew I'd been writing prompts wrong for the last two years. PromptOT - Prompt Management Platform Compose prompts from typed blocks, version safely, and deliver to your apps via API. The prompt management platform built for AI engineering teams. promptot.com Prompts are code, but we treat them like Notion docs A typical system prompt for anything useful crams five things into one string: You are a friendly support agent for Acme. Use this knowledge: {{kb}}. Follow escalation rules. Never share internal ticket IDs. Reply in plain text, two to four paragraphs. That's a role, context, instructions, guardrails, and an output format all jammed together. When the PM wants to soften the tone, they're editing the same string an engineer uses to update the knowledge base. When security adds a guardrail, it lands inches from the response format. One bad edit and every reply ships broken. We wouldn't write code this way. So why are prompts always a 200-line const somewhere in lib/ ? What I built PromptOT is a prompt management platform. The core idea is small: typed blocks instead of flat strings. You break a prompt into pieces. Each piece has a type — role, context, instructions, guardrails, output_format, custom. Each one is independently editable, can be toggled on or off, and has its own version history. The compiler joins them into a single prompt string at delivery time. Block 1 — role : " You are a support agent for Acme..." Block 2 — context : " Knowledge base: {{kb}}..." Block 3 — instructions : " 1. Acknowledge the issue..." Block 4 — guardrails : " Never share inte

IndexCache: Killing the Indexer's O(NL²) Bottleneck in DeepSeek Sparse Attention Notes from my notebook on GLM-5.2 / DeepSeek Sparse Attention (DSA), reconstructed from the IndexCache paper (Bai, Dong et al., Tsinghua + Z.ai, 2026) — the mechanism behind GLM-5.2's "IndexShare." 1. Why this exists — the bottleneck nobody talks about DSA's whole pitch is: don't do full O(L²) attention, instead let a cheap lightning indexer look at all preceding tokens and pick the top-k (k=2048) that actually matter, then do real attention only on those. That drops core attention from O(L²) → O(Lk). Great — except I missed this the first time I read DSA: the indexer itself is still O(L²) . It has to score every preceding token against the query to decide who's in the top-k. So across N layers you've traded one O(L²) cost for N separate O(L²) costs — total O(NL²). At long context this indexer becomes the dominant cost, not the attention it was supposed to fix. Adding the indexer is "DSA on steroids" because it kills DSA's one real bottleneck (full attention) — but in doing so, it grows its own. The indexer is cheap per-FLOP (few heads, low-rank, FP8) but it still runs at every single layer. The fix the paper proposes isn't a smarter indexer — it's don't run it every layer at all. 2. The core insight: adjacent layers pick almost the same tokens If you measure pairwise overlap between the top-k token sets selected by each layer's indexer, adjacent layers share 70–100% of their picks. The heatmap even shows block structure — clusters of layers (e.g. layers 3–5, 17–30, etc.) that all converge on roughly the same "important" tokens. So most of the O(NL²) indexer cost is redundant computation of the same answer. This motivates IndexCache : split the N layers into two roles — F (Full) layers — run their own indexer, compute fresh top-k, cache it. S (Shared) layers — skip the indexer entirely, just reuse the nearest preceding F layer's cached top-k. The first layer is always F (has to seed the

Wix-owned vibe coding platform Base44 has started rolling out its own AI model — with hopes that it will eventually outperform frontier models.

How I ended up running a local LLM to generate my git commit messages