标签：#softwareengineering

Everyone talks about prompts memory RAG But production issues were actually loops false completion replay retries wrong tool non deterministic execution Here are the top 7. That's what led us to build Failproof AI not because we wanted another framework, but because we kept seeing the same reliability problems across every framework.

i've been that person. standing in front of leadership with an 18-month architecture diagram, explaining why we need six months of infrastructure before a user touches a single feature. and it made sense. for 25 years it made sense. writing boilerplate was expensive. every feature came with a tax — database migrations, routing config, auth wiring. build a shared platform first, pay that tax once. the roadmap justified the investment. then i saw a stat that wouldn't leave me alone. roughly 60% of features on a six-month roadmap are obsolete by launch. not slightly off. obsolete. the customer's problem shifted. the market moved. you spent six months building a precise answer to a question nobody asks anymore. the longer you invest before showing something real, the more expensive it is to admit you were wrong. so you don't. you ship the wrong thing and call it "on schedule." i've done it. i've watched it happen. AI didn't create this problem. but agents are making it impossible to ignore. the 82-point gap mckinsey's 2025 survey: 88% of organizations use AI. only 6% see real bottom-line impact. that 82-point gap isn't about tools. everyone has the same tools. but something shifted in their may 2026 report. they describe agents working overnight — enriching requirements, generating code, packaging outputs for morning review. they call it the "24-hour sprint." leading organizations see 3-5x productivity with 60% smaller teams. a product owner logs in at 9am and finds a feature went from requirements to tested code overnight. nobody worked late. agents did. that's not autocomplete. that's a different delivery model. and here's what most teams miss: it only works when the work is small, bounded, and complete. agents need to know where a task starts and ends. horizontal platform architectures don't give them that. the codebase is the prompt jeremy d. miller built wolverine for .NET. in june 2026 he wrote: "the structure of your codebase is now, effectively, part of the prom

The cohesion paper series is now published in full — five papers that build a chain from the concept of cohesion to the Independent Variation Principle (IVP) . The chain: On the Nature of Cohesion — defines cohesion as a $2k$-tuple: for $k$ partitioning rules, $k$ (purity, completeness) pairs. Proves the knowledge-embodiment theorem: maximal cohesion under a rule coincides with exact knowledge embodiment under that rule. Shows that every published algorithmic cohesion metric measures a structural proxy (method-call overlap, shared-field density), not cohesion as defined by a principle. DOI: 10.5281/zenodo.20785752 Causal Cohesion — instantiates the schema under one concrete rule — change-driver-assignment identity: elements belong together iff $\Gamma(e_1) = \Gamma(e_2)$. Develops the metric $H_\text{causal}(M) = (\text{purity}(M), \text{completeness}(M))$, a two-dimensional score that fills one slot of the $2k$-tuple. DOI: 10.5281/zenodo.20785881 Four Necessary Conditions for Optimal Modularization — from the schema plus the objective of minimizing change propagation, proves four conditions — Admissibility, Element Form, Separation, Unification — are necessary and jointly exhaustive, uniquely pinning the $\Gamma$-equality partition $E / \tilde{\Gamma}$. DOI: 10.5281/zenodo.21362420 Why Minimizing Change Propagation Minimizes Maintenance Cost — decomposes total maintenance cost into access, alignment, cognitive, and domain-fixed components. Proves that minimizing change propagation cost is equivalent to minimizing total maintenance cost under an explicit coefficient condition, justifying the objective paper 5 assumed. DOI: 10.5281/zenodo.21362542 The Independent Variation Principle — synthesizes the chain into a single structural principle and examines the premises (change drivers, functional model, change isolation), preconditions (driver independence, decisional autonomy), and scope boundary. DOI: 10.5281/zenodo.21362618 Two derivations Last month's preprint — Der

If you ask me what part of the system I protect the most, the answer is the database. I've been writing software alone for twenty-four years, and across every platform I've built, the rule has stayed the same: the web servers can take whatever you throw at them, the batches can be rebuilt, but the database has to stay idle on purpose. Not because I love idle databases, but because the day a database actually starts to struggle is a day with very few good options. This article is about what "keep the database idle on purpose" actually means in practice, and about one particular kind of graph that, in my experience, almost nobody is watching. The three layers and what each of them gets I think of a production system as having three tiers, and each tier gets a different rule. The web server tier can be horizontally scaled. If load grows, you add machines. If something is wrong, you take a machine out of the pool, and the others handle it. Failures here are visible immediately, and they're cheap to recover from. The batch server tier can be scaled up or out depending on the work. A batch that's too slow can be split. A batch that crashes can be retried. End users don't see batch servers, so a stuck batch is a problem for me and not for them. Some headroom up here is fine. The database tier is the one I treat completely differently. The database is not where you absorb load. The database is what you protect from load. The reason is simple: the other tiers can be rebuilt or re-scaled. The database is the irreplaceable record. If it slows down, everything slows down. If it falls over, you don't have many minutes before the rest of the stack notices. So my rule for the database is: keep it idle. Not idle in the sense of "doing nothing." Idle in the sense of "running well below its capacity, at all times, so that any extra load it picks up has somewhere to go." For more than a decade I ran a large appliance-grade database where I kept the load average below 1 at all times. N

For decades, software was designed around one assumption: A human would be the one using it. That...

Episode 1 established the mindset: failure is normal, not a sign of bad engineering. Episode 2 gets specific — you can't detect or handle a failure you can't even name. Saturday, Round 2 👦 Nephew: Uncle, last time you convinced me failure is basically guaranteed. Fine, I accept it. So what actually fails ? 👨‍🦳 Uncle: You tell me. Start listing things that could go wrong in your app right now. 👦 Nephew: Uh... the server could crash. The database could go down. My code could have a bug. 👨‍🦳 Uncle: Keep going. 👦 Nephew: The network? Someone could deploy the wrong thing? Payment gateway dies mid-checkout? 👨‍🦳 Uncle: You just named six of the seven categories without trying. You already know this. You've just never sorted it. 1. Hardware Failure 2. Software Failure 3. Network Failure 4. Database Failure 5. Third-Party Failure 6. Human Error 7. Resource Exhaustion 👦 Nephew: Then why do we need the list at all, if I already know it instinctively? 👨‍🦳 Uncle: Because "instinctively" isn't fast enough at 2 AM. Let's trace each one properly. Part 1 — Hardware Failure 👦 Nephew: This one's obvious anyway — I deploy to AWS. The cloud hides hardware failure from me. 👨‍🦳 Uncle: Does it? 👦 Nephew: ...doesn't it? That's the whole point of paying for EC2 instead of buying a server. 👨‍🦳 Uncle: Let's trace it. Your app sits on an EC2 instance. What's underneath the instance? 👦 Nephew: Virtual machine stuff, I guess? 👨‍🦳 Uncle: And underneath that ? 👦 Nephew: ...an actual physical machine somewhere. In a data center. 👨‍🦳 Uncle: There it is. Your app | "Virtual" server (EC2/Droplet) | ACTUAL physical hardware somewhere in a data center | Still capable of failing — just less visible to you 👦 Nephew: So it's not hidden. It's just one layer further away than I thought. 👨‍🦳 Uncle: Exactly. AWS absorbs a lot of it — that's part of what you're paying for — but disks still fail, instances still get abruptly terminated, whole availability zones still go down. That's Hardware Failure . Hardware Fa

Your AI agent is confident. It points to line 42 of PaymentService.java . "There's your null pointer exception." You check. Line 42 is a comment. The code was refactored 14 commits ago. The production crash happened 3 hours ago . Your agent just spent 45 minutes debugging ghosts . The Problem: Agents Are Stuck in the Present Every AI coding agent today — Claude Code, Cursor, Copilot, Cody, you name it — operates on the same assumption: The code that matters is at HEAD . But production bugs don't live at HEAD . They live in the commit that was running when the crash happened. That commit is buried under hotfixes, refactors, dependency updates, and feature merges that landed after the incident. HEAD (now) ← Agent analyzes THIS │ ├─ feat: add new payment provider ├─ refactor: extract UserService ├─ fix: handle edge case in checkout ├─ chore: update dependencies │ ▼ a1b2c3d (3 hours ago) ← Bug ACTUALLY lives HERE Your agent confidently finds bugs in code that didn't exist when the crash occurred . The Insight: Git Already Has Time Travel We don't need a time machine. Git has had one for years: git worktree . # Get the commit from 3 hours ago git log --before = "3 hours ago" -1 --format = "%H" # → a1b2c3d4e5f6... # Create an isolated, read-only snapshot at that commit git worktree add /tmp/debug-a1b2c3d a1b2c3d # Now analyze the historical codebase cat /tmp/debug-a1b2c3d/src/PaymentService.java # Clean up when done git worktree remove --force /tmp/debug-a1b2c3d This gives you: ✅ Isolated — doesn't touch your working directory ✅ Parallel — can have multiple historical snapshots simultaneously ✅ Disposable — cleanup is one command ✅ Zero deps — pure Git, works everywhere The Missing Piece: Teaching Agents When to Time-Travel Agents already know git log , git show , git diff , cat , grep . They can analyze code perfectly. What they struggle with : Fuzzy time → commit resolution — "last night", "v2.4.1", "the deploy before the hotfix" Worktree lifecycle management — create,

PR volume went up, ticket quality didn't, and the gap got filled with LLMs on both sides of the review: bots reviewing, bots replying, bots occasionally arguing with bots about priorities that only existed in a teammate's head. Our CEO named the actual problem, and it's bigger than code review.

The biggest losses in software history were, with one deliberate exception, not attacks. They were silent, correlated, self-inflicted — and they teach the exact risk autonomous AI agents are about to make expensive again. At 9:30 in the morning on August 1, 2012, Knight Capital Group was one of the largest trading firms in the United States, executing a sixth of all the volume on the New York Stock Exchange. By 10:15 it was, for practical purposes, finished. In those forty-five minutes a piece of its own trading software (not a hacker's, its own) fired more than four million unwanted orders into the market, accumulating roughly $7 billion in positions the firm never meant to hold and a loss of about $440 million by the time humans understood what their machine was doing. The cause, documented in the SEC's administrative proceeding, was almost insultingly small: a deployment that updated seven of eight servers. The eighth still carried a dormant piece of code called Power Peg, retired years earlier, and the new release reused the old feature flag that woke it up. No one attacked Knight Capital. The market data was accurate, the exchange functioned perfectly, and every system reported itself healthy while the company bled ten million dollars a minute. That shape (no adversary, no alarm, one change propagating everywhere at once) turns out to be the shape of almost every entry on the list below. We've written before about the biggest bug-bounty payouts in history , the ledger of what it costs when someone does attack. This is the other ledger, the bigger one: what software has cost when nobody attacked at all. Every figure below states what it counts, and comes from a primary or authoritative source (inquiry boards, SEC filings, statutory inquiries) linked at the end. The ledger 1. CrowdStrike outage (2024) — roughly $5.4 billion in direct losses to Fortune 500 companies alone (estimate). One faulty content update to the Falcon Sensor security agent blue-screened Windo

I am interested in agentic coding for the same reason I care about good engineering process in general: I want work to move forward in a way that is inspectable, repeatable, and resilient once the task gets messy. A lot of AI-assisted coding still feels like improvisation. You ask for something, get a result, adjust the prompt, try again, and hope the useful reasoning is still somewhere in the scrollback. That can work for tiny edits. It gets much less convincing when the task starts touching architecture, tests, review, or pull requests. What I want instead is a workflow where the model helps me think and execute, but inside a structure I can inspect afterwards. I want artifacts, gates, and something I can resume tomorrow without reconstructing the entire mental state from memory. That is why I use po8rewq/agentic-skills . It gives me a practical way to do agentic coding as an engineering workflow rather than as a long sequence of chat turns. A task moves through requirements, architecture, implementation, checks, review, and pull request creation. Each stage leaves something I can read, verify, and challenge. What makes this interesting to me The interesting part is not just that there is a CLI. Plenty of tools have a CLI. What matters to me is that it turns AI-assisted coding into a staged system: requirements force the task to become explicit architecture makes risks visible before code is written implementation happens against a plan instead of against a vague prompt checks and review happen as part of the flow, not as an afterthought runs are resumable, so interruptions do not destroy context That changes the feel of the work quite a bit. Instead of asking "what should I prompt next?", I am usually asking "what stage is this task in, and what should exist before I move on?" Where this really clicked for me was when I noticed I was spending less energy trying to preserve context in my head and more energy evaluating actual outputs. What the repository actually

Enterprises are now talking about AI coding agents in a very predictable way. The first question is usually: "How much more code can it help us generate?" It is not a wrong question. But if that is the only question, the ROI calculation will probably be wrong. Because enterprises are not really buying "more code." They are buying: faster delivery less rework lower maintenance cost better developer experience more stable software quality more controllable security and compliance risk faster translation from product capability to business value Code generation is an input. It is not the outcome. That distinction matters. An AI coding agent can help developers write functions, fix bugs, add tests, generate documentation, understand codebases, and refactor legacy systems. That sounds powerful. But the enterprise question is not: "How many lines of code did it generate today?" The better question is: Did that code reach production faster? Did incidents go down? Did the team spend less time on repetitive work? Did customers get value sooner? If the answer is unclear, generating 100,000 lines of code a day may simply mean producing technical debt faster. The short version: AI coding agent ROI does not end inside the IDE Many teams start measuring AI coding tools with the most obvious numbers: code suggestion acceptance rate lines of code generated number of active users number of prompts time saved on individual tasks These metrics are useful. But they mostly show that the tool is being used. They do not prove that the enterprise is getting value. Enterprise ROI has to be measured across software delivery, quality, risk, and business outcomes. In other words, an AI coding agent is not just a point solution for individual efficiency. It affects the entire software value stream: Request -> Design -> Coding -> Review -> Testing -> Deployment -> Monitoring -> Feedback -> Business outcome If you calculate value only inside the "coding" box, you miss the bigger picture. Why "amo

Software architecture debates have a problem that most other engineering disciplines don't: the alternative was never built. When a bridge fails, the failure is physical, attributable, and measurable against every other bridge that didn't. The engineering decisions that caused it can be isolated, traced, and corrected — not just in theory, but in the next bridge, because the material itself produces feedback that no amount of professional opinion can override. Steel deflects. Concrete cracks. Physics doesn't care what the architect believed. Software produces no equivalent feedback. A system built around the wrong abstractions compiles, runs, ships, and passes its tests just as readily as one built around the right ones. A bug introduced by a misaligned domain model looks identical, from the outside, to a bug introduced by a typo. A feature that took three times longer than it should have, because the structure made it harder than the business logic warranted, produces no artifact that distinguishes it from a feature that was simply difficult. The cost is real. The cause is invisible. This is the unfalsifiability problem, and it runs deeper than "we can't measure everything." It means that when a system becomes expensive to change, the diagnosis almost always lands on the wrong variable. The domain is complex. The requirements changed. The previous team was careless. Almost never: the structure was wrong, and the structure was wrong because nobody ever built the other version of it to compare against. That version doesn't exist, it never will, and every architectural argument in the industry is conducted in its absence. This would be a purely philosophical problem if there were nothing to do about it. There is something to do about it — but it requires accepting that the standard metric for software quality, whether it works, is measuring the wrong thing entirely. The Metric That Hides the Problem The natural substitute for "is this good engineering" is "does it wor

Reading Lilian Weng's harness engineering survey as a reliability engineer — what self-improving harness papers actually show, and the three invariants every working loop converges on.

A lot of AI memory systems start with the same temptation: "Just save the useful thing." That sounds harmless until the knowledge base becomes a junk drawer. Half the notes are too specific, a few are duplicates, some are obsolete, and nobody knows which ones the agent should trust. In ai-assistant-dot-files , the memory system is deliberately slower. It uses a promotion lifecycle: Capture -> Candidate -> Audit -> Approve -> Index -> Retrieve -> Expire That lifecycle is documented in docs/runbooks/memory-engineering.md , and the important word is not "capture." It is "candidate." Nothing writes directly to memory The framework has a durable memory layer: Knowledge Items in shared/knowledge/ , ADRs in docs/adrs/ , the domain dictionary, team topology, a feature archive, and a registry at shared/memory-registry.json . But a lesson from a delivery does not jump straight into shared/knowledge/ . It first becomes a Candidate Record. That record has required fields: Source Type Evidence Tags Expiration condition Then memory-engineer audits it: Is it reusable? Is it already covered? Is it too speculative? Does it belong as a Knowledge Item, or should it become a rule change, prompt edit, or ADR instead? Only after that does a human approve the destination. The design is intentionally similar to code review. Durable memory changes future behavior, so they deserve a paper trail. Rejection is a feature One of my favorite parts of the memory runbook is that it has explicit rejection rules. Do not promote a memory when it is: a one-off already covered too speculative That makes "zero candidates promoted this cycle" a healthy result, not a failure. This is where memory engineering starts to look less like note-taking and more like gardening. The point is not to preserve every leaf. The point is to keep the soil useful. Expiration matters The lifecycle also includes expiration. A Knowledge Item can become stale when the underlying code, agent, or pattern changes. It can be supers

I hesitated to write this. Not because I don’t have an opinion about AI in software engineering, but...

When I wrote about shifting the engineering process left — spec sessions, autonomous agents, humans reviewing output rather than writing code — a question kept coming up. Where does the Product Owner fit in all of this? It’s the right question. And I think the answer is more interesting than “the PO disappears.” Let’s start with acceptance criteria. We invented them to bridge a gap. The team needed to know when something was done. The PO needed confidence that what got built matched the intent. Acceptance criteria were the contract between the two. But if the Spec Session is where intent gets defined — by the whole team, together, before the agent runs — that gap closes. What the team agreed on in the room is the definition of done. The spec is the acceptance criteria. You don’t need a separate validation step because the planning and the agreement happened at the same time. The tighter the loop, the less ceremony you need around it. There’s a caveat though. The spec is a necessary contract. It’s not a sufficient one. Simon Martinelli’s work on the AI Unified Process validates the spec-driven approach technically. But his model is about the artifact — requirements at the center, AI generating everything else from them. How the team actually builds shared understanding before the spec exists isn’t something it addresses. That’s not a criticism. It’s just a different question. A spec written after a real Spec Session — where the team worked through edge cases together, disagreed, got to resolution — is different from a spec written by one person and signed off asynchronously. Same artifact. Different quality of shared understanding. That distinction matters when the agent hits an edge case the spec didn’t anticipate. So what’s actually left for a dedicated PO? Two things. And they’re very different. The first is product thinking — challenging intent, representing user needs, asking why before the agent runs with something. That’s valuable. But it doesn’t require a ded

One of the biggest takeaways from Chapter 3 of AI Engineering was realizing that building an AI model is only part of the challenge. Figuring out 𝗵𝗼𝘄 𝘁𝗼 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝗶𝘁 𝗳𝗮𝗶𝗿𝗹𝘆 𝗮𝗻𝗱 𝗮𝗰𝗰𝘂𝗿𝗮𝘁𝗲𝗹𝘆 can be just as difficult. With traditional software, it's usually easy to tell whether something works. If a calculation is wrong or a test fails, you know there's a bug. But AI doesn't always work that way. A model can generate multiple reasonable answers to the same question, making it much harder to determine which one is actually better. That made me think: 𝗛𝗼𝘄 𝗱𝗼 𝘄𝗲 𝗸𝗻𝗼𝘄 𝗶𝗳 𝗮𝗻 𝗔𝗜 𝗺𝗼𝗱𝗲𝗹 𝗶𝘀 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗶𝗺𝗽𝗿𝗼𝘃𝗶𝗻𝗴? 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀 𝗡𝗲𝗲𝗱 𝘁𝗼 𝗞𝗲𝗲𝗽 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 Reading this section made me realize how difficult it is for evaluation benchmarks to keep up with the pace of AI development. The chapter explains that GLUE (General Language Understanding Evaluation) was introduced in 2018 to measure how well language models performed on common natural language tasks. But within about a year, models had already become so good at it that researchers introduced SuperGLUE in 2019 as a more difficult benchmark. GLUE evaluates tasks such as: Question answering Sentiment analysis Sentence similarity Text classification The chapter also mentions newer benchmarks like: SuperGLUE MMLU (Massive Multitask Language Understanding) MMLU-Pro Each one was introduced because the previous benchmark was no longer challenging enough. What I found interesting is that a model getting a higher benchmark score doesn't always mean it understands language better. Sometimes it simply means the model has become very good at solving that particular benchmark. 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗘𝗻𝘁𝗿𝗼𝗽𝘆 𝗮𝗻𝗱 𝗣𝗲𝗿𝗽𝗹𝗲𝘅𝗶𝘁𝘆 Another section I really enjoyed was the explanation of entropy and perplexity. The chapter explains entropy as a measure of how much information a token carries and how difficult it is to predict the next token in a sequence. Perplexity measures uncertainty. If a model is very uncertain about what comes next, its perplexity will be higher. If

I recently worked through the classic Word Break problem in an interview. My approach was solid from the start — recursion with memoization, a breakable helper that tests every prefix and recurses on the rest. The logic was right. What slowed me down was everything around the logic. Here's the solution I landed on: class Solution { public: bool wordBreak ( string s , vector < string >& wordDict ) { unordered_set < string > dict ( wordDict . begin (), wordDict . end ()); unordered_map < size_t , bool > memo ; return breakable ( s , dict , memo , 0 ); // missed: breakable was a free function defined below -> "not declared in this scope" } private : // missed: had int here, compared against s.length() (size_t) -> sign-compare warnings bool breakable ( const string & s , const unordered_set < string >& dict , unordered_map < size_t , bool >& memo , size_t starting ) { if ( starting == s . length ()) return true ; if ( memo . count ( starting )) return memo [ starting ]; for ( size_t e = starting + 1 ; e <= s . length (); e ++ ) { string word = s . substr ( starting , e - starting ); // missed: shadowed an outer `word`, and had substr(starting, e) instead of e - starting if ( dict . count ( word ) && breakable ( s , dict , memo , e )) { memo [ starting ] = true ; // missed: wrote == instead of =, so success was never cached return true ; } } memo [ starting ] = false ; // missed: this line, so failures were never cached and memoization broke down return false ; } }; The real lesson Most of what tripped me up was syntax and scope — a function declared in the wrong place, signed/unsigned mismatches, a shadowed variable, == where I meant = . None of these were about the algorithm. But because I spent my time chasing them, I had less room to focus on the one thing that actually matters in this problem: the logical correctness of the memoization. The takeaway I'm keeping: get the syntax and scoping clean early, so the debugging budget goes toward logic, not typos. That's the

In the previous article, we argued that a Locator is neither a UI control nor a search mechanism. It is an architectural concept that resolves business references within a specific business context and transforms them into canonical business identities. If that is true, however, an obvious question immediately follows. How can a Locator be described? Not how it is implemented. Not how it is executed. But how it can be described independently of any particular implementation. This question is more important than it may seem at first glance. If two different development teams decide to implement a Locator, how can they be sure they are implementing the same concept? How can we discuss a Locator without referring to a particular library, programming language, or framework? The answer is that, like any mature architectural concept, a Locator must first be separated from its implementation. This distinction appears repeatedly throughout the history of software engineering. Relational theory existed before relational database systems. SQL does not describe how a query will be executed. It describes only the result we want to obtain. The choice of indexes, execution plans, and optimization algorithms is the responsibility of the query optimizer. Exactly the same principle applies here. A Locator should not be defined by the mechanism that implements it. It should be possible to describe it independently of that mechanism. This naturally leads to three distinct levels of abstraction. Locator Pattern │ ▼ Locator Definition │ ▼ Execution Engine The Locator Pattern is the architectural concept itself. The Locator Definition is the declarative description of a particular resolution process. The Execution Engine is the component responsible for interpreting that description and performing the actual resolution. Each level has a different responsibility, and they should never be confused with one another. This means that a Locator Definition is not an implementation artifact. It

Hello, DEV Community! 👋 Hi everyone! My name is Sulemana Abdallah , and I'm excited to be part of the DEV Community. I'm passionate about software development and enjoy building modern, responsive web applications using: HTML CSS JavaScript TypeScript React Python My goal is to become a skilled Full-Stack Developer and Software Engineer while continuously learning and building real-world projects. I joined DEV to: Learn from experienced developers. Share my projects and progress. Write about what I learn. Connect with developers from around the world. I'm looking forward to growing with this amazing community. Thanks for reading! 🚀