标签：#platformengineering

When I wrote about shifting the engineering process left — spec sessions, autonomous agents, humans reviewing output rather than writing code — a question kept coming up. Where does the Product Owner fit in all of this? It’s the right question. And I think the answer is more interesting than “the PO disappears.” Let’s start with acceptance criteria. We invented them to bridge a gap. The team needed to know when something was done. The PO needed confidence that what got built matched the intent. Acceptance criteria were the contract between the two. But if the Spec Session is where intent gets defined — by the whole team, together, before the agent runs — that gap closes. What the team agreed on in the room is the definition of done. The spec is the acceptance criteria. You don’t need a separate validation step because the planning and the agreement happened at the same time. The tighter the loop, the less ceremony you need around it. There’s a caveat though. The spec is a necessary contract. It’s not a sufficient one. Simon Martinelli’s work on the AI Unified Process validates the spec-driven approach technically. But his model is about the artifact — requirements at the center, AI generating everything else from them. How the team actually builds shared understanding before the spec exists isn’t something it addresses. That’s not a criticism. It’s just a different question. A spec written after a real Spec Session — where the team worked through edge cases together, disagreed, got to resolution — is different from a spec written by one person and signed off asynchronously. Same artifact. Different quality of shared understanding. That distinction matters when the agent hits an edge case the spec didn’t anticipate. So what’s actually left for a dedicated PO? Two things. And they’re very different. The first is product thinking — challenging intent, representing user needs, asking why before the agent runs with something. That’s valuable. But it doesn’t require a ded

An astrolabe doesn’t map every star. It gives you a way to find your position relative to the ones that hold still. That’s the instrument I reach for when someone asks which AI tool they should be using. The honest answer is that the tools will be different in six months. The layers won’t. I spent a week trying to make sense of a handful of names that kept showing up in the same conversations. Tessl . Goose . Archestra . Kestra . Modelplane . RAG , MCP , half a dozen others orbiting nearby. Each one has its own pitch, its own funding round, its own reason it’s the thing you should adopt next. Taken together they read like noise. Taken apart, they sit on different floors of the same building. The agent loop again, the one I keep coming back to. Once you place each tool on a floor, the noise turns into a map. Tessl sits left of the loop , at the intent layer. Turn a spec into something an agent runs against directly. This is the one tool on the list that pushes back instead of going along with it. A well-formed spec is not the same thing as a team that agrees on what the spec means. The Agora produces the second thing as a byproduct of producing the first. Tessl produces the first and assumes the second follows. It doesn’t, automatically. That’s the whole argument. RAG and MCP are plumbing. Protocol, not position. They carry context into the loop and don’t take a side in any argument about who should be in the room when the spec gets written. They’re also the one floor with an actual standard. MCP, A2A , ACP , all under Linux Foundation governance now, joint working groups, cross-protocol commitments. Passing data between systems is a solved problem with decades of precedent behind it, so it standardized almost on contact. Nothing else on this floor plan has that. Governance, orchestration, the harness, the spec layer: every vendor is still building its own version and calling it the obvious one. The standard showed up first at the floor that mattered least to this ar

Anthropic's engineers ship eight times more code than they did a few years ago. And they had to start scheduling lunches so people would talk to each other. Fiona Fung, who leads the Claude Code team, said it on Lenny's Podcast last week. Working with agents all day had started to feel isolating. The team was fast, but they'd stopped running into each other. So they added pairwise programming lunches and hackathons — rituals to put back the thing that used to happen on its own. Eight times the output. Scheduled conversation. That ratio is worth sitting with. Whatever goes missing here doesn't show up in the metrics. It doesn't throw an error. It just quietly stops being available. Here's the part that bugs me most. Ask an AI whether your approach is sound and it mostly tells you it is. Not because it's lying — because it's answering the prompt. No stake in the outcome, no history with the system, no memory of the last three times this exact idea was tried and quietly failed. A colleague pushing back is a different thing. They've got context you never typed into the window, because they were there when it was earned. They're going to maintain this too. They might be wrong — but wrong in a direction you hadn't thought of. An agent can't disagree with you like that. It agrees faster. Same with scope. The agent builds what you ask for, all of it, thoroughly. It won't mention that the third feature is the one nobody will use, or that "good enough" happened two iterations ago, or that something next door already solves most of this. Knowing when to stop comes from someone who's watched a codebase rot under a hundred individually-reasonable decisions. And it only knows what you put in front of it. The person who worked on payments remembers the edge case you're about to recreate. The junior who joined three months ago still sees the thing everyone stopped noticing. That gap — between what's in the window and what isn't — is where the expensive mistakes live. Then the part

I noticed something a few months ago. I was talking less to my colleagues. Not because anything was wrong. I had a question, I described it to an AI, I got something useful back. Why loop in a human if the loop is already closed? It took a while to name what was actually happening. There's a version of the AI story where the interesting work disappears. The agent implements. The spec session produces the plan. Humans review the output. What's left? Ticket hygiene and rubber stamping. Engineering as a series of approvals. I think that's wrong. But I understand why it feels true. Here's what I think is actually happening instead. The agent produces the increment. But the agent doesn't decide what the increment should move toward. It doesn't know whether this library is the right bet for the next three years. It doesn't know which of two implementation approaches leaves options open and which quietly closes them. It doesn't know whether the architectural call made today creates a problem nobody will notice until the system is under load eighteen months from now. That work — giving the project direction, validating trade-offs, deciding what the system becomes — isn't specable. You can't write a ticket for it. And it's not going away. The craft didn't disappear. It moved. Direction is the word I keep coming back to. The agent executes well. It implements against a spec. It generates options when you ask for them. But it doesn't carry a point of view about where the system should go. It doesn't have a stake in the decision. It will implement the wrong architectural direction just as confidently as the right one, if that's what the spec says. Someone has to hold the direction. Someone has to know enough about the codebase's history, the team's constraints, and the product's trajectory to say: not that library, we've been down that road. Not that pattern, it doesn't survive the load we're heading toward. This approach now, that refactor later, in this order, for these reaso

"A fool with a tool is still a fool." — often attributed to Grady Booch I keep coming back to this quote when I watch teams adopt AI. In my last post ( https://schrottner.at/2026/06/18/The-Wrong-End-of-the-Problem.html ) I wrote about shifting the engineering process left — spec sessions, autonomous agents, humans reviewing output rather than writing it. A few people asked the obvious follow-up: if an agent implements and an AI reviews, why do I need a team at all? It's a fair question. And I think the answer is in that quote. The agent validates against your prompt. That's it. If your thinking is muddled, the output will be muddled — just faster and at greater cost. An agent doesn't tell you that you're solving the wrong problem. It solves whatever problem you gave it, thoroughly and without complaint. Most AI usage right now treats AI as a tool. Which means the quality of the output is bounded by the quality of the thinking that went into the prompt. A fool with a tool is still a fool. The tool just makes the foolishness more expensive. The team is the check on intent. Not after the agent has burned three sprints on the wrong thing — before it starts. That's what mob planning actually is, when you think about it. Not a meeting. Not process overhead. It's the place where bad ideas get caught before they get expensive. Where someone asks "wait, why are we building this" before an agent runs with it for a week. But there's something else happening in that room that I think gets underestimated. It's where the learning happens. Not just prompting. System thinking. Architectural patterns. How to decompose a problem. Why a certain approach fits this codebase and another doesn't. How a senior frames a problem before an agent ever touches it — the mental model that makes the output actually good. Right now that knowledge isn't transferring. Everyone is heads-down with their own tools, developing their own habits in isolation. Engineer A gets dramatically better output than

Everyone is asking whether AI can write code. That question is already answered. The more important question is: Can AI understand the system it is changing? The biggest limitation of AI coding tools isn't code generation. It's system understanding. That is no longer the interesting question. AI can already generate APIs, tests, database migrations, infrastructure files, and entire services. The better question is: Does your AI understand the system it is changing? For most engineering teams, the answer is no. And that is where many AI-assisted workflows quietly fail. The illusion of understanding Ask an AI assistant to: create a new endpoint add a background worker generate a service layer write a migration Most models will produce something that looks correct. The code compiles. The tests may even pass. But production systems are not collections of files. They are collections of relationships. The real questions are: Which service owns this capability? Which projects depend on it? Which runtime executes it? Which release gates are affected? Which verification steps must pass? What breaks if this change is wrong? These questions are rarely visible in source code. They exist in architecture, operational knowledge, deployment rules, contracts, and team conventions. That is why an AI agent can generate valid code and still make the wrong change. Bigger context windows won't solve this The common response is: Give the model more context. But more context is not the same as better context. A million tokens of source code still do not explicitly answer: What projects exist? Which commands are safe? What evidence is trusted? What is currently blocked? What is ready for release? The issue is not missing tokens. The issue is missing structure. The missing layer Most AI tools understand: files functions repositories Production systems require understanding: ownership architecture dependencies operational boundaries verification requirements change impact This is the gap betw

GitOps policy drift is what happens when a control plane keeps a policy perfectly reconciled long after the reason for that policy has stopped being true. Every commit is applied. Every pull request is merged cleanly. Every dashboard reads green. And the rule being enforced no longer reflects anything anyone would choose to enforce today — it just hasn't been told to stop. That gap is the subject of this post. Not configuration drift — the thing GitOps was built to kill — but a second, quieter failure mode that lives one layer above it: the policy is right by every technical measure and wrong by every practical one, and nothing in the reconciliation loop is capable of telling the difference. The Promise GitOps Actually Kept GitOps earned its place in the infrastructure as code architecture stack by solving a real and expensive problem: state drift. Before declarative reconciliation, infrastructure diverged from its source of truth constantly — a console change here, an emergency hotfix there, a manual override nobody logged. The git repository said one thing. Production said another. Reconciling the two was a forensic exercise. GitOps closed that gap with a simple, durable mechanism: a controller that continuously compares declared state to actual state and corrects the difference without waiting for a human to notice. That's not a small win. It's the reason platform teams can run infrastructure at a scale that would have been operationally unmanageable a decade ago, and it's why GitOps controllers sit at the center of nearly every modern infrastructure as code architecture built since. This post isn't an argument against that mechanism. It's an argument that the mechanism's success created a blind spot nobody designed for. What GitOps Never Promised to Solve Here's the boundary GitOps was never built to cross: reconciliation proves that declared state and actual state match. It says nothing about whether the declared state should still exist in its current form. A

How GPU scheduling complexity and MLOps integration are forcing platform teams to rearchitect Kubernetes clusters before operational debt becomes insurmountable. As AI workloads consume roughly 40% of enterprise Kubernetes clusters by 2026, the platform's default scheduler is proving fundamentally mismatched with the topology-aware, gang-scheduled demands of GPU-intensive training and inference. Platform engineering teams that invest now in purpose-built GPU scheduling layers, multi-tenant partitioning, and FinOps-driven autoscaling will separate themselves from organizations drowning in 30-45% GPU utilization rates and mounting infrastructure costs. Why the Default Kubernetes Scheduler Fails GPU Workloads Kubernetes was designed for stateless, CPU-bound services, and its pod-by-pod bin-packing scheduler has no native awareness of GPU topology, NUMA boundaries, or NVLink interconnect bandwidth. This becomes a critical failure point with NVIDIA H100 SXM5 nodes, where achieving full-bandwidth tensor parallelism requires all 8 GPUs on a node to be scheduled as a single atomic unit. The default scheduler cannot guarantee this co-placement, meaning distributed PyTorch FSDP or MPI training jobs frequently land on suboptimal node configurations, wasting expensive NVLink bandwidth and forcing teams to over-provision GPU capacity. Idle GPU memory stranded across partially-utilized nodes is the primary driver behind the 30-45% utilization rates reported in 2025 surveys by Gradient Dissent and Weights and Biases, representing millions of dollars in annual wasted spend for mid-to-large enterprises running mixed AI workloads. Building the GPU Scheduling Stack: Volcano, KAI Scheduler, and MIG Platform teams are converging on a layered scheduling architecture that replaces or augments the default Kubernetes scheduler with GPU-aware primitives. Volcano has become the dominant choice for distributed training workloads, using its PodGroup abstraction to enforce gang scheduling across

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI ** AI in the Stack #1** Byte size summary After reading this article, you'll have a framework for evaluating AI tools in platform engineering contexts — not by capability type, but by where in your workflow the tool actually changes the outcome. You'll understand why the tools that sound most compelling are still hype, where genuine productivity gains exist today, and what governance infrastructure you need in place before any AI component gets near production. This article is the foundation for the series; subsequent articles implement each touch point against real OpenShift infrastructure. The story I spent months selling IBM's AI and data science portfolio before I truly understood what I was selling. I knew the pitch. Predictive analytics. Optimization. Decision intelligence. I could walk a room through the business value without breaking a sweat. CPLEX for scheduling, Watson for insights — I had the slides, the talking points, the customer stories. Then I sat in on a data scientist demo. Not a sales demo. An actual working session — models being trained, outputs being interrogated, assumptions being challenged in real time. And somewhere in that room, watching someone do the thing I'd been describing from the outside, something clicked — and not in a good way. The models were impressive. The theory was solid. But I kept asking myself the same quiet question: where does this go next? Because most of what I saw never made it anywhere near production. It lived in notebooks. In slide decks. In proof-of-concept environments that were never ready to cross the line into something real. I'd been selling outcomes — optimised schedules, smarter decisions, reduced costs — without a clear path to how you'd actually get there. And underneath all of it, something else bothered me that nobody was talking about loudly enough: the data going into these models was often messy, unvalidated, and ungoverned. Bias wasn't

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI AI in the Stack #2 ⚡ Byte Size Summary RAG inserts a retrieval layer between your existing runbooks and an LLM — answers come from your documentation, not generic training data, with source citations included. This article builds a complete FastAPI service with /ingest , /query , and /health endpoints, using OpenAI embeddings and Chroma as the vector store. Everything is cloneable from GitHub. The goal is not to replace your runbooks. It is to make them queryable at the moment an incident is happening. I have never met a platform team with bad runbooks. I have met plenty of platform teams where the runbooks exist, are reasonably well written, are stored somewhere sensible — and are still completely useless at 2am when something is on fire. Not because the content is wrong. Because nobody can find the right one fast enough. The search in Confluence returns fourteen results and none of them are titled the way the engineer is thinking about the problem. The person on call is junior and doesn't know the runbook exists. The runbook was written for a slightly different version of the service and nobody updated it. The runbook problem is not a writing problem. It is a retrieval problem. That is exactly the problem RAG was built to solve — and it is one of the highest-ROI first applications of AI in a platform engineering context. Not because it is technically impressive. Because it closes a gap that costs your team hours every month. This article builds a working pipeline. By the end you will have a FastAPI service that takes a natural language question — "why is my pod stuck in CrashLoopBackOff after a config change?" — and returns an answer grounded in your actual runbooks, with the source document cited. Everything is in the GitHub repo agentic-devops What RAG Is — Without the Hype RAG stands for Retrieval-Augmented Generation. Instead of asking an LLM a question and hoping its training data contains the answ

The monorepo vs polyrepo argument is old enough that Buildkite was comparing it to the Vim and Emacs wars back in 2024. It should have been settled, or at least gone quiet. Instead, in the space of six months, an AI coding vendor re-litigated it for the agent era, a benchmark firm published PR cycle-time data across hundreds of organisations, and half the platform engineering threads I read found their way back to it. Something pulled the question out of retirement. I think the something is worth naming, because it is not really about repositories at all. I maintain a product whose entire reason to exist is that most organisations run polyrepos, so I want to be upfront about where I sit before arguing anything. Riftmap parses cross-repo dependencies. If everyone migrated to a monorepo tomorrow, a good part of my roadmap would evaporate. Read what follows with that in mind, and check the sources, all of which are linked. With that declared: I think both camps in this debate are arguing about a proxy. The real variable underneath, the one that decides whether your team ships confidently or plays dependency archaeology at 2am, is something the standard pros-and-cons lists never name. This post walks the honest trade-offs first, because they are real and you deserve a straight answer to the question you searched for. Then it gets to the variable. What each side buys you A monorepo is one repository holding many projects. A polyrepo (or multi-repo) setup gives each project, service, or module its own repository. Both are proven at every scale that matters: Google and Meta run famous monorepos, Amazon and Netflix run famous polyrepos, and none of them are wrong. The monorepo's case The strongest monorepo argument has always been atomic cross-project change. Uber's iOS team moved to a monorepo largely for this: when an API contract and all of its clients live in one repo, a breaking change is one commit, one review, one revert path. No choreographed pull requests across si

The Uncomfortable Truth Here's a test: when your deployment fails in production, what happens to your main branch? If the answer is "the broken code is already merged" — congratulations, you're doing CI/CD with a Git trigger. That's not GitOps. It's a pipeline that happens to watch a branch. I've spent years building platform engineering systems at enterprise scale — identity management frameworks, infrastructure-as-code pipelines, AI agent platforms that manage operational code. And I keep seeing the same mistake: teams adopt "GitOps" by adding a deployment step after merge, then wonder why they get drift. True GitOps has one non-negotiable rule: main always equals production. If a deployment fails, main doesn't change. Period. This isn't just my opinion — it's the logical extension of OpenGitOps principles : declarative desired state, versioned in Git, automatically reconciled. The enforcement mechanism I'm describing is how you make those principles real rather than aspirational. The Anti-Pattern Everyone Runs The most common "GitOps" setup I see in enterprise teams looks like this: Developer opens PR CI runs tests Reviewer approves PR merges to main Deployment triggers from main ❌ Deployment fails main now contains code that isn't in production This is merge-then-deploy . It's standard CI/CD with extra steps. The moment you merge before confirming a successful deployment, you've broken the core GitOps contract: Git as the single source of truth for what's actually running. The result? Drift. Stale state in main . A branch that lies about what's deployed. Every subsequent PR is now based on a broken foundation. The Enforcement Pattern: Deploy Before Merge The fix isn't philosophical — it's mechanical. GitHub's Merge Queue gives you exactly the right primitive: Developer opens PR CI runs tests (standard checks) Reviewer approves → PR enters the merge queue Merge queue trigger runs a dry-run deployment against the target environment If dry-run passes → queue trigge

If you manage PostgreSQL databases across multiple environments, you've probably done this: SSH to the DB host (or connect via psql ) Run CREATE USER jsmith CONNECTION LIMIT 20 PASSWORD '...' Slack the password to the developer Forget to log it anywhere Repeat for every environment, every onboarding, every access request It's tedious, error-prone, and leaves zero audit trail. Here's a better way. What I Built pg-user-api is a lightweight Flask REST API that wraps PostgreSQL user provisioning in clean HTTP endpoints. You register your databases once in a SQLite inventory, then any tooling — CI pipelines, internal portals, Ansible playbooks, or a plain curl — can create and manage users across environments without ever touching psql . GitHub: pcraavi/PostgreSQL-user-creation-API The Problem It Solves In teams that span dev, QA, UAT, and prod, you end up with different patterns of users: App service accounts — named after the host/port combo ( web01_8080 ) Kubernetes workload accounts — named after env prefix + farm ( dv_gearservice ) Individual dev/QA accounts — low connection limits, scoped to non-prod Read-only analyst accounts — prod only, no DDL DBA accounts — CREATEDB CREATEROLE LOGIN , rarely provisioned Each type has different CONNECTION LIMIT values, privilege levels, and naming conventions. Encoding these patterns in an API means the rules are consistent, repeatable, and auditable. Architecture The project is intentionally small — five Python files and a requirements list: pg_user_api/ ├── app.py # Flask app — all endpoints ├── auth.py # HTTP Basic Auth (constant-time compare) ├── database.py # SQLite registry + audit log ├── notifications.py # Notification stubs (Webex / Slack / Email) ├── seed_db.py # One-time setup: creates DB + sample records └── requirements.txt Two credential pairs, clearly separated: PG_API_USER / PG_API_PASS — who can call this API (your team/tooling) PG_ADMIN_USER / PG_ADMIN_PASS — the PostgreSQL DBA role that executes DDL The DBA cr

Platform team sprint planning in 2026 begins with budget allocation, not architecture review. The first question is no longer "what do we need to build?" — it's "what can we afford to run?" This is not FinOps adoption. This is authority displacement. The platform team became a finance team because the control plane for infrastructure decisions migrated from architecture governance to budget governance. Cost constraints don't inform architectural decisions anymore — they dictate them. And when financial systems gain veto authority over technical systems, resilience becomes the variable that adjusts. Platform team cost governance is now the primary control surface. Architecture is secondary. How We Got Here The timeline is sharper than most organizations admit. 2018–2022 was the cloud adoption phase. Platform teams built for scale. Multi-region resilience was standard. Observability was deep. Auto-scaling was elastic. Architectural requirements shaped cost models. The budget followed the design. 2023–2024 brought FinOps as a cost visibility layer. Teams could finally see where money was going. Dashboards got built. Anomaly detection got configured. Attribution models got refined. But visibility was still separate from authority. The FinOps team reported. The platform team decided. 2025–2026 is when cost governance moved from reporting to gating. The turning point: platform teams stopped asking "can we build this?" and started asking "can we afford this?" Engineering roadmaps became cost roadmaps. Feature requests now come with budget allocation approvals. Architecture reviews now include CFO sign-off gates. This shift introduced Budget-Normalized Architecture — systems designed around predictable monthly spend targets instead of operational resilience targets. The architecture no longer optimizes for failure domains, latency requirements, or recovery objectives. It optimizes for staying under the cost ceiling. Cost governance expanded because engineering governance fa