标签：#DevOps

Um 02:47 Uhr stoppte mein System ein Deployment. Kein Mensch war wach. Es war ein Dienstagmorgen, als mein Guard-System anschlug. Nicht wegen eines fehlgeschlagenen Tests. Nicht wegen eines Syntaxfehlers. Ein Agent hatte versucht, einen Commit zu pushen, der einen AWS-API-Schlüssel enthielt. Der Schlüssel steckte in einer Konfigurationsdatei, die eigentlich nie ins Repository sollte. Der Deployment-Prozess wurde blockiert. Um 02:47 Uhr. Kein Mensch hätte das um diese Zeit gesehen. Der Schlüssel wäre live gegangen. Das war kein Einzelfall. Es war der 47. Vorfall in 14 Monaten, den mein System automatisch abgefangen hatte, bevor er Schaden anrichten konnte. Und er hat mir klarer als je zuvor gezeigt, warum das Regelwerk wichtiger ist als das Modell selbst. Was ein Guard-System wirklich ist Die meisten, die über KI-Sicherheit sprechen, meinen Alignment, Halluzinationen oder Trainingsdaten. Das sind echte Probleme, aber sie liegen auf einer anderen Ebene. Ich rede von etwas Handwerklichem: einem System, das verhindert, dass ein KI-Agent im laufenden Betrieb Fehler macht, die Menschen Geld oder Daten kosten. Mein System läuft auf einem Prinzip, das ich GRIP nenne: Guards, Rules, Isolation, Protocol. Jeder Agent, der in meinem Stack läuft, durchläuft vor jeder kritischen Aktion eine Prüfkette. Nicht als Empfehlung. Als harter Block. Das bedeutet konkret: Der Agent darf nicht weiter, bis das Problem behoben ist. Kein Fallback, kein "try anyway", kein Override ohne explizite Freigabe. # Beispiel: Pre-Commit Guard gegen Secrets #!/bin/bash STAGED_FILES = $( git diff --cached --name-only ) for FILE in $STAGED_FILES ; do if grep -rE "(AKIA[0-9A-Z]{16}|sk-[a-zA-Z0-9]{32,}|ghp_[a-zA-Z0-9]{36})" " $FILE " 2>/dev/null ; then echo "GUARD BLOCK: Potential secret detected in $FILE " echo "Deployment halted. Remove secret before proceeding." exit 1 fi done Das ist kein ausgeklügeltes KI-Modell. Das ist ein Shell-Skript, das seit Monaten zuverlässig seinen Job macht. 176 Regeln und wie

I had 87% coverage, and we still broke the billing flow on launch day. Not because of a gap in the percentage. Because 87% was covering the wrong things. The tests were written to pass a gate, not to catch a failure. That is a more common story than most teams admit. And the reason it keeps happening is not that engineers are careless. It is that the incentive structure you created made it the rational outcome. The series checkpoint The first three articles in this series built the investment case for testing and then dismantled the received wisdom about how to execute it. We've made the economic argument for automation. We've restructured when quality checks happen across the SDLC. We've replaced the pyramid model with something shaped by risk rather than by code hierarchy. Now, when someone asks: how do you know if it is working? The answer most teams give is their coverage percentage. This article is about why that answer is structurally broken, and why fixing it is a management decision before it is a tooling decision. What coverage percentage actually measures Coverage percentage tracks which lines of your code were executed during a test run. If a line ran, it counts as covered. That is the complete definition. It does not measure whether the test asserted anything meaningful about that line. It does not measure whether both branches of a conditional were exercised. It does not measure whether the specific inputs that cause failures were ever tried. A test that calls a payment function and checks assert response is not None covers the same lines as a test that validates the transaction ID, amount, currency, error code, and retry behaviour. The coverage tool treats them identically. The research on this is unambiguous. A 2017 study by Kochhar et al. examined the correlation between code coverage and actual bug rates across 100 large open-source Java projects. The finding: the coverage of existing test suites has an insignificant correlation with the number of b

We run an automated publishing pipeline entirely on GitHub Actions cron schedules — no server, no queue, just workflows that wake up, do one thing, and commit the result. It mostly works. But there is one behaviour of scheduled workflows that the docs mention in a single quiet sentence and that will silently halve your job frequency if you design around the cron expression instead of around reality: Scheduled workflows do not fire as often as you declare. What we measured We had a feedback-watcher workflow declared at four runs per hour: on : schedule : - cron : ' 7,22,37,52 * * * *' Measured over days, it actually fired one to two times per hour — not four, and not at the declared minutes. Roughly hourly on most days, at inconsistent offsets from the declared slots. We later redeclared it at two runs per hour ( 7,37 * * * * ) — measured result: still one to two runs per hour. The declared frequency changed by 2x; the delivered frequency barely moved. This is not an outage and not a misconfiguration. GitHub's own documentation says the schedule event can be delayed during periods of high load , and that high load times include the start of every hour — which is precisely where naive cron expressions cluster — and adds: "If the load is sufficiently high enough, some queued jobs may be dropped." What the docs understate is the magnitude: in our observation, on a private repo, "delayed" in practice meant "throttled to a fraction of the declared rate, indefinitely." What this breaks The failure mode is subtle because nothing goes red. Every run that happens succeeds. The runs that don't happen leave no trace — no log, no failure email, nothing. You only notice if something downstream depends on the frequency: We had promised a "reply within 15 minutes" SLA on incoming feedback, initially backed by the 4x/hour schedule. The schedule couldn't hold it, so for a while we ran a local 15-minute scheduler as the primary path and kept the workflow as fallback. When we later rel

Classic Machine Learning Through the Eyes of an SRE — Part 7 Picture a client health metric that has been flat at 2 out of 10 for six months. Ask PCA to compress your client-health data and that metric will contribute almost nothing to the directions PCA decides to keep. Not because PCA is broken. Because PCA treats variance as importance, and a signal that barely moves contributes almost no variance. Reduce the data far enough and the independent information it carried is simply not there anymore. But a CSAT frozen at 2/10 is not noise. It is a crisis nobody is escalating. And after compression, it may no longer be available to anything downstream. That is the bet, and in ops data it is frequently wrong. The critical signals are often the quiet ones. There is a cheaper version of the same failure that catches most people first. PCA measures variance in whatever units your features happen to be in, so a metric ranging from 0 to 10,000 can dominate one ranging from 1 to 5 purely because it is bigger. Standardize before you compress, or your first principal component may just be an elaborate way of saying "ticket count." Same class of bug as unscaled features in K-Means and SVM, and it fails just as quietly. What PCA actually is Third answer-finding strategy in the unsupervised set, using the same shorthand as the last two articles. K-Means SEARCHES: iterate and hope. DBSCAN DEFINES: declare a rule and traverse. PCA SOLVES: an eigendecomposition or SVD gives a direct solution rather than an iterative local search. No convergence to babysit, no restarts, no local optima to escape. Two caveats on the word "direct," both worth knowing. Many libraries will use randomized SVD on large matrices, which is approximate and stochastic. And even with an exact solver, eigenvectors are only defined up to sign, so a component can come back inverted between runs or across implementations. The variance explained is identical either way, which is precisely why nobody notices. Hold ont

LangSmith is excellent for debugging live AI systems. But keeping every trace in its extended-retention tier can turn observability into a surprisingly large line item. Today we merged a new archive workflow into langsmith-cli that changes that tradeoff: keep LangSmith for live debugging, continuously archive verified traces to organization-owned private S3, and query the retained Parquet directly with DuckDB. In other words, you can preserve your complete trace history without placing every trace on LangSmith's extended-retention tier. The cost-overrun risk LangSmith currently documents two trace-retention tiers: Tier Retention Published trace price Base 14 days 0.05¢ Extended 400 days 0.50¢ total The 0.45¢ extended-retention upgrade makes an extended trace cost 10× as much as a base trace. That difference becomes material at production volume: Monthly traces Base, 14 days Extended, 400 days Added retention cost 100,000 $50 $500 $450 1,000,000 $500 $5,000 $4,500 10,000,000 $5,000 $50,000 $45,000 These examples use the published per-trace rates before free allowances, plan terms, negotiated pricing, or taxes. Always check the official LangSmith usage and billing documentation before making budget decisions. There is another subtle risk: online evaluators and automation rules can upgrade matching traces when retention extension is enabled. A rule that matches one run upgrades the whole trace, and a thread-level rule can upgrade every trace in that thread. LangSmith currently enables retention extension by default for new online evaluators and automation rules, although you can opt out. At scale, an innocent-looking evaluator or rule can therefore create a much larger bill than expected. The new langsmith-cli archive workflow The new workflow separates live observability from long-term retention: LangSmith live traces (14 days) │ ├── D+2 primary export ───────┐ └── D+12 reconciliation ──────┤ deduplicate by run ID ▼ private S3 / Parquet │ ▼ runs ... --archive (DuckDB)

Every one of my publishing lanes went dark for four days, and every script involved exited with status 0. Nothing had crashed. The files themselves had quietly stopped existing on disk — macOS had uploaded them to iCloud and deleted the local copies to "optimize storage." Why This Matters What it means for automation to depend on its environment When you run 160+ launchd jobs around the clock, the execution environment itself becomes a failure source before your script logic does. Ports get exhausted, processes orphan and pile up, memory never frees — I wrote about that class of resource leak last time. This is a completely different kind of total failure that happened the very next day. The files had become fatal to read . Not a bug in my code. Not a filesystem bug. An unintended side effect of a mechanism macOS runs under the name "optimization." What optimize-storage actually does macOS's "Optimize Storage" (System Settings → General → Storage → Optimize Storage), on a machine with iCloud Drive enabled, uploads files under Desktop and Documents to iCloud and deletes the local copies when free disk space gets tight . In Finder they still look like normal icons, but there is no local data — they are in a "dataless" state. Click one and it downloads automatically. For a human user, that's an acceptable tradeoff. The problem is automation scripts. python3 's open() , pathlib.Path.read_text() , cat , jq , cp — all of them die instantly on a dataless file with Errno 11: EDEADLK: Resource deadlock avoided . The name "Resource deadlock" makes you suspect a deadlock, but this is a POSIX errno code that macOS repurposes to mean "waiting for a file download." No lock is contended. No thread is stuck. The mere fact that "the data isn't local" surfaces to the process as a fatal error code. You can also get EAGAIN (resource temporarily unavailable). That one shows up as a race right after a download starts. The actual damage: four days of zero posts On August 6, 2026, note's a

I Gave Five Graph Databases 256MB of RAM Each. Here's What Broke. CognoDB Cloud's free tier gives you a graph database instance with half a CPU core and 256MB of RAM. That's not a lot. It's also, honestly, a pretty realistic starting point a lot of real side projects and early-stage products live exactly there, on whatever the free tier happens to give them, and find out the hard way what their database does under pressure. So I decided to actually find out. I took CognoDB and lined it up against four other graph databases Neo4j AuraDB, FalkorDB, and ArangoDB gave every single one of them the same tiny resource budget, threw the same 198,050-edge dataset at all of them, and ran the same queries. No cherry-picking, no "best case" numbers. Just: here's a small VM's worth of resources, go. One of the databases I originally planned to include never even made it into the results. It crashed on startup. Not "slow to start" a full segfault, reproducibly, across two different versions, with nothing I threw at it fixing it. More on that below, because it's honestly one of the more interesting parts of this whole thing. The setup, quickly Five candidates going in: CognoDB (mandatory, since that's the actual point of this), Neo4j AuraDB Free, Memgraph, FalkorDB, and ArangoDB. Same dataset for all of them a real social-graph-shaped dataset from Stanford's SNAP collection, ~18.7k nodes and ~198k edges, sized specifically to fit inside every platform's free tier without anyone getting an unfair advantage. Same queries too: I wrote every single query 1-hop, 2-hop, 3-hop traversals, point lookups, filtered lookups, aggregations exactly once, then translated each one into whatever query language a given platform actually speaks. No platform ever got a "friendlier" version of a query than another. And everyone ran under the same 0.5 vCPU / 256MB RAM ceiling, whether that was their real cloud free tier or a Docker container I capped by hand to match. The one that didn't survive Memgra

You said you'd remove that flag after launch. You lied. It's been six months and the flag is still in appsettings.json , the if statement is still in your controller, and nobody remembers which state is "on." This is how codebases turn into haunted houses. Why Cleanup Matters Dead feature flags are technical debt with teeth . They add branches to your code that nobody tests. They confuse new developers who don't know the history. They inflate configuration files and make deployments harder to reason about. And they compound. Every flag you don't clean up makes the next cleanup harder because the cognitive load of understanding the system keeps increasing. The cost of removing a flag is lowest immediately after the feature ships, while everyone still remembers what the thing does. Six months later? Good luck. Track Every Flag You can't clean up what you can't find. Maintain a registry of every active feature flag with: Name Purpose Owner Date created Expected removal date This can be a spreadsheet, an issue tracker, internal documentation, or a dedicated feature flag management system. The format doesn't matter nearly as much as the habit. When you add a flag, add it to the registry. When you remove a flag, remove it from the registry. If your registry contains flags with no owner or no removal date, congratulations: you've found your next cleanup project. Set Expiry Dates Every flag should have a planned removal date when it's created. For example: Release toggles: Remove shortly after the feature ships. Two weeks is a reasonable default. Experiment toggles: Remove when the experiment concludes. Ops toggles: May be permanent by design. Permission toggles: May also be permanent, but document that explicitly. If a flag has been alive longer than its planned expiry and nobody deliberately extended it, it's already a zombie. Treat it accordingly. Make Cleanup Part of the Process Flag cleanup doesn't happen unless someone owns it. Add a cleanup step to your feature compl

I've been running Claude Code as a daily driver for about three months now. It writes Ansible I'd have taken a week to write. It reads a codebase faster than I do. It is, genuinely, very good. It also once wanted to force-push to main , and it wanted to for an extremely good reason. Sit with that for a second, because it's the whole post. The rebase was stuck. Force-pushing would have unstuck it. Every link in that chain of reasoning is sound. The agent wasn't being careless, wasn't hallucinating, wasn't "drifting" or whatever we're calling it this month. It made a locally correct decision with a non-local consequence, which is the exact category of mistake that human code review is worst at catching — because the diff looks fine . It could see the command. It could not see the crater. The thing I stopped doing For a while my answer was to read everything. Every diff, every command, eyes on the screen, hand hovering over Ctrl-C like a man watching a toddler near a staircase. This does not scale, and the reason it doesn't is embarrassing when you say it out loud: reviewing output scales with how much the agent writes. That number is going exactly one direction, and it isn't down. So I flipped it. Instead of reviewing what it produces, I started writing down what it must never do. And here's the good news that took me way too long to notice: that list is short . Not "short for a security policy" short. Short like you can fit it on a napkin. Here's mine: A credential it read an hour ago gets inlined into a source file. A rebase gets stuck, and the fastest route to a green terminal is git push --force origin main . rm -rf "$BUILD_DIR/" runs on the one machine where BUILD_DIR never got set. A version bump gets typed straight into package-lock.json , because that's the file the version number is visibly in. A failing test quietly grows a .skip and CI goes green. Someone runs cat .env "just to see which variables exist." That last one is my favourite, and I'll come back to

Today I shipped contenox 1.0.0. Not by pushing a tag on top of a thousand commits, but as a single commit into an empty repository: the whole tree, one signed tag, binaries built from that tag by CI. The 957 commits that got me there are still public, in the old repository, as history. They are no longer how the project is published. This post is about why, and about what went wrong before I had finished reading the result back. What a commit used to mean GitHub's workflow rests on four assumptions so old that nobody states them any more: A commit is a unit of human intent. Someone decided something and typed it. A pull request is a unit of review. A human reads the diff, because a human wrote it. History is provenance. Who changed what, when, and — through the message — why. Timestamps are labor. The contribution graph on your profile is a diary. All four were true in 2008. For a tree that agents write, none of them survive contact. What my repository actually looked like Some numbers from a tree you can inspect yourself: 957 commits in just over a year, most of them named Checkpoint , Fix tests , Snapshot WiP . Dozens on a busy day. The production Go grew from 17,267 hand-written lines to 134,040 agent-assisted ones. Measured, not estimated. The median file stayed the same size; the number of files and packages did not. At one point 530 uncommitted paths sat in a single working tree. Inside that blob, the file that carried the repository's own conventions had been deleted. Nobody noticed for days, because nobody reviews a 530-file diff. A commit stream like that is not history. It is a log. Reading it tells you nothing about what a human decided — the decisions happened in prompts, in agent declarations, in a policy file — and it tells you one thing with great precision: when the work happened. If you also do client work, a public commit stream is a timesheet you never agreed to publish. Review had quietly inverted, too. I was no longer reviewing commits. I was re

Did you know there's a new major version in town for Puppet Core? You might have heard about it through the grapevine or in the Are You Ready for Puppet 9? webinar that @gpatton and I recently hosted. The wait is over and Puppet Core 9.0.0 is now available alongside Puppet Core 8.21.0. Puppet Core 9 introduces significant runtime and platform changes, moving to Ruby 4.0, OpenSSL 3.5, and other changes, but the essential Puppet under the hood is largely unchanged from Puppet 8. The majority of upgrade effort will center on Ruby 4 compatibility and runtime dependency changes rather than Puppet language changes. If you are staying on the Puppet Core 8.x release track, the latest Puppet Core 8.21 delivers the basic support fixes and security improvements you might need without the major dependency changes found in Puppet Core 9. What matters most for the admins Before upgrading to Puppet Core 9: Test custom facts, functions, types, and providers against Ruby 4.0. Validate any Forge modules you use for Ruby 4 compatibility. Review integrations that depend on OpenSSL behavior. Verify any workflows that still rely on SHA-1. Confirm managed nodes are running supported operating systems. Review any custom code that depends on PSON or multi_json . Check deferred function behavior if you have custom types or providers. Perforce will be rolling out updates to Puppetlabs modules on the Forge based on their priority tier and dependencies. The first batch of these should be rolling out soon. Puppet Core 9.0 highlights These are a few highlights I pulled from the release notes. Make sure to reference the full 9.0 release notes to get all the details about what has changed! Ruby updated to 4.0.5: With a new Ruby baseline some deprecated syntax from older Ruby versions will no longer be compatible. This is the primary focus area for upgrades as you will want to validate your custom code and modules. The latest PDK 3.8.0 introduced some Ruby 4 validators to help you update your syntax

James Arthur shares why sync is the next frontier in frontend architecture. He explains how extending reactivity to the server with Electric and TanStack DB replaces imperative fetching with declarative data bindings. Learn how query-driven sync and local optimistic updates enable engineering leaders to build insanely fast, collaborative, and agentic applications using their existing stack. By James Arthur

Building in public You explain your deploy setup to your assistant. It helps. Tomorrow you explain the same setup again. And the day after. You are not training it. You are re-typing. The tool nobody asked for I did not set out to build a product. I set out to stop repeating myself. My setup is four servers with names that mean nothing to anyone else, a tunnel with a numbering scheme I keep getting wrong, and a dozen small traps that only exist because of decisions I made two years ago. Every new session started from zero. So I gave the assistant a place to write things down, and a way to read them back before it started working. Two calls: one to save what was learned, one to recall it. That was the whole idea. For six weeks it had exactly one user. Nobody else could have used it, because I had not written a single line of documentation. Six weeks of being my own only customer That stretch turned out to be the most valuable part, and not because of what got built. Because of what got measured. When you are the only user, every rough edge lands on you within a day. A recall that returns the wrong thing costs you the next hour. A save that silently drops a field costs you the next week, when you go looking for it. I kept a count of the times the memory actually prevented a mistake. Not a feeling, a count. After six weeks it was high enough that I stopped arguing with myself about whether the thing was worth the effort. The uncomfortable part: several of those saved lessons were about mistakes I had already made twice. The tool did not make me smarter. It made me stop paying for the same lesson. The moment it stopped being a personal tool The thought that changed it was not a market analysis. It was smaller and more honest: if I find this useful, and my setup is not special, then somebody else is retyping their own servers right now. That is a weak argument on its own. Plenty of internal tools are useful precisely because they fit one person. So I looked for the part

Flux has introduced Flux Mirror, a CLI plugin that mirrors container images, Helm charts and OCI artifacts between registries from a declarative configuration. The plugin is part of the Flux v2.9 CLI plugin system and is presented as a way to keep Kubernetes clusters reconciling only from registries that teams operate themselves. By Matt Saunders

When building an application, we usually focus on the frontend, backend, APIs, and database. But there is another important question: Where should the application actually run? Two common approaches are physical servers and cloud/virtual servers. Understanding the difference is important because infrastructure decisions affect scalability, availability, security, maintenance, and cost. What Is a Server? A server is a computer system that runs applications, processes requests, communicates with databases, and provides information to users. A typical request might look like: User → Internet → Application Server → Backend → Database → Response Depending on the application, the server may handle authentication, APIs, user data, file processing, notifications, and other backend operations. In simple terms, the server provides the execution environment behind the application. Physical Server: More Control, Less Flexibility A physical server is a dedicated machine used to run applications. For example: 16 CPU cores + 64 GB RAM + 2 TB SSD Advantages: • Dedicated hardware • Predictable performance • Greater hardware-level control • Suitable for stable workloads Limitations: • Higher initial investment • Hardware maintenance • Hardware failures can cause downtime • Scaling requires additional or upgraded hardware If an application suddenly grows beyond the capacity of the machine, increasing capacity may require purchasing and configuring new hardware. Cloud / Virtual Server: Infrastructure That Can Adapt A cloud server is a virtual server running on physical infrastructure inside a cloud data center. For example: 4 vCPU + 16 GB RAM + SSD Instead of purchasing the entire physical machine, resources can be provisioned according to the application's requirements. Cloud environments also provide different scaling approaches. Scale Up: Increase the resources of an existing server. 4 vCPU → 8 vCPU → 16 vCPU Scale Out: Add additional application instances. Application Server 1 + Ap

I run self-hosted Chatwoot as the WhatsApp inbox for a dozen or so small Israeli businesses. Two servers, a few thousand conversations a week, a drip-sequence engine bolted on the side. Chatwoot is good software. The self-hosting docs will get you to a running container. What they will not tell you is which failures actually happen at month six, when you have real customers and real volume. These five all bit me in production, and none of them looked like what they were. 1. Your disk fills from somewhere Postgres never sees I got a disk alert at 86 percent and immediately went looking at the database. That was the wrong place. DB (postgres): 680 MB chatwoot_storage_data: 17 GB Attachments live in ActiveStorage, on a Docker volume, not in Postgres. Every image, voice note, and PDF a customer sends is a file on disk, and none of it shows up when you check database size. If your monitoring watches the DB, it will report everything is fine right up until the container cannot write. The growth curve is a function of how many accounts you host, not how busy any one of them is. Mine sat at roughly 0.05 GB a month until I onboarded seven new businesses over two months, and then it hit 16 GB a month. Check the right volume: docker system df -v | grep chatwoot_storage_data 2. Forty-four percent of my outbound storage was duplicate files This is the part that surprised me. When I actually measured what was on that volume, almost half the outbound media was byte-identical copies of the same file. One 14.5 MB video was stored 48 separate times. One image was stored 325 times. Chatwoot creates a new blob and a new file on disk on every send, even when the bytes are identical. That is correct behavior for a chat app where every message owns its attachment. It becomes expensive the moment you have anything that fans one file out to many conversations. In my case it was not campaigns at all, it was the drip engine sending the same media to 48 separate conversations as ordinary outbo

A tutorial for running two or more python bots on Railway inside one container and one service, with independent crash recovery for each. Deploying Multiple Python Bots to a Single Railway Container If you're running more than one Python bot — say, a Telegram ingestion bot and a Discord notification bot that share a database — deploying each as its own Railway service means double the hosting cost and double the configuration for something that's logically one unit. This tutorial covers deploying both bots inside a single Railway container, with each one still getting fully independent crash recovery. Table of Contents Why Two Services Is Usually Overkill The Naive Fix and Why It Falls Short Step 1: Install StayPresent Step 2: Structure Your Project Step 3: Configure Multiple Bots in One Entry Point Step 4: Read Railway's Assigned Port Step 5: Deploy as a Single Railway Service Verifying Both Bots Are Running FAQs Conclusion Why Two Services Is Usually Overkill Railway (like most PaaS platforms) charges per service, and each service needs its own configuration, environment variables, and deployment pipeline. If two bots are closely related — sharing a database, a queue, or just conceptually belonging to the same project — running them as two separate Railway services duplicates all of that for no real benefit. The Naive Fix and Why It Falls Short A common first instinct is a shell script: python telegram_bot.py & python discord_bot.py & wait This runs both, but there's no real process supervision here — if telegram_bot.py crashes, nothing restarts it, and you still haven't solved Railway's HTTP port requirement, since neither script opens one. Step 1: Install StayPresent pip install staypresent[prod] # requirements.txt staypresent[prod] Step 2: Structure Your Project project/ ├── main.py ├── telegram_bot.py ├── discord_bot.py ├── requirements.txt Both bot scripts stay exactly as they are — nothing about their internal logic needs to change. Step 3: Configure Multipl

Most engineering organizations now pay several vendors for AI coding assistants, each one bills differently, and no single person in the company can answer the simplest question: what did our AI coding tools actually cost this month, and what did we get for it? This guide is the practical answer — the metrics that matter, the ways teams track spend, a step-by-step setup, and an honest maturity model for governing it. The short answer To track AI code assistant spend across every vendor, pull cost and usage from each tool's admin or billing API, normalize it into one model — because every vendor bills on a different unit and a different clock — and map it to your teams and cost centers. The four approaches teams use are manual spreadsheets, each vendor's native dashboard, an open-source usage CLI, and a dedicated AI spend management platform. Only the last gives finance, engineering, and IT one live number plus forecasting, anomaly detection, and per-developer and per-pull-request cost. If you only do three things: inventory every assistant in use, including shadow tools bought on personal cards; connect each vendor read-only and normalize to a common cost model; and instrument the leading indicators — premium-model mix, token or credit runway, and idle seats — because they move before the invoice does. What "AI code assistant spend" means AI code assistant spend is the total cost an organization pays across all of its AI coding tools — commonly GitHub Copilot, Cursor, Anthropic Claude, OpenAI, and others teams connect — including per-seat license fees, metered token or credit consumption, premium-model surcharges, and the hidden cost of idle or duplicate licenses. It sits at the application layer, which distinguishes it from general cloud cost (compute, storage, networking), and it concerns money and utilization, which distinguishes it from AI model governance and its focus on model risk and compliance. Why it's genuinely hard to track (and got harder in 2026) There

Docker VMM (virtual machine monitor) is Docker's new, first-party virtualization layer for Docker Desktop, replacing third-party virtualization components with an engine that Docker can directly control and optimize specifically for container workloads. The public beta launched with Docker Desktop 4.86 for Mac and Windows. By Sergio De Simone

Short version: An Application or Network Load Balancer costs ~$0.0225/hour, about $16/month, just to exist , plus capacity units. Classic Load Balancers run ~$18/month. Load balancers outlive the services behind them: the app gets torn down, the ALB keeps billing. Here's how to find load balancers with no real traffic or no healthy targets, and remove them safely. Why idle load balancers linger The hourly base charge is fixed - an ALB with zero requests bills the same ~$16/month as a busy one. Load balancers are usually created early (with an app or an IaC module) and deleted last, if ever. A handful of abandoned ALBs from old environments is real, recurring money. Step 1 - List load balancers and their traffic aws elbv2 describe-load-balancers \ --query 'LoadBalancers[].{Name:LoadBalancerName,Type:Type,ARN:LoadBalancerArn}' \ --output table For an ALB, check request volume over the last 7 days (the metric dimension is the tail of the ARN, e.g. app/my-alb/50dc6c495c0c9188 ): aws cloudwatch get-metric-statistics \ --namespace AWS/ApplicationELB \ --metric-name RequestCount \ --dimensions Name = LoadBalancer,Value = app/my-alb/50dc6c495c0c9188 \ --start-time " $( date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ ) " \ --end-time " $( date -u +%Y-%m-%dT%H:%M:%SZ ) " \ --period 86400 --statistics Sum \ --query 'Datapoints[].Sum' Near-zero request counts over a week is a strong idle signal. (For NLBs, use the AWS/NetworkELB namespace and ActiveFlowCount .) Step 2 - Check for empty or unhealthy target groups A load balancer with no healthy targets is doing nothing useful: for tg in $( aws elbv2 describe-target-groups \ --load-balancer-arn <lb-arn> \ --query 'TargetGroups[].TargetGroupArn' --output text ) ; do echo "== $tg ==" aws elbv2 describe-target-health --target-group-arn " $tg " \ --query 'TargetHealthDescriptions[].TargetHealth.State' --output text done Empty output (no targets) or all unhealthy alongside near-zero requests is a confident "delete me." Step 3 - Delete saf