标签：#Engineering

About a year ago, I was building a terminal-based workflow manager called Glyph.Flow. It was mostly a learning project. I wanted to understand Python better, experiment with Textual, think about commands, state, configuration, logging, and all the small architectural decisions that suddenly appear when a script stops being a script. Somewhere between then and now, the workflows became a little more real. For my Master's thesis, I built a data pipeline to construct and process a cross-national research database from multiple sources. It had a clear purpose: take heterogeneous input data, transform it consistently, validate important assumptions, and produce the dataset I needed for the analysis. And it worked. But this is no longer enough. I am not rebuilding it because the original system failed. I am rebuilding it because the question changed: My Master's thesis needed a pipeline. My PhD will need research infrastructure. And I am slowly discovering that these are not the same thing. A pipeline can be finished There is something comfortable about building software for a well-defined research project. You know the research question. You know most of the variables you need. You know which datasets are involved. You can define the transformations, produce the outputs, validate them, run the analysis, and eventually say: Done. Of course, research is never really that clean. Data sources change. Weird edge cases appear. A country disappears from one dataset. Another source changes a variable name. An indicator turns out to mean something slightly different than you thought. But there is still a boundary around the problem. A PhD changes that boundary. Now I have to think about a system that may need to survive several years of research, new questions I have not formulated yet, datasets I have not discovered yet, and methodological decisions I will probably reconsider more than once. Suddenly, "Does it work?" becomes a surprisingly weak design criterion. The more useful

Many engineers believe Common Table Expressions (CTEs) are always faster than subqueries. In modern SQL Server (and PostgreSQL), that is a myth . Here is what actually happens under the hood: 1. Inlining & The Query Optimizer By default, the SQL optimizer treats standard CTEs and derived tables (subqueries) almost identically: The engine expands both into the same relational tree. They generate the exact same execution plan and I/O cost . -- Pattern A: Derived Table (Subquery) SELECT DeptID , EmpName , Salary FROM ( SELECT DeptID , EmpName , Salary , DENSE_RANK () OVER ( PARTITION BY DeptID ORDER BY Salary DESC ) AS rnk FROM Employees ) RankedData WHERE rnk <= 2 ; -- Pattern B: Common Table Expression (CTE) WITH RankedData AS ( SELECT DeptID , EmpName , Salary , DENSE_RANK () OVER ( PARTITION BY DeptID ORDER BY Salary DESC ) AS rnk FROM Employees ) SELECT DeptID , EmpName , Salary FROM RankedData WHERE rnk <= 2 ; 2. When CTEs Truly Win: Readability & Pipeline Stacking: You can chain 5 CTEs sequentially without deeply nested pyramid brackets. In-Place Deduplication: In SQL Server, you can run DELETE directly on a CTE, and it deletes duplicate rows straight from the real underlying table! WITH DuplicateCleaner AS ( SELECT CustomerID , Email , ROW_NUMBER () OVER ( PARTITION BY Email ORDER BY RegistrationDate ASC ) AS rn FROM Customers WHERE Email IS NOT NULL ) DELETE FROM DuplicateCleaner WHERE rn > 1 ; -- ✅ Clean in-place deletion! 3. The Big Trap (Spooling Overhead): If you reference the same CTE multiple times in a query (e.g. CTE_A JOIN CTE_A ), SQL Server may execute the underlying CTE query multiple times or create a Lazy Spool in tempdb . -> Fix: For heavy multi-million row reuse, use a Temporary Table ( #TempTable ) with an explicit Clustered Index instead! 💡 How do you choose between CTEs, Temp Tables, and Subqueries in your pipelines? 💼 Connect on LinkedIn: linkedin.com/in/arpitmbangre

If you run ClickHouse in production, you're probably on 26.3 LTS. And now 26.8 LTS has been announced, which means the LTS-to-LTS upgrade conversation starts again. Here's the thing most release posts skip: this is not a one-release hop. Going from 26.3 LTS to 26.8 LTS means crossing 26.4, 26.5, 26.6 and 26.7 as well. Every breaking change in those four releases applies to you, and some of the ones most likely to ruin your day aren't in 26.8 at all. So instead of writing another "here are the 26.8 features" post, I wanted to write the thing I'd actually want before scheduling this upgrade: what breaks, what silently changes, what order to do things in, and what you get for the trouble. A note on release timing As of writing (27 August 2026), 26.8 has been announced but is not fully released yet. The release branch is cut and versioned (v26.8.1.1-lts), but the tag and Docker images have not been published yet, and the upstream changelog still marks the 26.8 section as in progress. By the time you read this, the tag has probably landed. Check for yourself: curl -s https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/utils/list-versions/version_date.tsv \ | awk -F '\t' '$1 ~ /^v26\.8\./ {print "26.8 is released - newest: " $1 " (" $2 ")"; f=1; exit} END {if (!f) print "26.8 not released yet"}' version_date.tsv is the list ClickHouse maintains of every released version and its date, so this is the most direct answer available - no auth, no rate limit, nothing to download. As of writing it prints 26.8 not released yet . Worth knowing: the Docker image will lag whatever that command tells you. The Docker Official Images repo trails the GitHub tags by a few patch versions - clickhouse:lts currently resolves to 26.3.20.7 even though 26.3.24.4 has already shipped. So don't treat a missing image as evidence the release hasn't happened. Either way, the timing works in your favour. Historically ClickHouse LTS releases pick up several patch releases quickly - 26.7 had

Meta has detailed MTIA 300, its first in-house accelerator optimized for training ranking and recommendation models. By Matt Foster

Insert Molding Design: How to Place Metal Inserts Without Disaster — 8 Years of Structure Design Notes Every structure designer has been burned by inserts at some point — a nut seated crooked, an insert causing sink marks or cracks, pull-out force too low, an insert washed away by melt during injection. I've tripped on all of these myself. Insert molding sounds simple: drop a metal part into the mold and inject plastic around it. But metal and plastic have thermal expansion coefficients an order of magnitude apart, and every detail — shrinkage, grip force, locating method — can turn into a disaster. This article walks through the key design principles of insert molding, from insert types and locating structures to wall thickness and defect prevention. All of it is experience paid for with real money on real projects. Three Common Types of Inserts Insert molding falls into three categories by purpose, each with completely different design priorities. 1. Thread Inserts (Nut Inserts) The most common type. Tapping threads directly into plastic fails fast — fine threads under M3 strip after a few cycles — so metal nuts are embedded in the plastic. Copper inserts dominate because copper conducts heat well (fast heat dissipation during molding), has moderate hardness, and gives clean threads after tapping. We made a portable Bluetooth speaker with an ABS housing whose four corner posts needed M2.5 screws. Tapping the plastic posts directly stripped after three cycles. We switched to embedded M2.5×4mm copper nuts and measured over 45N pull-out, still stable after 500 screw cycles. The key: leave at least 1.5mm of plastic wall around the nut's outer diameter, or the area sinks and bubbles after cooling. 2. Locating / Support Inserts These locate, support, or conduct magnetism — locating pins in motor brackets, magnetic cores in sensor housings. The biggest challenge is insert positioning accuracy and post-molding offset. In 2024 we made a smart lock panel embedding a stainle

You have done everything right. You made the economic case for automation and got the investment approved. You distributed quality checks across the SDLC instead of piling them at the end. You replaced pyramid thinking with risk-weighted coverage. You stopped reporting a coverage percentage that was lying to you. Six months later, your engineers have started ignoring test failures. Not because they are careless. Because ignoring test failures became the rational choice. This article is about how that happens, why it happens to teams that know better, and why it is the final form of Test Debt. What is flakiness? A flaky test is a test that fails intermittently without any change to the code it covers. It sometimes passes and sometimes fails, with no consistent pattern. The most common root causes are timing issues in async operations, test-order dependencies, shared mutable state, and coupling to external services. All of these are fixable. The fixable nature of the problem is not what makes it interesting. What makes it interesting is that teams fix very little of it, and teams with strong engineers who care about quality fix very little of it. The reason is not the technical difficulty. The scale The numbers are worth stating clearly, because they establish what is actually at stake here: At Google , approximately 16% of tests show some form of flakiness, and 84% of transitions from passing to failing involve a flaky test rather than a genuine regression. At Microsoft , roughly 25% of test failures in large-scale CI systems are caused by flakiness, not actual code defects. The average time a developer spends per flaky test investigation: 30 minutes, before determining it was not a real failure. Atlassian estimated 150,000 developer hours per year consumed by flaky test investigation before they built automated detection tooling. Slack's mobile test failure rate reached 56.76% before they intervened. More than half of all test failures were noise. These are not team

Tired of Claude Code generating bizarre, overly dramatic jargon like "load-bearing spine"? You can fix this by enforcing Simplified Technical English (STE) in your system instructions or .claudemd files. This 1970s aerospace standard restricts vocabulary, forcing your AI agent to communicate in clear, direct, and highly actionable prose. "The load-bearing spine has hit a ceiling, and that is a significant foot gun with a large blast radius." If you have spent any time recently working with AI coding agents, you have probably stared at your terminal reading absolute gibberish like this, wondering: What on earth are you trying to tell me? I asked a straightforward technical question, and instead of a direct answer, I got a theatrical performance. It is incredibly tiring to translate AI metaphors back into plain English just to figure out which line of code actually broke. Fortunately, there is a remarkably elegant fix for this. The solution does not involve complex prompt engineering; instead, it leverages a fifty-year-old aerospace standard: Simplified Technical English (STE) . Why does Claude Code output weird technical jargon? AI models generate overly dramatic jargon because they are trained on vast internet corpuses where technical writing is often cluttered, metaphorical, and performative. To sound authoritative, the model indexes on complex vocabulary and metaphorical hand-waving instead of simple, direct statements. Imagine a scenario where your team is debugging a database lock. A human engineer would say, "The transaction is blocked." An AI model, eager to please and sound sophisticated, might describe it as a "temporal execution bottleneck causing systemic architectural paralysis." This happens because reinforcement learning from human feedback (RLHF) often rewards models for sounding smart and comprehensive. Without strict stylistic constraints, the agent defaults to verbose, metaphorical explanations that add cognitive load rather than solving your proble

In virtually every software engineering team, the temptation of the 'quick and dirty' fix surfaces sooner or later. The sprint deadline is looming, stakeholders are eager for a release, and a code snippet exists that barely passes the happy path. The logic is undocumented, edge cases remain unaddressed, and the design is brittle, yet the ticket can technically be moved to 'Done'. In the short term, everyone appears satisfied: the feature ships and the milestone is recorded. But before long, the consequences arrive: subtle bugs surface in production, extending the codebase becomes perilous, and teammates spend frustrating hours attempting to decipher undocumented logic. What began as a brief shortcut solidifies into technical debt and team friction. At the core of Christian ethics lies the command to love your neighbour as yourself. While that principle is often discussed in abstract theological terms, in modern software engineering it takes on direct, tangible significance. Who is your neighbour in a development team? Your neighbour is the colleague who will maintain, debug, or extend your pull request six months from now. Your neighbour is the junior engineer looking to existing code for guidance. And your neighbour is the end user relying on the system to function reliably and securely. When you deliberately invest effort in clear naming conventions, modular architecture, comprehensive documentation, and thorough automated tests, you provide genuine service to your peers. You choose to carry the cognitive burden today so that someone else does not suffer tomorrow. That is Christian care translated into code. Craftsmanship extends beyond syntax; it shapes the cultural atmosphere of an engineering team: Honesty regarding technical debt: Having the courage to articulate when architectural shortcuts threaten system sustainability, rather than passively allowing brittle code into production. Constructive peer reviews: Conducting code reviews with the intention of mento

One thing I learned from working with experienced engineers is that solving a problem and approaching a problem are two different skills. During one of my projects, I had the opportunity to work closely with Microsoft engineers. Since I was working independently, whenever I faced an issue, I would first spend time exploring it myself. I would check the data, logs, code, test different possibilities, and eventually figure out a solution. But sometimes, when I discussed the same issue with them, I was surprised by how differently they approached it. Instead of immediately looking for a fix, they would pause and ask a few simple but thoughtful questions. Those questions often narrowed the scope of the problem quickly and helped uncover the root cause much faster than trial and error. Over time, I started adopting that mindset. I learned that spending more time understanding why something is happening often leads to a better outcome than rushing into how to fix it. I also picked up many small but valuable engineering habits from everyday discussions, habits that continue to help me in my work today. Courses and certifications definitely help us learn new technologies. But some of the best learning in my career has simply come from working with skilled people, observing how they think, and applying those learnings in my own way. Grateful for the experiences, mentorship, and the people who generously shared their knowledge along the way. Learning #ProblemSolving #CareerGrowth #DataEngineering #GrowthMindset #ProfessionalDevelopment

MyZubster Is Not Trying to Build Another App — We're Exploring a Verifiable Digital Ecosystem For years, software development has largely followed the same pattern: User → Application → Database → Service AI changed part of that equation. IoT changed another part. Blockchain introduced new models for provenance and ownership. But there is still a difficult problem connecting all of them: How can a digital system verify what actually happened in the real world? This is one of the questions driving the development of MyZubster. MyZubster is an Italian open-source digital ecosystem currently under development. It hasn't reached its final public form yet. And that's important. Because we're not presenting a finished platform. We're documenting how the architecture evolves. From application to ecosystem Calling MyZubster simply an "app" increasingly feels incomplete. The architecture we're exploring connects several layers: MYZUBSTER ┌─────────────────┐ │ REAL WORLD │ │ people / places │ │ devices / events│ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ DATA │ │ sensors / users │ │ external sources│ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ PROVENANCE │ │ source / time │ │ context / proof │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ AI │ │ interpretation │ │ automation │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ EVIDENCE │ │ verification │ │ reproducibility │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ DIGITAL SERVICES│ └─────────────────┘ The goal isn't to put every technology imaginable into one application. The interesting part is the connection between these layers. AI needs evidence Generative AI can produce extraordinary outputs. But generation and verification are fundamentally different operations. An AI system can say: "This intervention reduced water consumption by 30%." But where did that number come from? What sensor produced the original measurement? What period was compared? What methodology was used? Was the dataset modified? Can somebody repro

When learning Angular, Spring, and Node.js, I often came across terms like SOLID, Dependency Injection (DI), Inversion of Control (IoC), IoC Container, and Composition . At first, these concepts can feel like they are all the same thing. They are not. The key realization is: SOLID is about how we design software. Composition is about how we build larger systems from smaller pieces. Dependency Injection is a technique for providing those pieces. IoC containers automate that process. Understanding this relationship makes Angular, Spring, and Node.js architectures much easier to reason about. 1. SOLID Is a Design Principle, Not a Framework Feature SOLID is a collection of software design principles. For example, Single Responsibility Principle (SRP) says that a component should have a focused responsibility. Instead of having one class responsible for HTTP handling, database access, validation, email, and payment processing, we can separate those responsibilities: Controller ↓ Service ↓ Repository ↓ Database Each part has a focused job. Similarly, the Open/Closed Principle (OCP) encourages us to design components that can be extended without constantly modifying their existing implementation. These principles don't require Angular, Spring, or an IoC container. You can follow SOLID in plain JavaScript. 2. Composition Is the Bigger Idea Composition means: Build a larger behavior by combining smaller, focused pieces. This works in both functional and object-oriented programming. In functional programming: function A ↓ function B ↓ function C A larger function can be created by composing smaller functions. In object-oriented programming: OrderService │ ├── PaymentService └── EmailService OrderService is composed using other objects. The important relationship is often: HAS-A rather than IS-A For example: OrderService HAS-A PaymentService rather than: OrderService IS-A PaymentService This is one reason composition is often preferred over deep inheritance hierarchies. 3. Dep

Diagrid Catalyst 2.0 applies Dapr-based recovery, signed workflow history and execution attestation across several agent frameworks. Architects should compare it with framework-native durability and established workflow engines, while evaluating benchmark evidence and operational trade-offs. By Mark Silvester

Variant Multiplier already let an editor swap one section of a winning ad and keep the rest. The next request from a real production job — replacing product SL-603 with SL-808, a different hearing-aid SKU, across an entire finished ad — was a different shape of problem. It's not "change one section," it's "change every mention of the product, everywhere it appears, while keeping literally everything else the same." Two direct quotes from the editor drove the whole five-PR arc: the transcript editing was too rigid for word-by-word changes, and separately, "the music, voice, etc. should retain the same, we should keep the quality the same, and not make it do a lot of changes." If a re-render can degrade something the editor explicitly asked to keep untouched, the render path is wrong for the job — no matter how good the model is. The cheap fix first: let editors actually edit PR #67 shipped before any product-swap work started, because it was the cheap, high-value half of the same feedback: "I am just able to select word by word here but I am not really able to change the whole sentence a lot easier," and separately, "I'm able to double click on these words and then just type it in." Both were UI gaps in the transcript editor, not pipeline gaps — selecting by sentence or scene instead of only by word, and retyping a line verbatim instead of only substituting individual words. Shipping this first, standalone, meant the harder product-swap work that followed didn't also have to carry an unrelated UX fix in its diff. A product catalog the tool never had PR #69, stacked directly on top of the transcript work, is pure groundwork with no user-visible feature of its own: a product catalog, because Variant Multiplier had no concept of "a product" at all before this. The editor's own framing made the requirement explicit: "have a product selection right here, for Pro Bluetooth, for [the other SKU], and maybe other tons of products" going forward. The catalog data itself is mai

Every AI video pipeline eventually has to answer an unglamorous question: what did we actually pay for that clip? On the main video-generation service, the answer for months had been "a hardcoded constant." That's fine until the vendor changes its own pricing, or a code path pays for the same synthesis twice, or a voice engine mints a clone, bills for it, and never sends it downstream. Over a ten-PR run I audited and rebuilt the voice and lip-sync pipeline from the billing layer up, then used the vendor's own SKU tiers to cut cost 7x without touching output quality. A cost model built from hardcoded constants isn't a cost model. It's a guess that happens to compile. Billing what the vendor actually charges PR #224 was workstream one of three from a sibling-tool audit: port the cost-accounting fixes that Presenter Generation and Variant Multiplier had already found, verifying each one against this repo's own code rather than assuming the same defect existed in the same place. Anthropic returns exact token counts on every response. Nothing in the pipeline read them — every charge was a hardcoded per-call constant, so the ledger and the vendor invoice diverged the moment usage drifted from whatever number had been typed in at launch. The same PR closed a second gap: two editor-facing routes could spend money — kicking off a generation, retrying a step — outside any run . A run is the unit everything else (budgets, audit trail, the cost ledger) is keyed to. A spend with no run attached is a spend the ledger can't even see, which is worse than a wrong number. Paying twice for a take the model returns unchanged PR #225 found the sibling bug's twin: some vendor calls return the exact same asset on a retry — no new synthesis happened — and the pipeline billed a second time anyway because "call succeeded" and "call did new work" were treated as the same fact. The fix is the boring, correct kind: hash the output, and only charge when the hash changes from the take you already

On the AI video ad platform I work on, every scene goes through the same painful loop: write a prompt, send it to an AI video model provider, wait two minutes, open the result, squint at the frame, and decide what went wrong. Camera too wide. Product missing from the hero shot. Color palette drifted warm when the brand brief says cool neutrals. Avatar looks like a different person than scene three. That loop was manual, slow, and expensive. Each regeneration burns GPU credits. Operators were becoming prompt engineers by accident — and still missing subtle failures until stitch time, when fixing scene four means re-rendering everything downstream. The insight behind vision-in-the-loop prompt authoring is simple: the model that wrote the prompt can also look at its own output and rewrite the prompt with surgical fixes. Not a full replan — a per-scene correction grounded in the actual generated frame, not the operator's memory of what they hoped would appear. The manual loop we were trying to kill Before this work shipped, the swipe iteration flow looked like this: Plan — Claude generates a scene-by-scene script with visual prompts Generate — each scene renders independently through an AI video model provider Review — operator opens the portal, compares frames to the reference ad Rewrite — operator edits prompts in a text field, often guessing at what the model misread Regenerate — repeat until acceptable or budget exhausted Steps three and four are where throughput dies. An experienced operator can spot "product not visible" in three seconds, but translating that into prompt language — "medium close-up, product centered in lower third, shallow depth of field" — takes another minute per scene. Multiply by twelve scenes and three swipe iterations, and a single ad creative consumes an hour of human attention that should be spent on brand strategy, not frame inspection. The generated frame is ground truth. The original prompt is a hypothesis. Vision-in-the-loop closes the

You do not have a camera. You have a machine that dreams a short motion out of a single still image, and it dreams badly the moment you ask it for something the still does not already contain. I learned this across a 10-episode series, and every rule below was paid for in failed generations. None of it is theory. The medium's real physics A real camera moves through a space that exists whether or not you point at it. The model has no space. It has one flat image and a statistical guess about what "zoom out" tends to look like in its training data. When the frame widens, the model is not revealing more of a room that was always there. It is inventing pixels to fill the new area, drawn from everything it has ever seen. That single fact reorganizes everything you know about directing: There is no coverage. Every "angle" is a separate generation from a separate still. Continuity is not captured; it is engineered, frame by frame. Nothing survives the cut for free. The model does not know that shot 12 and shot 13 are the same character in the same room. Anything you want to persist (damage state, light, color) must be re-declared or re-anchored every single time. The model abhors an empty frame. Its deepest reflex is to resolve ambiguity: a silhouette becomes a face, fog becomes a mountain range, a clean retro interior grows drips and cobwebs because "analog" reads as "abandoned". Spawn pressure is constant. Background figures flicker into existence in any populated-looking scene. Every motion prompt in my pipeline ends with an anti-spawn guard: "Do not add extra characters. Keep everything as pictured." Drop that guard and the figures come back. A widening or traveling frame is an invitation for the model to hallucinate. Direct this camera and you are not choosing what to show. You are choosing what to withhold from its imagination. The classical grammar, re-pointed If you carry film vocabulary, it all still applies. The mechanism just changes completely. Classical tool

For a long time, whenever we hear 'User Experience', we instinctively think of UI/UX designers, product designers, or maybe frontend engineers. Why? Because we tend to think users interact first with a graphical or command-line interface, while the backend engine plays little to no role in how they experience the product. The first half is correct. The second half, incorrect. A user doesn't experience your frontend in isolation. They experience the entire system. As I continue to compound my experience building products as a backend-leaning engineer, I've found it increasingly necessary to think beyond whether an endpoint works or whether an architecture is technically sound. I have to ask: How does this technical decision affect the user's experience? Here's how. 1. API Response Times Become UX A user doesn't care that your endpoint executes 17 database queries, that your service is making five downstream requests, or that your server is experiencing a cold start. They care that they clicked “Pay” three seconds ago and nothing has happened. Eventually, they may refresh the page, click the button again, or abandon the application altogether. The frontend can add a beautiful loading animation, but it cannot completely hide a system that is fundamentally slow. 2. Error Messages Become UX One of the easiest ways to see the relationship between backend engineering and UX is through errors. Imagine trying to make a payment and receiving: 400 Bad Request Technically, something has gone wrong. But the user has learned almost nothing. Compare that with: “Your payment could not be completed because your card was declined. Please try another payment method.” Good backend error handling should therefore answer three questions: What happened? Why did it happen? What can the user do about it? 3. API Design Becomes UX API design can feel very far removed from UX. After all, users don't see JSON responses. But, developers build products using those responses. The decisions we make

Have you ever stopped to think about the risk of having a pipeline where any merge into the main branch deploys straight to production without a single safety gate? For a long time, our workflow here was that classic setup almost every developer has used at some point: merge on main triggering an SSH script with git pull and pm2 restart It worked for day-to-day tasks, but it gave a false sense of stability lol The reality check hit when I found a critical blind spot in the automation: remote SSH scripts were running without strict error handling. In other words, if a git pull caused a conflict or a database migration failed halfway through, the script simply ignored the failure, ran to the end, and GitHub Actions marked the pipeline as green The absolute worst-case scenario for monitoring: the pipeline reported that everything went smoothly, while production was already completely down On top of that, the execution order was inverted: database migrations were running before the application build. If TypeScript threw a type error right after, the database schema had already advanced while the new code never booted. And since Prisma has no native down migrations, rolling back meant a high-risk manual intervention I decided to stop everything and redesign our delivery pipeline from scratch, starting from one clear premise: a tag is a release, a merge is not Today, nothing touches the production server without an annotated SemVer tag, going through 6 tightly coupled stages: Strict tag validation: only accepts annotated tags matching vX.Y.Z, ensuring author, timestamp, and audit trail for every single release Quality gates across PR and Release: automated tests with Vitest, strict typechecking, builds, and migration validation against a clean database via workflow_call Decoupled backups: an independent daily scheduled routine combined with a mandatory safety snapshot right before touching production Real migration dry-run: the most valuable gate, where the pipeline resto

Ian Nowland discusses why conventional CI/CD practices break down for stateful, core infrastructure. Drawing from his leadership at AWS and Datadog, he shares actionable techniques for safe progressive deployments, synthetic testing in production, and mitigating blast radius in complex software platforms. By Ian Nowland

Data has become one of the most valuable assets for modern enterprises, powering everything from business intelligence dashboards to machine learning models and generative AI applications. However, the biggest challenge organizations face today is not collecting data — it is trusting it. Enterprise data often contains duplicate records, missing values, inconsistent schemas, outdated information, and inaccurate entries that silently reduce the quality of analytics and AI predictions. These hidden data quality issues can lead to poor business decisions, increased operational costs, compliance risks, and unreliable AI outcomes. While most organizations implement basic validation rules, traditional data quality frameworks are largely rule-based, difficult to maintain, and unable to detect complex anomalies that continuously evolve across modern cloud data platforms. This article introduces the Data Trust Score Engine, an AI-powered cloud-native solution designed to automatically measure and improve enterprise data reliability. Instead of relying solely on manual validation or predefined rules, the platform combines metadata intelligence, large-scale analytics, and machine learning to calculate a dynamic Trust Score (0–100) for every dataset. The score is generated by evaluating multiple quality dimensions, including data completeness, consistency, uniqueness, freshness, schema compliance, null-value distribution, statistical anomalies, and AI-detected outliers. As a result, organizations can quickly identify fake, duplicate, corrupted, or low-quality datasets before they impact reporting, business intelligence, or downstream AI models. Learn about Medium’s values The solution is built entirely on Google Cloud Platform (GCP) using BigQuery as the scalable analytical data warehouse, Data Catalog for centralized metadata management and governance, and Vertex AI for intelligent anomaly detection and predictive quality analysis. BigQuery processes billions of records efficie