标签：#dbt

I've spent more than a decade building data pipelines, and the part nobody warns you about isn't the pipeline logic. It's the tuning. Executor memory, shuffle partitions, cluster size, thread counts. You pick numbers, ship it, and a few weeks later something breaks in a way that's obviously tuning-related but not obviously what to change . The pattern repeats enough times that you start recognizing it before you've even opened the logs. Job's slow, thousands of tiny shuffle tasks, someone way overestimated the partition count. Job dies on OOM, memory's set for last quarter's data volume, nobody updated it since. Cloud bill jumps, a cluster's been sized for peak load and just sits there mostly idle the other 20 hours a day. Every senior data engineer has this pattern-matching running in their head. It's tribal knowledge, and it lives in one or two people's heads on most teams, which means it doesn't scale and it definitely doesn't survive someone leaving. So I built a small tool to make that pattern-matching explicit instead of tribal: it reads your pipeline's config alongside its actual run metrics and tells you what's likely wrong, with the reasoning shown, not just a suggested number. Why rules instead of a model The obvious move in 2026 is to reach for an ML model. I didn't, and it wasn't because I don't think ML has a place here eventually. It's that for this specific problem, a handful of threshold rules already gets you most of the value, and they're something you can actually audit. If a rule fires, I can point at the exact condition and the exact number: average heap usage 28%, peak 47%, five runs, no OOM errors, therefore memory's over-provisioned, shrink it by roughly a fifth. That's checkable. You can look at your own metrics and see whether the reasoning holds. A model's confidence score doesn't give you that, and for something that's about to change a production config, I want the person approving it to be able to say "yes, I see why" rather than "the m

Hello everyone! Following up on my previous post , Day 1 of my Modern Data Stack migration was an absolute rollercoaster of refactoring and deep data auditing. I’m moving our legacy system (spreadsheets and Qlik) into a robust pipeline using Python, ClickHouse, and dbt . Here is what went down over the last 24 hours. 1. From Messy Scripts to a Single, Parameterized Extraction Engine 🛠️ In the legacy setup, each company had its own folder, its own .env file, and its own duplicated Python extraction script. It was a maintenance nightmare. Yesterday, I completely refactored this structure: Centralized Configuration: Merged all separate environments into a single, global .env file at the root level, mapping all 8+ companies and their branches. Eliminated Code Duplication (DRY): Instead of having identical extraction logic copied across folders, I built a single, unified codebase. Now, we have one universal script for Sales, one for Stock, one for Orders, etc. The behavior changes dynamically based on the company argument we pass to the CLI (e.g., python -m extract.run extract --source company1 ). To speed up this refactoring, I used Claude to generate the initial application skeleton. Since the AI already had the context of our legacy extraction logic, translating it into this new clean architecture was incredibly smooth. 2. Highs and Lows: The Data Parity Challenge With the pipeline modernized, I ran the pilot ingestion for Company #1 . To minimize friction for our downstream BI consumers, I kept the ClickHouse Bronze tables structured 1:1 with the legacy CSV schemas. The Good News: The data ingestion into the Bronze layer worked flawlessly. Moving up to the Silver layer (where we do data cleaning and domain-specific transformations), everything validated beautifully. Row counts matched perfectly. The "Fun" Part (The $2 Million Gap): When I materialized the Gold layer (our consolidated group business models), I hit a massive wall. The new pipeline reported $2 million U