标签：#dataengineering

Half the market is arguing about whether RAG or a semantic layer is the right foundation for enterprise AI. They are not competing. They answer different questions, and most teams need both. Two shapes of question Every question an agent receives breaks into one of two forms: "What did we say about X?" — lives in contracts, policies, tickets, docs. Unstructured. RAG was built for this. "What is true about X?" — lives in your warehouse and governed metrics. Structured. A semantic layer was built for this. Treating them as rivals is how teams end up with a system that can quote the pricing policy but cannot tell you this quarter's realised price. Where each one breaks RAG Semantic layer Good at Retrieving relevant prose Resolving definitions and joins Fails on Aggregation, math, current state Anything not modelled as data Permissions Flattened at ingest, rebuilt at query time Compiled per person, per query Answer stability Varies with retrieval ranking Identical by construction Audit story Cites a chunk Reproduces the exact SQL The permissions row is the one that ends pilots. A retrieval index that ingested everything has, by construction, assembled your most sensitive object — and reconstructing entitlement at query time is guesswork. The layer that actually decides Neither a document chunk nor a metric definition is worth much until something compiles it into a governed query and runs it. That is the piece most architectures are missing: intent → context resolution → constrained planning → governed execution . RAG can feed the first step. It cannot perform the last three. Point an agent at raw tables and the best models score in the low teens on real enterprise data. Give the same model compiled, governed context and it clears the high nineties. The retrieval quality was never the bottleneck. The full breakdown — the precise division of labour, why hybrid architectures win, and how compile-time governance closes the gap RAG cannot — is here: 👉 RAG vs. Semantic Layer

We live in an era where our wrists track every heartbeat, step, and sleep cycle. Yet, most of this "Quantified Self" data sits rotting in massive .xml or .json export files that are impossible to read. What if you could simply ask your AI, "How did my resting heart rate trend during the week I was stressed about the product launch?" In this tutorial, we are building a Quantified Self RAG (Retrieval-Augmented Generation) pipeline . We will take fragmented health data from Apple HealthKit and Google Health Connect, process it using DuckDB , and vectorize it into Pinecone using LangChain . By the end of this guide, you’ll have a production-grade Health Data RAG system capable of high-performance natural language queries over your personal biometrics. The Architecture: From Raw Logs to Vector Insights Handling health data at scale requires a robust ETL (Extract, Transform, Load) process. Vectorizing every single heart rate measurement (which can occur every few seconds) is inefficient and expensive. We need to downsample and summarize before embedding. graph TD A[Apple Health/Google Health] -->|Export XML/JSON| B[Raw Data Storage] B --> C{DuckDB Processing} C -->|Cleaning & Downsampling| D[Structured Parquet/JSON] D --> E[LangChain Document Loader] E --> F[OpenAI Embeddings] F --> G[Pinecone Vector Database] H[User: 'Why was my sleep poor last Tuesday?'] --> I[LangChain RAG Chain] G --> I I --> J[LLM Contextual Answer] Prerequisites 🛠️ To follow along, you'll need: Python 3.10+ Tech Stack : Pinecone , LangChain , DuckDB , OpenAI , and Pandas . An export of your health data (Apple Health export.xml or Google Takeout). Step 1: Efficient Data Crunching with DuckDB Apple Health exports are notoriously large XML files. Loading them directly into memory with standard Python is a recipe for a crash. We use DuckDB for its blazing-fast analytical capabilities to filter and downsample our data. import duckdb # Load and parse the XML (simplified logic) # Note: In a real scenario,

You've learned not to trust the first answer. So you read it carefully. You spot two problems. You fix them yourself, ship it, and move on. That's a reasonable way to work, and it's what separates an engineer who uses these tools well from one who copies and pastes. But there's a step you skipped. You never asked Claude to find the problems, and asking produces a different kind of output than reviewing does. 🔍 What actually happens Here's a specific case. You ask for an incremental load. You get something clean: a watermark column, a filter on records newer than the last run, an upsert into the target. You review it. You notice it assumes source records arrive in order, which yours don't, so you add a buffer window. You notice it doesn't handle the first run when the watermark is null, so you add a default. Two fixes, maybe fifteen minutes, and now it's correct. What you didn't find was the third problem: the upsert assumes a stable business key, and in your source system that key gets reassigned when records are merged. That one surfaces in production six weeks later as duplicate rows nobody can explain. You caught the problems you were looking for. You didn't catch the one you weren't. This is the normal outcome of self-review. You check against your own mental list of things that go wrong, and your list is good but finite. The problems that hurt are the ones outside it. 🧠 Why intermediates specifically miss this Beginners don't review AI output much at all, so this isn't their failure mode yet. Seniors have usually developed the habit after being caught out by something their own review missed. Intermediates sit in an awkward middle. They have learned, correctly, that AI output needs questioning. And they have concluded, understandably, that the questioning is entirely their job. That conclusion makes sense. Reviewing is what you do with a junior's pull request. It's what you do with your own code before you push. Review is a human activity performed on work some

The answer arrives clean, well-formatted, and certain. You run it. The config flag doesn't exist. Nothing in the response told you the difference between the parts it was sure about and the parts it filled in. The syntax was right. The explanation was reasonable. The flag had a sensible name and a clear purpose. It just wasn't real. This is a calibration problem, not a trust problem. "Don't trust AI" is useless advice. It tells you to doubt everything, which is the same as doubting nothing, because you can't actually work that way. What you need instead is a way to tell which parts of an answer are grounded and which were filled in. There are patterns to look for, and once you know them, they are hard to miss. 🔍 Why this affects beginners specifically This isn't about intelligence. It's about experience. An engineer with a few years behind them has already lost time to a function that didn't exist. They have wasted an afternoon on a config parameter that turned out to belong to a different engine. They have built up a collection of "that looked right but wasn't" memories, and those memories now activate automatically. A small feeling that says check this one before they run it. Beginners haven't had those afternoons yet. The pattern recognition that comes from repeated failures doesn't exist, because the failures haven't happened. Everything arrives in the same confident tone, and without past mistakes to compare against, there is no internal signal separating a grounded answer from a made-up one. That's not a character flaw. It's missing experience, and it can be partly replaced with three concrete things to look for. ✅ The three signs These aren't perfect. They are a minimum standard, and they catch the common cases. I'll be clear about what they miss at the end. Sign 1: Very specific details with no source When the answer includes a precise detail (a specific config flag, a particular function signature, an exact version number) and doesn't say where that detail

We've all been there: staring at a blood test report from three years ago, trying to remember if that "slightly elevated" glucose level was a one-time thing or a trend. Our health data is scattered across messy PDFs, fitness tracker exports, and physical medical folders. In the era of AI, why are we still manually digging through folders? 📂 Today, we are building the Ultimate Personal Health Knowledge Base . By leveraging Retrieval-Augmented Generation (RAG) , we will transform fragmented medical reports and logs into a searchable, private, and intelligent second brain. We’ll be using LlamaIndex for orchestration, Unstructured.io for parsing those pesky PDFs, and ChromaDB for local vector storage. If you're looking for advanced architectural patterns or production-grade data engineering strategies beyond this tutorial, I highly recommend checking out the deep dives over at WellAlly Tech Blog , which served as a major inspiration for this build. 🚀 The Architecture 🏗️ The goal is to create a pipeline that ingests raw data, vectorizes it, and allows for Hybrid Search —combining semantic meaning with keyword precision (crucial for medical terms!). graph TD A[Raw Health Data: PDFs, CSVs, MD] --> B(Unstructured.io Parser) B --> C{Chunking & Cleaning} C --> D[Sentence-Transformers] D --> E[(ChromaDB Vector Store)] F[User Query: Is my cholesterol improving?] --> G[LlamaIndex Query Engine] E <--> G G --> H[LLM: Local or OpenAI] H --> I[Actionable Health Insight] Prerequisites 🛠️ To follow along, you’ll need a Python environment with the following stack: Unstructured.io : To handle "dirty" PDF and image-based reports. ChromaDB : Our lightweight, open-source vector database. Sentence-Transformers : To generate local embeddings without sending data to the cloud. LlamaIndex : The glue that connects our data to the LLM. pip install llama-index chromadb unstructured sentence-transformers llama-index-vector-stores-chroma Step 1: Ingesting Messy Medical Reports 📄 Medical reports are

Most "data lineage" you've seen answers a schema question: table B comes from table A , or column B.total comes from columns A.price and A.qty . That's genuinely useful, and tools like OpenLineage do it well. But notice what it doesn't tell you: it says which columns can influence an output. It never says which values actually did . That gap is the whole subject of this post. I built a small, self-contained reference pipeline that captures provenance at the row and field level — "the value in this destination row, this field, was computed from these specific source (row, field) pairs" — and I want to walk through two things: why you'd ever want provenance at that granularity, and why it's genuinely hard once you commit to it. Repo (dbt-core + DuckDB, no server, no cloud, runs on a clean checkout): https://github.com/stevenblough/row-level-prov The one distinction everything follows from Here's the sentence the entire project turns on: Column-level lineage is a schema-sized, static fact you can derive from code. Value-level provenance is a data-sized, dynamic fact you must capture at execution. Put it in complexity terms and the consequences become obvious: Column lineage is O(schema) . It scales with how many columns you have. You can compute it by parsing SQL, offline, without ever looking at a single row. Value provenance is O(rows × fan-in) . It scales with your data volume times how many source values feed each output value. It does not exist anywhere until the query runs, and it can only be captured there , piggybacked on the query that actually produced the values. You cannot "reconstruct" value provenance later by re-querying the sources — the moment the source changes, you'd reconstruct a different answer than what really happened. That single exponent change ( schema → rows × fan-in ) is why value-level provenance has an entire class of problems that column lineage never faces. Why bother? The reasons for this level of granularity Granularity is expensive,

TL;DR I designed a proposed AWS Transform Custom community skill that prepares Glue 2.0, 3.0, and 4.0 repositories for Glue 5.0. It separates safe mechanical transformations from changes that require human evidence, generates a migration report, and preserves already-compatible files unchanged. Because I didn't have live atx access, the benchmarks in this post are explicitly labeled manually simulated, not agent-executed. The proposal is open as issue #75 — not yet merged, not yet a pull request. The missing data-engineering transformation AWS Transform Custom can apply agent-driven code transformations across a single repository — or thousands of them at once, via AWS Batch and Fargate. As of July 30, 2026, its public sample repository, aws-samples/aws-transform-custom-samples , contained three community-contributed transformations: an EKS version-upgrade-readiness skill, a JBoss-to-Spring-Boot migration, and a Kubernetes readiness migration. None of them touched data engineering. Given that most of my day-to-day work sits across AWS data engineering, Databricks, and Delta Lake, that gap was the obvious thing to fill. What an AWS Transform Custom "skill" looks like Before writing anything, I studied the deepest existing example, jboss-to-springboot , since the pattern it establishes is effectively an unwritten spec for the other two skills as well: README.md — the problem, what the skill does, and how to invoke it via the atx CLI. This is also where the repo draws a clear line: these are readiness transformations. They modify repository artifacts — code and infrastructure-as-code — but they don't deploy jobs, call AWS APIs to change running resources, or claim data-level equivalence. That distinction matters throughout everything below. SKILL.md — the agent-facing definition: YAML frontmatter with trigger keywords, an Objective, explicit Non-Goals , Constraints, worked before/after examples, a "signal in source code → reference file" routing table, and a numbered V

Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fails. The answer used to default to Airflow because there wasn't a real alternative. Now there are three reasonable defaults, and they optimize for different things. Picking wrong doesn't break anything on day one — it shows up eighteen months later as either an operations team drowning in scheduler maintenance or an engineering team fighting a platform that won't do what they need it to. Here's the actual tradeoff, not the vendor pitch version. Databricks Workflows: the path of least resistance, if you're all-in on Databricks Databricks Workflows is the orchestrator built into the platform. Jobs, clusters, Unity Catalog permissions, and Workflows all share the same control plane, which means you're not maintaining a separate scheduler, not managing a second set of credentials, and not debugging why an external system can't see a table that Unity Catalog says it can. Task dependencies, retries, cluster reuse across tasks, and job-level alerting all come for free. The cost is exactly what you'd expect from a platform-native tool: it orchestrates Databricks well and everything else poorly. There's no first-class way to trigger a task in your orchestration DAG that waits on a Salesforce export, calls an internal API, or coordinates a dbt run against a warehouse that isn't Databricks SQL. You can bolt these in with webhooks and external scripts, but you're fighting the tool rather than using it. Workflows also doesn't give you the asset-lineage or testing story that Dagster does — it schedules tasks, not data assets. If your data platform genuinely is Databricks end to end — ingestion, transformation, ML, serving — Workflows removes an entire category of operational overhead you'd otherwise be paying for nothing. Teams in this position who reach for Airflow anyway usually do it out of habit, not need, and end up running two sch

If you have ever tried to find a new physician through a search box, you already know the frustration: outdated phone numbers, doctors who left the practice two years ago, and "accepting new patients" labels that turn out to be fiction. Anyone who has read the candid breakdown in Online Doctor Directories: A User's Guide to a Very Imperfect Tool will recognize the pattern immediately, because the core problem is not laziness on anyone's part — it is a data engineering problem hiding inside a healthcare product. And for those of us who build software for a living, it is a fascinating case study in what happens when stale data meets high-stakes decisions. The Root Cause Is a Data Pipeline, Not a Design Flaw Most doctor directories aggregate information from insurance networks, state licensing boards, hospital affiliations, and self-reported provider profiles. Each of these sources updates on its own schedule, uses its own identifiers, and defines fields differently. One system records a physician under her maiden name; another lists the clinic's billing address instead of the practice location; a third still shows a specialty she stopped practicing in 2019. The result is a classic entity-resolution nightmare. Without a reliable primary key shared across sources, merge logic has to guess whether "J. Martinez, Internal Medicine, Suite 400" and "Julia Martinez-Reyes, IM" are the same human. Get it wrong in either direction and the user suffers: duplicates erode trust, while over-aggressive merging attaches one doctor's malpractice history to a stranger with a similar name. If you have ever built a CRM deduplication service or wrestled with customer identity graphs, you have fought this exact battle — just with lower stakes. Staleness compounds the problem. Physicians change practices constantly. A directory that syncs quarterly is, by definition, wrong about a meaningful slice of its records at any given moment. Harvard Health has pointed out that an ongoing physician sh

Modern electrical grids increasingly rely on distributed sensors installed across conductors, towers, poles, substations, and remote line sections. These devices can measure: Conductor temperature Current and voltage Mechanical tension Line sag Vibration Weather conditions Fault passage Switch and recloser states Collecting these measurements is relatively straightforward. Building a reliable data pipeline around them is much harder. Power infrastructure often operates in locations with unstable connectivity, limited bandwidth, and strict requirements for alarm delivery. A useful architecture must therefore do more than move telemetry from sensors to a cloud database. It must determine which data is urgent, validate measurements, preserve event order, survive network outages, and integrate the results with operational utility systems. This article explores how to design that pipeline. The Basic Architecture A practical grid-monitoring data flow may look like this: Field Sensors | v Protocol Adapters | v Edge Data Model | +----> Local Rules and Fault Detection | +----> Local Time-Series Buffer | +----> Event Queue | v Central IoT or Utility Platform | +----> SCADA +----> GIS +----> OMS +----> Analytics +----> Maintenance Systems The edge gateway sits between field equipment and central applications. Its job is not limited to protocol conversion. It also acts as a local data-processing and reliability layer. Why Cloud-Only Processing Is Risky Imagine a utility operating 5,000 field sensors. Each device reports one measurement every second. That produces: 5,000 measurements per second 300,000 measurements per minute 18,000,000 measurements per hour Most of those measurements will describe normal operating conditions. Sending every individual value to a central platform creates unnecessary: Bandwidth consumption Storage growth Processing overhead Communication costs Dependence on network availability More importantly, cloud-only logic can stop working when the connectio

Every time this comes up, someone credits the same setup: Athena on Iceberg is where "the code is the spec" — where you open Git and read the whole system, catalog to transforms to schema, without logging into anything. In my experience they're not wrong. What I want to argue is that they're right for mostly the wrong reason. The reason people reach for is the engine — serverless, open format, nothing to provision. But the thing that actually keeps code as the spec, when it does, is something you could have applied to almost any engine. And the thing that breaks it, when it breaks, has nothing to do with Iceberg at all. So here's the split I've landed on, for now: whether "code is the spec" holds is about 90% discipline and 10% engine . Athena/Iceberg earns that 10% honestly — but 10% is all it earns, and I keep watching people mistake it for the whole thing. (Just where my own tinkering has led — not advice.) I should say up front that I'm still in the middle of this. What the best declarative setup for an agent actually looks like — how you turn a system into a spec it can read and act on without guessing — is something I'm actively testing, not something I've settled. Read what follows as a working idea at a particular moment, written down partly so I can find out where it's wrong. To see where the 10% actually lives, it helps to notice that a stateful system always keeps two copies of itself. The spec has two copies One copy is declared : the code you wrote, the schema you committed, the transforms in dbt, the catalog in Terraform. The other is realized : the state the engine accumulates while running — statistics, physical layout, caches, maintenance history, tuning knobs someone set at 2am. "Code is the spec" is really a claim about the distance between those two. When the declared copy explains almost everything about the realized one, you can reason about the system by reading Git. When it doesn't, you can't. Some of that state is declarable — you can pin a

Suspect-condition workflows are deceptively simple to prototype and surprisingly hard to make defensible . Anyone can flag "this member might have HCC X." Building a system whose output survives a RADV audit is a different problem. This is a walkthrough of the three stages and the engineering decisions that matter at each. Stage 1: Identify Identification is pattern detection over a member's clinical record — labs, medications, prior diagnoses, utilization. Model it as a set of rules or features that emit candidate HCCs: def identify_suspects ( member ): suspects = [] if member [ " labs " ]. get ( " a1c " , 0 ) >= 9.0 and " insulin " in member [ " meds " ]: suspects . append ({ " hcc " : " HCC38 " , " trigger " : " a1c>=9 + insulin " }) if member . get ( " egfr " ) and member [ " egfr " ] < 30 : suspects . append ({ " hcc " : " HCC326 " , " trigger " : " egfr<30 " }) return suspects The temptation is to maximize recall here — flag everything. Resist it. Every unvalidated suspect you generate is downstream work and downstream risk. Stage 2: Validate (the stage that actually matters) Validation attaches evidence to each suspect and scores its defensibility. This is the difference between a documentation opportunity and an audit liability. def validate ( suspect , member ): evidence = collect_evidence ( suspect [ " hcc " ], member ) # labs, rx, prior dx suspect [ " evidence " ] = evidence suspect [ " confidence " ] = score_evidence ( evidence ) suspect [ " defensible " ] = suspect [ " confidence " ] >= 0.7 return suspect Key design rule: a suspect with an empty evidence array should never reach a coder. Make that a hard gate, not a soft warning. Under CMS-HCC V28 and current audit posture, a captured-but-unsupported diagnosis can be extrapolated across a contract into a real clawback — so "defensible by default" is the right engineering stance. Stage 3: Capture Capture routes validated suspects to the right human with the evidence inline, so the clinician or coder can

Your data warehouse has a table called orders . It has columns like amount , status , created_at , and customer_id . Now three people ask "What was Q1 revenue?" The analyst writes SELECT SUM(amount) FROM orders WHERE created_at BETWEEN '2026-01-01' AND '2026-03-31' . The data engineer adds WHERE status = 'completed' . Finance excludes refunds and trial conversions. Three queries, three numbers, one question. Nobody is wrong. They just defined "revenue" differently. Multiply this by every metric in your organization, every team that queries the warehouse, and every tool that displays a number. That's the problem. A semantic layer solves it by defining each metric once, in one place, and serving that definition to every consumer. What is a semantic layer? A semantic layer is a metadata layer between your data warehouse and every tool that queries it. It defines business metrics, maps them to SQL, and exposes them through APIs. Instead of every consumer writing its own query, they all reference the same definition. When someone asks for "revenue," the semantic layer knows that means: SUM ( CASE WHEN status != 'refunded' AND type != 'trial' THEN amount ELSE 0 END ) That definition lives in one place. Dashboards, APIs, AI agents, and ad-hoc queries all use it. Change the definition once and every consumer gets the updated calculation. No Slack thread asking "which number is right." No detective work tracing a wrong number back to a stale query in a notebook somewhere. The core components of a semantic layer: Metrics (measures). The numbers you aggregate: revenue, order count, average deal size. Each metric has a fixed SQL definition. Dimensions. The columns you filter and group by: date, status, category, region. Dimensions define the axes of analysis. Relationships (joins). How tables connect: orders belong to customers, products belong to categories. Defined once, reused by every query. Access rules. Who can see what. Row-level security, tenant isolation, role-based ac

"We need real-time analytics" is one of the most common requests in data engineering. It's also one of the most misunderstood. When the VP of Sales says "real-time," they usually mean "faster than the dashboard that refreshes overnight." When the CTO says it, they might mean sub-second event streaming. The gap between those two definitions is a 6-month infrastructure project. Most teams don't need true real-time. They need fast enough. And "fast enough" is achievable with pre-aggregation caching at a fraction of the complexity and cost of a streaming architecture. What is real-time analytics? Real-time analytics means querying data with minimal latency between when an event happens and when it's visible in your analytics. The spectrum: Freshness Latency Architecture Use case True real-time < 1 second Event streaming (Kafka, Flink) Fraud detection, stock trading, live monitoring Near real-time 1-60 seconds Micro-batch or streaming Operational dashboards, alerting Frequent refresh 1-60 minutes Scheduled refresh + caching KPI dashboards, AI agent queries Batch Hours to daily Scheduled ETL Board reports, monthly summaries Most analytics use cases fall in the "frequent refresh" category. Revenue by region doesn't need sub-second freshness. Active users in the last hour doesn't need event streaming. A pre-aggregation cache that refreshes every 15 minutes covers 90% of what teams call "real-time." When you actually need real-time True real-time analytics (sub-second latency from event to query result) is worth the infrastructure investment when: Fraud detection. Every second of delay is potential fraud that slips through. Live monitoring. Server health, API error rates, active user counts for live products. Trading and pricing. Financial instruments where stale data means wrong prices. Live events. Streaming metrics during a product launch, marketing campaign, or live broadcast. If you're in one of these categories, you need an event streaming architecture: Kafka, Flink, M

"We're using Kafka with exactly-once semantics, so we don't have to worry about duplicates." I've heard this in architecture reviews, design docs, and postmortem explanations. It represents a misunderstanding of what Kafka's exactly-once guarantee actually covers, and the gap between the promise and the reality has caused real production incidents. What Kafka's Exactly-Once Actually Covers Kafka's exactly-once semantics (EOS), introduced in 0.11.0, operates at two levels: Producer idempotence ( enable.idempotence=true ): The producer assigns each message a sequence number. The broker deduplicates messages with the same producer ID and sequence number. This prevents duplicates caused by producer retries — the message lands in the Kafka partition exactly once, regardless of retry count. Transactions ( transactional.id ): Allows a producer to write to multiple partitions atomically. Either all writes commit or none do. Combined with isolation.level=read_committed on consumers, readers only see committed transactions. Together, these give you exactly-once message delivery within the Kafka cluster. What Exactly-Once Does Not Cover Here's the boundary that engineers miss: Kafka's exactly-once guarantee is scoped to the Kafka cluster. The moment your consumer does anything outside Kafka — writes to a database, calls a REST API, publishes to a cloud queue — you're outside the transaction boundary. Consider a typical consumer: consumer . poll ( records ); for ( record : records ) { database . save ( process ( record )); // External write — outside Kafka transaction } consumer . commitSync (); If the application crashes after database.save() but before commitSync() , Kafka re-delivers the message. The consumer reprocesses it. The database now has two writes for the same event. Enabling producer idempotence on the consumer's Kafka writes does not fix this. The Patterns That Actually Give You End-to-End Safety Idempotent Consumers Design consumer processing logic to be idempote

Risk stratification sounds like a data-science buzzword until you have to build the thing. For a Medicare Advantage plan, it's a concrete pipeline: take a population of members, score each one's clinical and financial risk, and rank them so care management and documentation teams know who to touch first. Here's how I'd architect it. The core idea Population health risk stratification = scoring + segmentation. You compute a per-member risk signal, then bucket members into tiers (e.g., rising-risk, high-risk, catastrophic) so finite resources go where they move outcomes and revenue most. The mistake teams make is treating it as a single ML model. In practice you want a layered signal: a stable, explainable base (RAF + chronic conditions) plus optional predictive overlays. Explainability matters because care managers won't act on a black-box score, and auditors won't accept one. Step 1: Build the member feature record { "member_id" : "SYNTH-77310" , "age" : 73 , "hccs" : [ "HCC37_1" , "HCC85" , "HCC18" ], "raf" : 1.842 , "gaps" : [ "a1c_overdue" , "no_pcp_visit_180d" ], "utilization" : { "ed_visits_12m" : 3 , "inpatient_12m" : 1 } } The RAF here is your defensible, model-grounded risk anchor under CMS-HCC V28. Everything else is supplemental signal. Step 2: Score and tier def risk_tier ( member ): base = member [ " raf " ] util = 0.15 * member [ " utilization " ][ " ed_visits_12m " ] \ + 0.30 * member [ " utilization " ][ " inpatient_12m " ] score = base + util if score >= 3.0 : return " catastrophic " if score >= 1.8 : return " high " if score >= 1.0 : return " rising " return " stable " Keep the weights transparent and tunable. The point isn't a perfect model; it's a defensible, reproducible ranking your operational teams trust. Step 3: Make "rising-risk" actionable The tier that quietly drives the most ROI is rising-risk — members trending toward high cost who still have open documentation and care gaps. Surface their specific gaps (overdue labs, undocumented chroni

TL;DR Traditional SaaS streaming requires exporting sensitive data to a vendor cloud, creating security risks and egress costs. BYOC reverses this model by running the data plane inside the customer’s cloud while the vendor manages the control plane. This keeps data within the enterprise perimeter while still providing a managed platform. Condense builds on this model with AI-driven automation, unified monitoring, and marketplace deployment, enabling private, compliant, and cost-efficient real-time data streaming. The enterprise data landscape is currently defined by a conflict between real-time AI data streaming utility and the strict requirements of data sovereignty . For years, the standard SaaS model forced a compromise. To access premium analytics, companies had to export sensitive telemetry to a vendor cloud. This created massive cloud egress costs and introduced significant security vulnerabilities. Bring Your Own Cloud (BYOC) for data streaming platforms has emerged as the professional solution to this dilemma. It allows a business to keep data within its own perimeter while benefiting from a fully managed, high-performance ecosystem. The BYOC Architecture: Privacy by Design An experienced analyst views BYOC as a clean separation of concerns. The architecture splits the environment into two distinct layers to ensure raw data never leaves the authorized environment. SaaS Control Plane: This is the management layer hosted by the provider. It handles the brain of the operation. It manages orchestration, user access, and pipeline configuration without ever seeing the actual data packets. Private Data Plane: This is the muscle. The managed Kafka clusters , Kubernetes (K8s) nodes, and storage engines like ClickHouse live inside the customer Virtual Private Cloud (VPC) . By keeping the data plane inside the customer perimeter, telemetry collection remains private. This architecture is the most direct path to satisfying internal security audits and global regulatory

Cargar 112.647 filas en una tabla de hechos debería tardar segundos. A mí me tardaba más de 25 minutos, y acababa cancelando la query. Los datos estaban bien, el SQL estaba bien, las dimensiones se poblaban sin problema. El culpable era otro, y descubrirlo fue la parte más instructiva de todo el proyecto. Todo esto surgió construyendo un Data Warehouse en estrella sobre datos reales de e-commerce: no una tabla bonita para hacer un SELECT * , sino un modelo dimensional completo, reproducible desde cero, capaz de responder preguntas de negocio de verdad. El dataset Trabajé con el Brazilian E-Commerce Public Dataset by Olist : pedidos reales de un marketplace brasileño entre septiembre de 2016 y octubre de 2018. Son 9 CSV relacionados entre sí: 99.441 pedidos y 112.650 líneas de venta 103.886 pagos y 104.719 reseñas 32.951 productos, 3.095 vendedores 1.000.163 registros de geolocalización Y con trampas de datos reales que hay que ver antes de que te muerdan: Un pedido puede tener varios pagos y varias reseñas. Si los unes tal cual a la tabla de hechos, duplicas ventas . Es el error clásico y silencioso: los totales salen inflados y nadie se entera. customer_id no es un cliente. Olist crea uno por cada pedido; la persona real es customer_unique_id . Contar mal aquí te cambia el KPI: hay 99.441 cuentas frente a 96.096 personas. El CSV de productos trae una errata en la cabecera ( product_name_lenght , con "lenght"). Si tu esquema la escribe bien y cargas por interfaz gráfica (que empareja por nombre ), esas columnas se quedan vacías sin que nadie avise. El proceso Monté una arquitectura en capas: CSV → staging → modelo dimensional → vistas → análisis , todo en cuatro scripts ejecutables en orden y idempotentes (el esquema se recrea desde cero, se puede relanzar mil veces). El modelo es un star schema : una tabla de hechos fact_sales al grano de línea de producto dentro de un pedido , y cinco dimensiones (cliente, producto, vendedor, pago y fecha), con claves sustitutas,

We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana Soumyadeep Dey Soumyadeep Dey Soumyadeep Dey Follow Jul 12 We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana # devchallenge # weekendchallenge # snowflake 5 reactions 1 comment 16 min read

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기 이전 글에서는 같은 source_hash 가 다시 들어왔을 때 기존 successful run을 재사용하는 idempotency를 다뤘다. 하지만 재처리에는 두 종류가 있다. 1. 같은 입력이 다시 들어온 경우 -> skip이 맞다. 2. 같은 business_date의 정정 입력이 들어온 경우 -> skip하면 안 된다. -> 같은 날짜의 gold 결과를 중복 없이 교체해야 한다. manufacturing-data-platform-mini 의 B5 slice는 두 번째 문제를 아주 작게 다룬다. 전체 Spark pipeline을 만든 것이 아니다. gold_daily_metrics Iceberg table 하나를 local Spark에서 만들고, business_date partition overwrite와 snapshot evidence만 검증했다. Scenario 이미 아래 gold row가 있다. business_date=2026-06-29 plant-a / line-1 / gearbox-a units_produced=120 defect_count=3 나중에 같은 business_date=2026-06-29 에 대한 정정 source가 들어온다. 운영자가 원하는 것은 append가 아니다. 원하지 않는 상태: 2026-06-29 old row 2026-06-29 corrected row -> 같은 날짜 결과가 중복됨 원하는 상태: 2026-06-29 corrected row만 남음 2026-06-30 같은 다른 날짜 partition은 그대로 유지됨 재처리 전후 snapshot evidence가 남음 그래서 이 slice의 질문은 이렇다. 같은 business_date의 정정 source를 처리할 때, gold table에서 해당 날짜 partition만 중복 없이 교체하고, 어떤 run이 어떤 Iceberg snapshot을 만들었는지 남길 수 있는가? Decision Pressure Slice1의 CSV pipeline은 already-successful source를 안전하게 skip할 수 있다. dataset_id + business_date + source_hash 이 key가 같으면 같은 입력이다. 다시 계산해도 같은 결과이므로 기존 run을 재사용한다. 하지만 source_hash 가 달라졌다면 의미가 다르다. same business_date different source_hash 이건 retry가 아니라 correction이다. CSV run-folder 방식에서는 새 run output을 만들 수는 있지만, "현재 gold table에서 해당 날짜를 원자적으로 교체한다"는 table-level 의미가 약하다. Iceberg를 붙이는 이유는 여기 있다. source_hash -> 같은 입력인지 판단하는 idempotency key business_date partition -> 정정 시 교체할 gold table 범위 snapshot_id -> table commit의 evidence 즉 Spark/Iceberg는 도구 이름을 추가하려고 붙인 것이 아니라, 재처리 상태 전이를 더 명확히 표현하기 위해 붙였다. Options Option 장점 문제 판단 same source면 항상 재계산 단순함 retry 때 불필요한 commit이 계속 생김 제외 corrected source를 append 구현 쉬움 같은 날짜 gold row가 중복될 수 있음 제외 whole-table overwrite 단순함 다른 날짜 partition까지 지울 위험 제외 business_date partition overwrite correction 범위가 명확함 Spark/Iceberg 설정과 test가 필요 선택 MERGE/upsert 강력함 이번 skeleton에 과함 backlog 이번 구현은 DataFrameWriterV2.overwritePartitions() 를 사용했다. corrected_d