标签：#dataengineering

TL;DR Traditional SaaS streaming requires exporting sensitive data to a vendor cloud, creating security risks and egress costs. BYOC reverses this model by running the data plane inside the customer’s cloud while the vendor manages the control plane. This keeps data within the enterprise perimeter while still providing a managed platform. Condense builds on this model with AI-driven automation, unified monitoring, and marketplace deployment, enabling private, compliant, and cost-efficient real-time data streaming. The enterprise data landscape is currently defined by a conflict between real-time AI data streaming utility and the strict requirements of data sovereignty . For years, the standard SaaS model forced a compromise. To access premium analytics, companies had to export sensitive telemetry to a vendor cloud. This created massive cloud egress costs and introduced significant security vulnerabilities. Bring Your Own Cloud (BYOC) for data streaming platforms has emerged as the professional solution to this dilemma. It allows a business to keep data within its own perimeter while benefiting from a fully managed, high-performance ecosystem. The BYOC Architecture: Privacy by Design An experienced analyst views BYOC as a clean separation of concerns. The architecture splits the environment into two distinct layers to ensure raw data never leaves the authorized environment. SaaS Control Plane: This is the management layer hosted by the provider. It handles the brain of the operation. It manages orchestration, user access, and pipeline configuration without ever seeing the actual data packets. Private Data Plane: This is the muscle. The managed Kafka clusters , Kubernetes (K8s) nodes, and storage engines like ClickHouse live inside the customer Virtual Private Cloud (VPC) . By keeping the data plane inside the customer perimeter, telemetry collection remains private. This architecture is the most direct path to satisfying internal security audits and global regulatory

Cargar 112.647 filas en una tabla de hechos debería tardar segundos. A mí me tardaba más de 25 minutos, y acababa cancelando la query. Los datos estaban bien, el SQL estaba bien, las dimensiones se poblaban sin problema. El culpable era otro, y descubrirlo fue la parte más instructiva de todo el proyecto. Todo esto surgió construyendo un Data Warehouse en estrella sobre datos reales de e-commerce: no una tabla bonita para hacer un SELECT * , sino un modelo dimensional completo, reproducible desde cero, capaz de responder preguntas de negocio de verdad. El dataset Trabajé con el Brazilian E-Commerce Public Dataset by Olist : pedidos reales de un marketplace brasileño entre septiembre de 2016 y octubre de 2018. Son 9 CSV relacionados entre sí: 99.441 pedidos y 112.650 líneas de venta 103.886 pagos y 104.719 reseñas 32.951 productos, 3.095 vendedores 1.000.163 registros de geolocalización Y con trampas de datos reales que hay que ver antes de que te muerdan: Un pedido puede tener varios pagos y varias reseñas. Si los unes tal cual a la tabla de hechos, duplicas ventas . Es el error clásico y silencioso: los totales salen inflados y nadie se entera. customer_id no es un cliente. Olist crea uno por cada pedido; la persona real es customer_unique_id . Contar mal aquí te cambia el KPI: hay 99.441 cuentas frente a 96.096 personas. El CSV de productos trae una errata en la cabecera ( product_name_lenght , con "lenght"). Si tu esquema la escribe bien y cargas por interfaz gráfica (que empareja por nombre ), esas columnas se quedan vacías sin que nadie avise. El proceso Monté una arquitectura en capas: CSV → staging → modelo dimensional → vistas → análisis , todo en cuatro scripts ejecutables en orden y idempotentes (el esquema se recrea desde cero, se puede relanzar mil veces). El modelo es un star schema : una tabla de hechos fact_sales al grano de línea de producto dentro de un pedido , y cinco dimensiones (cliente, producto, vendedor, pago y fecha), con claves sustitutas,

We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana Soumyadeep Dey Soumyadeep Dey Soumyadeep Dey Follow Jul 12 We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana # devchallenge # weekendchallenge # snowflake 5 reactions 1 comment 16 min read

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기 이전 글에서는 같은 source_hash 가 다시 들어왔을 때 기존 successful run을 재사용하는 idempotency를 다뤘다. 하지만 재처리에는 두 종류가 있다. 1. 같은 입력이 다시 들어온 경우 -> skip이 맞다. 2. 같은 business_date의 정정 입력이 들어온 경우 -> skip하면 안 된다. -> 같은 날짜의 gold 결과를 중복 없이 교체해야 한다. manufacturing-data-platform-mini 의 B5 slice는 두 번째 문제를 아주 작게 다룬다. 전체 Spark pipeline을 만든 것이 아니다. gold_daily_metrics Iceberg table 하나를 local Spark에서 만들고, business_date partition overwrite와 snapshot evidence만 검증했다. Scenario 이미 아래 gold row가 있다. business_date=2026-06-29 plant-a / line-1 / gearbox-a units_produced=120 defect_count=3 나중에 같은 business_date=2026-06-29 에 대한 정정 source가 들어온다. 운영자가 원하는 것은 append가 아니다. 원하지 않는 상태: 2026-06-29 old row 2026-06-29 corrected row -> 같은 날짜 결과가 중복됨 원하는 상태: 2026-06-29 corrected row만 남음 2026-06-30 같은 다른 날짜 partition은 그대로 유지됨 재처리 전후 snapshot evidence가 남음 그래서 이 slice의 질문은 이렇다. 같은 business_date의 정정 source를 처리할 때, gold table에서 해당 날짜 partition만 중복 없이 교체하고, 어떤 run이 어떤 Iceberg snapshot을 만들었는지 남길 수 있는가? Decision Pressure Slice1의 CSV pipeline은 already-successful source를 안전하게 skip할 수 있다. dataset_id + business_date + source_hash 이 key가 같으면 같은 입력이다. 다시 계산해도 같은 결과이므로 기존 run을 재사용한다. 하지만 source_hash 가 달라졌다면 의미가 다르다. same business_date different source_hash 이건 retry가 아니라 correction이다. CSV run-folder 방식에서는 새 run output을 만들 수는 있지만, "현재 gold table에서 해당 날짜를 원자적으로 교체한다"는 table-level 의미가 약하다. Iceberg를 붙이는 이유는 여기 있다. source_hash -> 같은 입력인지 판단하는 idempotency key business_date partition -> 정정 시 교체할 gold table 범위 snapshot_id -> table commit의 evidence 즉 Spark/Iceberg는 도구 이름을 추가하려고 붙인 것이 아니라, 재처리 상태 전이를 더 명확히 표현하기 위해 붙였다. Options Option 장점 문제 판단 same source면 항상 재계산 단순함 retry 때 불필요한 commit이 계속 생김 제외 corrected source를 append 구현 쉬움 같은 날짜 gold row가 중복될 수 있음 제외 whole-table overwrite 단순함 다른 날짜 partition까지 지울 위험 제외 business_date partition overwrite correction 범위가 명확함 Spark/Iceberg 설정과 test가 필요 선택 MERGE/upsert 강력함 이번 skeleton에 과함 backlog 이번 구현은 DataFrameWriterV2.overwritePartitions() 를 사용했다. corrected_d

wide CSV 여러 개를 EAV로 모아 gold mart 만들기 현실의 데이터 소스는 한 가지 모양으로 오지 않는다. 같은 의미의 값도 어떤 파일에서는 생산수량 , 다른 파일에서는 units , 또 다른 파일에서는 made 로 올 수 있다. 온도도 어떤 곳은 섭씨, 어떤 곳은 화씨일 수 있다. 이걸 매번 pipeline code에 if source == ... 로 박기 시작하면 source가 늘 때마다 코드가 지저분해진다. manufacturing-data-platform-mini 의 EAV mini slice는 이 문제를 작게 다룬다. 여러 wide CSV를 mapping config로 표준 attribute에 맞춘 뒤, EAV long format으로 모으고, 다시 gold metric mart로 pivot/aggregate한다. 데이터는 모두 synthetic이고, 회사 코드·고객 데이터·실제 schema는 쓰지 않았다. 1. Scenario 서로 다른 공장/라인/벤더에서 비슷한 제조 지표 파일이 들어온다. 예: plant_a.csv: 설비ID, 생산수량, 불량수, 온도C, 압력kPa plant_b.csv: machine_id, output_units, defects, temp_f, pressure_bar vendor_d.csv: unit_name, made, scrap, deg_c, kpa 비즈니스적으로는 같은 지표를 보고 싶다. units_produced defect_count temperature_c pressure_kpa 문제는 source마다 column name과 unit이 다르다는 점이다. 2. Decision Pressure 단순 구현은 source마다 코드를 늘린다. if source == "plant_a": 생산수량을 units_produced로 읽는다 if source == "plant_b": output_units를 units_produced로 읽는다 temp_f를 섭씨로 변환한다 if source == "vendor_d": made를 units_produced로 읽는다 이 방식은 작게는 빨라 보이지만 source가 늘수록 문제가 된다. 새 파일 형식마다 pipeline code를 고쳐야 한다. column mapping과 transform logic이 섞인다. unit conversion이 흩어진다. quality check가 source별로 갈라진다. gold mart grain을 설명하기 어려워진다. 그래서 mapping은 config로 빼고, pipeline은 표준 attribute를 처리하게 만들었다. 3. Options option result risk source별 hard-coded parser 처음엔 빠름 source가 늘 때 code change 반복 모든 source를 wide table 하나로 합치기 보기 쉬움 sparse/heterogeneous column 폭발 EAV long format 이종 attribute를 표준 형태로 모음 pivot/quality 설계가 필요 full mapping DSL/rules engine 유연함 mini project에는 과함 이 프로젝트의 선택은 단순한 JSON mapping + EAV long + gold pivot이다. 4. Decision 각 source는 JSON config로 자신의 column을 표준 attribute에 매핑한다. source column -> standard attribute output_units -> units_produced temp_f -> temperature_c with f_to_c pressure_bar -> pressure_kpa with bar_to_kpa pipeline 흐름: wide CSVs -> mapping configs -> EAV long rows -> gold entity_daily_metrics -> quality checks -> catalog/lineage EAV row의 핵심 shape: entity_id business_date attribute value v

schema drift를 fail이 아니라 warn으로 둔 이유 데이터 파이프라인에서 source schema가 바뀌는 순간은 애매하다. 무조건 무시하면 운영자는 입력 구조가 바뀐 사실을 모른다. 반대로 모든 schema 변화를 실패로 처리하면, 정상적인 컬럼 추가까지 daily run을 막아버린다. manufacturing-data-platform-mini 에서는 이 문제를 작게 다뤘다. synthetic manufacturing CSV의 실제 header를 기준으로 schema_hash 를 만들고, previous successful run과 비교해 달라졌으면 schema_drift quality check를 warn 으로 남긴다. 단, required column이 빠져 silver/gold contract를 만들 수 없는 경우는 현재 ValueError 로 빠르게 실패한다. 1. Scenario 어느 날 source CSV에 새 컬럼이 추가된다. 기존 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date 새 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date,operator_id operator_id 는 아직 silver/gold mart에서 쓰지 않는다. 하지만 source 구조가 바뀐 사실은 기록되어야 한다. 2. Decision Pressure schema drift에서 중요한 질문은 단순히 "바뀌었나?"가 아니다. 바뀐 것을 운영자가 알 수 있는가? 정상적인 컬럼 추가 때문에 pipeline을 멈춰야 하는가? downstream gold mart contract가 조용히 바뀌지는 않는가? 이전 successful run과 지금 run의 schema identity를 비교할 수 있는가? 초기 구현에서는 한 가지 실제 버그가 있었다. schema_hash 가 고정된 required column 목록에 너무 묶여 있어서, 추가 컬럼이 들어와도 hash가 바뀌지 않았다. 즉 operator_id 가 추가되어도 drift가 보이지 않았다. 이 문제를 고치기 위해 read_rows 가 실제 CSV header를 반환하고, 그 실제 header 기준으로 schema_hash 를 계산하도록 바꿨다. 3. Options option result risk ignore drift pipeline은 계속 돈다 source 변화가 보이지 않음 fail every drift 변화에 강하게 반응 정상적인 컬럼 추가도 막음 warn and continue 변화가 보이고 run도 계속됨 warning을 inspect해야 함 auto-evolve silver/gold 새 컬럼을 바로 사용 가능 downstream contract가 조용히 바뀔 수 있음 full schema registry production에 가까움 mini slice에는 무거움 이 프로젝트의 선택은 warn and continue 다. 4. Decision 현재 contract는 이렇다. previous successful run이 없으면: schema_drift = pass baseline schema established current schema_hash == previous successful schema_hash: schema_drift = pass current schema_hash != previous successful schema_hash: schema_drift = warn quality_passed는 true 유지 run/lineage record에 previous/current schema_hash 저장 required column missing: V

source_hash로 같은 입력 재처리를 안전하게 skip하기 작은 데이터 파이프라인도 한 번만 실행된다고 가정하면 금방 거짓말이 된다. 실제로는 같은 파일을 다시 실행할 수 있다. 실패한 run을 재시도할 수도 있고, 과거 날짜를 backfill할 수도 있고, 운영자가 실수로 같은 입력을 다시 넣을 수도 있다. 이때 결과가 중복되면 gold metric은 더 이상 믿을 수 없다. 이 글은 개인 포트폴리오 프로젝트 manufacturing-data-platform-mini 에서 source_hash 를 이용해 같은 입력 재처리를 안전하게 skip하도록 만든 작은 설계 판단을 정리한 글이다. 데이터는 모두 synthetic이며, production platform이 아니라 검증 가능한 mini slice다. 1. Scenario 같은 business_date 의 제조/로봇 이벤트 파일을 다시 처리해야 하는 상황이 있다. 예: retry: 앞 run이 중간에 실패해서 다시 실행한다. backfill: 과거 날짜를 다시 채운다. operator mistake: 같은 파일을 실수로 다시 실행한다. 단순히 매번 append하면 같은 날짜의 gold metric이 중복될 수 있다. 2. Decision Pressure 단순 CSV pipeline은 보통 이렇게 끝난다. CSV 읽기 -> silver 만들기 -> gold 집계 -> 결과 저장 하지만 운영 관점에서는 질문이 생긴다. 이 입력은 전에 처리한 파일과 같은가? 같은 파일을 다시 돌리면 중복 output이 생기나? 다른 파일로 같은 날짜를 다시 돌리면 어떻게 구분하나? 어떤 run이 어떤 source에서 만들어졌나? 그래서 재실행을 판단할 identity가 필요했다. 3. Options option result problem always append 모든 run 결과를 계속 추가 같은 입력 재실행 시 중복 always overwrite 결과를 항상 덮어씀 이전 결과/원인 추적이 약함 skip by business_date only 같은 날짜면 무조건 skip 정정 파일을 반영할 수 없음 skip by dataset_id + business_date + source_hash 같은 입력만 no-op 정정 파일은 새 run으로 처리 가능 이 프로젝트의 Slice1은 마지막 선택지를 쓴다. 4. Decision 현재 mini pipeline은 입력 파일의 content hash를 source_hash 로 계산한다. idempotency key: dataset_id + business_date + source_hash 이미 성공한 run이 있으면 새로 처리하지 않고 기존 run을 재사용한다. same dataset_id same business_date same source_hash prior successful run exists => status = skipped 이 선택은 작지만 중요하다. 같은 파일 재실행: skip -> 중복 없음 같은 날짜의 정정 파일: source_hash가 다름 -> skip하지 않고 새 run으로 처리 가능 단, 여기서 조심해야 할 경계가 있다. Slice1은 다른 source_hash 를 새 run으로 처리할 수 있지만, 이전 gold partition을 원자적으로 교체하는 Iceberg-style overwrite까지 구현한 것은 아니다. 그 문제는 다음 Slice2의 business_date partition overwrite 주제다. 5. Evidence 관련 코드와 검증 evidence: src/manufacturing_data_platform/pipeline/lakehouse.py tests/test_lakehouse_pipeline.py VERIFICATION_LOG.md README.md 검증 로그: 2026-07-08 publication readiness check: pytest: 33 passed lakehouse JSON CLI: passed, status=processed, quality_passed=true EAV JS

Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The sentence is right. The trouble starts on the next page, because there are two fundamentally different ways to build that single place, and the industry has spent years arguing about them as if they were rivals. Road one is consolidation: bring the data together. Land everything in one governed store, in this era an open lakehouse, Apache Iceberg tables on object storage, and point every consumer at it. Road two is federation: leave the data where it lives and bring the access together instead. A query engine that speaks to your databases, warehouses, lakes, and applications in place, presenting one surface over many sources, with no copies made. I work at Dremio, a company whose platform is built on the conviction that this is a false choice, that the right architecture uses both roads with judgment, and I will declare that bias now and then earn it with an honest treatment. Because the truth practitioners live is messier than either camp's marketing: federation without a lakehouse hits performance and scale ceilings, a lakehouse without federation spends years and fortunes migrating the long tail, and the teams that win treat the two as phases and partners rather than competitors. So this article is the full playbook. What federation and the lakehouse each actually are, mechanically. The honest strengths and limits of each, including the failure modes their advocates gloss over. A concrete decision framework for when each one carries a workload. The lifecycle pattern that connects them, federate first, promote deliberately. And the unified architectures, mine included, that put both behind one governed door, which matters more than ever now that the consumers walking through that door increasingly are AI agents. Why Unify at All: The Cost of the Status Quo Before the two roads, the destination deserves a paragraph, because

Let's be real: our personal medical history is a mess. It’s a chaotic mix of PDF lab results, grainy scans of prescriptions, and cryptic Electronic Medical Records (EMR) scattered across different hospital portals. If you’ve ever tried to remember exactly when a specific symptom started or how your cholesterol has trended over the last decade, you know the "search" struggle is real. In this guide, we are moving beyond simple folders. We are architecting a Personal Health Knowledge Base using a modern Vector Database and RAG (Retrieval-Augmented Generation) pipeline. We’ll leverage Qdrant for high-performance similarity search, Unstructured.io for complex document parsing, and Sentence-Transformers to turn 10 years of medical jargon into searchable embeddings. By the end of this post, you'll have a system capable of cross-year symptom correlation and instant medical history retrieval. The Architecture: From Pixels to Insights 🏗️ The biggest challenge with medical records isn't storage; it's ingestion . Medical PDFs are notoriously difficult to parse because they often contain nested tables and checkboxes. Our pipeline handles this by isolating the layout before embedding. graph TD A[Raw Medical Data: PDFs, Scans, EMRs] --> B[Unstructured.io: Partitioning & OCR] B --> C[Text Chunking & Cleaning] C --> D[Sentence-Transformers: Vector Embedding] D --> E[(Qdrant Vector DB)] F[User Query: 'Show me my blood sugar trends since 2015'] --> G[FastAPI Interface] G --> H[Query Embedding] H --> I[Vector Search in Qdrant] I --> J[Contextual Results + LLM Synthesis] J --> K[Actionable Health Insight] Prerequisites 🛠️ To follow along, you'll need: Python 3.9+ Unstructured.io : For the heavy lifting of PDF/Image parsing. Qdrant : Our vector engine (run it via Docker: docker run -p 6333:6333 qdrant/qdrant ). Sentence-Transformers : To generate local embeddings without sending sensitive data to the cloud. FastAPI : To wrap it all in a slick API. Step 1: Parsing the Chaos with Unstructu

Original Japanese article : Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). Most of the time, when working with Iceberg tables, we reach for PyIceberg or Spark. I'm no exception, and honestly there were parts of the PyIceberg configuration — rest.sigv4-enabled , rest.signing-name , warehouse — that I understood only vaguely. Iceberg defines a standard called the Iceberg REST Catalog Open API specification , and AWS implements it through two separate endpoints: The AWS Glue Iceberg REST endpoint ( https://glue.<region>.amazonaws.com/iceberg ) The Amazon S3 Tables Iceberg REST endpoint ( https://s3tables.<region>.amazonaws.com/iceberg ) If two implementations follow the same spec, sending the same requests to both and comparing the results should reveal what's actually different between them. In this article, I'll bypass clients like PyIceberg entirely and hit the REST API directly to explore the differences between the two endpoints. To state the conclusion up front: Even though both implement the same Iceberg REST Catalog specification, Glue is designed as an "entry point to multiple catalogs," while S3 Tables is designed as an "entry point to a single table bucket." That difference is visible just by looking at the URL paths. I previously wrote about the relationship between S3 Tables and Glue Data Catalog in another article — worth a read alongside this one: Does Amazon S3 Tables Replace AWS Glue Data Catalog? Understanding Their Relationship What Is the Iceberg REST Catalog? The Iceberg REST Catalog is a specification that standardizes Iceberg catalog operations as an HTTP API. It's published as an OpenAPI definition (YAML), and any catalog that conforms to it can be accessed the same way from clients such as PyIceberg, Spark, and Trino. The key points of the spec are: URL paths follow a pattern like GET /v1/{prefix}/namespaces , where {prefix} is a free-form segment Clients first call

Most "Debezium vs managed tool" articles get the question wrong. They frame it as a product bake-off, feature grid included, and declare a winner. But if you've actually run change data capture in production, you know the real decision isn't which tool captures a transaction log better. They mostly read the same logs the same way. The real decision is who operates everything that sits around the capture, and whether that work is a good use of your team's time. That's a build-vs-buy question, not a product question. This post is a framework for answering it for your own situation. First, let's kill an outdated assumption A lot of Debezium criticism floating around is two or three years stale, and if you repeat it in 2026 you'll get corrected fast. So let's set the record straight before we compare anything. Debezium is no longer just “the thing you run with Kafka Connect.” In recent Debezium 3.x releases, the project has become much more flexible than the old tutorials suggest. Today, you have several deployment options: Kafka Connect , the classic setup, which gives you the Kafka ecosystem, distributed fault tolerance, durable schema history, and access to Kafka Connect sink connectors. Debezium Server , a standalone application that streams changes to systems like Amazon Kinesis, Google Cloud Pub/Sub, Apache Pulsar, Redis Streams, or NATS JetStream without requiring Kafka. Debezium Management Platform , which builds on Debezium Server and the Debezium Operator to provide a higher-level way to configure and manage CDC pipelines in Kubernetes-style environments. Embedded usage , where you run Debezium Engine inside your own application. Recent Debezium releases also added framework support such as the Quarkus extension. A few more things are worth knowing so the comparison is fair: Kafka 4.x runs in KRaft mode, and ZooKeeper mode has been removed. “You need to babysit ZooKeeper” is no longer true for a modern Kafka deployment. Debezium's default remains at-least-once

We’ve all been there: staring at a stack of printed lab results or a folder full of cryptic report_final_v2_NEW.pdf files, trying to remember if our cholesterol was higher or lower two years ago. For developers, this isn't just a filing problem—it's a data engineering challenge. In the world of healthcare, data is messy, siloed, and often locked in "unstructured" formats. To build a truly personal Electronic Health Record (EHR) system, we need more than just a folder; we need a RAG (Retrieval-Augmented Generation) pipeline that can parse PDFs, map them to the FHIR (Fast Healthcare Interoperability Resources) standard, and provide natural language insights. In this guide, we’ll leverage Unstructured.io , Milvus , and DuckDB to turn chaotic medical PDFs into a queryable, structured knowledge base. The Architecture: From Raw Pixels to Structured Insights Before we dive into the code, let’s look at how the data flows from a messy lab report to a structured answer. graph TD A[Unstructured PDF Reports] --> B[Unstructured.io Partitioning] B --> C{Data Split} C -->|Textual Context| D[Milvus Vector DB] C -->|Tabular Data| E[DuckDB Structured Storage] D --> F[LangChain RAG Engine] E --> F G[User Query: Is my glucose trending up?] --> F F --> H[FHIR-Formatted Response] Why this stack? Unstructured.io : The gold standard for handling "ugly" PDFs (tables, headers, and nested lists). Milvus : A high-performance vector database built for scale. DuckDB : Perfect for running complex analytical SQL queries on the extracted "structured" parts of our medical data. FHIR Standard : To ensure our data follows global healthcare interoperability rules. Prerequisites Make sure you have your environment ready: pip install langchain milvus unstructured[pdf] duckdb openai Step 1: Extraction with Unstructured.io Medical PDFs often contain complex tables. Standard PDF parsers usually fail here. We’ll use unstructured to partition the document into logical elements. from unstructured.partition.pdf

Introduction As data volumes continue to grow, running aggregation queries directly on raw datasets becomes increasingly expensive. Business dashboards, analytics platforms, and reporting systems often execute the same calculations repeatedly—such as total sales, daily active users, page views, or revenue trends. While ClickHouse® is designed to process analytical workloads at remarkable speed, repeatedly scanning billions of records still consumes valuable CPU, memory, and storage resources. This is where AggregatingMergeTree proves its value. Rather than calculating aggregates every time a query is executed, AggregatingMergeTree stores intermediate aggregation states that are merged automatically in the background. This approach allows analytical queries to read compact, pre-aggregated datasets, resulting in dramatically faster response times and reduced infrastructure costs. In this guide, you'll learn how AggregatingMergeTree works, why aggregate states matter, how to build an automated aggregation pipeline using Materialized Views, and when this engine is the right choice for your ClickHouse® workloads. What is AggregatingMergeTree? AggregatingMergeTree is a specialized ClickHouse® table engine designed to store aggregate function states instead of raw records. Unlike the standard MergeTree engine, which stores every inserted row, AggregatingMergeTree keeps partially aggregated values that ClickHouse combines during background merge operations. This significantly reduces the amount of data that must be processed when generating analytical reports. Because much of the computational work happens during data ingestion, dashboards and reporting applications can retrieve summarized information much more efficiently. Typical scenarios include: Sales reporting Website traffic analytics Financial summaries IoT sensor monitoring Business KPI dashboards Application observability metrics Why Use AggregatingMergeTree? Imagine an online marketplace processing millions of tr

📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Clusters The BOOTSTRAP_TIMEOUT Mystery Fixing It with AWS PrivateLink How We Structure the Terraform Before the build story, here's the destination. This is the target-state data architecture we designed the whole platform toward — the three principles that shaped every later decision, and the Unity Catalog governance model that keeps production data safe from human hands. The rest of this series is a build log: workspaces, RBAC, compute, the networking rabbit hole, the Terraform layout. But every one of those decisions was made in service of a target picture we drew first . This post is that picture — the "to-be" architecture, not the scaffolding we happened to have up on any given week. It's built on three things Databricks basically hands you if you lean into them: the Lakehouse (one store, ACID tables, no separate warehouse to sync), the Medallion architecture (raw → cleaned → integrated → business, each layer a promotion), and Unity Catalog as the single governance plane across all of it. The interesting part isn't reciting those three buzzwords — it's the specific way we wire them so that prod, dev, and analyst sandboxes never step on each other. Three principles, and everything follows Almost every concrete rule later in this series is a consequence of one of these three. 1. Nobody touches production by hand. Create, update, delete in prod data happens only through an automated, code-reviewed pipeline running as a service principal. Human accounts don't get write on prod — not analysts, not engineers, not admins. The blast radius of a bad afternoon is capped at whatever a person can do with read-only. This one principle is why the whole "promote" flow later exists. 2. Never copy production to look at it. If an analyst wants to explore the gold layer, they read it in p

Introduction As applications grow, traditional relational databases such as MySQL may struggle with analytical workloads involving millions of records and complex aggregations. While MySQL excels at Online Transaction Processing (OLTP), ClickHouse® is purpose-built for Online Analytical Processing (OLAP), enabling lightning-fast analytical queries on massive datasets. Migrating data from MySQL to ClickHouse® allows organizations to build high-performance reporting systems, dashboards, and real-time analytics without impacting transactional workloads. In this guide, you'll learn several approaches to migrate data from MySQL to ClickHouse®, along with their advantages, limitations, and ideal use cases. Why Migrate from MySQL to ClickHouse®? MySQL and ClickHouse® are designed for different workloads. Feature MySQL ClickHouse® Storage Model Row-based Columnar Best For Transactions (OLTP) Analytics (OLAP) Query Speed Fast for row lookups Extremely fast for large scans Aggregation Performance Moderate Extremely fast Scalability Primarily Vertical Optimized for analytical scaling Typical Use Cases Applications and transactional systems Reporting, dashboards, and analytics Migrating from MySQL to ClickHouse® makes sense when: Analytical queries are becoming slow in MySQL. You need real-time dashboards over large datasets. Reporting queries are impacting your production database. You regularly process millions or billions of rows. Migration Architecture MySQL │ ▼ Export / Synchronization │ ▼ Data Transformation │ ▼ ClickHouse® │ ▼ Dashboards / Analytics Migration Methods There are multiple ways to migrate data depending on your requirements. Method 1: CSV Export and Import (Recommended for Beginners) This is the simplest approach for performing a one-time migration of historical data. Step 1: Export Data from MySQL Run the following command inside MySQL: SELECT * INTO OUTFILE '/tmp/employees.csv' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY ' \n ' FROM employ

I Built a Free Apache Kafka Course from Scratch — Here's the Full Curriculum (and What I Got Wrong) I spent months building a free Apache Kafka course covering everything from first principles to a real-time analytics platform final project. No paywall. No "premium tier." 9 modules, 470 minutes of content, completely free. Here's the full syllabus, the Python code that actually works, and the honest mistakes I made building the curriculum — so you don't repeat them. Why I Built This Every time someone asked me "how do I learn Kafka?", I sent them to the same 3 places: The official Confluent docs (dense, assumes you already know what you're doing) A $15 Udemy course that spends Module 1 explaining what a computer is A YouTube playlist where half the videos are deleted None of them answered the real question beginners have: why does Kafka exist, and what problem does it actually solve before I write a single line of code? That's the gap I built for. The Problem With Most Kafka Tutorials Most tutorials start with: "Kafka is a distributed event streaming platform..." And then they immediately show you a Docker Compose file with 6 services. Beginners copy-paste it, something breaks, they don't know why, they quit. The real problem is that Kafka is an answer to a specific architectural problem — and if you don't understand the problem first, the solution makes no sense. So Module 1 and 2 of this course don't touch Kafka at all. They build the problem statement from scratch. The Full Syllabus (9 Modules, 470 Minutes) Module 1: Introduction to Kafka — 35 min Not "what is Kafka" — but why event streaming exists at all. What breaks in traditional request-response architectures at scale. Module 2: The Problem Statement — 30 min A real-world scenario: you're building an e-commerce platform. Orders, inventory, notifications, analytics — all tightly coupled. What happens when one service goes down? This module makes the pain visceral before Kafka enters the picture. Module 3: How

As developers, data engineers, or analysts, we’ve all been there: you download a massive database export, a logging stack dump, or a transaction archive, only to find it's a multi-gigabyte JSON file. You try to import it into a spreadsheet or run it through a standard online converter, and boom—your browser tab freezes, crashes, or shows the dreaded "Out of Memory" screen. Even worse, if you try to use standard cloud-based online tools, you might have to wait for a 500MB upload to complete, only to hit a rigid file-size cap or, worse, compromise sensitive data privacy by uploading corporate logs or database records to a third-party server. In this guide, we will explore: Why large JSON files crash standard parsers (the V8 heap limit problem). How streaming architectures solve this by reading data chunk-by-chunk. NDJSON (JSON Lines) vs. JSON Arrays and how to stream them. A browser-native, 100% offline tool to convert large JSON to CSV instantly: Parsify's Large JSON Stream Converter . How to implement your own basic browser-based JSON streaming parser in JavaScript. 1. The Anatomy of a Memory Crash (Why JSON.parse Fails) If you are using JavaScript or Node.js, the simplest way to read and parse a JSON file is to load the file into memory and run JSON.parse(). const fs = require ( ' fs ' ); // Naive approach: Will crash on a 1GB+ file fs . readFile ( ' database-dump.json ' , ' utf8 ' , ( err , data ) => { if ( err ) throw err ; // POINT OF FAILURE: V8 Heap Out of Memory const records = JSON . parse ( data ); records . forEach ( record => { // Process record... }); }); This works fine for small config files. But once your JSON file reaches 100MB, 500MB, or 1GB+, this approach is guaranteed to trigger a fatal crash: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory Why does this happen? The String Duplication Overhead: When you load a 1GB file into memory, you first allocate ~1GB of RAM for the raw text string. The

Apache Iceberg looked like the answer to everything when we first adopted it. Open format, ACID transactions, time travel, schema evolution. We migrated our Hive tables, ran a few queries, and felt good about life. Three months later, our S3 costs doubled. Queries that used to take 10 seconds were taking 4 minutes. Metadata operations were timing out. Nobody on the team could explain why. That was the beginning of a real education in how Iceberg actually behaves in production. This post covers what I wish someone had told us before we went all-in. The Small Files Problem Is Not Optional Iceberg is append-friendly by design. Every micro-batch write, every streaming insert, every incremental load creates new Parquet files. Each file also gets its own metadata entry. After a week of hourly loads, you might have 10,000 files in a single partition where you wanted 20. The result: Iceberg's metadata layer has to plan queries across thousands of file manifests. Planning takes longer than execution. Your 10-second query becomes a 4-minute query, and your users start filing tickets. Fix: automate compaction from day one. In Spark, compaction is called rewrite_data_files . The basic call looks like this: -- Run this on a schedule, not on-demand CALL iceberg_catalog . system . rewrite_data_files ( table => 'analytics.events' , strategy => 'binpack' , options => map ( 'target-file-size-bytes' , '134217728' , -- 128MB target per file 'min-input-files' , '5' -- only compact if 5+ small files exist ) ) Target file size of 128MB to 512MB is the practical sweet spot. Smaller than that, you still have too many files. Larger, and your query engines cannot parallelize reads efficiently. If you are not using Spark, PyIceberg exposes compaction through the table maintenance API (as of 0.7.x). For Flink or Trino-only shops, schedule compaction as a separate Spark job. Yes, it is annoying, but it is the right call. Hidden Partitioning Is the Feature You Are Probably Ignoring Old Hive parti

Introduction When a query runs in ClickHouse®, the database does much more than simply read data and return results. Before execution begins, ClickHouse® parses the SQL statement, analyzes it, applies optimizations, and builds an execution plan that determines the most efficient way to process the query. Understanding query execution plans is one of the most valuable skills for anyone working with ClickHouse®. They provide visibility into how queries are executed, helping you identify bottlenecks, validate optimization efforts, and troubleshoot performance issues. In this article, we'll explore how ClickHouse® generates execution plans, the different EXPLAIN modes, and how to interpret them for better query optimization. Why Query Execution Plans Matter A SQL query defines what data you want, but it doesn't explain how the database retrieves it. Consider the following query: SELECT country , count () FROM events GROUP BY country ; Although the query looks simple, ClickHouse® must determine: Which data parts to read Whether primary indexes can reduce the scan If data skipping indexes can be used How aggregation should be performed Whether parallel execution is possible How intermediate results should be merged A query execution plan provides answers to these questions, making it an essential tool for performance tuning. The ClickHouse Query Lifecycle Every query passes through several stages before producing results. The lifecycle typically looks like this: SQL Query │ ▼ Parser │ ▼ Analyzer │ ▼ Optimizer │ ▼ Query Plan │ ▼ Execution Pipeline │ ▼ Results Each stage plays an important role: Parser validates SQL syntax. Analyzer resolves tables, columns, and expressions. Optimizer applies query optimizations. Query Plan determines the logical execution steps. Pipeline distributes work across multiple threads. Execution processes the data and returns the results. Understanding this workflow makes execution plans much easier to interpret. Introducing the EXPLAIN Statement

I got a take-home assignment last year from a company I was genuinely excited about. "Should take about four hours," the recruiter said. Build an ingestion pipeline, model the data, write tests, document your design decisions, and prepare a 15-minute presentation walkthrough for the panel. Four hours. I laughed, closed my laptop, and started on it the next morning like it was a sprint. Sixteen hours later I had something I was proud of. Clean pipeline, solid tests, real documentation. I submitted it on a Sunday night. Monday I got a form rejection. No notes. No feedback. Not even which stage I failed. Just "we've decided to move forward with other candidates" and a link to their Glassdoor page. That was the moment I stopped pretending take-homes are assessments. They're consulting gigs. Unpaid ones. The Scope Creep Nobody Talks About Five years ago, a data engineering take-home was a focused exercise. Model this dataset into a star schema. Write a few SQL transforms. Maybe a short README. Two to four hours, tops. Bounded, reasonable, and actually useful for evaluating how someone thinks about data. That version is dead. Today, 68% of companies use take-home tests, up 12% year over year. And the scope has quietly ballooned into something unrecognizable. Full pipeline implementations. Test suites with coverage thresholds. Documentation that reads like a design doc. A presentation follow-up where you defend your architecture to a panel. We're talking 10 to 20 hours of work, routinely, for a role you haven't been offered. Industry best practice caps take-homes at 90 minutes of expected effort. The reality? Candidates consistently take 2x longer than company estimates to reach submission quality. That "four-hour" assignment is an eight-hour assignment. That "weekend project" is a week of evenings. And 25% of companies are still handing these out like they're reasonable asks. Here's the part that makes my eye twitch: 71% of engineering leaders openly say take-homes no lon