标签：#data

The Cybersecurity and Infrastructure Security Agency (CISA) has issued a postmortem on a data leak in which a contractor published dozens of internal CISA credentials -- including AWS Govcloud keys -- in a public GitHub repository for almost six months before being notified by KrebsOnSecurity. Experts say the gaps identified in the agency's initial response provide important lessons that all security teams should absorb.

A panel on data ownership challenged the definition of "ownership," arguing it must extend beyond simple account control to include structural independence, interoperability, and community governance. Speakers like Zenna Fiscella, Paul Frazee, Boris Mann, and Robin Berjon emphasised the need for shared standards, unbundled platforms, and better tools to support user sovereignty. By Olimpiu Pop

If you ask me what part of the system I protect the most, the answer is the database. I've been writing software alone for twenty-four years, and across every platform I've built, the rule has stayed the same: the web servers can take whatever you throw at them, the batches can be rebuilt, but the database has to stay idle on purpose. Not because I love idle databases, but because the day a database actually starts to struggle is a day with very few good options. This article is about what "keep the database idle on purpose" actually means in practice, and about one particular kind of graph that, in my experience, almost nobody is watching. The three layers and what each of them gets I think of a production system as having three tiers, and each tier gets a different rule. The web server tier can be horizontally scaled. If load grows, you add machines. If something is wrong, you take a machine out of the pool, and the others handle it. Failures here are visible immediately, and they're cheap to recover from. The batch server tier can be scaled up or out depending on the work. A batch that's too slow can be split. A batch that crashes can be retried. End users don't see batch servers, so a stuck batch is a problem for me and not for them. Some headroom up here is fine. The database tier is the one I treat completely differently. The database is not where you absorb load. The database is what you protect from load. The reason is simple: the other tiers can be rebuilt or re-scaled. The database is the irreplaceable record. If it slows down, everything slows down. If it falls over, you don't have many minutes before the rest of the stack notices. So my rule for the database is: keep it idle. Not idle in the sense of "doing nothing." Idle in the sense of "running well below its capacity, at all times, so that any extra load it picks up has somewhere to go." For more than a decade I ran a large appliance-grade database where I kept the load average below 1 at all times. N

Hackathons are intense. You've got a brilliant idea, a tight deadline, and often, limited sleep. The last thing you want is to spend half your precious time wrestling with database boilerplate, ORM setup, or SQL query syntax. This guide will walk you through building a functional Node.js backend for your hackathon project, focusing on speed and minimal friction, so you can spend more time on your core idea. The Hackathon Backend Challenge Typically, setting up a database and its interaction layer involves several steps: Schema Definition: Deciding on tables/collections, fields, types, and relationships. ORM/Driver Setup: Installing and configuring your database driver or ORM (e.g., Mongoose, Sequelize). Model Creation: Translating your schema into code, often with verbose syntax. Query Writing: Crafting SELECT , INSERT , UPDATE , DELETE statements or ORM methods for every data operation. Debugging: Fixing typos, schema mismatches, and complex join logic. This process, while fundamental, eats up valuable time that could be spent on features, UI, or even sleep. For a hackathon, you need to iterate rapidly, and database interactions should be the least of your worries. Strategy 1: Embrace Simplicity For many hackathon projects, you don't need highly optimized, production-grade queries from day one. You need functional queries that work quickly. Focus on getting data in and out reliably. Strategy 2: Natural Language for Data Modeling Instead of writing verbose schema definitions, think about how you'd describe your data to a non-technical person. For example, if you're building a task management app, you might say: "We need a collection of tasks. Each task has a title, a description, a due date, and a status (like 'pending' or 'completed'). Each task belongs to one user." This natural language description contains all the essential information for a data model, including relationships and field types. Strategy 3: Expressive Querying Similarly, when you need to fetch dat

Jake Lazaroff discussed the AT Protocol as a framework for distributed applications beyond social networking. He emphasised a local-first architecture where users maintain data in PDSs while leveraging shared infrastructure for synchronisation and updates. The presentation included experiments showcasing collaborative tools and highlighted the benefits of reduced reliance on app-specific backends. By Olimpiu Pop

Introdução Validar dados é uma responsabilidade que pode ficar na aplicação, no banco de dados, ou em ambos. Deixar tudo na aplicação é arriscado: diferentes sistemas podem acessar o mesmo banco, migrações podem rodar diretamente, um bug pode deixar passar um valor inválido. Constraints são regras definidas no próprio banco de dados — uma camada de proteção que age independente de quem está escrevendo os dados. PRIMARY KEY A chave primária identifica cada linha de forma única. Ela combina duas restrições implicitamente: NOT NULL e UNIQUE . Nenhuma linha pode ter o mesmo valor de chave primária, e nenhuma pode tê-la nula. CREATE TABLE clientes ( id INT PRIMARY KEY , nome VARCHAR ( 100 ) NOT NULL ); Quando a chave primária envolve mais de uma coluna, ela é declarada separadamente: CREATE TABLE matriculas ( aluno_id INT , curso_id INT , PRIMARY KEY ( aluno_id , curso_id ) ); Na maioria dos bancos, é comum usar uma chave primária auto-incremental para não precisar gerenciar os IDs manualmente: -- PostgreSQL id SERIAL PRIMARY KEY -- MySQL id INT AUTO_INCREMENTPRIMARY KEY -- SQL padrão (suportado por ambos) id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY FOREIGN KEY A chave estrangeira garante integridade referencial : um valor só pode existir numa coluna se ele existir como chave primária na tabela referenciada. É o que torna os relacionamentos entre tabelas confiáveis. CREATE TABLE pedidos ( id INT PRIMARY KEY , cliente_idINT REFERENCES clientes ( id ) ); Tentar inserir um pedido com cliente_id = 99 quando não existe cliente com esse id resulta em erro imediato. O banco rejeita a operação antes mesmo de ela chegar ao disco. O comportamento quando o registro referenciado é deletado pode ser configurado: CREATE TABLE pedidos ( id INT PRIMARY KEY , cliente_id INT REFERENCES clientes ( id ) ON DELETE CASCADE -- deleta os pedidos junto com o cliente ON UPDATE CASCADE -- atualiza o cliente_id se o id do cliente mudar ); As opções disponíveis são: Opção Comportamento RESTRICT

Introdução Consultas individuais respondem perguntas como "qual o email do cliente 42?". Mas as perguntas mais valiosas em qualquer sistema são de outro tipo: "qual o produto mais vendido?", "qual a receita média por pedido?", "quantos clientes se cadastraram esse mês?". Para responder isso, o SQL oferece as funções de agregação — operações que recebem um conjunto de linhas e devolvem um único valor resumido. Para os exemplos a seguir, considere esta tabela: pedidos: | id | cliente | produto | categoria | quantidade | valor | |----|------------|-------------|--------------|------------|--------| | 1 | Ana Lima | Notebook | Eletrônicos | 1 | 3500.00| | 2 | Ana Lima | Mouse | Periféricos | 2 | 80.00| | 3 | Bruno Melo | Teclado | Periféricos | 1 | 150.00| | 4 | Bruno Melo | Notebook | Eletrônicos | 1 | 3500.00| | 5 | Carla Nunes| Monitor | Eletrônicos | 2 | 1200.00| | 6 | Carla Nunes| Mouse | Periféricos | 1 | 80.00| As Funções de Agregação COUNT Conta o número de linhas — ou de valores não nulos em uma coluna específica. -- Total de pedidos SELECT COUNT ( * ) AS total_pedidosFROM pedidos ; -- Resultado: 6 -- Clientes distintos que fizeram pedidos SELECT COUNT ( DISTINCT cliente ) AS clientes_unicosFROM pedidos ; -- Resultado: 3 COUNT(*) conta todas as linhas, incluindo as que têm nulos. COUNT(coluna) conta apenas as linhas onde aquela coluna não é nula. COUNT(DISTINCT coluna) conta valores únicos — útil para saber quantos clientes, produtos ou categorias distintos aparecem no resultado. SUM Soma os valores de uma coluna numérica. -- Receita total SELECT SUM ( valor ) AS receita_total FROM pedidos ; -- Resultado: 8510.00 -- Total de itens vendidos SELECT SUM ( quantidade ) AS itens_vendidos FROM pedidos ; -- Resultado: 8 AVG Calcula a média aritmética dos valores. -- Valor médio por pedido SELECT AVG ( valor ) AS ticket_medio FROM pedidos ; -- Resultado: 1418.33 AVG ignora valores nulos automaticamente — calcula a média apenas sobre os registros que têm valor preenchid

Cargar 112.647 filas en una tabla de hechos debería tardar segundos. A mí me tardaba más de 25 minutos, y acababa cancelando la query. Los datos estaban bien, el SQL estaba bien, las dimensiones se poblaban sin problema. El culpable era otro, y descubrirlo fue la parte más instructiva de todo el proyecto. Todo esto surgió construyendo un Data Warehouse en estrella sobre datos reales de e-commerce: no una tabla bonita para hacer un SELECT * , sino un modelo dimensional completo, reproducible desde cero, capaz de responder preguntas de negocio de verdad. El dataset Trabajé con el Brazilian E-Commerce Public Dataset by Olist : pedidos reales de un marketplace brasileño entre septiembre de 2016 y octubre de 2018. Son 9 CSV relacionados entre sí: 99.441 pedidos y 112.650 líneas de venta 103.886 pagos y 104.719 reseñas 32.951 productos, 3.095 vendedores 1.000.163 registros de geolocalización Y con trampas de datos reales que hay que ver antes de que te muerdan: Un pedido puede tener varios pagos y varias reseñas. Si los unes tal cual a la tabla de hechos, duplicas ventas . Es el error clásico y silencioso: los totales salen inflados y nadie se entera. customer_id no es un cliente. Olist crea uno por cada pedido; la persona real es customer_unique_id . Contar mal aquí te cambia el KPI: hay 99.441 cuentas frente a 96.096 personas. El CSV de productos trae una errata en la cabecera ( product_name_lenght , con "lenght"). Si tu esquema la escribe bien y cargas por interfaz gráfica (que empareja por nombre ), esas columnas se quedan vacías sin que nadie avise. El proceso Monté una arquitectura en capas: CSV → staging → modelo dimensional → vistas → análisis , todo en cuatro scripts ejecutables en orden y idempotentes (el esquema se recrea desde cero, se puede relanzar mil veces). El modelo es un star schema : una tabla de hechos fact_sales al grano de línea de producto dentro de un pedido , y cinco dimensiones (cliente, producto, vendedor, pago y fecha), con claves sustitutas,

No hacking — a passive scan only looks at what your browser already downloads when it opens a page. Here's what I found: 6 of 15 load their Supabase database directly client-side. The public API key sits in the page source. That's fine if Row-Level Security is configured right — but it's one wrong setting away from "anyone can read the whole table." 14 of 15 ship no Content-Security-Policy — a simple, high-value hardening against script injection, almost always missing. Is this theoretical? No. Two apps I audited with the owner's permission: A social app: the profiles table — user names, cities, and a password hash — readable by a logged-out stranger. Closed in an afternoon. A paid learning app: 155 paid study sheets and 4,872 answers were readable by anyone, with no account and no subscription — its entire paid catalogue, a single API call away. The paywall lived only in the front-end; the database served everything to everyone. Loading Supabase in the browser isn't the mistake. Not enforcing access in the database (RLS) is. And the tools you build with won't tell you — they'll happily ship it. If you built something on Lovable / Bolt / Replit with real users (or paying ones), it's worth 60 seconds to check what a stranger can already see. I made a free tool that runs the surface check (passive, no signup): sealdy.dev Happy to answer questions on how RLS leaks happen and how to lock them down.

If you’ve ever struggled with CALCULATE() or wondered why SUMX() behaves differently from SUM() , this guide is for you. DAX (Data Analysis Expressions) is the language that powers Power BI , Analysis Services , and Power Pivot — enabling dynamic calculations, filtering, and time intelligence. Below is a categorized cheat sheet of essential DAX functions , plus examples showing how to use each in real-world Power BI scenarios. Filtering & Context These functions control how filters are applied and evaluated in your calculations. Function Example Description CALCULATE() CALCULATE(SUM(Sales[Amount]), Region[Name] = "Nairobi") Changes filter context to calculate total sales for Nairobi. FILTER() FILTER(Sales, Sales[Amount] > 10000) Returns a table filtered by condition. ALL() CALCULATE(SUM(Sales[Amount]), ALL(Region)) Ignores filters on Region. REMOVEFILTERS() CALCULATE(SUM(Sales[Amount]), REMOVEFILTERS(Region)) Removes filters from Region. VALUES() VALUES(Customer[City]) Returns unique list of cities. SELECTEDVALUE() SELECTEDVALUE(Product[Category], "All") Returns selected category or “All” if none. TREATAS() TREATAS(VALUES(Temp[City]), Customer[City]) Applies one table’s values as filters on another. KEEPFILTERS() CALCULATE(SUM(Sales[Amount]), KEEPFILTERS(Product[Category] = "Electronics")) Keeps existing filters and adds new ones. ALLSELECTED() CALCULATE(SUM(Sales[Amount]), ALLSELECTED(Region)) Respects user selections in visuals. ALLEXCEPT() CALCULATE(SUM(Sales[Amount]), ALLEXCEPT(Sales, Sales[Year])) Removes all filters except Year. Aggregation Summarize or aggregate data across rows or columns. Function Example Description SUM() SUM(Sales[Amount]) Adds all sales amounts. AVERAGE() AVERAGE(Sales[Amount]) Calculates mean value. COUNT() COUNT(Customer[ID]) Counts non-blank entries. COUNTROWS() COUNTROWS(Sales) Counts rows in a table. DISTINCTCOUNT() DISTINCTCOUNT(Customer[ID]) Counts unique customers. MIN() MIN(Sales[Amount]) Finds smallest sale. MAX() MAX(Sales[Amo

We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana Soumyadeep Dey Soumyadeep Dey Soumyadeep Dey Follow Jul 12 We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana # devchallenge # weekendchallenge # snowflake 5 reactions 1 comment 16 min read

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기 이전 글에서는 같은 source_hash 가 다시 들어왔을 때 기존 successful run을 재사용하는 idempotency를 다뤘다. 하지만 재처리에는 두 종류가 있다. 1. 같은 입력이 다시 들어온 경우 -> skip이 맞다. 2. 같은 business_date의 정정 입력이 들어온 경우 -> skip하면 안 된다. -> 같은 날짜의 gold 결과를 중복 없이 교체해야 한다. manufacturing-data-platform-mini 의 B5 slice는 두 번째 문제를 아주 작게 다룬다. 전체 Spark pipeline을 만든 것이 아니다. gold_daily_metrics Iceberg table 하나를 local Spark에서 만들고, business_date partition overwrite와 snapshot evidence만 검증했다. Scenario 이미 아래 gold row가 있다. business_date=2026-06-29 plant-a / line-1 / gearbox-a units_produced=120 defect_count=3 나중에 같은 business_date=2026-06-29 에 대한 정정 source가 들어온다. 운영자가 원하는 것은 append가 아니다. 원하지 않는 상태: 2026-06-29 old row 2026-06-29 corrected row -> 같은 날짜 결과가 중복됨 원하는 상태: 2026-06-29 corrected row만 남음 2026-06-30 같은 다른 날짜 partition은 그대로 유지됨 재처리 전후 snapshot evidence가 남음 그래서 이 slice의 질문은 이렇다. 같은 business_date의 정정 source를 처리할 때, gold table에서 해당 날짜 partition만 중복 없이 교체하고, 어떤 run이 어떤 Iceberg snapshot을 만들었는지 남길 수 있는가? Decision Pressure Slice1의 CSV pipeline은 already-successful source를 안전하게 skip할 수 있다. dataset_id + business_date + source_hash 이 key가 같으면 같은 입력이다. 다시 계산해도 같은 결과이므로 기존 run을 재사용한다. 하지만 source_hash 가 달라졌다면 의미가 다르다. same business_date different source_hash 이건 retry가 아니라 correction이다. CSV run-folder 방식에서는 새 run output을 만들 수는 있지만, "현재 gold table에서 해당 날짜를 원자적으로 교체한다"는 table-level 의미가 약하다. Iceberg를 붙이는 이유는 여기 있다. source_hash -> 같은 입력인지 판단하는 idempotency key business_date partition -> 정정 시 교체할 gold table 범위 snapshot_id -> table commit의 evidence 즉 Spark/Iceberg는 도구 이름을 추가하려고 붙인 것이 아니라, 재처리 상태 전이를 더 명확히 표현하기 위해 붙였다. Options Option 장점 문제 판단 same source면 항상 재계산 단순함 retry 때 불필요한 commit이 계속 생김 제외 corrected source를 append 구현 쉬움 같은 날짜 gold row가 중복될 수 있음 제외 whole-table overwrite 단순함 다른 날짜 partition까지 지울 위험 제외 business_date partition overwrite correction 범위가 명확함 Spark/Iceberg 설정과 test가 필요 선택 MERGE/upsert 강력함 이번 skeleton에 과함 backlog 이번 구현은 DataFrameWriterV2.overwritePartitions() 를 사용했다. corrected_d

wide CSV 여러 개를 EAV로 모아 gold mart 만들기 현실의 데이터 소스는 한 가지 모양으로 오지 않는다. 같은 의미의 값도 어떤 파일에서는 생산수량 , 다른 파일에서는 units , 또 다른 파일에서는 made 로 올 수 있다. 온도도 어떤 곳은 섭씨, 어떤 곳은 화씨일 수 있다. 이걸 매번 pipeline code에 if source == ... 로 박기 시작하면 source가 늘 때마다 코드가 지저분해진다. manufacturing-data-platform-mini 의 EAV mini slice는 이 문제를 작게 다룬다. 여러 wide CSV를 mapping config로 표준 attribute에 맞춘 뒤, EAV long format으로 모으고, 다시 gold metric mart로 pivot/aggregate한다. 데이터는 모두 synthetic이고, 회사 코드·고객 데이터·실제 schema는 쓰지 않았다. 1. Scenario 서로 다른 공장/라인/벤더에서 비슷한 제조 지표 파일이 들어온다. 예: plant_a.csv: 설비ID, 생산수량, 불량수, 온도C, 압력kPa plant_b.csv: machine_id, output_units, defects, temp_f, pressure_bar vendor_d.csv: unit_name, made, scrap, deg_c, kpa 비즈니스적으로는 같은 지표를 보고 싶다. units_produced defect_count temperature_c pressure_kpa 문제는 source마다 column name과 unit이 다르다는 점이다. 2. Decision Pressure 단순 구현은 source마다 코드를 늘린다. if source == "plant_a": 생산수량을 units_produced로 읽는다 if source == "plant_b": output_units를 units_produced로 읽는다 temp_f를 섭씨로 변환한다 if source == "vendor_d": made를 units_produced로 읽는다 이 방식은 작게는 빨라 보이지만 source가 늘수록 문제가 된다. 새 파일 형식마다 pipeline code를 고쳐야 한다. column mapping과 transform logic이 섞인다. unit conversion이 흩어진다. quality check가 source별로 갈라진다. gold mart grain을 설명하기 어려워진다. 그래서 mapping은 config로 빼고, pipeline은 표준 attribute를 처리하게 만들었다. 3. Options option result risk source별 hard-coded parser 처음엔 빠름 source가 늘 때 code change 반복 모든 source를 wide table 하나로 합치기 보기 쉬움 sparse/heterogeneous column 폭발 EAV long format 이종 attribute를 표준 형태로 모음 pivot/quality 설계가 필요 full mapping DSL/rules engine 유연함 mini project에는 과함 이 프로젝트의 선택은 단순한 JSON mapping + EAV long + gold pivot이다. 4. Decision 각 source는 JSON config로 자신의 column을 표준 attribute에 매핑한다. source column -> standard attribute output_units -> units_produced temp_f -> temperature_c with f_to_c pressure_bar -> pressure_kpa with bar_to_kpa pipeline 흐름: wide CSVs -> mapping configs -> EAV long rows -> gold entity_daily_metrics -> quality checks -> catalog/lineage EAV row의 핵심 shape: entity_id business_date attribute value v

schema drift를 fail이 아니라 warn으로 둔 이유 데이터 파이프라인에서 source schema가 바뀌는 순간은 애매하다. 무조건 무시하면 운영자는 입력 구조가 바뀐 사실을 모른다. 반대로 모든 schema 변화를 실패로 처리하면, 정상적인 컬럼 추가까지 daily run을 막아버린다. manufacturing-data-platform-mini 에서는 이 문제를 작게 다뤘다. synthetic manufacturing CSV의 실제 header를 기준으로 schema_hash 를 만들고, previous successful run과 비교해 달라졌으면 schema_drift quality check를 warn 으로 남긴다. 단, required column이 빠져 silver/gold contract를 만들 수 없는 경우는 현재 ValueError 로 빠르게 실패한다. 1. Scenario 어느 날 source CSV에 새 컬럼이 추가된다. 기존 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date 새 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date,operator_id operator_id 는 아직 silver/gold mart에서 쓰지 않는다. 하지만 source 구조가 바뀐 사실은 기록되어야 한다. 2. Decision Pressure schema drift에서 중요한 질문은 단순히 "바뀌었나?"가 아니다. 바뀐 것을 운영자가 알 수 있는가? 정상적인 컬럼 추가 때문에 pipeline을 멈춰야 하는가? downstream gold mart contract가 조용히 바뀌지는 않는가? 이전 successful run과 지금 run의 schema identity를 비교할 수 있는가? 초기 구현에서는 한 가지 실제 버그가 있었다. schema_hash 가 고정된 required column 목록에 너무 묶여 있어서, 추가 컬럼이 들어와도 hash가 바뀌지 않았다. 즉 operator_id 가 추가되어도 drift가 보이지 않았다. 이 문제를 고치기 위해 read_rows 가 실제 CSV header를 반환하고, 그 실제 header 기준으로 schema_hash 를 계산하도록 바꿨다. 3. Options option result risk ignore drift pipeline은 계속 돈다 source 변화가 보이지 않음 fail every drift 변화에 강하게 반응 정상적인 컬럼 추가도 막음 warn and continue 변화가 보이고 run도 계속됨 warning을 inspect해야 함 auto-evolve silver/gold 새 컬럼을 바로 사용 가능 downstream contract가 조용히 바뀔 수 있음 full schema registry production에 가까움 mini slice에는 무거움 이 프로젝트의 선택은 warn and continue 다. 4. Decision 현재 contract는 이렇다. previous successful run이 없으면: schema_drift = pass baseline schema established current schema_hash == previous successful schema_hash: schema_drift = pass current schema_hash != previous successful schema_hash: schema_drift = warn quality_passed는 true 유지 run/lineage record에 previous/current schema_hash 저장 required column missing: V

source_hash로 같은 입력 재처리를 안전하게 skip하기 작은 데이터 파이프라인도 한 번만 실행된다고 가정하면 금방 거짓말이 된다. 실제로는 같은 파일을 다시 실행할 수 있다. 실패한 run을 재시도할 수도 있고, 과거 날짜를 backfill할 수도 있고, 운영자가 실수로 같은 입력을 다시 넣을 수도 있다. 이때 결과가 중복되면 gold metric은 더 이상 믿을 수 없다. 이 글은 개인 포트폴리오 프로젝트 manufacturing-data-platform-mini 에서 source_hash 를 이용해 같은 입력 재처리를 안전하게 skip하도록 만든 작은 설계 판단을 정리한 글이다. 데이터는 모두 synthetic이며, production platform이 아니라 검증 가능한 mini slice다. 1. Scenario 같은 business_date 의 제조/로봇 이벤트 파일을 다시 처리해야 하는 상황이 있다. 예: retry: 앞 run이 중간에 실패해서 다시 실행한다. backfill: 과거 날짜를 다시 채운다. operator mistake: 같은 파일을 실수로 다시 실행한다. 단순히 매번 append하면 같은 날짜의 gold metric이 중복될 수 있다. 2. Decision Pressure 단순 CSV pipeline은 보통 이렇게 끝난다. CSV 읽기 -> silver 만들기 -> gold 집계 -> 결과 저장 하지만 운영 관점에서는 질문이 생긴다. 이 입력은 전에 처리한 파일과 같은가? 같은 파일을 다시 돌리면 중복 output이 생기나? 다른 파일로 같은 날짜를 다시 돌리면 어떻게 구분하나? 어떤 run이 어떤 source에서 만들어졌나? 그래서 재실행을 판단할 identity가 필요했다. 3. Options option result problem always append 모든 run 결과를 계속 추가 같은 입력 재실행 시 중복 always overwrite 결과를 항상 덮어씀 이전 결과/원인 추적이 약함 skip by business_date only 같은 날짜면 무조건 skip 정정 파일을 반영할 수 없음 skip by dataset_id + business_date + source_hash 같은 입력만 no-op 정정 파일은 새 run으로 처리 가능 이 프로젝트의 Slice1은 마지막 선택지를 쓴다. 4. Decision 현재 mini pipeline은 입력 파일의 content hash를 source_hash 로 계산한다. idempotency key: dataset_id + business_date + source_hash 이미 성공한 run이 있으면 새로 처리하지 않고 기존 run을 재사용한다. same dataset_id same business_date same source_hash prior successful run exists => status = skipped 이 선택은 작지만 중요하다. 같은 파일 재실행: skip -> 중복 없음 같은 날짜의 정정 파일: source_hash가 다름 -> skip하지 않고 새 run으로 처리 가능 단, 여기서 조심해야 할 경계가 있다. Slice1은 다른 source_hash 를 새 run으로 처리할 수 있지만, 이전 gold partition을 원자적으로 교체하는 Iceberg-style overwrite까지 구현한 것은 아니다. 그 문제는 다음 Slice2의 business_date partition overwrite 주제다. 5. Evidence 관련 코드와 검증 evidence: src/manufacturing_data_platform/pipeline/lakehouse.py tests/test_lakehouse_pipeline.py VERIFICATION_LOG.md README.md 검증 로그: 2026-07-08 publication readiness check: pytest: 33 passed lakehouse JSON CLI: passed, status=processed, quality_passed=true EAV JS

Losing PostgreSQL Gains? Blame Inline JSONB!! PostgreSQL's jsonb is a favorite among developers for its flexibility - but it hides a dark side. When used carelessly, especially in-line within rows under 2KB, it can silently destroy performance, even if you're using indexes. Here's why. 🔍 The Hidden Cost of JSONB (Inline Storage) PostgreSQL stores table rows in 8KB pages, packing as many tuples as possible. For a typical row with 10–12 columns, and small text/integers, 40–100 rows can easily fit per page. Typically row count = Page Size(8kb) / row size + row metadata (30-50 bytes approx.) But the game changes when you add jsonb. Example CREATE TABLE events ( id serial PRIMARY KEY, user_id int, action text, metadata jsonb ); Suppose metadata which is a jsonb column contains: { "ip": "127.0.0.1", "device": "Android", "country": "IN" } This JSON might be just 100–500 bytes, so PostgreSQL stores it in-line inside the same page (no TOASTing). Result Each row size jumps from ~80 bytes → ~200–400 bytes Row count per page drops from 100 → 20–40 Index scan still needs to read each page for matching rows More pages = more I/O, slower performance 🔢 Real Benchmark Insight Performance comparisonEven with a GIN or B-tree index on the JSONB column, PostgreSQL still needs to scan all matching pages to retrieve the full tuple. 🧠 Why Index Doesn't Save You Say you index a JSONB key like: CREATE INDEX ON events ((metadata->>'ip')); And query: SELECT * FROM events WHERE metadata->>'ip' = '127.0.0.1'; PostgreSQL will: Use the index to find matching tuples Still need to fetch the row from disk Because JSONB is in-line, many pages are touched More page fetches = more IO = slower queries 🩹 What You Can Do ✅ Force TOAST: Add padding to make JSONB exceed 2KB: UPDATE events SET metadata = metadata || jsonb_build_object('padding', repeat('x', 2000)); ✅ Split into separate table: If JSONB is rarely queried ✅ Stick to well defined schema and avoid using jsonb unless absolutely necessary. 🧾 TL;DR

In Part 2 , we learned why AI sometimes hallucinates. One of the biggest reasons is that an LLM can only answer based on what it learned during training and the information available in its context window. We also introduced grounding -providing the model with reliable information at runtime instead of expecting it to know everything. But that raises an important question: Where does that information come from? Modern AI applications don't simply dump an entire database or a thousand-page PDF into the prompt. Instead, they first identify the most relevant pieces of information and only send those to the model. In this article, we'll learn how that works. Running Example Let's continue building our AI-powered Travel Planner . So far, it can answer general travel questions using the knowledge it learned during training. Now we want to make it much smarter by uploading several documents into our application: Lonely Planet's Japan travel guide A PDF containing train schedules A document listing recommended local restaurants Hotel information Internal travel policies for our company Together, these documents contain hundreds of pages. Now the user asks: I'm staying near Tokyo Station. Which ramen restaurant from our travel guide is within walking distance and is known for vegetarian options? Somewhere in those hundreds of pages is the answer. The challenge is no longer generating text-it's finding the right information first. The Problem: An LLM Can't Read Your Entire Knowledge Base Every Time A common misconception is that AI applications simply send all their documents to the model. Imagine our travel guide contains 450 pages, thousands of restaurant listings, hotel descriptions, transportation details, and sightseeing recommendations. Sending all of that to the LLM every time someone asks "Where should I eat tonight?" creates several problems. First, many documents are simply too large to fit inside the model's context window. Second, even if they did fit, making the

Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The sentence is right. The trouble starts on the next page, because there are two fundamentally different ways to build that single place, and the industry has spent years arguing about them as if they were rivals. Road one is consolidation: bring the data together. Land everything in one governed store, in this era an open lakehouse, Apache Iceberg tables on object storage, and point every consumer at it. Road two is federation: leave the data where it lives and bring the access together instead. A query engine that speaks to your databases, warehouses, lakes, and applications in place, presenting one surface over many sources, with no copies made. I work at Dremio, a company whose platform is built on the conviction that this is a false choice, that the right architecture uses both roads with judgment, and I will declare that bias now and then earn it with an honest treatment. Because the truth practitioners live is messier than either camp's marketing: federation without a lakehouse hits performance and scale ceilings, a lakehouse without federation spends years and fortunes migrating the long tail, and the teams that win treat the two as phases and partners rather than competitors. So this article is the full playbook. What federation and the lakehouse each actually are, mechanically. The honest strengths and limits of each, including the failure modes their advocates gloss over. A concrete decision framework for when each one carries a workload. The lifecycle pattern that connects them, federate first, promote deliberately. And the unified architectures, mine included, that put both behind one governed door, which matters more than ever now that the consumers walking through that door increasingly are AI agents. Why Unify at All: The Cost of the Status Quo Before the two roads, the destination deserves a paragraph, because

series: Prediksi Survival Klub Debutan Kenapa Project Ini? Setiap musim, klub yang promosi ke liga top (Premier League, La Liga, dst.) menghadapi risiko besar: sekitar 2 dari 3 klub yang naik biasanya kembali terdegradasi di musim pertama mereka. Saya penasaran — bisakah performa di beberapa laga awal musim memberi sinyal dini soal peluang klub tersebut bertahan? Ini jadi project portofolio pertama saya sebagai data scientist yang baru mulai (0-1 tahun pengalaman). Saya sengaja pilih topik yang saya suka (sepak bola) supaya prosesnya tetap enjoyable, bukan cuma "tutorial project" generik. Rencana Project Pertanyaan utama: Berdasarkan performa 8 laga pertama musim debut, seberapa besar peluang klub promosi bertahan hingga musim berikutnya (tidak degradasi)? Data yang dipakai: football-data.co.uk — data hasil pertandingan tiap musim sejak 1993/1994 Wikipedia (halaman musim liga) — daftar klub promosi & klasemen akhir musim Tech stack: pandas , requests untuk data collection scikit-learn untuk modeling (mulai dari Logistic Regression sebagai baseline) imbalanced-learn untuk handle class imbalance Streamlit + Plotly untuk dashboard interaktif Deploy ke Streamlit Community Cloud Timeline (Build in Public) Saya bikin timeline ini publik supaya ada tekanan yang sehat untuk benar-benar menyelesaikannya, bukan cuma jadi ide yang menguap: Checkpoint Target Tanggal Yang Harus Selesai Part 1 (post ini) 11 Juli 2026 Kickoff, rencana, environment siap Part 2 15 Juli 2026 Dataset jadi, push ke GitHub Part 3 17 Juli 2026 EDA selesai, insight awal Part 4 24 Juli 2026 Model final dipilih + evaluasi Part 5 31 Juli 2026 Dashboard live di Streamlit Cloud Part 6 (final) 8 Agustus 2026 Project selesai, recap lengkap Tantangan yang Sudah Saya Antisipasi Data leakage — fitur harus dihitung dari laga awal musim saja, bukan seluruh musim, biar model beneran memprediksi bukan "menyontek" hasil akhir Dataset kecil — kemungkinan hanya ~60-100 sampel klub, jadi saya mulai dari model sederhana (Lo

By the team at MailTester Ninja — a real-time email verification API that stores nothing. Everyone says "IPv6 is here." For the web, mostly true. For email , it is a mirage. We resolved the MX records of 50,000 of the most-linked domains and checked whether any of their mail servers publish an AAAA record, meaning they can actually receive over IPv6. No sending, no personal data, just DNS. 55.2% of mail-enabled domains have at least one IPv6-capable MX. That sounds healthy. It is not, because two companies carry almost the whole number: Email provider IPv6 MX Other / self-hosted ██░░░░░░░░ 18.4% Google Workspace / Gmail ██████████ 100% Microsoft 365 / Outlook █████████░ 91.3% Proofpoint ░░░░░░░░░░ 0.6% Mimecast ░░░░░░░░░░ 0% Tencent QQ ░░░░░░░░░░ 4.2% Namecheap ░░░░░░░░░░ 0.2% Cisco IronPort ░░░░░░░░░░ 4.5% Zoho ░░░░░░░░░░ 0% Barracuda ░░░░░░░░░░ 0% Google ( 100% ) and Microsoft ( 91.3% ) run IPv6 on nearly every inbox. Remove those two, the providers that already anchor most of the world's mail, and IPv6 email adoption falls from 55.2% to 12.9% . The enterprise security gateways that gate corporate mail, such as Proofpoint, Mimecast and Barracuda, are effectively not on IPv6 at all. Why it matters for deliverability. IPv6-only sending is a dead end. It reaches Gmail and Outlook and little else. Dual-stack is not optional. IPv4 is still the backbone of email, and that is where blocklists, FCrDNS and IP reputation are mature. The takeaway: IPv6 email is not adopted. Google and Microsoft adopted it for you. Plan your sending for an IPv4 world with two big IPv6 exceptions. Check any domain yourself — our free deliverability analyzer shows a domain's MX / SPF / DMARC in one click (no signup, nothing stored). Need to confirm whether a specific mailbox actually exists and is deliverable? That is exactly what MailTester Ninja's email verifier does in real time — and we store no data. Source: MailTester Ninja's open Email Infrastructure Index — a live DNS scan of 50,000 of