标签：#Data

Introduction When a query runs in ClickHouse®, the database does much more than simply read data and return results. Before execution begins, ClickHouse® parses the SQL statement, analyzes it, applies optimizations, and builds an execution plan that determines the most efficient way to process the query. Understanding query execution plans is one of the most valuable skills for anyone working with ClickHouse®. They provide visibility into how queries are executed, helping you identify bottlenecks, validate optimization efforts, and troubleshoot performance issues. In this article, we'll explore how ClickHouse® generates execution plans, the different EXPLAIN modes, and how to interpret them for better query optimization. Why Query Execution Plans Matter A SQL query defines what data you want, but it doesn't explain how the database retrieves it. Consider the following query: SELECT country , count () FROM events GROUP BY country ; Although the query looks simple, ClickHouse® must determine: Which data parts to read Whether primary indexes can reduce the scan If data skipping indexes can be used How aggregation should be performed Whether parallel execution is possible How intermediate results should be merged A query execution plan provides answers to these questions, making it an essential tool for performance tuning. The ClickHouse Query Lifecycle Every query passes through several stages before producing results. The lifecycle typically looks like this: SQL Query │ ▼ Parser │ ▼ Analyzer │ ▼ Optimizer │ ▼ Query Plan │ ▼ Execution Pipeline │ ▼ Results Each stage plays an important role: Parser validates SQL syntax. Analyzer resolves tables, columns, and expressions. Optimizer applies query optimizations. Query Plan determines the logical execution steps. Pipeline distributes work across multiple threads. Execution processes the data and returns the results. Understanding this workflow makes execution plans much easier to interpret. Introducing the EXPLAIN Statement

Introduction Every serious backend developer eventually faces the same problem: you need to make multiple changes to a database as part of a single business operation, and you need all of them to succeed or none of them to go through. Partial updates are worse than no updates at all - they leave your data in an inconsistent state that can be nearly impossible to debug in production. This is not a new problem. Enterprise developers have been solving it for decades, and Martin Fowler documented the canonical solution in his 2002 book Patterns of Enterprise Application Architecture : the Unit of Work pattern. In this article we are going to go deep on what Unit of Work is, why it exists, how it works internally, and how to build a clean, production-quality implementation from scratch in Python using only the standard library. By the end you will have a working implementation you can adapt to any project, and a solid understanding of how popular frameworks like SQLAlchemy and Django ORM implement this pattern under the hood. The full source code is available on GitHub: 👉 github.com/diegocastillo12/unit-of-work-python - ## Background: What is the Unit of Work Pattern? The Unit of Work pattern is part of Martin Fowler's catalog of Patterns of Enterprise Application Architecture (PoEAA), a collection of battle-tested solutions for common problems in enterprise software design. Fowler defines it as follows: > "A Unit of Work maintains a list of objects affected by a business transaction and coordinates the writing out of changes and the resolution of concurrency problems." Let's unpack that definition carefully. "Maintains a list of objects affected by a business transaction" - this means the Unit of Work acts as a tracker. When your business logic creates a new object, modifies an existing one, or marks one for deletion, it does not immediately write to the database. Instead, it registers the change with the Unit of Work, which keeps an in-memory list of everything that ne

The Quest Begins (The "Why") Honestly, I still remember the first time I stared at the Daily Temperatures problem on LeetCode and felt like I was trying to crack a vault with a toothpick. The brute‑force solution — two nested loops, checking every future day for a warmer temperature — was simple to write, but it timed out on the larger test cases. I spent an hour tweaking loops, adding early breaks, and even trying to memoize results, only to watch the same red “Time Limit Exceeded” banner flash again. I was frustrated, but more than that, I was curious. Why did this problem feel so repetitive ? Every element seemed to be asking the same question: “What’s the next greater value to my right?” If I could answer that for each index in a single pass, the whole thing would collapse into O(n). That’s when I remembered a weird little data structure I’d seen in a textbook — the monotonic stack — and realized it might be the magic wand I needed. The Revelation (The Insight) Here’s the thing: a monotonic stack isn’t just a stack with a funny name; it’s a way to capture relationships between elements without ever looking backward more than once . Imagine you’re walking through a line of people sorted by height, and you want to know, for each person, who is the first taller person standing ahead of them. If you keep a stack of people whose heights are strictly decreasing as you move from left to right, then whenever you see a new person taller than the one on top of the stack, you’ve just found the answer for that stacked person: the current person is their “next greater.” You pop them off, record the distance, and keep going. Because each index is pushed once and popped at most once , the total work is linear. The same idea works for “next smaller,” “previous greater,” or any problem where you need the nearest element that satisfies a monotonic condition. The stack does the heavy lifting of remembering candidates that could still be relevant, discarding the ones that are alrea

I got a take-home assignment last year from a company I was genuinely excited about. "Should take about four hours," the recruiter said. Build an ingestion pipeline, model the data, write tests, document your design decisions, and prepare a 15-minute presentation walkthrough for the panel. Four hours. I laughed, closed my laptop, and started on it the next morning like it was a sprint. Sixteen hours later I had something I was proud of. Clean pipeline, solid tests, real documentation. I submitted it on a Sunday night. Monday I got a form rejection. No notes. No feedback. Not even which stage I failed. Just "we've decided to move forward with other candidates" and a link to their Glassdoor page. That was the moment I stopped pretending take-homes are assessments. They're consulting gigs. Unpaid ones. The Scope Creep Nobody Talks About Five years ago, a data engineering take-home was a focused exercise. Model this dataset into a star schema. Write a few SQL transforms. Maybe a short README. Two to four hours, tops. Bounded, reasonable, and actually useful for evaluating how someone thinks about data. That version is dead. Today, 68% of companies use take-home tests, up 12% year over year. And the scope has quietly ballooned into something unrecognizable. Full pipeline implementations. Test suites with coverage thresholds. Documentation that reads like a design doc. A presentation follow-up where you defend your architecture to a panel. We're talking 10 to 20 hours of work, routinely, for a role you haven't been offered. Industry best practice caps take-homes at 90 minutes of expected effort. The reality? Candidates consistently take 2x longer than company estimates to reach submission quality. That "four-hour" assignment is an eight-hour assignment. That "weekend project" is a week of evenings. And 25% of companies are still handing these out like they're reasonable asks. Here's the part that makes my eye twitch: 71% of engineering leaders openly say take-homes no lon

It's unclear why Klue had not revoked the credential after the limited pilot, which hackers then used to breach a system holding keys for accessing customers' data.

NocoDB is an open-source no-code platform that puts a spreadsheet-style UI on top of a relational database, with grid, form, Kanban, and gallery views plus a REST API. This guide deploys NocoDB using Docker Compose with a PostgreSQL backend and Traefik handling automatic HTTPS, then exercises the API with a sample base. By the end, you'll have NocoDB serving a base over HTTPS with API access at your domain. Set Up the Directory Structure 1. Create the project directories: $ mkdir -p ~/nocodb/ { data,pgdata,letsencrypt } $ cd ~/nocodb 2. Create the environment file: $ nano .env DOMAIN = nocodb.example.com LETSENCRYPT_EMAIL = admin@example.com POSTGRES_DB = postgres POSTGRES_PASSWORD = strong_password POSTGRES_USER = postgres Deploy with Docker Compose 1. Create the Compose manifest: $ nano docker-compose.yaml services : traefik : image : traefik:v3.6 container_name : traefik command : - " --providers.docker=true" - " --providers.docker.exposedbydefault=false" - " --entrypoints.web.address=:80" - " --entrypoints.websecure.address=:443" - " --entrypoints.web.http.redirections.entrypoint.to=websecure" - " --entrypoints.web.http.redirections.entrypoint.scheme=https" - " --certificatesresolvers.letsencrypt.acme.httpchallenge=true" - " --certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web" - " --certificatesresolvers.letsencrypt.acme.email=${LETSENCRYPT_EMAIL}" - " --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json" ports : - " 80:80" - " 443:443" volumes : - " ./letsencrypt:/letsencrypt" - " /var/run/docker.sock:/var/run/docker.sock:ro" restart : unless-stopped db : image : postgres:16 container_name : nocodb-db hostname : root_db environment : POSTGRES_DB : ${POSTGRES_DB} POSTGRES_USER : ${POSTGRES_USER} POSTGRES_PASSWORD : ${POSTGRES_PASSWORD} volumes : - " ./pgdata:/var/lib/postgresql/data" healthcheck : test : [ " CMD" , " pg_isready" , " -U" , " ${POSTGRES_USER}" , " -d" , " ${POSTGRES_DB}" ] interval : 10s timeout : 5s retries :

This is the second data breach to affect LastPass customers in recent years, after one of the password manager's tech partners was recently breached.

Payment data pipelines fail in ways that ruin a payments engineer’s week, and the failures rhyme. The dashboards froze. Fraud scores arrived after the transaction had already cleared. Settlement reports came in stale. Nobody slept. The frustrating part is that the same data architecture had run fine for years. So, what changed? The honest answer is that batch thinking does not survive contact with real-time payments. A lot of banks built their data foundations in an era when nightly jobs were good enough. Load the warehouse overnight, run the reports in the morning, move on. That rhythm worked when money moved slowly. It does not work when a customer expects an instant confirmation and a fraud engine has milliseconds to make a call. Here is where things crack. Real-time payment rails push a constant stream of events instead of a tidy nightly dump. Your pipeline now has to ingest, transform, and serve data while transactions are still happening. Add ISO 20022 into the mix and the pressure climbs. ISO 20022 messages are rich. They carry far more structured detail than the old formats, which is wonderful for analytics and miserable for a pipeline that was never designed to parse that much context at speed. This is not a fringe concern either. Swift reported that by the time its MT/ISO 20022 coexistence period closed in November 2025, around 80% of daily traffic was already running on the ISO 20022 format, with more than 3.1 million of these messages exchanged every day. The rich-data era is the default now, not the roadmap. Then there is the fraud-scoring window. Fraud models need fresh features. Account behaviour over the last few minutes, velocity checks, device signals. If your pipeline takes thirty seconds to surface that data, the fraud decision is already too late. You are essentially detecting fraud after the loss. That gap between when data is created and when it becomes usable is the silent killer in most payment systems. And the cost of getting it wrong runs

I had a long Google AI Studio (Gemini) session that I wanted to keep. I selected the conversation in the browser, copied it, and pasted it into a text file. File size: a few hundred KB. "OK, that's safe." Later, I exported the same session as JSON. File size: a few MB. More than half of the data had silently disappeared. What was missing I checked what the manual copy had dropped. The system prompt The instruction I had originally given the model — the system prompt — was completely gone. Manual copy captures only the user/assistant turns visible in the conversation pane. The instruction context that shaped the entire session does not get copied. The tail of long responses When a Gemini response is long, the browser shows a "Show more" button. If you copy without expanding it, the response gets cut mid-sentence. Out of 8 sessions I checked, 3 had responses truncated this way. Newlines inside code blocks Newlines inside code blocks got mangled in several places. Responses containing JSON or YAML had indentation that no longer parsed. The reasoning trace For some models, the model's reasoning trace is stored separately from the visible response. Manual copy doesn't capture it at all. How to export as JSON Google AI Studio has a session export feature. In the session view, click the ... menu at the top right Select "Export" Choose JSON format and download The JSON contains the full data, including the system prompt. Measured: manual copy vs. JSON export I compared 8 sessions. Session Manual copy JSON Loss rate A tens of KB ~150 KB ~70% B ~90 KB ~200 KB 50-60% C ~30 KB ~100 KB 60-70% D ~50 KB ~180 KB ~70% E ~60 KB ~240 KB ~70% F ~20 KB ~70 KB ~60% G ~20 KB ~50 KB 50-60% H ~10 KB ~30 KB ~60% Total a few hundred KB ~1 MB 60-70% Average loss rate, 60-70%. The manual copy was, on every session, missing most of what was in the actual session state. Why I didn't notice If you open the manually-copied file, the conversation reads fine. As long as the start and end connect, a m

I spent the last few months trying to understand why reconciliation errors keep appearing in high-throughput pipelines. Here is what I found. In the race to process millions of transactions daily, modern fintech ecosystems have achieved a genuine miracle of scale. But beneath the surface of that velocity lies a structural problem most engineering teams aren't measuring: causality violations in async event pipelines. Most teams assume that if a transaction shows "Success" in the database, the job is done. At high concurrency levels, that assumption breaks quietly. When "Eventual Consistency" Becomes "Eventual Loss" In distributed systems, Kafka partitions and database shards experience micro-millisecond timing gaps. When a network retry delays a validation webhook, the downstream ledger can commit a wallet update before the validation that should have preceded it completes. To the user, the app glitches. To the engineering team, it's a reconciliation ticket. To the CFO, it's untracked operational cost. The Reconciliation Tax I built a simulation modelling this exact failure mode across 5,000 concurrent transactions. With an 8% network retry probability, conservative for high-traffic payment rails, the causality violation rate was 8.3%. At one million daily transactions, that's over 80,000 unvalidated commits every day requiring manual review. The operational cost compounds across three dimensions: engineering hours spent patching database state, fraud model accuracy degrading on out-of-order training data, and audit trails that cannot demonstrate strict causal sequence to regulators. The Fix The solution is enforcing strict event ordering at the ingestion layer before state commits happen, not better monitoring after the fact. When safeguards including partition-aware routing, exponential backoff, and idempotency controls were added to the same simulation, the violation rate dropped to 0%. Full simulation code and methodology: github.com/yakuburoseline1-gif/cif-simul

Here's the scene I've watched play out on a dozen teams. Accuracy plateaus. Someone rips out the logistic regression, drops in XGBoost, and waits for the jump. It doesn't come — or it comes with two points you can't explain to anyone. So the week disappears into hyperparameter tuning, and you end up with a slower, heavier, less interpretable model that's barely better than where you started. The model was almost never the bottleneck. The features were. This post is the long, practical version of that argument. We'll define the two camps in plain language, run real code, look at when boosting genuinely wins, and then walk through the failure mode nobody warns you about — the one where the fancy model is "winning" because it's quietly cheating. A note before we start: keep your examples generic. We'll predict a numeric target — think demand, a quantity, a score on a tabular dataset. The principles are the same everywhere, and you should validate them on your own data. The two camps, in plain terms Linear / logistic regression fits a straight-line relationship: each feature gets a weight (a coefficient), and the prediction is a weighted sum. Logistic regression is the same idea bent for classification — it outputs a probability. from sklearn.linear_model import LogisticRegression model = LogisticRegression ( max_iter = 1000 ) model . fit ( X_train , y_train ) # the whole model, readable in one line per feature: for name , weight in zip ( feature_names , model . coef_ [ 0 ]): print ( f " { name : < 20 } { weight : + . 3 f } " ) That loop is the entire model. A positive weight means "more of this pushes the prediction up," and you can hand that table to a stakeholder and defend every number. The cost: it assumes the relationship is roughly linear and that features act independently. Real data often isn't that polite. Gradient boosting (XGBoost, LightGBM, sklearn's GradientBoostingClassifier ) builds hundreds of small decision trees, each one correcting the mistakes of th

DuckDB 1.5.2, PostgreSQL Internal Stats, and SQLite Virtual Table xUpdate Deep Dive Today's Highlights This week brings a stable new patch release for DuckDB, enhancing performance and adding DuckLake support. We also delve into PostgreSQL's internal statistics for better tuning and explore advanced SQLite virtual table implementation via xUpdate . Announcing DuckDB 1.5.2 (DuckDB Blog) Source: https://duckdb.org/2026/04/13/announcing-duckdb-152.html DuckDB has released version 1.5.2, a patch update focusing on stability and performance. This release includes critical bugfixes that improve the reliability of the in-process analytical database, addressing various edge cases and enhancing overall robustness. Key enhancements also target performance bottlenecks, ensuring faster query execution for diverse analytical workloads. A significant new feature in this version is the official support for the DuckLake v1.0 lakehouse format. This integration positions DuckDB as a more robust tool for handling modern data architectures, allowing users to efficiently query and manage data stored in a lakehouse paradigm directly within their applications or analytical workflows. This update makes DuckDB even more compelling for embedded analytics and data pipeline use cases, providing a flexible and high-performance option for developers. Comment: Always good to see performance improvements and bug fixes for an embedded analytics powerhouse like DuckDB. DuckLake v1.0 support is a big step for managing structured data in lakehouse environments directly from DuckDB, enhancing its utility for complex data architectures. pg_stats: How Postgres Internal Stats Work (Planet PostgreSQL) Source: https://postgr.es/p/9mG This article from Planet PostgreSQL delves into the intricate mechanisms behind PostgreSQL's internal statistics, specifically focusing on pg_stats . Understanding how Postgres collects and utilizes these statistics is fundamental for effective database performance tuning and q

Nvidia announced a new cooling system that cuts water use inside the data center. But it does nothing to address AI's biggest water use — fossil fuel power plants.

আমার colleague এখন প্ল্যান দেখতে পারছে। Scan types বুঝতে পারছে। Join types বুঝতে পারছে। Estimate আর actual এর gap দেখতে পারছে। BUFFERS ও দেখছে। কিন্তু সে প্রশ্ন করল। এসব দেখে কি করব? Step by step কোন পথে যাব? আমি বললাম। পাঁচটা step আছে। অর্ডার অনুযায়ী। পর্ব ২ এ আমি বলেছিলাম scan types, join types, estimate আর actual এর gap। BUFFERS কি। এবার আসি সমাধান এ। Diagnostic Workflow আপনার কাছে একটা slow query এসেছে। কিভাবে debug করবেন? এই পাঁচটা প্রশ্ন করুন অর্ডার অনুযায়ী। ৯০% slow query প্রথম বা দ্বিতীয় ধাপেই solve হয়ে যায়। ১. Deepest Seq Scan দেখুন Table বড় কি না? Filter selective কি না? Missing index থাকলে add করুন। আজই শুরু করুন যখন একটা Seq Scan দেখবেন big table এ, প্রথমে WHERE clause টা check করুন। Selective কি না? ৫% এর কম row return হওয়ার কথা? যদি তাই হয়, index missing। CREATE INDEX idx_name ON table(column) run করুন। ২. Join types দেখুন কোনো Nested Loop আছে কিন্তু দুই পাশেই বড় table? Hash Join force করুন বা ডান পাশে index add করুন। আজই শুরু করুন Nested Loop দেখলে ডান পাশের table এ index check করুন। যদি না থাকে, create করুন। Index থাকা সত্ত্বেও planner Nested Loop use করছে? SET enable_nestloop = off temporarily disable করে দেখুন। Hash Join আসবে কি না। ৩. Row estimates দেখুন Estimate vs actual ১০x এর বেশি difference? ANALYZE table দিন বা predicate rewrite করুন। আজই শুরু করুন rows=1 estimate কিন্তু rows=100000 actual দেখলে ANALYZE tablename run করুন। Statistics refresh হবে। তারপর plan আবার দেখুন। যদি তাও না আসে, WHERE clause rewrite করুন। Function call থাকলে remove করুন। Type mismatch থাকলে fix করুন। ৪. BUFFERS add করুন কোনো node এ অনেক disk reads? Caching investigate করুন। আজই শুরু করুন EXPLAIN (ANALYZE, BUFFERS) run করে দেখুন shared read high কোথায়। সেই node টাই bottleneck। Index add করলে reads কমবে। Pre-warm cache করতে পারেন। Data pre-load করতে পারেন। ৫. Sorts আর hashes দেখুন কোনো spill-to-disk আছে? work_mem raise করুন বা sort eliminate করুন। আজই শুরু করুন Plan এ external merge Disk: 421MB দেখলে spill-to-disk হয়েছে। SET work_mem = '256MB' temporarily rais

Alina Krasavina explains how Delivery Hero successfully deprecated Google Analytics and migrated to an internal user tracking platform. She discusses how a simplistic, highly scalable architecture allowed them to handle 10 times more load while capturing 97% of tracking data. By Alina Krasavina

Huntress, HackerOne, Jamf, Recorded Future, and Tanium are among the cybersecurity companies that had data stolen following an earlier breach at market research firm Klue.

In this article, the author explores data poisoning as a threat to machine learning systems, covering techniques such as label flipping, backdoors, clean-label poisoning, and gradient manipulation. The article reviews real-world incidents, discusses the challenges of detecting poisoned data, and presents practical defenses, tools, and operational practices for securing ML training pipelines. By Igor Maljkovic

Big Tech is throwing big money into data center buildouts. As national opposition to the facilities grows, some workers are beginning to question whether it’s worth it.

If you're learning Data Analytics and looking to build a strong portfolio, working on real-world...

Let me confess something a little creepy. I have a habit of peeking at other people's dev posts. Not stealing the writing — relax. I run a tiny read-only job that fetches the public pages on dev.to, Zenn, and Qiita and counts only the boring parts: titles, post times, like counts. Who published what, at what hour, and how far it traveled. Then it tallies the lot. The reason is petty: my own posts weren't landing. The content is already in my hands — so I wanted to know how much the rest, the when and how you publish , actually moves the needle. By the numbers, not by gut. So I counted across three platforms. And the conditions that make a post fly turned out to be roughly mirror images between Japan (Zenn / Qiita) and the English-speaking world (dev.to). Here's the story. First, my most important disclaimer This post is full of numbers, so let me put up a guardrail before any of them. This is correlation, not causation . A result like "weekend posts don't do well" could mean the weekend itself is bad — or it could mean people who post on weekends are just dashing something off on the side. The data can't separate those. Please read it that way. Also, I only keep aggregate numbers I computed myself . I don't store or reuse anyone's article body (read-only GET, count the features, throw the page away). I peek, but only at the overall shape . Nobody gets singled out here. With that out of the way — four findings I enjoyed. 1. The best hour to publish is just your readers' time zone This one came out cleanest. On Qiita , posts published in the morning win (+32pt in the GOOD group). Midday is +14pt. Evening is -32pt, late night -14pt. Zenn likes midday too (+27pt). Late night is -15pt. dev.to is the exact opposite. Late night Japan time scores +7pt — Japanese evening is actually weak. The trick is obvious once you see it. dev.to's readers are English-speaking, mostly US. Late night in Japan is the US working day. Zenn and Qiita readers are in Japan, so the Japanese morni