标签：#Data

Murat Demirbas discusses the shift toward disaggregated cloud database architectures driven by cloud economics. He explains how decoupling compute from storage enables elastic scaling, cost efficiency, and fault isolation. He shares how classical Paxos roles foreshadowed disaggregation, while analyzing network tradeoffs, shared-memory evolution, and self-assembling database designs. By Murat Demirbas

Despite the ongoing AI boom, LinkedIn is holding the line on compute spending. Instead, it’s challenging engineers to make every GPU count.

How I applied Exploratory Data Analysis, Feature Engineering, Pipelines, and Ensemble Models to solve a real-world machine learning problem—and the lessons I learned along the way. Introduction There comes a point in every machine learning learner's journey when watching tutorials and completing small practice exercises are no longer enough. After spending weeks understanding statistics, exploratory data analysis (EDA), feature engineering, preprocessing techniques, and classical machine learning algorithms, I wanted to answer one question: Can I apply everything I've learned to a real machine learning competition? That's when I decided to participate in a Kaggle Playground competition. Unlike classroom datasets, Kaggle competitions force you to think like a machine learning engineer. You're responsible for understanding messy data, building preprocessing pipelines, selecting models, evaluating performance, debugging errors, and finally creating a submission that competes with thousands of participants. This article documents my complete journey—from loading the dataset to building production-style preprocessing pipelines and training multiple ensemble models. Along the way, I'll also share the challenges I faced, what worked well, and the lessons I'll carry into future competitions. Why Kaggle? Learning machine learning isn't just about knowing algorithms. Real-world ML requires answering questions like: Which features are useful? How should missing values be handled? Should categorical variables be one-hot encoded or ordinal encoded? Which preprocessing steps belong inside a pipeline? How do different ensemble models compare? Kaggle provides an environment where all of these questions matter. Instead of building a model that works only inside a notebook, you're solving a problem under realistic constraints and evaluating your solution on unseen data. Competition Goal The objective of this Playground competition was to predict the target class based on a combinatio

Introduction When building a Flutter application, you'll often need to store data on the user's device. For example: Saving user preferences Storing login information Caching API responses Creating offline applications Building note-taking or to-do apps While there are several local storage solutions available, Hive is one of the fastest and easiest local storage for Flutter developers. In this tutorial, you'll learn Hive from scratch by building a simple example. No prior database knowledge is required. What is Hive? Hive is a lightweight, NoSQL database written entirely in Dart. It stores data directly on the device, making it perfect for Flutter applications. Why use Hive? Extremely fast Works offline No native platform code required Simple API Easy to learn Great for small and medium-sized applications Think of Hive as a collection of boxes where each box stores your application's data. Hive ├── User Box ├── Settings Box ├── Notes Box └── Products Box Each Box is similar to a table in traditional databases. Step 1: Create a Flutter Project Create a new Flutter project. flutter create hive_demo Open the project. cd hive_demo Step 2: Install Hive Open pubspec.yaml and add the following packages. dependencies : flutter : sdk : flutter hive : ^2.2.3 hive_flutter : ^1.1.0 Then install them. flutter pub get Step 3: Initialize Hive Before using Hive, initialize it inside main() . import 'package:flutter/material.dart' ; import 'package:hive_flutter/hive_flutter.dart' ; void main () async { WidgetsFlutterBinding . ensureInitialized (); await Hive . initFlutter (); await Hive . openBox ( 'settings' ); runApp ( const MyApp ()); } Here we open a box called settings . Step 4: Understanding Boxes A Box is where Hive stores data. Imagine this box: Settings Box theme -> dark username -> Alex loggedIn -> true Keys are on the left. Values are on the right. Step 5: Save Data Saving data is incredibly simple. var box = Hive . box ( 'settings' ); box . put ( 'username' , 'John' );

At TechCrunch Disrupt 2026, the AI Stage is back to dig into the single hottest topic in the community for the past few years, presented by Google for Startups.

On May 20, 2026, AWS ended support for DMS Fleet Advisor. Fleet Advisor answered a question every migration team asks first: what is actually in my database estate, and how hard will it be to move? It was free. It was fully managed. It was backed by the largest cloud provider on earth. It still lost. AWS's official notice says only: "After careful consideration, we decided to end support for AWS DMS Fleet Advisor." No reason given. But you don't need one — the documentation tells you. Here is what Fleet Advisor required before it would tell you a single thing about your databases: Install a standalone data collector in your local environment Create an Amazon S3 bucket Create IAM policies, roles, and users — via CloudFormation, which was the recommended path Create database users with the minimum required permissions on every source Establish network access from the collector to each database server Then you'd meet the ceilings: recommendations for up to 100 databases at a time, one-to-one target mapping only, no multitenant server support. Now picture running that gauntlet inside a bank. You are a Business Solution Architect. You have been asked to scope a migration. You do not yet have approval for the migration — that approval is what the assessment is for . And to produce the assessment, you must first request production database credentials, get an agent binary through software approval, provision an S3 bucket, and get an IAM stack past a security review. That is a six-week procurement conversation to answer a question you were hoping to answer this week. AWS's replacement recommendation is Migration Evaluator — a consulting-led engagement. Read that as the finding it is: AWS looked at self-serve migration assessment, and concluded that humans and services do it better than a product. I think they were half right. And the half they got wrong is the interesting part. The lesson: friction is a competitor, and it usually wins We talk about developer tools as if the

Missing data is not the same thing as a convenient default. That sounds obvious, yet temporal software regularly converts an empty time field into midnight, noon, the current time, or the start of a day. The interface may look complete after that conversion, but the program has silently changed an unknown fact into a known one. This matters anywhere an hour can change the result: medical timelines, transport schedules, legal deadlines, astronomical calculations, historical records, and calendrical systems. I encountered the problem while working with a BaZi calculation pipeline. A BaZi chart can use year, month, day, and hour components. If the birth time is absent, the honest result is a three-component analysis with hour-dependent conclusions withheld. Inserting noon would make the output look richer while making its provenance weaker. The useful engineering question is not “Which fallback time should we choose?” It is “How do we keep uncertainty visible through every layer of the system?” The public calculation evidence repository provides the concrete calendar-domain fixtures referenced below. The rest of this article focuses on the reusable software boundary behind them. Model knowledge, not just a string A common input model makes absence too easy to erase: const birthTime = form . time || " 12:00 " ; After this line runs, downstream code cannot tell whether noon came from the user or the fallback. Validation, analytics, caching, and the result renderer all see the same string. The information loss happens before the calculation begins. A small discriminated union keeps the two states separate: /** * @typedef {{ kind: "known", localTime: string, source: "user" }} * KnownTime * @typedef {{ kind: "unknown" }} UnknownTime * @typedef {KnownTime | UnknownTime} BirthTime */ function parseBirthTime ( value ) { const normalized = value ?. trim (); return normalized ? { kind : " known " , localTime : normalized , source : " user " } : { kind : " unknown " }; } This typ

Daniel Doubrovkine explains why traditional LeetCode whiteboard interviews fail to evaluate senior engineering talent. He discusses his own experience bombing basic algorithm tests despite decades of leadership, and shares actionable frameworks for redefining the interview loop. Discover how evaluating human judgment, system design, and hands-on AI collaboration yields far better hiring signals. By Daniel Doubrovkine

This article presents a defense-in-depth approach for securing Model Context Protocol (MCP) deployments in production. It outlines four architectural control layers: safe execution, management infrastructure, outbound trust, and semantic integrity, arguing that production security requires enforcement beyond the gateway at the earliest trustworthy control points. By Nik Kale

I had a simple-sounding question: does ChatGPT recommend this business? You'd think you just ask it. Ask ChatGPT "best personal injury law firm in NYC", see if the business is named, record yes or no. That works exactly once. Ask again an hour later and you might get a different answer. Not slightly different — potentially a completely different set of firms and a completely different set of cited sources. Which means the naive version of this measurement is worthless. You're not measuring visibility, you're sampling a distribution once and calling it a fact. This is the same problem anyone gets when they try to test an LLM-backed feature. Your normal testing instinct — same input, assert on output — just doesn't apply. So here's how I ended up designing around it, and the numbers that came out, which surprised me. The setup I wanted to compare four assistants (GPT-4o, Claude Haiku 4.5, Gemini 2.5 Flash, Perplexity Sonar, all with web search on) across 10 buyer-intent questions in one vertical. Something like: "Best personal injury law firm in New York City?" "Top immigration lawyers in Mumbai?" For each response I recorded two things: which businesses got named, and which URLs got cited. The cited sources come from each API's own citation metadata, so that part is structured — no scraping the prose. First pass, the results looked dramatic. The four assistants barely agreed on anything. Different firms, different sources, almost no overlap. Great finding. Except I couldn't publish it, because there was an obvious objection I couldn't answer: Maybe they weren't disagreeing with each other. Maybe each one was just disagreeing with itself. If a single assistant returns wildly different sources run to run, then "these four models cite different things" is a meaningless statement. You'd be measuring noise and calling it signal. The control The fix is the same idea as a control group. Measure the thing you're worried about, separately, and see if it explains your result.

How a Missing Database Index Turned an 8ms API into an 8-Second Nightmare Every developer...

Any application that ships across borders needs a way to name a country. You reach for a two-letter code, write US , JP , DE , and move on. Then a support ticket arrives. A user in Belfast picked "United Kingdom" and your shipping API rejected UK . Someone in Pristina found no option at all. Your analytics dashboard shows a country called AN that dissolved in 2010. These bugs share one root: ISO 3166-1 alpha-2 carries more rules than its two characters suggest. Let's walk through the parts that break real applications, and how to model country data so the next revision of the standard does not break yours. Key takeaways ISO 3166-1 defines 249 officially assigned alpha-2 codes. UK is not one of them. The United Kingdom is GB . Four other status categories exist: user-assigned, exceptionally reserved, transitionally reserved, and indeterminately reserved. They follow different rules. Kosovo uses XK , a code from the user-assigned range that ISO has never officially assigned. Codes get recycled. CS meant Czechoslovakia, then Serbia and Montenegro. Country names change far more often than their codes. Store the code, resolve the name at render time. What alpha-2 covers ISO 3166 splits into three parts. Part 1 names countries and their dependent territories. Part 2 names subdivisions inside them. Part 3 records codes that fell out of use. Part 1 gives you three code sets for the same entity: Format Japan Notes Alpha-2 JP Two letters. Used by ccTLDs, BCP 47 language tags, payment APIs. Alpha-3 JPN Three letters. Easier to read on its own. Numeric-3 392 Digits from UN M49. Script-independent, survives alphabet changes. Alpha-2 is the set you meet most often. Two characters fit anywhere, and the Internet Assigned Numbers Authority (IANA) draws the country-code top-level domains straight from the alpha-2 list, which puts these codes in front of everyone who ever registered a domain. That reach explains the misuse. Five kinds of code The 249 official codes get the attention.

Every team I've watched ship a broken measurement system broke it the same way. Not with bad math — with an org chart problem that happened to live in code. The entity making the claim ended up being the entity that decided whether the claim was right. Once you have the shape in your head you start seeing it everywhere. Three roles, not two Most engineers think about measurement as two roles: the thing that acts, and the thing that grades it. That's one role short. There are three: Player — makes the claim. Your model, your service, your PR. Scorer — applies the rubric. Your eval harness, your test suite, your metrics dashboard. Settler — determines what actually happened. Production outcomes. Reality. The scorer is a proxy. The settler is the thing the proxy is trying to approximate. The rule: be the scorer, never the settler. When the player captures the settler, the loop closes on itself and the system can no longer be wrong — which sounds like success and is actually the failure. What it looks like in code Tuning on the test set. You check test accuracy, adjust hyperparameters, check again. Twenty iterations later the test set is training data with extra steps. The player is now selecting its own settler. That's what overfitting is , structurally — not a math failure, a role-collapse failure. LLM-as-judge from the same family. Your generator is GPT-flavored and your judge is GPT-flavored. They share pretraining data, failure modes, and blind spots. The judge doesn't rate quality — it rates similarity to what it would have produced. Correlated error is invisible to averaging; running it 1,000 times makes you more confident of the same wrong answer. Benchmark contamination. The model scores 94% on the benchmark that's in its training data. Nobody lied. The settler just quietly moved inside the player. Self-reported health. A service that returns its own health check is a claimant ruling on its own claim. If the process is wedged, the check is wedged too, and your

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't. I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong. Why clustering shrinks your real sample size Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half. The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately: Deff = 1 + (m̄ - 1) · ICC where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is: n_eff = n / Deff That is the number of independent examples your clustered set is actually worth. The number Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher). Deff = 1 + (5 - 1) · 0.3 = 2.2 n_eff = 200 / 2.2 ≈ 91 Your 200-example eval is worth about 91 independent examples. The correct confidence interval

Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fails. The answer used to default to Airflow because there wasn't a real alternative. Now there are three reasonable defaults, and they optimize for different things. Picking wrong doesn't break anything on day one — it shows up eighteen months later as either an operations team drowning in scheduler maintenance or an engineering team fighting a platform that won't do what they need it to. Here's the actual tradeoff, not the vendor pitch version. Databricks Workflows: the path of least resistance, if you're all-in on Databricks Databricks Workflows is the orchestrator built into the platform. Jobs, clusters, Unity Catalog permissions, and Workflows all share the same control plane, which means you're not maintaining a separate scheduler, not managing a second set of credentials, and not debugging why an external system can't see a table that Unity Catalog says it can. Task dependencies, retries, cluster reuse across tasks, and job-level alerting all come for free. The cost is exactly what you'd expect from a platform-native tool: it orchestrates Databricks well and everything else poorly. There's no first-class way to trigger a task in your orchestration DAG that waits on a Salesforce export, calls an internal API, or coordinates a dbt run against a warehouse that isn't Databricks SQL. You can bolt these in with webhooks and external scripts, but you're fighting the tool rather than using it. Workflows also doesn't give you the asset-lineage or testing story that Dagster does — it schedules tasks, not data assets. If your data platform genuinely is Databricks end to end — ingestion, transformation, ML, serving — Workflows removes an entire category of operational overhead you'd otherwise be paying for nothing. Teams in this position who reach for Airflow anyway usually do it out of habit, not need, and end up running two sch

Before we start This is a series about the intermediate results of an ongoing effort, not a finished story. It isn't an academic paper — it's a record of real engineering work and the insights that emerged along the way. Also, it's not about AI generating code. The AI angle here is about investigation and research — a careful, governed use of AI as a tool, not an autopilot — something I'll come back to in the final part. A word on why now, with the project still unfinished: details fade — the small technical decisions, the intermediate observations, the context in which a given call was made. Writing this down while the work is still ongoing is partly how I keep that context from slipping away. And that context matters: it's a reminder that every past decision, mine or anyone else's, was made for reasons that made sense at the time. One more note: none of this happened instead of product work. All of it ran alongside building new features and fixing bugs — the roadmap never paused for it. On confidentiality: I don't name the Customer, and I avoid any personal data or details a competitor could use. For the same reason, I don't mention anyone by name and refer to colleagues only by role. I won't name them, but I want to acknowledge up front that much of what follows was only possible thanks to the people I work with. The numbers are approximate and rounded — the point is the order of magnitude and the reasoning, not the exact figure. And a framing to carry through the series: at this scale, optimization is less a sprint than a marathon — yes, probably the most overused metaphor around, but here it genuinely fits: steady pacing beats sprinting, and you get there one careful step at a time. How I ended up here I'm a software engineer, and I've spent most of my career close to backends and databases. I've also led teams as a technical team lead — though over time I've deliberately shifted back toward more hands-on technical roles, which is where I'm most effective and m

The final condition will shock you! (It won’t. It’s taxes.)

The largest grid operator in the U.S. says it will cut power to large data centers to prevent blackouts starting next year.

If you have ever tried to find a new physician through a search box, you already know the frustration: outdated phone numbers, doctors who left the practice two years ago, and "accepting new patients" labels that turn out to be fiction. Anyone who has read the candid breakdown in Online Doctor Directories: A User's Guide to a Very Imperfect Tool will recognize the pattern immediately, because the core problem is not laziness on anyone's part — it is a data engineering problem hiding inside a healthcare product. And for those of us who build software for a living, it is a fascinating case study in what happens when stale data meets high-stakes decisions. The Root Cause Is a Data Pipeline, Not a Design Flaw Most doctor directories aggregate information from insurance networks, state licensing boards, hospital affiliations, and self-reported provider profiles. Each of these sources updates on its own schedule, uses its own identifiers, and defines fields differently. One system records a physician under her maiden name; another lists the clinic's billing address instead of the practice location; a third still shows a specialty she stopped practicing in 2019. The result is a classic entity-resolution nightmare. Without a reliable primary key shared across sources, merge logic has to guess whether "J. Martinez, Internal Medicine, Suite 400" and "Julia Martinez-Reyes, IM" are the same human. Get it wrong in either direction and the user suffers: duplicates erode trust, while over-aggressive merging attaches one doctor's malpractice history to a stranger with a similar name. If you have ever built a CRM deduplication service or wrestled with customer identity graphs, you have fought this exact battle — just with lower stakes. Staleness compounds the problem. Physicians change practices constantly. A directory that syncs quarterly is, by definition, wrong about a meaningful slice of its records at any given moment. Harvard Health has pointed out that an ongoing physician sh

Grafana Labs has expanded the capabilities of Grafana Assistant, enabling its AI-powered observability assistant to query and correlate data across more than 30 different data sources through natural language. By Craig Risi