标签：#Data

If you have ever tried to find a new physician through a search box, you already know the frustration: outdated phone numbers, doctors who left the practice two years ago, and "accepting new patients" labels that turn out to be fiction. Anyone who has read the candid breakdown in Online Doctor Directories: A User's Guide to a Very Imperfect Tool will recognize the pattern immediately, because the core problem is not laziness on anyone's part — it is a data engineering problem hiding inside a healthcare product. And for those of us who build software for a living, it is a fascinating case study in what happens when stale data meets high-stakes decisions. The Root Cause Is a Data Pipeline, Not a Design Flaw Most doctor directories aggregate information from insurance networks, state licensing boards, hospital affiliations, and self-reported provider profiles. Each of these sources updates on its own schedule, uses its own identifiers, and defines fields differently. One system records a physician under her maiden name; another lists the clinic's billing address instead of the practice location; a third still shows a specialty she stopped practicing in 2019. The result is a classic entity-resolution nightmare. Without a reliable primary key shared across sources, merge logic has to guess whether "J. Martinez, Internal Medicine, Suite 400" and "Julia Martinez-Reyes, IM" are the same human. Get it wrong in either direction and the user suffers: duplicates erode trust, while over-aggressive merging attaches one doctor's malpractice history to a stranger with a similar name. If you have ever built a CRM deduplication service or wrestled with customer identity graphs, you have fought this exact battle — just with lower stakes. Staleness compounds the problem. Physicians change practices constantly. A directory that syncs quarterly is, by definition, wrong about a meaningful slice of its records at any given moment. Harvard Health has pointed out that an ongoing physician sh

Grafana Labs has expanded the capabilities of Grafana Assistant, enabling its AI-powered observability assistant to query and correlate data across more than 30 different data sources through natural language. By Craig Risi

Ben Greene discusses how software engineers can adapt and thrive in an era of rapid AI code automation. Drawing on his startup experience, he explains key mindsets like starting simple, maintaining code comprehension, attacking hard problems first, and focusing on customer impact. He shares why human empathy, agency, and practical problem-solving remain irreplaceable when code is automated. By Ben Greene

Modern electrical grids increasingly rely on distributed sensors installed across conductors, towers, poles, substations, and remote line sections. These devices can measure: Conductor temperature Current and voltage Mechanical tension Line sag Vibration Weather conditions Fault passage Switch and recloser states Collecting these measurements is relatively straightforward. Building a reliable data pipeline around them is much harder. Power infrastructure often operates in locations with unstable connectivity, limited bandwidth, and strict requirements for alarm delivery. A useful architecture must therefore do more than move telemetry from sensors to a cloud database. It must determine which data is urgent, validate measurements, preserve event order, survive network outages, and integrate the results with operational utility systems. This article explores how to design that pipeline. The Basic Architecture A practical grid-monitoring data flow may look like this: Field Sensors | v Protocol Adapters | v Edge Data Model | +----> Local Rules and Fault Detection | +----> Local Time-Series Buffer | +----> Event Queue | v Central IoT or Utility Platform | +----> SCADA +----> GIS +----> OMS +----> Analytics +----> Maintenance Systems The edge gateway sits between field equipment and central applications. Its job is not limited to protocol conversion. It also acts as a local data-processing and reliability layer. Why Cloud-Only Processing Is Risky Imagine a utility operating 5,000 field sensors. Each device reports one measurement every second. That produces: 5,000 measurements per second 300,000 measurements per minute 18,000,000 measurements per hour Most of those measurements will describe normal operating conditions. Sending every individual value to a central platform creates unnecessary: Bandwidth consumption Storage growth Processing overhead Communication costs Dependence on network availability More importantly, cloud-only logic can stop working when the connectio

AWS released a public preview of the GuardDuty investigation agent, which correlates findings, 90-day activity logs, and resource topologies into structured reports with risk ratings, confidence scores, and MITRE ATT&CK classification. It is reachable through the AWS MCP Server, so investigations can run from agentic tooling. Preview quotas cap usage at 10 investigations per account per day. By Steef-Jan Wiggers

Published on Aug 18, 2025 A New Era of AI-Powered Coding Begins I have installed Cursor on my laptop this weekend, and I am amazed at how much it speeds up my coding. I have a new debugging buddy!! This week, I have made several updates to the Portfolio website. The Challenge: When OpenAI Falls Short In my previous post, I shared the excitement of implementing a chatbot based on ChatGPT for my portfolio website. The initial experience was promising - I successfully created content embeddings and integrated them with OpenAI's API. However, as many developers know, relying on a single service provider can lead to unexpected roadblocks. When my OpenAI account encountered issues, I faced a critical decision: abandon the chat functionality or find an alternative solution. I chose the latter, embarking on a journey that would transform my portfolio's AI capabilities and teach me valuable lessons about building robust, fallback-ready systems. The Migration: Embracing Open Source AI The transition from OpenAI to Hugging Face wasn't just a simple API swap - it was a complete architectural evolution. Here's what I learned: 1. Model Selection Complexity Finding the right model on Hugging Face proved more challenging than expected. After testing several options: microsoft/DialoGPT-medium - No inference provider available gpt2 and distilgpt2 - Limited conversational capabilities Qwen/Qwen3-4B - Perfect fit with the nebius provider 2. Database Architecture Evolution The migration also prompted a database upgrade from MongoDB to Neon PostgreSQL. This wasn't just about changing providers - it was about building a more scalable, production-ready foundation for my portfolio. Technical Implementation: Building Resilience Streaming Responses for Better UX One of the most significant improvements was implementing streaming text responses. Instead of waiting for complete AI responses, users now see text appear word-by-word, creating a ChatGPT-like experience: // Streaming implementation

Telecom expects AI revenue from dark fiber deals and retrofitted data centers.

INTODUCTION Creating Greenwood academy database is essential for managing the students, subject and exam results efficiently. PostgreSQL, a powerful open-source relational database system, offers the perfect foundation for such a project. The main areas areas in SQL covered in this projects are : 1. DDL (Data Definition Language) DDL commands define, modify, and change the physical structure of database objects like tables and schemas. The first step is to create a greenwood academy schema using the create command. create schema greenwood_academy ; set search_path to greenwood_academy ; Next is to crete tables in the schema; The schema has 3 tables students,subject and exam results. create table greenwood_academy . students ( student_id INT PRIMARY key , first_name VARCHAR ( 50 ) NOT null , last_name VARCHAR ( 50 ) NOT null , gender VARCHAR ( 1 ), date_of_birth DATE , class VARCHAR ( 10 ), city VARCHAR ( 50 ) ); create table greenwood_academy . subject ( subject_id INT PRIMARY key , subject_name VARCHAR ( 100 ) NOT null unique , department VARCHAR ( 50 ), teacher_name VARCHAR ( 100 ), credits INT ); create table greenwood_academy . exam_results ( result_id INT PRIMARY key , student_id INT NOT null , subject_id INT NOT null , marks INT NOT null , exam_date DATE , grade VARCHAR ( 2 ) ); ALTER - This command changes the structure of tables in a database. Core Actions You Can Perform Add columns : Insert a new column and its data type into a table. The school realised that the nthey forgot to add phone numbers in the students table. The following command is used to add the data alter table greenwood_academy . students add column phone_number VARCHAR ( 20 ); Rename colums : Change the name of a table or a column. The column credit has to be changed to credit hours alter table greenwood_academy . subject rename column credits to credit_hours ; Drop columns : Delete an unwanted column from a table. Later the school relised that the phone number column is nolonger needed. a

SQL is quite flexible, making it easy to write a single query that works for two situations: one without a parameter and a WHERE clause, and another with a parameter for filtering, all in the same SQL query. For example, I came across a benchmark comparing MongoDB and PostgreSQL that shows how to handle pagination effectively—by avoiding OFFSET and instead using the last value to fetch the next set of results. The first page includes a WHERE clause along with ORDER BY and LIMIT, while the following pages add an extra WHERE condition. In the MongoDB version of this benchmark, the filter is handled within the application, which leads to two separate queries for these scenarios. export async function getOrders ( cursor ) { const match = cursor ? { timestamp : { $gt : new Date ( cursor ) } } : {}; const rows = await orders . aggregate ([ { $match : match }, { $sort : { timestamp : 1 } }, { $limit : PAGE_SIZE }, ]) We can do the same in PostgreSQL using a single prepared statement. SQL is such a powerful language that it often feels tempting to write it this way: SELECT * FROM orders WHERE $ 1 :: timestamptz IS NULL OR "timestamp" > $ 1 ORDER BY "timestamp" ASC LIMIT $ { PAGE_SIZE } If $1 is NULL, it skips the second condition in the OR clause and retrieves all rows without filters, resulting in a broad fetch. When $1 has a value, it filters the results using that specific value, enabling a more targeted search. However, using a generic query can sometimes lead to a less-than-ideal execution plan that's not perfectly tailored for each specific situation. I gave it a try: drop table if exists orders ; create table orders ( order_id text primary key , "timestamp" timestamptz not null ); create index idx_orders_timestamp on orders ( "timestamp" ); insert into orders select 'ORD-' || g , '2025-01-01' :: timestamptz + g * interval '1 minute' from generate_series ( 1 , 5000000 ) as g ; analyze orders ; prepare getorders ( timestamptz , int ) as select * from orders where $ 1 :

Most experiments that come back "no clear winner" were unwinnable on the day they launched. The data could not resolve an effect that size, and no amount of extra runtime was going to change that. You can find this out in about two minutes, before you spend anything, with one formula and a resampling pass over your own data. Here is the check, in three steps. Step 1. Compute the smallest lift your data can see For a two-arm test on a conversion rate, the smallest lift detectable at 95% confidence and 80% power is a one-liner: from math import sqrt Z_ALPHA = 1.96 # two-sided 95% Z_BETA = 0.84 # 80% power def mde ( baseline_cvr : float , n_per_arm : int ) -> tuple [ float , float ]: """ Minimum detectable effect: absolute (pp) and relative (%). """ se = sqrt ( 2 * baseline_cvr * ( 1 - baseline_cvr ) / n_per_arm ) abs_lift = ( Z_ALPHA + Z_BETA ) * se return abs_lift * 100 , abs_lift / baseline_cvr * 100 At a 3% conversion rate: clicks per arm smallest lift you can detect 5,000 +32% relative 20,000 +16% relative 100,000 +7% relative Read the middle row twice. Twenty thousand clicks per arm is a serious amount of traffic for a mid-market account, and a real 15% improvement still lands inside the confidence interval. The report will say "inconclusive," and the team will read that as a verdict on the idea. It is a verdict on the instrument. Invert the same formula and the planning question gets easier: at 3% baseline, detecting a 10% lift needs about 51,000 clicks per arm, and detecting a 5% lift needs about 203,000. If your account produces 8,000 clicks a month, you now know the honest answer to "how long should we run this." Step 2. Stop assuming your conversions are independent The formula above treats every click as an independent coin flip with the same probability. Account data does not behave that way, and the gap is not small. In a corpus of 31 advertiser accounts I maintain for diagnostic work (9.46 million search term rows, roughly $133M of spend, September 2024

Johannes Schickling explores how he moved beyond the traditional three-tier web stack to a local-first approach. He shares his experience transitioning from Prisma to developing Overtone—a music curation app leveraging client-side event sourcing and SQLite. The discussion highlights the trade-offs between event sourcing and CRDTs. By Johannes Schickling

Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating Meta description: Peeking at a fixed-sample A/B test inflates false positives. Sequential testing lets you check results repeatedly and stop early without cheating. TL;DR Fixed-sample testing assumes you'll wait for a pre-calculated sample size before looking at results. Checking early and stopping the moment you see significance — "peeking" — quietly inflates your real false-positive rate, often far above the 5% you think you're getting. Abraham Wald's Sequential Probability Ratio Test (SPRT), developed for wartime quality control, is the mathematically rigorous alternative: a procedure built to be checked repeatedly, with pre-calculated boundaries that keep the false-positive rate honest by construction. The difference between the SPRT and peeking isn't willpower — it's that the SPRT's stopping rule is part of the math from the start, so stopping early doesn't cost you anything in error-rate control. Sequential design is the right call when traffic is limited, the cost of running a test too long is high, or the business genuinely can't commit to waiting for a fixed horizon — not a substitute for rigor, but a different kind of rigor suited to a different constraint. This is a methodology choice, not a shortcut — and it's one input into the broader question of how much certainty a given bet needs, covered in the Confidence Tier Model . Every experimentation program eventually hits the same moment: a test has been live for four days, the dashboard shows a lift, and someone — a stakeholder, a PM, sometimes you — asks "can we call it?" The honest answer depends entirely on what kind of test you designed, and most teams don't have a clean answer, because most teams designed a fixed-sample test and are now trying to read it like a sequential one. Those are not interchangeable. Knowing the difference, and choosing deliberately between them before the test starts, is the actual skill — not "wait lon

The Confidence Tier Model: How to Decide When Your Data Isn't Enough Meta description: Most testing programs are built for traffic they don't have. Three confidence tiers — proven, directional, speculative — each with its own bet-sizing rule. TL;DR Fixed-sample A/B testing assumes you can wait for statistical significance. Most teams can't — traffic is too thin, or the market is moving too fast to wait. The fix isn't lowering your standards. It's replacing the binary "significant / not significant" gate with three explicit confidence tiers — Proven, Directional, Speculative — each with its own evidence bar and its own bet-sizing rule. Underpowered tests systematically overestimate effect size (the "winner's curse" ). A confidence tier that accounts for this is more honest than a p-value that pretends otherwise. The way to move a learning up a tier isn't more of the same test — it's triangulation: stacking correlated, individually-weak signals until they converge. This is a methodology choice, not a compromise. Teams that name their confidence tier explicitly make faster, more defensible decisions than teams that either wait for certainty they'll never reach, or ship everything with false confidence. A product manager says: "Users want better deals." A brand marketer says: "TV is driving more direct demand." A performance marketer says: "This channel has a strong ROAS." Finance says: "But is this incremental?" Product says: "Will this hurt user trust?" Leadership says: "Should we scale this?" Six people, six kinds of evidence, and a decision that needs to get made this quarter — not whenever a test finally clears p<0.05. This is the actual job: not running tests, but converting six competing claims into one evidence base leadership can act on. Most experimentation methodology is written for a world where you have the traffic to wait for a clean answer. Most companies don't live in that world. The problem classic A/B testing doesn't solve Fixed-sample significance tes

Regression and regularization are both important concepts in machine learning and statistics, but they solve different problems. Regression is primarily used to model relationships and make predictions. Regularization is used to improve a model's ability to generalize by controlling its complexity. Regression This is a statistical and machine learning technique used to predict a continuous numerical outcome based on one or more input variables. For example, we might want to predict: A house's price based on its size and location A student's exam score based on study hours A company's sales based on advertising spending Simple Linear Regression In simple linear regression, we model the relationship between an input variable (x) and an output (y): $$ y = \beta_0 + \beta_1x + \epsilon $$ Where: (y) is the predicted outcome (\beta_0) is the intercept (\beta_1) is the coefficient or slope (x) is the input variable (\epsilon) represents the error The model learns values for (\beta_0) and (\beta_1) that make its predictions as close as possible to the actual values. Multiple Linear Regression In multiple linear regression, several predictors are used: $$ y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \cdots + \beta_px_p + \epsilon $$ The goal is typically to minimize the sum of squared errors (SSE) : $$ \text{SSE} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 $$ This approach is known as Ordinary Least Squares (OLS) . Regularization Regularization is a technique used to prevent a machine learning model from becoming too complex. A model can perform extremely well on training data but poorly on new, unseen data. This problem is called overfitting . Regularization addresses overfitting by adding a penalty for large model coefficients to the model's objective function. Instead of minimizing only the prediction error, the model minimizes: $$ \text{Prediction Error} + \text{Complexity Penalty} $$ The penalty discourages the model from relying too heavily on individual features. The Main Types o

The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew , and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformation Diagnosing and fixing data skew with salting and adaptive query execution (AQE) Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant files Governing access to the whole pipeline with Unity Catalog Architecture overview Pipeline shape — a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage — this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1 — Set up governed tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics ; CREATE SCHEMA IF NOT EXISTS retail_analytics . events ; CREATE TABLE IF NOT EXISTS retail_analytics . events . raw_orders ( order_id STRING , customer_id STRING , product_id STRING , quantity INT , event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders' ; CREATE TABLE IF NOT EXISTS retail_analytics . events . dim_products ( product_id STRING , category STRING , unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products' ; GRANT SELECT ON TABLE retail_analytics . events . raw_orders TO `analysts` ; Step 2 — Read and force a broadcast join for the small dimension table dim_products is s

The moment that started it A colleague of mine was cleaning up some old records in a staging environment. Same query he'd run a dozen times before, except this time he was connected to production. He hit F5. No WHERE clause. 47,000 rows gone in milliseconds . We recovered from a backup, lost about two hours, and nobody got fired. But it stuck with me:** SSMS will let you delete an entire production table with the same amount of friction as running a SELECT 1**. No pause, no confirmation, nothing. The tool that DBAs and backend developers spend all day in has zero built-in protection against the single most common way people destroy data. So I built one. What SQL Guard does SQL Guard is a free SSMS extension (18 through 22) that inspects your query the moment you press F5, and pauses execution if it matches a known destructive pattern: DELETE without WHERE UPDATE without WHERE TRUNCATE TABLE DROP TABLE / DROP DATABASE / DROP PROCEDURE ALTER DATABASE Dangerous EXEC calls MERGE without a filter When one of these fires, you get a dialog showing exactly what object would be affected, with three options: cancel, run anyway, or run and ignore for the rest of the session. The point isn't to block you — it's to make the action conscious. Most accidental damage happens because muscle memory took over, not because someone genuinely meant to wipe a table. How the detection works Nothing exotic here, and honestly that's by design. SQL Guard runs a lightweight pattern-matching pass over the query text before it hits the connection. No parsing tree, no round-trip to the server, no measurable latency — it runs in under a millisecond, so you never notice it on normal queries. The core idea is simple: certain statements are only safe when scoped by a WHERE clause, and the extension checks for that clause's absence rather than trying to understand the full semantics of the query. That keeps false positives low and means it works the same way whether you're on SQL Server 2016 or the la

"What if satellite data could help protect the livelihoods of millions who depend on Africa's largest lake?" Our team JONAM had the privilege of participating in the Kijani Space Hackathon, where we proudly secured 3rd place while tackling Challenge 2: Sustainable Fisheries & Blue Economy. Rather than building another dashboard, we wanted to solve a real problem affecting millions of people around Lake Victoria: declining fish stocks caused by worsening water quality. Lake Victoria supports millions of people through fishing, transportation, agriculture, and tourism. However, over the years the lake has experienced: Increasing water pollution Poor water quality Frequent algal blooms Reduced fish breeding habitats Declining fish populations For fishing communities, these are not just environmental issues—they directly affect livelihoods, food security, and local economies. Our question became: Can Earth observation data help communities understand where water conditions are becoming unsuitable for fish before the problem becomes critical? Our Solution: JONAM JONAM is an AI-powered web application that combines satellite-derived environmental data with machine learning to monitor water quality and provide insights into conditions that may contribute to declining fish stocks. Instead of relying solely on manual sampling—which is expensive and only covers small areas—our platform continuously analyses satellite observations covering the entire lake. Why Copernicus? To build JONAM, we integrated the KijaniBox API, which provides access to environmental datasets from the Copernicus Programme. Copernicus is the European Union's Earth observation programme. It uses a constellation of Sentinel satellites together with in-situ observations to monitor Earth's atmosphere, land, and oceans. For our project, we focused specifically on live water telemetry variables available through the KijaniBox platform. Water Temperature Satellites measure the thermal radiation emitted from th

When I first heard the term ETL (Extract, Transform, Load), I thought it sounded like something only advanced data engineers dealt with. But as I’ve been learning, I realized ETL is the backbone of almost every data project. It’s the process that makes raw data usable, and without it, analysis can quickly fall apart. The first lesson was short but powerful. ETL is about moving data from one place to another, transforming it along the way so it’s clean and ready for analysis. I remember thinking: “So this is how companies make sense of the chaos in their databases.” It felt like peeking behind the curtain of how insights are really built. Then came the part about Excel macros. At first, I was intimidated, macros sounded complicated. But once I tried them, I realized they’re like little helpers that automate repetitive cleaning tasks. Instead of manually fixing hundreds of rows, I could write a macro and let Excel do the heavy lifting. It felt like discovering a secret shortcut. I even laughed at myself when I realized how much time I had wasted before, manually correcting data. This was a turning point: I started to see how automation can save not just minutes, but hours. Finally, I explored Power Query. If macros are shortcuts, Power Query is like a full toolkit. It lets you connect to different data sources, transform them, and keep everything organized. I loved how visual it was dragging, dropping, and shaping data felt almost creative. I remember thinking: “This is what makes data cleaning less of a chore and more of a craft.” It gave me confidence that even messy datasets could be tamed. Learning ETL, macros, and Power Query taught me that data cleaning isn’t just technical, it’s about mindset. It’s about respecting the data, being patient, and finding smarter ways to work. I used to think cleaning data was boring, but now I see it as the foundation of every meaningful insight. Without clean data, analysis is just noise. ✨ Takeaway: If you’re starting out in dat

A close call in Northern Virginia revealed just how poorly data centers respond to grid disruptions. Here's how to fix the problem.

Museums are embracing data-driven curation and a shifting technology landscape.