标签：#analytics

DuckDB Labs has previewed DuckDB v2.0, codenamed "Cyanoptera." This release includes over 10000 commits and introduces a client/server mode, enabling network connections. Improvements also encompass extension portability, advanced data types, and a new parser. Performance enhancements include asynchronous I/O and storage optimisations. General availability is expected in fall 2026. By Olimpiu Pop

A week cleaning 290 booking records taught me more about silent failure than any error message ever has Last week I cleaned a deliberately messy dataset; 290 booking records from Safari Connect, Nairobi bus platform, 21 columns, 23 catalogued data problems. Class exercise, but the data was built from real failure modes. The problems I'd been warned about took an afternoon. The ones that cost me were the five that ran perfectly, returned plausible output, and were wrong. Every one of these produced a result. None produced an error. 1. The date heuristic that silently dropped five bookings The dataset had three date formats in one column: 2024-09-15 , 15/09/2024 ,and 09-25-2024 . Two of those are ambiguous - 01-18-2024 is unmistakably MM-DD-YYYY because there's no month 18, but 04-10-2024 could be either. The supplied guide handled it like this: UPDATE bookings_staging SET departure_date = TO_DATE ( departure_date , 'MM-DD-YYYY' ):: TEXT WHERE departure_date LIKE '%-%' AND LENGTH ( departure_date ) = 10 AND SPLIT_PART ( departure_date , '-' , 2 ):: INTEGER > 12 ; Read that last condition. If the second component is too large to be a month,this must be month-first. Reasonable logic - and it only fires when the day happens to be 13 or higher. Five rows had days between 1 and 12. They never converted. Then the next step filtered on ISO format: INSERT INTO bookings SELECT ... FROM bookings_staging WHERE departure_date SIMILAR TO '[0-9]{4}-[0-9]{2}-[0-9]{2}' ; ...and dropped them. No error. No warning. Five completed bookings and KES 3,840 of revenue gone from every downstream total. The guide's expected row count was written as "~280+", which is loose enough to hide it. The fix is to match on shape, not to infer from values: WHERE departure_date ~ '^ \d {2}- \d {2}- \d {4}$' Anchored patterns are mutually exclusive, so you can classify every row before touching any of it: SELECT CASE WHEN departure_date ~ '^ \d {4}- \d {2}- \d {2}$' THEN 'ISO' WHEN departure_date ~ '^ \d

Python's value comes not only from handling a great deal of data; its biggest asset comes from translating that data into meaningful business insight, and that business insight is used to make better business decisions. For businesses striving to increase customer satisfaction, enhance sales figures, and make smarter choices, a deep understanding of customer behavior is essential. Valuable business data includes customer transaction histories, website visits, product reviews, and responses to marketing efforts. When data such as this is analyzed, companies can effectively identify trends, understand preferences, and predict what their customers will do in the future. Python is the most popular when it comes to customer behavior analysis due to its comprehensive set of libraries, ranging from data cleaning, analysis, visualization, and machine learning; its flexibility makes it useful for new as well as seasoned data analysts. Why Analyze Customer Behavior? Customer behavior analysis assists businesses in answering key business questions such as: What are the products a customer buys most frequently? What spending figures do different customer groups have? Which customers are most likely to discontinue their service/products? What factors influence the customer's decision to purchase? Which marketing channels seem to receive the highest engagement? With answers like these, companies can implement targeted marketing campaigns, improve their product and services, customize experiences, and retain more customers. Key Python Libraries Some Python libraries that business data analysts use most frequently are: Pandas: Used for data cleaning, organizing, filtering, and manipulating datasets. NumPy: Provides a collection of high-level mathematical functions to perform numerical operations and work with arrays efficiently. Matplotlib: Enables users to create and plot static, animated, and interactive visualizations. Seaborn: An excellent library for plotting statistical graph

Imagine working as a Data Analyst in a healthcare Non-Governmental Organization (NGO) implementing HIV and AIDS programmes across several communities. The organization has limited resources. There may not be enough funding, healthcare workers, testing kits, transport, outreach teams, or community programmes to serve every community at the same intensity. This creates an important question: How can we use data and machine learning to direct limited programme resources to communities with the greatest need? This is where Machine Learning (ML) can become valuable. Rather than distributing resources equally across all communities, an NGO can use historical programme data to identify communities experiencing greater HIV-related service gaps or higher levels of need. Resources can then be prioritized based on evidence. What Is Machine Learning? Machine Learning is a branch of Artificial Intelligence that enables computers to learn patterns from data and use those patterns to make predictions or support decisions. Instead of manually creating rules for every situation, you provide the algorithm with historical data and allow it to identify relationships within that data. For example, the NGO could have this information about different communities: Community HIV Testing Coverage ART Coverage Missed Appointments Outreach Activities Community A 85% 90% 5% High Community B 52% 61% 25% Low Community C 70% 75% 15% Medium Community D 40% 55% 32% Low Looking at this data, Community D appears to have greater programme gaps than Community A. However, in a real programme, the decision should not be based on one indicator alone. Machine learning can analyse many variables simultaneously to identify communities that may require greater attention. Why Resource Allocation Matters in HIV Programmes HIV programmes operate in environments where resources are often limited. An NGO may have: A limited number of community health workers A fixed outreach budget Limited HIV testing supplies Limi

In modern Data Engineering, there is an unspoken fetish for "Real-Time." If you ask any business stakeholder how fast they need their dashboard to update, the default answer will always be: "As fast as possible." This drives well-intentioned engineers to design incredibly complex architectures. We spin up Kafka clusters, implement Flink, and wrestle with latency, late-arriving data, and tumbling windows. All to have data flowing in milliseconds. But the harsh reality is that the vast majority of companies are building Ferraris just to sit in rush-hour traffic. 1. The Actionability Gap (The Golden Question) The biggest mistake when choosing a streaming architecture isn't technical; it's a business mistake. Before implementing real-time pipelines, the only question that matters is: "Does the company have the operational capacity to make a decision in milliseconds?" If you are building a credit card fraud detection system or a live e-commerce recommendation engine, yes, every millisecond counts. But if the data is feeding a financial dashboard that the executive board only reviews during their Monday morning meeting, updating that screen every second is a colossal waste of money and effort. Real-time data has zero value if the human action is batch. 2. The Hidden Complexity and the Cloud Bill Batch processing is forgiving. If a pipeline fails at 3 AM, you trigger a rerun, and by 8 AM, everything is fine. Batch is cheap, predictable, and easy to debug. Streaming, on the other hand, is unforgiving. Handling application state, event duplication (exactly-once semantics), out-of-order events, and sudden traffic spikes requires a senior engineering team dedicated solely to keeping the infrastructure alive. Furthermore, the cloud bill for 24/7 continuous processing is orders of magnitude higher than spinning up your compute clusters on a schedule. 3. "Micro-Batch" Solves 99% of Your Problems There is a perfect middle ground that the hype industry tries to ignore: the micro-ba

Part of "AI, engineering and what survives production", a series on the parts of building with AI that hold up once real traffic hits them. There is a claim going round that you have probably absorbed by now: AI-assisted development is making codebases worse. Refactoring is down, duplication is up, we are all writing more and revising less. The numbers behind it are real, the samples are enormous, and I found I had started repeating the conclusion in conversation without ever having checked it. Then it occurred to me that those figures are averages taken across hundreds of millions of changes from thousands of organisations, not one of which is mine. So what is the rate in your repository? Nobody has told you, and on current evidence nobody is going to. I set out to find mine, assumed it would take an afternoon, and spent three days discovering that the answer is far harder to get at than the confident version suggests. So this is not a piece about what AI does to code. It is about how to ask that question of your own repository without arriving at a wrong answer, which turned out to be the genuinely difficult part. The tool I built to do it is git-habits : free, local, and it reads no source code whatsoever. What git can actually tell you Git history is a surprisingly rich behavioural record. Not of quality, about which it knows nothing at all, but of habits: how often you commit, how large those commits are, whether you go back and change what you wrote last month, and whether anybody still touches the old code. That is a narrower thing than quality and it is the thing the industry claims has changed, so it is the thing worth measuring. Four signals are computable from commit metadata alone, without opening a single source file: Moved lines. The share of changed lines sitting in files git detected as renamed or copied. It is the closest thing history offers to "somebody went back and reorganised this." Legacy touch. The share of changes landing on files nobody has

Data pipeline automation has moved from a technical aspiration to a business imperative. As organisations deploy more AI agents that need fresh, reliable data, the volume and complexity of data pipelines has grown beyond what manual management can sustain. AI-powered pipeline automation — using AI to build, monitor, and repair data pipelines — is the emerging solution. Key Insight: AI-automated data pipelines reduce pipeline development time by 60%, decrease pipeline failures by 45%, and enable data teams to manage 3x more pipelines per engineer compared to manual approaches. The Pipeline Scaling Challenge The average enterprise now maintains 1,500+ data pipelines, according to research by Barracuda Networks, and this number is growing 25% annually as organisations add new data sources, new AI use cases, and new reporting requirements. Each pipeline has an average of 4.2 transformation steps, 2.1 data quality checks, and connects an average of 2.8 systems. The total pipeline infrastructure is complex, fragile, and increasingly beyond the capacity of manual management. The scaling challenge manifests in three ways. First, pipeline development backlog — the average data team has a 3-6 month backlog of pipeline requests from business users. Second, pipeline failures — the average enterprise experiences 15-20 pipeline failures per week, each requiring manual investigation and repair that consumes 30-40% of the data engineering team's capacity. Third, pipeline maintenance — as source systems change (schema updates, API modifications, deprecated fields), pipelines break silently and produce incorrect results until someone notices. This 'silent failure' problem is particularly dangerous because it erodes trust in data without anyone being aware that anything is wrong. The root cause of these challenges is that data pipelines have been built as static, manually-maintained infrastructure. A pipeline is coded, tested, and deployed. When the source or destination changes, a hu

The ROI Black Hole in Social Marketing Consider a mid-market B2B software company whose social team manages campaigns across X, LinkedIn, Instagram, and TikTok from a single shared workspace. Each week the managers review platform-native dashboards that display rising follower counts, solid engagement rates on short-form video, and respectable click-throughs from carousel posts. They export weekly performance reports, paste the numbers into shared spreadsheets, and celebrate the month-over-month lift in impressions. Yet when the sales operations team asks which campaigns contributed to qualified pipeline, the social group cannot produce a single account-level match. Campaign links carry UTM strings, but many prospects arrive through mobile apps or shared links that strip those parameters, leaving the CRM with only anonymous referral domains and no usable journey data. The team attempts manual reconciliation by cross-referencing campaign dates with opportunity creation timestamps, but the exercise quickly collapses under volume. One campaign on LinkedIn might drive 400 clicks while another on TikTok drives 1,200, yet both appear in the CRM as undifferentiated social traffic. Without a consistent identifier that survives across platforms and into the marketing automation system, the social team cannot isolate which creative or audience segment produced the meetings that closed. Budget conversations therefore remain anchored to vanity metrics rather than incremental revenue, and executives grow increasingly skeptical of further platform spend. Medallion Architecture and the Absent Silver Layer Modern data platforms often organize information according to a medallion architecture that progresses through successive stages of refinement. The initial bronze layer captures raw event logs exactly as they arrive from each social API, preserving original timestamps, platform-specific identifiers, and unprocessed metadata. A subsequent silver layer then standardizes those recor

This guide is a collection or a summary on how to install and use a footprint of Apache Hadoop. I tried to follow an old version 2.7.1 guide that I created few years ago and adjusted this to use the latest version. Apache Hadoop 3.5.0 is used below; check the Apache releases page before future installations. These instructions target Linux (Ubuntu/Debian) for development or testing. Production clusters need Kerberos, network controls, encryption, monitoring, backups, and an upgrade plan. Do not expose HDFS or YARN ports to the internet. Native single-node installation Prerequisites sudo apt-get update sudo apt-get install -y openjdk-17-jdk openssh-client openssh-server pdsh curl tar java -version Hadoop requires Java and SSH; pdsh is recommended by the current Apache single-node documentation. Find JAVA_HOME if needed: readlink -f "$(command -v java)" | sed 's:/bin/java::' Download and install Pin the version for repeatable installs and verify Apache's SHA-512 checksum: export HADOOP_VERSION=3.5.0 cd /tmp curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz" curl -fLO "https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz.sha512" sha512sum -c "hadoop-${HADOOP_VERSION}.tar.gz.sha512" sudo tar -xzf "hadoop-${HADOOP_VERSION}.tar.gz" -C /opt sudo ln -sfn "/opt/hadoop-${HADOOP_VERSION}" /opt/hadoop sudo chown -R "$USER":"$USER" "/opt/hadoop-${HADOOP_VERSION}" Add this to ~/.bashrc, adjusting JAVA_HOME if necessary: export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 export HADOOP_HOME=/opt/hadoop export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop" export HADOOP_HDFS_HOME="$HADOOP_HOME" export HADOOP_YARN_HOME="$HADOOP_HOME" export HADOOP_MAPRED_HOME="$HADOOP_HOME" export PATH="$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin" Then load and verify it: source ~/.bashrc sed -i "s|^# export JAVA_HOME=.*|export JAVA_HOME=${JAVA_HOME}|" "$HADOOP_HOME/etc/hadoop/hadoop-env.sh" had

I watched my team get faster after we adopted AI-assisted coding, and honestly — I felt good about it. More tickets closed. Shorter cycle times. PR volume up. I remember thinking: this is what leverage looks like. That was the mistake. Not the tool. The assumption. Two weeks after a change shipped — passed every test, got through review, deployed cleanly — we found it had been quietly degrading a retry mechanism. It only broke under specific load conditions our test environments didn't replicate. Nobody caught it because it didn't look wrong. It worked. It just wasn't safe. When I asked the engineer to walk me through it, they could. The code made sense. They understood what each part did. But when I asked what would happen if the downstream service was slow — not down, just slow — there was a pause. Not because they weren't capable. Because they'd never needed to ask that question. The AI wrote the handling code, the tests passed, and the whole thing moved forward before that question ever came up. The dashboard wasn't lying. Things were shipping faster. It just wasn't showing me the part that mattered. What I didn't see coming Here's what actually surprised me: AI can make a team look more capable before it makes the team actually more capable. I used to learn by getting stuck. The 11 PM kind of stuck. Staring at a stack trace for three hours, genuinely questioning whether I understood any of this. That friction built something real — the instinct that says this probably works, but something feels off, and I should figure out what before we ship it. I built that through a migration that silently corrupted data for six hours. Through a caching layer that sailed through staging and failed on a Friday afternoon in production. That's not nostalgia for unnecessary suffering. The suffering was the mechanism. When the first draft is free, that mechanism stops. Learning used to happen inside the act of writing the code. Now it doesn't. And I'm genuinely not sure what repl

Metabase is an open-source BI tool for building charts and dashboards over MySQL, PostgreSQL, MongoDB, Redshift, and more. This guide deploys Metabase on Kubernetes, loads the Sakila sample dataset into MySQL, builds a dashboard, and secures it behind Nginx Ingress with cert-manager TLS. Prerequisites: a Kubernetes cluster with kubectl / helm configured, a Linux workstation, a reachable MySQL server, and a domain name. Load the Sakila Sample Database Sakila models a DVD rental store — films, actors, inventory, rentals. $ sudo apt install zip -y $ wget https://downloads.mysql.com/docs/sakila-db.zip $ unzip sakila-db.zip Connect to your MySQL server (replace host/port/user): $ mysql -h <HOST_ENDPOINT> -P <DATABASE_PORT> -u <ADMIN_USER> -p mysql > CREATE DATABASE sakila ; mysql > SOURCE sakila - db / sakila - schema . sql ; mysql > SOURCE sakila - db / sakila - data . sql ; Deploy Metabase $ nano metabase.yaml apiVersion : apps/v1 kind : Deployment metadata : name : metabase spec : selector : matchLabels : app : metabase replicas : 1 template : metadata : labels : app : metabase spec : containers : - name : metabase image : metabase/metabase:latest ports : - containerPort : 3000 protocol : TCP --- apiVersion : v1 kind : Service metadata : name : metabase-svc spec : type : LoadBalancer selector : app : metabase ports : - name : http port : 8080 targetPort : 3000 Your cloud provider may need a provider-specific LoadBalancer annotation here (e.g. to set the listener protocol) — check its Kubernetes docs if the default doesn't work. $ kubectl apply -f metabase.yaml $ kubectl get deployments $ kubectl get services Wait for metabase-svc to get an EXTERNAL-IP (can take a few minutes), then visit http://<external-ip>:8080 to confirm the Metabase welcome page loads. Connect Metabase to the Database Let's get started → pick language. Enter your name, email, company, and a password. Select your use case. Database engine: MySQL . Set a display name, then host/port/database/user/pa

Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating Meta description: Peeking at a fixed-sample A/B test inflates false positives. Sequential testing lets you check results repeatedly and stop early without cheating. TL;DR Fixed-sample testing assumes you'll wait for a pre-calculated sample size before looking at results. Checking early and stopping the moment you see significance — "peeking" — quietly inflates your real false-positive rate, often far above the 5% you think you're getting. Abraham Wald's Sequential Probability Ratio Test (SPRT), developed for wartime quality control, is the mathematically rigorous alternative: a procedure built to be checked repeatedly, with pre-calculated boundaries that keep the false-positive rate honest by construction. The difference between the SPRT and peeking isn't willpower — it's that the SPRT's stopping rule is part of the math from the start, so stopping early doesn't cost you anything in error-rate control. Sequential design is the right call when traffic is limited, the cost of running a test too long is high, or the business genuinely can't commit to waiting for a fixed horizon — not a substitute for rigor, but a different kind of rigor suited to a different constraint. This is a methodology choice, not a shortcut — and it's one input into the broader question of how much certainty a given bet needs, covered in the Confidence Tier Model . Every experimentation program eventually hits the same moment: a test has been live for four days, the dashboard shows a lift, and someone — a stakeholder, a PM, sometimes you — asks "can we call it?" The honest answer depends entirely on what kind of test you designed, and most teams don't have a clean answer, because most teams designed a fixed-sample test and are now trying to read it like a sequential one. Those are not interchangeable. Knowing the difference, and choosing deliberately between them before the test starts, is the actual skill — not "wait lon

The Confidence Tier Model: How to Decide When Your Data Isn't Enough Meta description: Most testing programs are built for traffic they don't have. Three confidence tiers — proven, directional, speculative — each with its own bet-sizing rule. TL;DR Fixed-sample A/B testing assumes you can wait for statistical significance. Most teams can't — traffic is too thin, or the market is moving too fast to wait. The fix isn't lowering your standards. It's replacing the binary "significant / not significant" gate with three explicit confidence tiers — Proven, Directional, Speculative — each with its own evidence bar and its own bet-sizing rule. Underpowered tests systematically overestimate effect size (the "winner's curse" ). A confidence tier that accounts for this is more honest than a p-value that pretends otherwise. The way to move a learning up a tier isn't more of the same test — it's triangulation: stacking correlated, individually-weak signals until they converge. This is a methodology choice, not a compromise. Teams that name their confidence tier explicitly make faster, more defensible decisions than teams that either wait for certainty they'll never reach, or ship everything with false confidence. A product manager says: "Users want better deals." A brand marketer says: "TV is driving more direct demand." A performance marketer says: "This channel has a strong ROAS." Finance says: "But is this incremental?" Product says: "Will this hurt user trust?" Leadership says: "Should we scale this?" Six people, six kinds of evidence, and a decision that needs to get made this quarter — not whenever a test finally clears p<0.05. This is the actual job: not running tests, but converting six competing claims into one evidence base leadership can act on. Most experimentation methodology is written for a world where you have the traffic to wait for a clean answer. Most companies don't live in that world. The problem classic A/B testing doesn't solve Fixed-sample significance tes

The dangerous part of AI analytics is not that a model may write a bad chart title. It is that one friendly question can turn into a warehouse query your user was never supposed to run. That risk is growing because builders are adding natural language analytics to products, dashboards, internal tools, support consoles, and agent workflows. Users want to ask, “Which accounts are slipping this month?” and get an answer. That is useful, and it is a permissions trap. If your AI analyst connects through one powerful service account, every customer question may inherit the same access. Your app may have perfect tenant checks in the UI, while the AI path quietly bypasses them. This guide shows how to design AI analytics row-level security so customers can ask useful questions without leaking rows, metrics, or private business context. Why this topic matters now Recent AI platform activity points in the same direction: builders are moving from “chat with documents” to “ask questions about live business data.” Developer pain points are consistent: safe natural language questions, tenant-scoped queries, auditable user identity, consistent metric definitions, and charts that do not expose raw tables. The search gap is clear. Many articles compare embedded analytics tools. Others explain database row-level security in isolation. Fewer walk through the product architecture for a customer-facing AI analyst that must handle tenant scope, natural language, semantic metrics, safe SQL, and audit evidence together. The core failure: one AI user, many real users Traditional analytics has a simple identity chain: Human user → app session → analytics permission → database query The database or BI layer knows who is asking. The app can apply tenant filters, role checks, and column restrictions. AI analytics often breaks that chain: Human user → app session → AI service → service account → database query Now the warehouse sees one identity: the AI service account. That account usually need

Ever chased a bug for days, only to discover the "bug" was actually the platform working exactly as designed? That happened to me building a client reporting pipeline. The lesson stuck. Here's what nobody tells you about pulling marketing data from multiple ad platforms: the hard part was never the dashboard. It was everything underneath it. The Setup That Looked Simple on Paper The brief sounded easy. Pull spend, clicks, and conversions from Google Ads and Meta. Store it. Display it in a chart. A junior dev could knock this out in a sprint, I figured. Reality disagreed. Google Ads API enforces operation quotas per developer token, and those quotas scale differently depending on account tier. Meanwhile, Meta's Marketing API throttles based on a rolling usage score tied to the ad account itself, not your app. Two platforms. Two completely different throttling philosophies. Neither documented in a way that made the actual limits obvious until you hit them in production. Where Things Actually Broke My first version polled every client account every hour. Fine for three clients. Then we onboarded client number twelve, and Meta started returning 429s intermittently. Not consistently — intermittently. That's the worst kind of bug. I initially assumed it was a code issue. Retry logic, maybe a race condition in my job scheduler. I spent three weeks going down that path. Eventually, I found the real cause: cumulative API call volume across all client accounts was tripping Meta's app-level rate limit, not the individual account limit. The fix wasn't more retries. It was a request queue with exponential backoff, plus a priority system so active dashboards refreshed before idle ones. Simple in hindsight. Expensive in dev hours. The Real Architecture Behind Multi-Platform Reporting If you're building this yourself, here's what a production-grade pipeline actually needs, based on what broke for me. A Queue, Not a Cron Job Don't just fire off API calls on a schedule and hope for t

I self-host Umami for my SaaS, ParserBee . The main reason I picked it: privacy-friendly, cookie-less, first-party analytics that adblockers supposedly leave alone because the script comes from your own domain. That last assumption turned out to be wrong, and the reason is the subdomain name itself. Writing it up because the fix is small and the failure mode is silent. The problem My Umami instance ran at umami.parserbee.com , so the tracker was loaded like this: <script defer src= "https://umami.parserbee.com/script.js" data-website-id= "..." ></script> The EasyPrivacy filter list (used by uBlock Origin, Brave Shields, AdGuard, and most other blockers) contains a rule that matches the Umami tracker by hostname pattern, along the lines of: ||umami.*/script.js It doesn't target a specific company's server. It targets any host whose subdomain is literally named umami serving a file called script.js . Which is exactly how most of us name things when self-hosting: umami.mydomain.com , plausible.mydomain.com , matomo.mydomain.com . The filter lists know this convention, and they have rules for it. The result: every visitor with an adblocker never loads the script. No errors on your side, no console noise, nothing in the Umami logs. The traffic just quietly never shows up. I only caught it because signups were arriving from campaigns that Umami claimed nobody clicked; the server logs and Stripe disagreed with the analytics, and the server logs were right. The fix Serve the same Umami instance from a second hostname that no filter list matches. No proxying, no renaming files, no changes to the Umami install itself. I added u.parserbee.com as an alias for the same service: 1. DNS record. A CNAME (or A record) for u.parserbee.com pointing at the same server as the existing umami.parserbee.com . 2. Reverse proxy. Add the new hostname to the existing Umami site config so both route to the same instance. I run Umami in Docker behind Coolify, where this is just adding a second d

I've been using Claude for a long time and mostly ignored Artifacts. Fine for a quick React demo. Not something I reached for. Then I needed to send an analysis to a few people at work, and it clicked. Or I'm just using it in a way nobody intended. Hard to say. The actual case I own the paywall backend at a Czech media house. The subscription offer on our news site is embedded as an iframe, and iframes are a bad neighbourhood: context isolation means the iframe has no access to the parent page's session, so user identity kept breaking and we kept patching it over postMessage. Every iframe is its own page view, so GA4 data was skewed and we had to build server-side tracking and session stitching to make the numbers mean anything. And ad blockers, CSP, and timeouts mean sometimes the thing just doesn't render, so we maintain a fallback UI in parallel. I wanted to propose we drop the iframe and ship a JS embed library instead, distributed through our internal npm registry. That's an architecture change, so it needs a document: what we fixed, why the iframe is still structurally wrong, what the alternative costs, what the numbers say. The numbers part came out of the same agent session, by the way. GA4 said roughly 0.14% of paywalled page views hit an error, about half of them iframe-blocked-by-browser. That's every 700th reader. Small number, real money. The boring problem You get a good answer out of the model. Now what? You paste it into a doc. Reformat it, because chat markdown does not survive the trip. Fix the tables. Decide whether it goes in Confluence or an email. Send it. Then someone asks a follow-up, you go back to the model, get a better answer, and now there are two versions of the truth and one of them is in someone's inbox. I've done the email version of exactly this document before. Outlook ate the markdown. I ended up hand-rolling plain text with unicode bullets and uppercase section headers like it was 1998. Half of that work is transport, not thinkin

Self-service BI was supposed to free the data team. Give everyone a BI tool, teach them to drag and drop, and they'll answer their own questions. The data team can stop building dashboards and focus on infrastructure. That's not what happened. What happened: everyone builds their own dashboards with their own metric definitions. Marketing's "active users" counts monthly logins. Product's "active users" counts weekly feature usage. Finance's "active users" counts paying customers. Three dashboards, three numbers, one term. The data team now spends their time reconciling conflicting metrics instead of building. Self-service BI without governed metrics is just self-service chaos. What is self-service BI? Self-service BI means non-technical users can query data, build visualizations, and generate reports without help from the data team. The tools (Metabase, Looker, Power BI, Tableau, Superset) provide drag-and-drop interfaces, visual query builders, and template libraries. The promise: democratize data access. Anyone can answer their own questions. The reality: it works for simple questions ("how many orders this week?") and breaks for anything requiring business logic ("what's our net revenue retention?"). Users either define metrics incorrectly or ask the data team anyway. The data team becomes a help desk for the self-service tool instead of a help desk for SQL. Where self-service BI goes wrong Metric sprawl Without a central definition, every self-service user creates their own version of key metrics. A company with 50 Metabase users might have 30 different "revenue" calculations saved across collections. Which one is right? The one that matches the board report. Which one matches the board report? Nobody knows without checking each one. The "someone who knows" bottleneck True self-service requires understanding the data model. Which table has revenue? What does status = 3 mean? Is the amount column in cents or dollars? Pre-tax or post-tax? Including shipping or exc

"We need real-time analytics" is one of the most common requests in data engineering. It's also one of the most misunderstood. When the VP of Sales says "real-time," they usually mean "faster than the dashboard that refreshes overnight." When the CTO says it, they might mean sub-second event streaming. The gap between those two definitions is a 6-month infrastructure project. Most teams don't need true real-time. They need fast enough. And "fast enough" is achievable with pre-aggregation caching at a fraction of the complexity and cost of a streaming architecture. What is real-time analytics? Real-time analytics means querying data with minimal latency between when an event happens and when it's visible in your analytics. The spectrum: Freshness Latency Architecture Use case True real-time < 1 second Event streaming (Kafka, Flink) Fraud detection, stock trading, live monitoring Near real-time 1-60 seconds Micro-batch or streaming Operational dashboards, alerting Frequent refresh 1-60 minutes Scheduled refresh + caching KPI dashboards, AI agent queries Batch Hours to daily Scheduled ETL Board reports, monthly summaries Most analytics use cases fall in the "frequent refresh" category. Revenue by region doesn't need sub-second freshness. Active users in the last hour doesn't need event streaming. A pre-aggregation cache that refreshes every 15 minutes covers 90% of what teams call "real-time." When you actually need real-time True real-time analytics (sub-second latency from event to query result) is worth the infrastructure investment when: Fraud detection. Every second of delay is potential fraud that slips through. Live monitoring. Server health, API error rates, active user counts for live products. Trading and pricing. Financial instruments where stale data means wrong prices. Live events. Streaming metrics during a product launch, marketing campaign, or live broadcast. If you're in one of these categories, you need an event streaming architecture: Kafka, Flink, M

Your company has a KPI dashboard. It was built six months ago by someone who has since moved teams. It shows revenue, churn, and a few product metrics. It loads slowly. The numbers don't match what finance reports. Nobody trusts it, but everyone screenshots it for the Monday standup. This is the state of KPI dashboards at most companies. Not because the tools are bad, but because the approach is wrong. A dashboard is a static view of a dynamic system. The moment someone builds it, it starts drifting from reality. What is a KPI dashboard? A KPI (Key Performance Indicator) dashboard is a visual display of an organization's most important metrics. Revenue, customer count, churn rate, conversion rate, average order value, NPS. The metrics that tell you whether the business is healthy. Traditional KPI dashboards live in a BI tool: Power BI, Tableau, Looker, Metabase, Grafana. An analyst builds the dashboard, connects it to a data source, and shares a link. People visit the dashboard (or receive a scheduled screenshot) to check the numbers. The concept is sound. The execution breaks for predictable reasons. KPI dashboard examples Before diving into what's broken, here's what teams typically build: SaaS executive dashboard. MRR, ARR, net revenue retention, churn rate, new customers this month, average contract value. Updated daily. Viewed by the CEO and board. The numbers must match the finance team's report exactly. Product analytics dashboard. Daily active users, feature adoption rates, conversion funnel stages, time to value. Updated hourly. Viewed by product managers. Often the first dashboard built and the first to drift from reality. Customer health dashboard (B2B). Per-customer usage metrics, support ticket volume, NPS, renewal risk score. Updated daily. Viewed by customer success. In embedded analytics use cases, the customer sees this too. Engineering operations dashboard. API error rates, p95 latency, deployment frequency, uptime. Updated real-time. Viewed by eng