标签：#Data

I wanted to see earthquakes the way they actually happen: as points lighting up on a spinning planet, in near real time. That became Earthquake: Live Seismic Monitor , an iOS app that renders a 3D globe of recent quakes straight from the USGS feed. No backend of my own - just the public data and the device. Here's how it comes together. The data source The USGS publishes earthquake data as GeoJSON feeds, updated continuously, at several time/magnitude cutoffs (past hour, past day, 2.5+, 4.5+, etc). Each feature has coordinates, magnitude, depth and time. That's everything you need to place a quake on a globe - no custom API required. The app polls the appropriate feed, diffs against what it already has, and updates the scene. Because USGS does the heavy lifting, the whole thing is effectively serverless from my side. Putting quakes on a globe The core mapping problem is turning (latitude, longitude) into a point on a sphere. Once you have that, each earthquake becomes a marker whose size and color encode magnitude and depth, so a glance tells you "big and shallow" vs "small and deep". Design decisions that mattered: Encode magnitude visually. Radius and color do more than any label. A magnitude 6 should look like a magnitude 6. Cluster sensibly. Active regions produce swarms; markers need to stay readable when dozens land in one area. Keep the globe interactive. Rotate, zoom, tap a quake for details. It should feel like an object, not a chart. Real-time without a server Every network-dependent app has to answer: what happens offline, and how fresh is "live"? My rules: Cache the last good feed so the globe still renders with no connection. Refresh on foreground and on an interval, and show the data's own timestamp so "live" is honest. Never block the UI on the network - render what you have, then update. Why no backend It's tempting to proxy the feed through your own server "for control". But USGS is reliable, public, and built for exactly this. Skipping a backend me

Most teams don't consciously choose their messaging infrastructure. They inherit it. Someone used Service Bus on the last project, it worked fine, and now it's the default answer for every async communication problem that comes up. Two years later, you're bending it into shapes it was never designed for, and the operational pain gets blamed on "distributed systems being hard" rather than on the actual culprit: a tool being asked to do a job it doesn't fit. The problem isn't that Service Bus, Event Grid, or Kafka are bad. It's that they solve genuinely different problems, and conflating them doesn't just create technical debt — it creates architectural liability that compounds over time. The Real Difference Is the Communication Contract, Not the Feature List When you put these three tools side by side in a comparison table, you'll find overlapping columns. All three move messages between systems. All three have some delivery guarantee story. That's where the surface-level comparison breaks down and people make bad decisions. The more useful question is: what contract does your system need to uphold with the data it moves? Service Bus is fundamentally about reliable, ordered processing with strong delivery guarantees. It's designed for the case where every message matters individually, where you need competing consumers pulling from a queue, where poison message handling and dead-lettering are first-class concerns. If you're coordinating business process steps or handling financial transactions where exactly-once semantics matter, this is the right shape of tool. Event Grid is about reactive routing. Something happened in your infrastructure or your application, and you want other things to respond to it. It's push-based, fan-out-friendly, and optimized for low-latency notification rather than high-volume throughput. It's not trying to be a buffer. If you're triggering downstream workflows in response to blob uploads, resource state changes, or custom application even

Companies that rely on Ceva Logistics for shipping their physical goods to customers say their personal data was taken during a recent cyberattack.

By the end of this guide you will have a published Tableau dashboard and a three-point story, built on a real dataset. It lives on a public URL you can put in an application. You build four small sheets. Each one makes exactly one point. You arrange them on a single screen, then walk a reader through them in three steps that end with a recommendation. Every step says what to click and what you should see afterwards. Four small sheets, rather than a wall of charts, because a dashboard has to argue for something. A screen holding everything you could build leaves the reader to work out what matters. Most readers will not do that work. Dashboard vs Story, in one line. A dashboard puts several charts on one screen so someone can explore. A story is a sequence of views with captions, clicked through in order, so someone is walked to a conclusion. Build both: the dashboard is what a hiring manager glances at, the story is what proves you can think. The original carries a diagram here. In words: Four separate worksheets stack on the left: a big single number, a set of vertical bars, a set of horizontal bars, and a scatter of circles. An arrow points right to one dashboard panel that holds all four of them arranged on a single screen: the number across the top, the two bar charts side by side in the middle, the scatter along the bottom. A second arrow points right to three story cards numbered one, two and three, each showing one of those views with a caption line above it. The worked example. Every instruction below is written against a real, free dataset: the Telco Customer Churn file on Kaggle, 7,043 customers, one row each. A finished analysis of it, including the Python script that shapes the data, is public at telco-churn-analysis . Swap in your own dataset and the steps do not change, only the field names do. Step 1: Shape the data before you open Tableau Tableau is a display layer. Deriving something inside it takes longer than deriving it upstream in SQL, Python or

This guide gives you a repeatable loop for the days when nothing works, and four true stories showing it used on real problems. Here is what a working day actually contains. A website's firewall blocks you for no reason. A table that visibly exists tells your script it does not. A query runs for thirty minutes with no end in sight. A fix you know is correct changes nothing at all. None of that means you are doing it wrong. That is the job. What separates people who ship analyses from people who stop is technical tenacity : staying methodical when the tools fight back. It is not a personality trait you either have or lack. It is a small procedure, and you can learn it in the next ten minutes. The diagnosis loop (tenacity is a method, not a mood) Think back to the last time a tool beat you for an hour. What was the first thing you did when it failed, and what did you do second? Most people can name the first move and not the second, and the second is where the method lives. Gritting your teeth and re-running the same thing harder is not tenacity; it's frustration with extra steps. What experienced people actually run is a loop: Step Move 1. Read the actual message Not "it's broken" — the words. Error messages name the symptom precisely, even when the cause is elsewhere. 2. Form ONE hypothesis "The table isn't in the file the script reads." Specific enough to be wrong. 3. Run the cheapest test of it Prefer checks that take seconds — list the tables, count the rows, print one value. 4. Verify from a second vantage point Don't ask the tool that's confusing you whether it's confused. Check the file from outside, the data from a different program, the value with a different query. 5. Change ONE thing, re-run Change three things and you'll never know which one mattered — or which one broke something new. 6. Timebox, then change strategy If the current approach has eaten 30 minutes with no progress, stopping is a decision, not a defeat. There's usually a second road. Four tr

COUNT looks like the simplest function in SQL, and it is the one that quietly trips up the most people in interviews and on the job. The confusion is almost always the same: COUNT(*) , COUNT(column) , and COUNT(DISTINCT column) look nearly identical but count three different things. Once you can say out loud what each one counts, a lot opens up. You can verify a data migration, find duplicates, and measure how complete a column is, all with the same little function. This guide is that explanation, with lots of small examples you can copy. The one-sentence version. COUNT(*) counts rows . COUNT(column) counts rows where that column is not NULL . COUNT(DISTINCT column) counts how many different non-NULL values that column has. Everything below is just that sentence, slowed down. The three forms of COUNT and what each one counts Picture one small table, customers , with a region column where two rows were never filled in: id name region 1 Maya North 2 Jordan South 3 Alex North 4 Sam NULL 5 Taylor NULL Now run the three forms on it: SELECT COUNT(*) AS all_rows, COUNT(region) AS rows_with_region, COUNT(DISTINCT region) AS different_regions FROM customers; all_rows rows_with_region different_regions 5 3 2 COUNT(*) = 5. Every row, no exceptions. The * means "the row itself," so NULLs never matter. COUNT(region) = 3. Only the rows where region has a value. Sam and Taylor are skipped because their region is NULL. COUNT(DISTINCT region) = 2. The different values are just North and South . The two Norths collapse to one, and NULL is not counted. The NULL rule that makes them disagree Predict it first. A table has 100 rows. Twenty of them have no email address. What does counting the email column give you? Say the number before you read on. Here is the whole trick in one line: COUNT(*) counts rows. COUNT(something) counts non-NULL values of that something. So the moment a column has any NULLs, COUNT(column) comes back smaller than COUNT(*) . That gap is not a bug, it is informat

This guide walks through five steps for working out which records are the same real thing, and merging them without wrecking your data. It runs on real chart data, and it includes the two times the rules came out wrong. Here is the problem in one example. Count the distinct artists in Billboard's public chart history and the number is wrong. "Elvis Presley" and "Elvis Presley With The Jordanaires" are the same man, and so are five other credit strings. One real-world entity , seven database strings . Every dataset with human-entered names has this. Customers who signed up twice. "IBM" against "I.B.M." against "International Business Machines". The same supplier in two systems, spelled two ways. The work of fixing it is called entity resolution . Matching across two datasets is record linkage . Removing duplicates inside one is deduplication . They are the same skill pointed at different situations, and it is one of the most common tasks an analyst actually gets handed. The vocabulary map Term Meaning Entity The real-world thing: one artist, one customer, one company Entity resolution Figuring out which records refer to the same entity Record linkage The same problem across two datasets. "Is row 5 in file A the same person as row 90 in file B?" Formalized by Fellegi & Sunter (1969) Deduplication The same problem inside one dataset Normalization / standardization Transforming values toward a canonical form (lowercasing, trimming, cutting suffixes) so equal things become equal strings Match key The cleaned column(s) you actually join on Match rate The share of records that found their counterpart. This is the number that keeps the whole exercise honest Clerical review Human eyes on the records the rules could not decide. This is a formal stage of the classic framework, not an admission of failure Step 1: measure the fragmentation before fixing anything The worked example is Billboard Hot 100 history, 1958 to present. The goal is one clean row per artist. Before writing

I have been building a thing that lets a language model propose an UPDATE , then executes it for real inside a transaction, measures the actual before and after values, and always rolls back. A human reads the measurement and decides. Only then does anything commit. The pitch is one sentence: what you approve is not the model's description of its SQL, it is what the database did when the SQL ran. Last week I found that the thing showing you that measurement was showing you a subset of it, and had been since the first release. The failure Real output, from @hyuga/llm-safe-sql@0.4.0 installed from npm. One row: name = 'Tanaka' , postcode = '00100' . UPDATE customers SET name='Sato', postcode='00100' WHERE id=1 What this touches customers — Customer records. The postcode is used for billing address and delivery. 1 row would change, across 1 column: name Measured by running the statement and rolling it back id = 1 name: 'Tanaka' -> 'Sato' One row, one column. postcode is not mentioned, and that is correct — it is being assigned the value it already holds, so nothing about it changes. The card is describing the diff accurately. Approve it. Then, before it is applied, somebody else notices the postcode is wrong and fixes it: UPDATE customers SET postcode = '90210' WHERE id = 1 ; Now apply the approved plan: Applied: UPDATE on customers, 1 row(s), at 2026-08-10T09:49:12.049Z. DB now: [{"name":"Sato","postcode":"00100"}] The fix is gone. Zero warnings. The word postcode never appeared on the approval card, never appeared in the audit record, and never appeared in the comparison the tool makes before it commits. One variable doing two jobs The diff was built like this: const changed : string [] = []; for ( const c of Object . keys ( before )) { if ( same ( before [ c ], after [ c ])) continue ; // drop what did not move if ( auto . has ( lower ( c ))) continue ; // drop what the DB maintains itself changed . push ( c ); } That is a correct answer to "what should the card sho

Kennedy Torkura discusses practical GenAI red teaming techniques to safeguard LLMs and knowledge bases against security threats like data poisoning and LLMjacking on AWS. He explains how engineering leaders and architects can bridge traditional cloud security with MITRE ATLAS frameworks to proactively identify vulnerabilities, implement guardrails, and secure production AI applications. By Kennedy Torkura

In Blog 5, we built a SQL parser. It can take this: INSERT INTO payments VALUES ( 500 , payment_1 , pending , 1 ) and turn it into a struct: InsertIntoTable { TableName : "payments" , ColumnValues : [] string { "500" , "payment_1" , "pending" , "1" }, } But this is still not enough for the storage engine. Our storage engine only knows how to store key-value pairs. It does not know what a table is. It does not know what a column is. It does not know that 500 is an integer, pending is a string, and 1 is a boolean. So, in this post we solve the missing bridge of persisting these in our key-value store. CREATE and INSERT are PUT operations This is the first major realisation. A key-value store is extensible to store literally anything. This is what we have been saying from the first post itself. But now we will be taking actual examples to prove that. CREATE TABLE Example Let's start with the create table example and see what should be the key and the value. Serialisation The key should be something that uniquely identifies the table, which is straightforward enough in this case as the table name . The value becomes everything else except the key, which is the schema of the table. So, in order to store the table name, we can append a reserved keyword as prefix like schema as a unique identifier. The structure of the key becomes _schema:<table_name> . The next question to answer is: How do we store a struct like below into our key value store where the value is always string? CreateTable { TableName : "payments" , ColumnDetails : [] Column { { ColumnName : "amount" , DataType : Int }, { ColumnName : "id" , DataType : String }, { ColumnName : "status" , DataType : String }, { ColumnName : "captured" , DataType : Bool }, }, PrimaryKeyColumnPosition : 1 , } One way is to serialise the entire struct into a string and store that directly. But in that case, deserialisation is a complex logic. JSON or struct serialisation and deserialisation is both space-heavy and compute inte

I thought this would be a simple task. I already had a Python Kafka producer running. Kafka was up in Docker. The topic existed, and I could send a message into it successfully. The next step sounded straightforward: Python Producer ↓ Kafka ↓ Spark Structured Streaming All I wanted Spark to do was read a JSON message from a Kafka topic. Instead, I ran into one error after another. At first, it looked like one problem: Spark cannot read Kafka. It was not one problem. It turned into a chain of failures across several different layers: Python / PySpark ↓ Spark runtime ↓ Kafka connector ↓ Hadoop / Windows ↓ Docker ↓ Kafka networking ↓ Ivy dependency resolution The useful part of this experience was not any single fix. It was learning how to separate the layers and stop treating every error as a problem in my Python code. This is the full debugging path. What I Was Building This was part of an financial data engineering project. The batch side of the project already looked roughly like this: Financial Data Source ↓ Python ingestion ↓ AWS S3 ↓ Snowflake ↓ dbt ↓ Financial anomaly models I wanted to add a streaming extension for newly arriving financial events. For the first version, I kept it intentionally simple: Python Kafka Producer ↓ Kafka topic: financial_events ↓ Spark Structured Streaming The producer sent a simulated financial event: { "company_id" : "COMPANY_001" , "company_name" : "Sample Company" , "report_type" : "quarterly_report" , "reporting_date" : "2026-08-08" , "event_id" : "FIN-20260808-001" , "source" : "simulated_financial_event" } Kafka accepted the message successfully. I could even read it with Kafka's console consumer. So Kafka itself was working. Then Spark entered the picture. Failure #1: PySpark Worked, but spark-submit Didn't I installed PySpark: pip install pyspark Then I installed Java 17 and verified it: java -version After reopening my terminal, Java was available. I tested Spark directly through Python: python -c "from pyspark.sql import S

we compared 8 web hosts in common crawl's domain-level link graph. 35 non-platform domains link to all 8, while 72% of linking domains appear for only one subject. how we pulled this for each subject domain, we pulled the top 2,000 referring domains by authority from common crawl release Apr-Jun 2026 (cc-main-2026-apr-may-jun). we intersected those lists, then removed platform, cdn, social, and other non-editorial domains from the clean overlap counts. the filter matters. hosting providers, cdns, url shorteners and the big social networks link to almost everything, so leaving them in would produce a universal list that is technically correct and useless for outreach. the counts below are after that filter unless a column says otherwise. everything here comes from the open common crawl webgraph, so you can reproduce it without a paid backlink tool. subject domains domain referring domains cg authority hostinger.com 36,900 61 siteground.com 13,882 60 bluehost.com 48,747 61 dreamhost.com 36,470 61 wpengine.com 100,000 64 cloudways.com 13,795 58 kinsta.com 13,963 62 namecheap.com 16,701 62 the overlap across the 8 subjects, we found 10,077 unique linking domains. the clean universal set contains 35 domains. metric value unique linking domains 10,077 link to one subject 72% link to all 8 35 overlap distribution overlap all domains non-platform link to all 8 42 35 link to 7 89 86 link to 6 133 131 link to 5 185 182 link to 4 336 335 link to 3 614 608 link to 2 1,432 1,430 link to just 1 7,246 7,240 the distribution is the interesting part. most linking domains sit in the bottom row: they mention one product and never come back. the rows above it are where outreach lives, because a site that already covers several products in a category has an editorial reason to cover another one. a short universal list usually means the category is covered by a handful of directories, review sites and integration hubs rather than by a broad press base. a long one means the category has r

Every API that sits behind an unreliable network eventually faces the same problem: a client sends a request, the connection drops before the response arrives, and the client has no idea whether the operation happened. Did the payment go through? Did the order get created twice? The client's only safe move is to retry — which means your server needs a story for what happens when the same "create this thing" request arrives more than once. That story is idempotency keys, and getting the details right is more subtle than it first looks. The core idea The client generates a unique token — typically a UUID — once per logical operation, and attaches it to every retry of that operation: POST /orders Idempotency-Key: 7c3fd9a2-df01-4b3e-9a55-1e5f9b6b6d55 {"sku": "WIDGET-1", "qty": 2} The server's job is to guarantee that no matter how many times a request with that key arrives, the side effect (charging a card, creating an order, sending an email) happens at most once, and every retry gets back the same response the original request would have produced. Note what this is not: it is not deduplicating by request body. Two requests with identical bodies but no key are legitimately two different orders for two widgets. The key is what marks them as "the same attempt," not the payload. The naive approach, and why it breaks A common first pass is a table like: CREATE TABLE idempotency_keys ( key TEXT PRIMARY KEY , response_body JSONB , status_code INT ); On each request: check if the key exists, and if so return the cached response; otherwise do the work and insert the result. This looks right and is wrong in a specific way: it has a race condition. Two retries can arrive concurrently (a client that timed out and fired a second attempt while the first was still in flight), both miss the cache check, and both execute the underlying operation. You've now charged the card twice. Making the check-and-do atomic The fix is to claim the key before doing the work, using the database's ow

Migrating from MongoDB Atlas to a self-hosted replica set bought us control and cut our bill. It also quietly removed something we had stopped thinking about: Atlas had been taking continuous backups for us the entire time. After the migration, production data for Prochesta lived in /var/db/mongo on a single VPS. No snapshots. No off-box copy. A rm -rf , a bad migration script, or a dead disk would have been the end of it. We had written "backups" as a follow-up task in the migration spec, which is the engineering equivalent of a sticky note on a bank vault. The requirement we actually cared about was narrower than "back up the database". Most real-world data loss at our scale isn't hardware failure — it's a deploy that writes garbage, or someone running an update without a filter. Recovering to last night doesn't help when the damage happened at 14:20 and you noticed at 14:50. We needed to recover to an arbitrary moment , not to a nightly snapshot. The constraint nobody mentions: Community has no $backupCursor We chose Percona Backup for MongoDB (PBM), and immediately hit the limitation that shapes every decision downstream. PBM offers physical backups — fast file-level copies that restore in minutes and barely touch the running server. They work by opening a backup cursor via the $backupCursor aggregation stage. That stage exists in Percona Server for MongoDB and in MongoDB Enterprise. It does not exist in MongoDB Community, which is what the official mongo:8.0 image ships. So on Community, PBM gives you logical backups only: every document read out through mongod , compressed, and shipped off-box. Two consequences, both accepted deliberately rather than discovered later: Backups cost CPU on the primary — and with a single-member replica set there's no secondary to offload the read to. Restores insert documents and rebuild indexes, so restore time grows with data size much faster than backup time does. At our current size that's minutes, not hours. It's also the t

AmaliTech Apprenticeship Program (AAP) launched in November 2025, with its first cohort starting on November 17th, 2025. It is self-paced, meaning apprentices move through the curriculum at their own speed rather than following a fixed lesson-by-lesson schedule, though attendance in the office is still required. It offers 5+ specializations, including Fullstack Development (Node.js/NestJS and React/Next.js or Angular), Python Backend & AI App Development, Backend Development with Java, Data Engineering, DevOps, and Quality Assurance. There are two entry paths, entry-level and mid-level, based on experience, and each spends a different amount of time in the program: entry-level apprentices spend 6–9 months, while mid-level apprentices spend 4–6 months. The program is intense: apprentices are required to be in the office 10 hours a day, Monday through Friday. In return, it offers solid compensation. Entry-level apprentices receive a stipend of 250k+ RWF, and mid-level apprentices receive 500k+ RWF. That's the program itself. So how do you actually join? Eligibility The biggest requirement: since this is an in-person program, you need to already be based in Rwanda or be willing to relocate. A background in software development. The Application Process Apply. Applications open every three months. Cohorts have run in November 2025, March 2026, June 2026, and September 2026, so you can expect the pattern to continue. Screening, then two assessments. If you pass the screening stage, you move on to: General Coding Assessment (GCA): the harder of the two, but manageable with preparation. It's done on CodeSignal , either in person or online. To prepare, practice DSA questions on competitive programming sites like LeetCode , Codewars , and CodeChef for 1–2 weeks, and you should be in good shape. Cognitive Test: taken the same day as the GCA, this evaluates problem-solving, pattern recognition, numerical analysis, and similar skills. Preparation helps here too. Watching a few Y

There’s no perfect way to transfer possession of your digital assets to your loved ones after you’re gone. But there are things you can do now to make the process easier for them.

The engineering team at Stripe recently described how they automated database incident recovery by modeling their global infrastructure as a graph. Using graph search algorithms together with state machines, the team computes and executes remediation plans automatically. By Renato Losio

I've spent more than a decade building data pipelines, and the part nobody warns you about isn't the pipeline logic. It's the tuning. Executor memory, shuffle partitions, cluster size, thread counts. You pick numbers, ship it, and a few weeks later something breaks in a way that's obviously tuning-related but not obviously what to change . The pattern repeats enough times that you start recognizing it before you've even opened the logs. Job's slow, thousands of tiny shuffle tasks, someone way overestimated the partition count. Job dies on OOM, memory's set for last quarter's data volume, nobody updated it since. Cloud bill jumps, a cluster's been sized for peak load and just sits there mostly idle the other 20 hours a day. Every senior data engineer has this pattern-matching running in their head. It's tribal knowledge, and it lives in one or two people's heads on most teams, which means it doesn't scale and it definitely doesn't survive someone leaving. So I built a small tool to make that pattern-matching explicit instead of tribal: it reads your pipeline's config alongside its actual run metrics and tells you what's likely wrong, with the reasoning shown, not just a suggested number. Why rules instead of a model The obvious move in 2026 is to reach for an ML model. I didn't, and it wasn't because I don't think ML has a place here eventually. It's that for this specific problem, a handful of threshold rules already gets you most of the value, and they're something you can actually audit. If a rule fires, I can point at the exact condition and the exact number: average heap usage 28%, peak 47%, five runs, no OOM errors, therefore memory's over-provisioned, shrink it by roughly a fifth. That's checkable. You can look at your own metrics and see whether the reasoning holds. A model's confidence score doesn't give you that, and for something that's about to change a production config, I want the person approving it to be able to say "yes, I see why" rather than "the m

When building a nested reply system, most developers worry about deep tree complexity or messy data structures. For Vlox , I took a different approach: keeping things flat, fast, and secure by reusing a single Mongoose schema with smart atomic limits. Here is a deep dive into how I engineered a production-ready, race-condition-safe reply mechanism using MongoDB transactions, strict type sanitization, and automated limits. How It Works 🛠️ User Action: A user clicks the reply icon and submits their reply. The Payload: Vlox's system sends 3 fields via the endpoint /api/v1/reply/comment/post/:id : id : The post ID (passed as a URL parameter). rootCommentId : The ID of the root comment being replied to. reply : The raw text entered by the user. Sanitization & Validation: The incoming reply is instantly converted to a trimmed string. It then passes through two critical validation checks: Existence Check: The reply must exist. (If a malicious actor sends a payload without a body, the string literally evaluates to "undefined" and gets blocked). Length Limit: The reply must be under 201 characters, enforcing the standard comment limit. Atomic Transactions: If the validation checks pass, the system initiates a Mongoose transaction to execute the following steps safely: Permission Check: It verifies if the user has permission to reply by checking the post's status via await schemas.Posts.findOne(hotQueries.find_user_post(id, req.session.userId)); . Creation: If permissions are valid, it creates a new reply. (Fun fact: It reuses the exact same schema as standard comments!) The Reply Schema Structure: The reply object functions just like a normal comment, with two distinct exceptions: It does not contain a repliesCount field. It includes an extra rootId field, which explicitly points to the ID of the root comment being replied to. Concurrency & Caps: To guarantee that a single comment never receives more than 10 replies while simultaneously incrementing the counter, the system r

Using a model to grade another model’s output is the only approach that scales to open-ended text. It is also the point at which your measurement device becomes a second stochastic system with opinions, and the difference between a useful judge and a number-generator is entirely in whether you validated it. A judge is an instrument, not an oracle Think of the judge the way a lab thinks about a thermometer. It has a reading, a bias, a precision, and a range over which it is trustworthy — and none of those are known until you check it against a reference. The reference is human labels. There is no way around this: a judge whose agreement with humans on your task is unknown produces numbers whose meaning is unknown, however many decimal places the harness prints. The good news is that the calibration is a one-off cost of a few hundred human labels, after which the judge runs for essentially free on every subsequent evaluation. That trade is what makes judges worth the trouble. What the published agreement figures say The standard reference is Zheng et al., 2023, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”. On their setup, a strong judge model agreed with human expert preferences at a rate above 80% — which the authors note is comparable to the agreement rate between two human experts on the same comparisons. That framing is the important part: the ceiling for a judge is not perfect agreement, it is human-human agreement, because the humans disagree with each other on genuinely ambiguous items. The same paper documents the failure modes that come with it — position bias, verbosity bias, self-enhancement bias, and weakness on maths and reasoning items where the judge must itself solve the problem to grade it. So the honest summary of the literature is: a well-constructed judge on general chat quality can approach human-level agreement, and it does so while carrying systematic biases that you have to design around. It is not evidence that your judge, on your