标签：#datascience

Design + Product Thinking: NYC’s Path to Reliable AI AI delivers value when it’s useful, trusted, and operational. For city services that affect millions, those qualities don’t happen by accident — they come from applying design thinking (who the service is for, how it’s used) together with product thinking (what outcome we’re trying to achieve and how we operate over time). This article explains why hiring designers and product managers matters for NYC’s digital and AI initiatives, summarizes the city’s PIT Crew program, and outlines how Flamelit applies outcome-focused delivery in the public sector. Why design and product roles matter Designers and product managers have distinct but complementary responsibilities that reduce common AI delivery failures: Designers (Design Thinking): center human needs, prototype user flows, and validate that interfaces and decision workflows are understandable and accessible. They surface usability and trust issues early, preventing technically accurate models from becoming unusable in practice. Product managers (Product Thinking): define the measurable outcomes, prioritize use cases, align stakeholders, and manage the lifecycle from discovery to ongoing operations. They ensure work is evaluated against mission impact, not just technical metrics. Together they prevent common failures: building technically impressive models that nobody trusts, deploying brittle systems without human review, or shipping features with unclear ownership that decay in production. PIT Crew and NYC hiring context NYC’s PIT Crew program is a city initiative designed to attract and staff product, engineering, and design talent for public service projects. It’s a practical recognition that public-sector digital transformation needs people skilled in user research, product management, and delivery. Read more about the PIT Crew and how it works here: https://www.nyc.gov/content/pitcrew/pages/ (open in a new tab). Hiring programs like PIT Crew help create the c

If you’ve ever struggled with CALCULATE() or wondered why SUMX() behaves differently from SUM() , this guide is for you. DAX (Data Analysis Expressions) is the language that powers Power BI , Analysis Services , and Power Pivot — enabling dynamic calculations, filtering, and time intelligence. Below is a categorized cheat sheet of essential DAX functions , plus examples showing how to use each in real-world Power BI scenarios. Filtering & Context These functions control how filters are applied and evaluated in your calculations. Function Example Description CALCULATE() CALCULATE(SUM(Sales[Amount]), Region[Name] = "Nairobi") Changes filter context to calculate total sales for Nairobi. FILTER() FILTER(Sales, Sales[Amount] > 10000) Returns a table filtered by condition. ALL() CALCULATE(SUM(Sales[Amount]), ALL(Region)) Ignores filters on Region. REMOVEFILTERS() CALCULATE(SUM(Sales[Amount]), REMOVEFILTERS(Region)) Removes filters from Region. VALUES() VALUES(Customer[City]) Returns unique list of cities. SELECTEDVALUE() SELECTEDVALUE(Product[Category], "All") Returns selected category or “All” if none. TREATAS() TREATAS(VALUES(Temp[City]), Customer[City]) Applies one table’s values as filters on another. KEEPFILTERS() CALCULATE(SUM(Sales[Amount]), KEEPFILTERS(Product[Category] = "Electronics")) Keeps existing filters and adds new ones. ALLSELECTED() CALCULATE(SUM(Sales[Amount]), ALLSELECTED(Region)) Respects user selections in visuals. ALLEXCEPT() CALCULATE(SUM(Sales[Amount]), ALLEXCEPT(Sales, Sales[Year])) Removes all filters except Year. Aggregation Summarize or aggregate data across rows or columns. Function Example Description SUM() SUM(Sales[Amount]) Adds all sales amounts. AVERAGE() AVERAGE(Sales[Amount]) Calculates mean value. COUNT() COUNT(Customer[ID]) Counts non-blank entries. COUNTROWS() COUNTROWS(Sales) Counts rows in a table. DISTINCTCOUNT() DISTINCTCOUNT(Customer[ID]) Counts unique customers. MIN() MIN(Sales[Amount]) Finds smallest sale. MAX() MAX(Sales[Amo

series: Prediksi Survival Klub Debutan Kenapa Project Ini? Setiap musim, klub yang promosi ke liga top (Premier League, La Liga, dst.) menghadapi risiko besar: sekitar 2 dari 3 klub yang naik biasanya kembali terdegradasi di musim pertama mereka. Saya penasaran — bisakah performa di beberapa laga awal musim memberi sinyal dini soal peluang klub tersebut bertahan? Ini jadi project portofolio pertama saya sebagai data scientist yang baru mulai (0-1 tahun pengalaman). Saya sengaja pilih topik yang saya suka (sepak bola) supaya prosesnya tetap enjoyable, bukan cuma "tutorial project" generik. Rencana Project Pertanyaan utama: Berdasarkan performa 8 laga pertama musim debut, seberapa besar peluang klub promosi bertahan hingga musim berikutnya (tidak degradasi)? Data yang dipakai: football-data.co.uk — data hasil pertandingan tiap musim sejak 1993/1994 Wikipedia (halaman musim liga) — daftar klub promosi & klasemen akhir musim Tech stack: pandas , requests untuk data collection scikit-learn untuk modeling (mulai dari Logistic Regression sebagai baseline) imbalanced-learn untuk handle class imbalance Streamlit + Plotly untuk dashboard interaktif Deploy ke Streamlit Community Cloud Timeline (Build in Public) Saya bikin timeline ini publik supaya ada tekanan yang sehat untuk benar-benar menyelesaikannya, bukan cuma jadi ide yang menguap: Checkpoint Target Tanggal Yang Harus Selesai Part 1 (post ini) 11 Juli 2026 Kickoff, rencana, environment siap Part 2 15 Juli 2026 Dataset jadi, push ke GitHub Part 3 17 Juli 2026 EDA selesai, insight awal Part 4 24 Juli 2026 Model final dipilih + evaluasi Part 5 31 Juli 2026 Dashboard live di Streamlit Cloud Part 6 (final) 8 Agustus 2026 Project selesai, recap lengkap Tantangan yang Sudah Saya Antisipasi Data leakage — fitur harus dihitung dari laga awal musim saja, bukan seluruh musim, biar model beneran memprediksi bukan "menyontek" hasil akhir Dataset kecil — kemungkinan hanya ~60-100 sampel klub, jadi saya mulai dari model sederhana (Lo

Adapted from an appendix of my MS thesis. Markov Chain Monte Carlo Almost as soon as computers were invented, they were used for simulation. Markov chain Monte Carlo (MCMC) was invested as Los Alamos, Metropolis et al (1953) simulated a liquid in equilibrium with its gas phase. Their tour de force was the realization that they did not need to simulate the exact dynamics, they only needed to simulate some Markov chain with the same equilibrium distribution. The Metropolis algorithm was widely used by chemists and physicists, but was not widely known among statisticians until after 1990. Hastings (1970) generalized the Metropolis algorithm, and simulations following his scheme are said to use the Metropolis-Hastings (MH) algorithm [1]. A special case of the MH algorithm was introduced by Geman et al (1984) discussing optimization to find the posterior mode rather than simulation. Algorithms following their scheme are said to use the Gibbs sampler. It took some time for the spatial statistics community to understand that the Gibbs sampler simulated the posterior distribution, thus enabling full Bayesian inference of all kinds. Gelfand et al (1990) made the wider Bayesian community aware of the Gibbs sampler, and then it was rapidly realized that most Bayesian inference could be done using MCMC, whereas very little could be done without MCMC. Green (1995) generalized the MH algorithm as much as it could be generalized [1]. Theoretical Foundations A sequence X 1 ​ , X 2 ​ , … of random elements of some set is a Markov chain if the conditional distribution of X n + 1 ​ given X 1 ​ , … , X n ​ depends on X n ​ only. The set in which the X i ​ take values is called the state space of the Markov chain. A Markov chain has stationary transition probabilities if the conditional distribution of X n + 1 ​ given X n ​ does not depend on n . This is the main kind of Markov chain of interest in MCMC. The joint distribution of a Markov chain is determined by the following [1]. The ma

I run a small AI trends site, and three weeks ago I started doing something simple: every day, snapshot the star count of every repo that crosses my GitHub trending scan for AI. No judgment, no curation, just append-only rows in a database. 611 repos and 2,671 data points later (June 19 to July 10), the picture of what's actually growing looks pretty different from what my feeds told me was hot. Here's what the data says. Before publishing this I re-checked every number below against GitHub's live API. Star counts drift by the hour, so treat them as of July 10. The top 10 risers, by raw stars gained Repo Gained Window From → To calesthio/OpenMontage +30,253 21 days 5,899 → 36,152 DeusData/codebase-memory-mcp +20,483 19 days 7,516 → 27,999 mattpocock/skills +19,053 15 days 137,485 → 156,538 obra/superpowers +16,887 20 days 232,908 → 249,795 NousResearch/hermes-agent +14,896 21 days 197,297 → 212,193 Panniantong/Agent-Reach +14,334 14 days 34,780 → 49,114 usestrix/strix +13,243 12 days 26,363 → 39,606 addyosmani/agent-skills +12,685 21 days 63,156 → 75,841 asgeirtj/system_prompts_leaks +11,720 21 days 43,415 → 55,135 msitarzewski/agency-agents +11,055 10 days 118,241 → 129,296 Windows differ because I only hold snapshots for the days a repo appeared in my scan; each row states its own real window. Three things in this data genuinely surprised me. 1. "Skills" are eating agent frameworks Four of the top ten are not agent frameworks. They are collections of packaged expertise that plug into an existing agent: obra/superpowers (still compounding at roughly 840 stars a day on a 250k base), mattpocock/skills, addyosmani/agent-skills, msitarzewski/agency-agents. A year ago this table would have been full of new frameworks. Now the framework layer looks settled and the growth is in what you load INTO the agent. The moat moved from orchestration code to encoded judgment. 2. The sharpest climbs are applications, not infrastructure The steepest sustained climb from a newcomer in

A friend messaged me a photo of a sealed booster box last month with one question: "worth it?" He'd already decided, really. The chase card in that set was all over his feed, so the box felt like a good deal. I asked him to send me the pull rates instead of the hype, and we spent twenty minutes turning "worth it?" into something we could actually compute. That exercise is a small, self-contained data problem. It's also a good example of how a clean-looking model can hand you a confident number that doesn't survive contact with reality. If you like building little estimators, this one is worth doing carefully, because the interesting part isn't the formula. It's everything the formula assumes. The formula is the easy part Expected value of a box is a weighted sum. Each card you can pull has a probability and a market value, and you multiply the two across every slot the box gives you. That's it. Undergrad probability. Here's a stripped-down version for a hypothetical set. I'm using made-up numbers so nobody mistakes this for real pull data — the point is the shape of the computation, not the specific set. # One "hit slot" in a box: probabilities cover the full outcome space. # Values are illustrative market estimates in USD. hit_table = [ { " name " : " Alt-art chase " , " p " : 0.0125 , " value " : 180.00 }, { " name " : " Secret rare " , " p " : 0.030 , " value " : 45.00 }, { " name " : " Full-art rare " , " p " : 0.100 , " value " : 8.00 }, { " name " : " Standard hit " , " p " : 0.400 , " value " : 0.55 }, { " name " : " No notable hit " , " p " : 0.4575 , " value " : 0.06 }, ] assert abs ( sum ( row [ " p " ] for row in hit_table ) - 1.0 ) < 1e-9 ev_per_slot = sum ( row [ " p " ] * row [ " value " ] for row in hit_table ) hit_slots_per_box = 36 # e.g. one meaningful slot per pack ev_box = ev_per_slot * hit_slots_per_box print ( f " EV per slot: $ { ev_per_slot : . 2 f } " ) # $4.65 print ( f " EV per box: $ { ev_box : . 2 f } " ) # $167.31 The box costs $150 sea

Every big social network locks audience data behind auth walls and anti-bot systems. Bluesky went the other way. The AT Protocol is open by design, so public profile data (bios, follower counts, full follower and following lists) is queryable through a documented API without logging in. The whole surface is basically two endpoints: GET https://api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=HANDLE GET https://api.bsky.app/xrpc/app.bsky.graph.getFollowers?actor=HANDLE&limit=100 There's also getProfiles for batching 25 handles per call. Follower lists paginate with a normal cursor , which still works on the graph endpoints. Search is a different story, cursor pagination 403s there now, but that's a topic for another post. For one-off lookups, curl is honestly all you need. Where it gets tedious Bulk. Thousands of profiles, follower exports that run into six figures, weekly snapshots for tracking. Pagination, rate-limit backoff, and stitching the pages together is boring code that has to run reliably. I packaged that part as an Apify actor: Bluesky Profile Scraper . Paste handles or profile URLs, optionally turn on follower/following export, and you get JSON or CSV back with a sourceProfile field linking each follower record to the profile it belongs to. $2 per 1,000 records, runs on a schedule if you want snapshots over time. What people use this for Vetting an influencer's real audience before paying them. Exporting who follows a competitor and what their bios say. Charting follower growth from weekly runs. And enrichment: find who's talking about you with a mentions monitor , then profile those authors to see their actual reach. Bluesky is the only major network right now where any of this is straightforward and stable. Worth using while it lasts.

Introduction I used to think "connecting to a database" was one skill. Turns out it's two: connecting to a database chilling quietly on your own laptop, and connecting to one living in the cloud, behind a login, in this case, an SSL certificate that will not let you in until you treat it with respect. This week I did both. Same tool (Power BI), same dataset, two very different vibes. Grab a coffee, here's the full walkthrough local PostgreSQL first, then Aiven's cloud version, side by side, screenshots and all. Part 1: Local PostgreSQL → Power BI Step 1 : Create a schema Nothing fancy, just giving my table a home: CREATE SCHEMA powerbi ; Step 2 : Import the dataset Right-click the new schema → Import Data in DBeaver, point it at your CSV, and let the wizard do its thing. Step 3 : Check the table landed properly A quick peek at the columns to make sure nothing got mangled on the way in. Step 4 : Connect Power BI In Power BI Desktop: Get Data → Database → PostgreSQL database. In the Server field, type localhost (or 127.0.0.1 ) and your database name. localhost Choose Import , hit OK, and log in with your local username and password. Click Load . That's it. That's the whole local experience. Part 2: Aiven PostgreSQL (Cloud) → Power BI Now for the part that actually taught me something. Step 1 : Grab your connection details Everything you need lives on Aiven's Overview page: Host, Port, Database name, User, SSL mode. Your service URI will look something like this (don't worry, this isn't a real password, Aiven masks it in the console): postgres : // avnadmin : •••••••• @ pg - xxxxxxxx - yourproject . c . aivencloud . com : 22016 / defaultdb ? sslmode = require Step 2 : Import the dataset into Aiven Same DBeaver wizard as before, just pointed at the Aiven connection instead of local. CREATE SCHEMA powerbi ; Step 3 : Aiven's certificate. Download the CA cert from the Overview page: Now here's the part that actually tripped me up: Power BI's PostgreSQL connector doesn't ha

Why Your SOE Resume Needs a Structural Overhaul Chinese state-owned enterprises (SOEs) often have deep hierarchical structures and a culture of collective achievement. But Western tech companies want to see individual impact, autonomy, and data-driven results. Continuing to lead with your former employer's prestige or your rank (e.g., "Senior Engineer Grade 7") wastes valuable space. The solution: reshape every section to answer the question "What did you personally accomplish with data?" The Core Shift: From Hierarchy to Impact In a Chinese SOE resume, it's tempting to list departments you led or teams you oversaw. In a Western senior data scientist resume, focus on the problems you defined, the algorithms you deployed, and the revenue, cost savings, or user metrics that improved. For example, instead of "Led the data analytics team of 10 people," write "Designed and deployed a demand-forecasting model that reduced inventory costs by 15% (¥12M annually)." Three Resume Sections That Require Full Rewriting Professional Summary: From 'Accomplished Engineer' to 'Data Science Leader' Start with your total years of experience, your technical stack, and the types of business problems you solve. Example: "Senior Data Scientist with 10+ years applying machine learning to supply chain and logistics. Expertise in Python, TensorFlow, and Spark. Reduced operational costs by 15-30% through predictive models deployed at [SOE name]." Work Experience: From Role Descriptions to Metric-Driven Bullets For each role, list 3-5 bullets. Every bullet should have a verb, a task, a technology (if relevant), and a quantified result. Avoid vague phrases like "responsible for." Use specific numbers: "Improved forecast accuracy from 70% to 85% by building an ensemble of ARIMA and XGBoost models." Education & Certifications: Emphasize Transferable Skills Your Chinese degree is fine, but add relevant certifications (AWS, TensorFlow, Coursera) to show adaptability. Consider a "Technical Skills" se

1. The Problem It Solves Logistic Regression is used when the outcome is a category rather than a number . Most commonly, it's used for binary classification , where the answer is either Yes or No , True or False , or 1 or 0 . Typical business problems include: Will a customer churn? Is this transaction fraudulent? Will a customer click an ad? Will a loan default? Is an email spam? Will a machine fail in the next 24 hours? Unlike Linear Regression, we're not trying to predict a continuous value. Instead, we're predicting the probability that an event belongs to a particular class. For example: A customer may have an 82% probability of churning . The business can then decide whether that probability is high enough to trigger an intervention. 2. Core Intuition Imagine you're trying to predict whether a customer will cancel their subscription. Suppose the only feature you have is how many times they opened your app this month. If you use a straight line like Linear Regression, the predictions quickly become unrealistic. A very active customer might end up with a -20% chance of churn . A completely inactive customer could end up with 140% . Probabilities obviously can't work like that. To fix this, Logistic Regression takes the linear equation and passes it through a mathematical function called the Sigmoid Function . Instead of producing a straight line, it creates an S-shaped curve . No matter how large or small the input becomes, the output always stays between 0 and 1 . That makes it perfect for probability estimation. 3. The Mathematical Model The model first calculates a linear score. Instead of using that score directly, it passes it through the Sigmoid function. Where: z = linear score p̂ = predicted probability The final output is always between 0 and 1 . For example: 0.08 → Very unlikely 0.32 → Low risk 0.65 → Moderate risk 0.94 → Very high probability Businesses can then choose a decision threshold. For example: Probability ≥ 0.50 → Predict Churn Probability

I recently completed an exploratory data analysis project on the NHANES (National Health and Nutrition Examination Survey) dataset from Kaggle. It's a real-world health survey collected by the CDC covering body measurements, lifestyle habits, and demographic data from thousands of US adults. In this article I'll walk you through exactly what I did — from loading and cleaning the data all the way to running statistical tests — and share what I found along the way. The Dataset The dataset has 5,735 rows and 28 columns , but for this project I focused on 8 columns that were relevant to the questions I wanted to answer: Column Description smoking Has the person smoked at least 100 cigarettes? gender Male or Female age Age in years education Highest level of education weight Weight in kg height Height in cm bmi Body Mass Index Step 1 — Loading and Selecting Columns import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns db = pd . read_csv ( ' NHANES.csv ' ) data = db . loc [:, ( ' SEQN ' , ' SMQ020 ' , ' RIAGENDR ' , ' RIDAGEYR ' , ' DMDEDUC2 ' , ' BMXWT ' , ' BMXHT ' , ' BMXBMI ' )] data = data . rename ( columns = { ' SEQN ' : ' id ' , ' SMQ020 ' : ' smoking ' , ' RIAGENDR ' : ' gender ' , ' RIDAGEYR ' : ' age ' , ' DMDEDUC2 ' : ' education ' , ' BMXWT ' : ' weight ' , ' BMXHT ' : ' height ' , ' BMXBMI ' : ' bmi ' }) One thing worth knowing about NHANES: all the columns come in as numeric codes. 1 means Male, 2 means Female. 1 means the person smoked, 2 means they didn't. You have to map these to readable labels before doing any analysis, otherwise your charts are meaningless. Step 2 — Cleaning the Data Drop the ID column and remove nulls data . drop ( ' id ' , axis = 1 , inplace = True ) data . dropna ( inplace = True ) This brought us from 5,735 rows down to 5,406 — about 6% lost, which is acceptable. Remove outliers using the IQR method The IQR (Interquartile Range) method flags values that fall too far outside the middle 50% of

The opening kickoff of Super Bowl LVII was still three weeks away when sharp bettors began their work. While casual fans were scrolling through prop bets and debating quarterback matchups on social media, a handful of disciplined bettors with sophisticated models were already identifying the first exploitable edges. Within hours, sportsbooks registered the shift: Kansas City opened at -2.5, but sharp action pushed the line to -3. By game day, it had settled at -2.5 again after public money flooded in on the Chiefs. This seemingly minor dance of numbers contains profound lessons about market efficiency, behavioral psychology, and where consistent value in sports betting actually exists. The difference between sharp money and public money isn't merely a matter of skill—it's a window into how financial markets process information in real time. For researchers, data scientists, and anyone interested in understanding how markets function under uncertainty, betting lines offer a peculiar advantage: instantaneous, objective outcomes. You can know within hours whether your hypothesis was correct. This article explores what line movement data reveals about market inefficiencies, the methodology behind detecting them, and what this teaches us about information asymmetry in competitive markets. The Hidden Market Beneath the Surface Most bettors see a line and make a decision: is this price fair or favorable? But they miss the crucial information happening before they ever see that number. Sportsbooks don't set lines based on game probability—they set them based on where they predict the money will flow. This distinction transforms betting markets into fascinating research subjects. Consider the structure: A sportsbook's primary goal isn't prediction; it's profit through balanced exposure. They're market makers, not forecasters. When sharp bettors arrive first with informational advantages, they move the line. When public money arrives later with no informational advantage but

When the referee checks their watch in the 85th minute, something predictable happens in soccer—but almost nobody is modeling it correctly. I spent three months analyzing 1,085 professional soccer matches using StatsBomb's open data, focusing specifically on goal-scoring patterns in the final 15 minutes of regulation play and stoppage time. What I found challenges the conventional wisdom that late goals are chaotic, random events determined purely by desperation and fortune. Instead, the data revealed a structured pattern that, when properly identified, has produced an 79.3% accuracy rate in backtesting across multiple leagues and seasons. The bookmakers aren't missing this pattern because the pattern doesn't exist—they're missing it because it requires looking at the problem completely differently than traditional sports analytics approaches it. The Setup: Why Late Goals Matter Before diving into methodology, let's establish why this question even matters. Late goals are the most emotionally charged moments in soccer. They're also economically significant. A goal in the 88th minute creates a cascade of outcomes: It flips match results It triggers goal-line drama and potential VAR decisions It creates dramatic shifts in market odds It validates or destroys betting positions The conventional narrative treats late goals as the result of two factors: increased urgency from trailing teams and increased vulnerability from leading teams. This is directionally correct but strategically useless. It's like saying "stock prices move when sentiment changes"—technically true, but not actionable. The real question isn't whether late goals happen more frequently. The real question is: which teams score them, under which specific conditions, with what measurable precursors? Methodology: Building the Dataset I used StatsBomb's open data repository, which contains event-level information from 1,085 professional matches across multiple seasons and competitions. StatsBomb's data inclu

The Night Everything Changed It was 87 minutes into a Premier League match. The score was 1-1. The home team had controlled possession for most of the second half, but their shots were consistently blocked or saved. Then something happened that's been happening for decades, yet nobody seems to adequately explain it: a late goal completely shifted the match outcome. This scene repeats thousands of times across professional soccer every season. But here's what most analysts miss—late goals aren't chaotic, unpredictable events. They follow patterns. Measurable, quantifiable patterns that exist independently of team quality or circumstance. Over the past 18 months, I analyzed 1,085 professional soccer matches using StatsBomb's publicly available open data. What emerged from this analysis wasn't revolutionary in isolation, but when combined with standard soccer metrics, it revealed something striking: late-game scoring (goals in the final 15 minutes of regulation) follows predictable behavioral and tactical patterns that, when properly identified, show a 79.3% correlation with specific pre-match and in-match conditions. This isn't about predicting individual goals with certainty. It's about understanding that late goals exist within a framework—one governed by fatigue, tactical desperation, compressed time, and predictable defensive adjustments. And once you see this framework, you can't unsee it. The Data Foundation Before diving into patterns, let me establish what we're working with. StatsBomb's open data includes detailed shot maps, pass completion sequences, player positioning, and event-by-event timelines from top-tier professional matches. When they made portions of this data publicly available, it created an unusual opportunity: examining thousands of matches with granular timing and contextual information. My analysis focused specifically on: 1,085 professional matches across five seasons (2017-2022) Shot events in the final 15 minutes of regulation (minutes 75-

The sportsbook odds for UFC 287 showed Sean Strickland at +340 against Dricus du Plessis. Most bettors saw a reasonable risk-reward opportunity. What they didn't see—what the market systematically misses—is that fighters in Strickland's exact statistical profile win substantially more often than their odds suggest. When Strickland knocked out du Plessis in the second round, it wasn't luck. It was a textbook case of market inefficiency that data reveals happens repeatedly in MMA. I spent six months building a comprehensive dataset of 500 UFC fights, cross-referencing striking accuracy, takedown defense, fight duration patterns, and historical betting odds against actual outcomes. What emerged was clear: the UFC betting market is inefficient in predictable ways. Certain underdog profiles generate consistent positive return on investment (ROI) that would be impossible if prices reflected true win probabilities. This isn't hindsight bias or cherry-picked examples. This is systematic analysis of where prediction markets get MMA wrong—and how you can identify it before the bell rings. The UFC Analytics Ecosystem: Why Data Matters More Than Ever Five years ago, serious MMA analytics barely existed outside Reddit threads and YouTube channels. Today, the landscape has transformed completely. UFCStats.com provides granular fight data that didn't exist in the sport's early years. Betting markets across DraftKings, FanDuel, and international books generate millions in handle. Meanwhile, fighter training data, coaching staff analytics, and institutional scouting reports are becoming increasingly sophisticated. Yet there's a persistent gap between information availability and information utilization . The casual bettor sees a -250 favorite and assumes the math is settled. Sportsbooks, operating on relatively thin margins and managing liability across thousands of bets, often make conservative assumptions. They price based on public perception, recent results, and popularity rathe

Part 4 of the Building Enterprise AI Automation Systems Series Introduction Most Named Entity Recognition (NER) tutorials end with a prediction. The model successfully extracts: COMPANY INVOICE CONTRACT PURCHASE_ORDER The article ends. The notebook prints a beautiful JSON response. Mission accomplished. Or so it seems. In real enterprise systems, extracting entities is only the beginning. Consider the following prediction: { "COMPANY" : "ALPHABRIDGE" , "INVOICE" : "MFG-INV-000157" } At first glance, everything looks correct. But from a business perspective, the system still knows almost nothing. Questions remain unanswered. Which ALPHABRIDGE? Which customer record? Which contract? Which invoice? Which business relationship? These questions belong to a completely different problem known as Entity Resolution. Entity Resolution transforms extracted text into business knowledge. Without it, AI understands words but not businesses. NER Finds Text Named Entity Recognition answers one question: "What pieces of text represent meaningful entities?" For example: PAYMENT FROM ALPHABRIDGE SOLUTIONS MFG-INV-000157 becomes { "COMPANY" : "ALPHABRIDGE SOLUTIONS" , "INVOICE" : "MFG-INV-000157" } This is extraction. Nothing more. The model has no idea whether: the company exists, the invoice exists, the invoice belongs to the company, the invoice has already been paid, the contract is still active. Extraction is syntax. Enterprise automation requires semantics. The Hidden Problem Imagine the following customer master. CUS-00001 ALPHABRIDGE SOLUTIONS Now imagine receiving these transaction narratives. PAYMENT FROM ALPHABRIDGE PAYMENT FROM ALPHABRIDGE LTD PAYMENT FROM ABS PAYMENT FROM ALPHA BRIDGE Humans immediately recognize these as the same customer. Machines do not. To a computer, every string is different. Without resolution, automation immediately breaks. What Entity Resolution Actually Does Entity Resolution answers a different question. Instead of asking: "What entity is this?"

Part 3 of the Building Enterprise AI Automation Systems Series Introduction Named Entity Recognition (NER) is one of the oldest problems in Natural Language Processing. Most tutorials introduce NER using examples like: Person Organization Location Date A sentence such as: Elon Musk founded SpaceX in California. becomes PERSON ORGANIZATION LOCATION While this is useful for learning NLP fundamentals, it has very little relevance to enterprise software. Businesses do not automate biographies. They automate operations. Enterprise documents contain an entirely different language. Invoices. Contracts. Purchase Orders. Bank Statements. Remittance Advice. Payment Narratives. ERP Exports. The entities that matter inside these documents are not "PERSON" or "LOCATION". Instead, they are business concepts such as: Customer Contract Invoice Purchase Order Payment Type Understanding these entities is the first step toward intelligent automation. In this article, we'll build a Financial Named Entity Recognition pipeline capable of transforming raw enterprise transaction narratives into structured business knowledge. The Difference Between Generic NER and Enterprise NER Traditional NER focuses on linguistic entities. Enterprise NER focuses on operational entities. Consider the following sentence. PART PMT ALPHABRIDGE SOLUTIONS MFG-INV-000157 A generic language model may identify: Organization and ignore everything else. From a business perspective, this is almost useless. What we actually need is: PAYMENT_TYPE COMPANY INVOICE The objective is not language understanding. The objective is business understanding. Step 1 — Designing the Business Taxonomy Before training any model, define what the model should learn. This is one of the most overlooked stages in machine learning projects. Many teams immediately begin annotation without first defining a taxonomy. As a result, annotations become inconsistent. Models become confused. Evaluation becomes unreliable. For our transaction intell

Part 2 of the Building Enterprise AI Automation Systems Series Introduction One of the biggest obstacles in enterprise AI is not choosing a model. It is finding data. Most tutorials assume that training data already exists. Reality is very different. Large organizations rarely share operational datasets. Financial transactions contain confidential information. Contracts contain sensitive agreements. Invoices reveal commercial relationships. Bank statements expose customer activity. For legal, regulatory, and competitive reasons, these datasets almost never become public. This creates a difficult problem for AI engineers. How do you build intelligent systems when the data you need cannot be accessed? The answer is synthetic data. Unfortunately, most synthetic datasets found online are little more than randomly generated CSV files. They contain names. Numbers. Dates. But they completely ignore something far more important: Business relationships. In this article, we'll explore how to design synthetic enterprise datasets that preserve real business logic and can be used for machine learning, automation, benchmarking, and AI engineering. Random Data Is Not Synthetic Data Many developers believe synthetic data simply means generating fake values. For example: Customer,Invoice,Amount John,INV001,500 Alice,INV002,1200 Bob,INV003,900 Technically, this is synthetic. Practically, it is useless. Why? Because enterprise systems are built around relationships. Invoices belong to contracts. Contracts belong to customers. Payments reference invoices. Purchase orders authorize invoices. Bank transactions settle invoices. Without these relationships, there is nothing meaningful to learn. A machine learning model trained on isolated records learns isolated patterns. Real enterprise automation requires connected data. Thinking Like an Enterprise System Before writing a single line of Python, ask one question: "How does the business actually operate?" Imagine a manufacturing company. A

The 2026 FIFA World Cup is reshaping competitive balance in ways that traditional 32-team analysis cannot predict. With 16 groups of 3 teams instead of 8 groups of 4, the mathematical probability of upsets—and the consequences of single matches—has fundamentally shifted. Early tournament data already shows this pattern emerging. The Format Change: A Statistical Earthquake The move from 32 to 48 teams introduces a critical structural change: Format Groups Teams/Group Matches/Team Elimination Threshold 2022 (Qatar) 8 4 3 Top 2 of 4 2026 (USA/CAN/MEX) 16 3 2 Top 2 of 3 This seemingly small difference creates massive implications. In a 3-team group, each team plays only 2 matches to determine their fate . Compare this to the 2022 format where teams had 3 chances to secure advancement. The upset probability multiplier: With two fewer matches per group, variance compounds. A single bad result—or a fortunate one—carries exponentially more weight. Early Tournament Evidence: The Data Doesn't Lie Let's examine the first week of actual 2026 results: Match Expected Result Actual Result Upset Indicator Portugal 5-0 Uzbekistan Portugal W Portugal W (5-0) Expected England 0-0 Ghana England W Draw Minor Upset France 3-0 Iraq France W France W (3-0) Expected Argentina 2-0 Austria Argentina W Argentina W (2-0) Expected Norway 3-2 Senegal Senegal slight favorite Norway W Major Upset Jordan 1-2 Algeria Algeria strong favorite Competitive Closer than xG Panama 0-1 Croatia Croatia W Croatia W Expected Colombia 1-0 Congo DR Colombia W Colombia W Expected Three critical takeaways: Norway's 3-2 victory over Senegal is statistically significant. Pre-tournament models favored Senegal slightly (ranked 18th globally vs Norway's 22nd). In a 4-team group, this result matters less; in a 3-team group, Norway essentially secures qualification with one match remaining. England's 0-0 with Ghana represents draw probability explosion. With only 2 group matches, a draw consumes 50% of your advancement op

Jump Tables, TLV Footers, and the Real Cost of Reading What You Don't Need You're paying for bytes you never read. A data engineer on a busy pipeline touches dozens of Parquet files a day: schema discovery, predicate pushdown, column pruning, metadata scrapes for a data catalog sync. In each case, the application needs maybe 200 KB of context from a file that is 4 GB on disk. Without a seekable archive format and a jump table to find the right frame, your HTTP client fetches the whole thing, and your cloud egress invoice reflects every unnecessary gigabyte. This post quantifies the problem, then walks through how HuskHoard uses seekable Zstd frames, a per-volume jump table, and TLV-encoded footer metadata to make partial egress a first-class citizen across multi-volume archives — disk, cloud, and LTO tape alike. The Problem, In Dollars S3 standard egress runs $0.09/GB. GCS is $0.08/GB. Even Cloudflare R2, which is free for egress from R2 to the internet , still costs you in latency and API call count when you cannot bound the range of bytes you need. Here is a representative read pattern for a cold analytics archive: Operation Bytes Needed Bytes Fetched (naïve) Ratio Schema discovery ~50 KB (Parquet footer) 1–8 GB (full file) ~1:16,000 Single column scan ~200 MB (one column chunk) 4 GB (full row group) 1:20 Data catalog sync (1M files) ~50 GB (footers only) ~4 PB (full files) 1:80,000 Selective restore (1 row group) ~128 MB 4 GB 1:32 On 100 TB of cold Parquet data with $0.09/GB egress: Full read for schema sync : 100 TB × $0.09 = $9,216 Partial read (footers only, avg 100 KB/file, 1M files) : ~100 GB × $0.09 = $9.00 Savings per catalog sync: $9,207 — 99.9% reduction Even a conservative column-scan scenario (pulling 15% of each file's bytes) cuts a $9,216 monthly read bill to $1,382 . The ceiling on savings is determined entirely by how precisely you can address the bytes you actually need. That precision is what frames and jump tables buy you. Zstd Frames: What They