标签：#datascience

I had a simple-sounding question: does ChatGPT recommend this business? You'd think you just ask it. Ask ChatGPT "best personal injury law firm in NYC", see if the business is named, record yes or no. That works exactly once. Ask again an hour later and you might get a different answer. Not slightly different — potentially a completely different set of firms and a completely different set of cited sources. Which means the naive version of this measurement is worthless. You're not measuring visibility, you're sampling a distribution once and calling it a fact. This is the same problem anyone gets when they try to test an LLM-backed feature. Your normal testing instinct — same input, assert on output — just doesn't apply. So here's how I ended up designing around it, and the numbers that came out, which surprised me. The setup I wanted to compare four assistants (GPT-4o, Claude Haiku 4.5, Gemini 2.5 Flash, Perplexity Sonar, all with web search on) across 10 buyer-intent questions in one vertical. Something like: "Best personal injury law firm in New York City?" "Top immigration lawyers in Mumbai?" For each response I recorded two things: which businesses got named, and which URLs got cited. The cited sources come from each API's own citation metadata, so that part is structured — no scraping the prose. First pass, the results looked dramatic. The four assistants barely agreed on anything. Different firms, different sources, almost no overlap. Great finding. Except I couldn't publish it, because there was an obvious objection I couldn't answer: Maybe they weren't disagreeing with each other. Maybe each one was just disagreeing with itself. If a single assistant returns wildly different sources run to run, then "these four models cite different things" is a meaningless statement. You'd be measuring noise and calling it signal. The control The fix is the same idea as a control group. Measure the thing you're worried about, separately, and see if it explains your result.

Every team I've watched ship a broken measurement system broke it the same way. Not with bad math — with an org chart problem that happened to live in code. The entity making the claim ended up being the entity that decided whether the claim was right. Once you have the shape in your head you start seeing it everywhere. Three roles, not two Most engineers think about measurement as two roles: the thing that acts, and the thing that grades it. That's one role short. There are three: Player — makes the claim. Your model, your service, your PR. Scorer — applies the rubric. Your eval harness, your test suite, your metrics dashboard. Settler — determines what actually happened. Production outcomes. Reality. The scorer is a proxy. The settler is the thing the proxy is trying to approximate. The rule: be the scorer, never the settler. When the player captures the settler, the loop closes on itself and the system can no longer be wrong — which sounds like success and is actually the failure. What it looks like in code Tuning on the test set. You check test accuracy, adjust hyperparameters, check again. Twenty iterations later the test set is training data with extra steps. The player is now selecting its own settler. That's what overfitting is , structurally — not a math failure, a role-collapse failure. LLM-as-judge from the same family. Your generator is GPT-flavored and your judge is GPT-flavored. They share pretraining data, failure modes, and blind spots. The judge doesn't rate quality — it rates similarity to what it would have produced. Correlated error is invisible to averaging; running it 1,000 times makes you more confident of the same wrong answer. Benchmark contamination. The model scores 94% on the benchmark that's in its training data. Nobody lied. The settler just quietly moved inside the player. Self-reported health. A service that returns its own health check is a claimant ruling on its own claim. If the process is wedged, the check is wedged too, and your

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't. I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong. Why clustering shrinks your real sample size Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half. The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately: Deff = 1 + (m̄ - 1) · ICC where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is: n_eff = n / Deff That is the number of independent examples your clustered set is actually worth. The number Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher). Deff = 1 + (5 - 1) · 0.3 = 2.2 n_eff = 200 / 2.2 ≈ 91 Your 200-example eval is worth about 91 independent examples. The correct confidence interval

Most experiments that come back "no clear winner" were unwinnable on the day they launched. The data could not resolve an effect that size, and no amount of extra runtime was going to change that. You can find this out in about two minutes, before you spend anything, with one formula and a resampling pass over your own data. Here is the check, in three steps. Step 1. Compute the smallest lift your data can see For a two-arm test on a conversion rate, the smallest lift detectable at 95% confidence and 80% power is a one-liner: from math import sqrt Z_ALPHA = 1.96 # two-sided 95% Z_BETA = 0.84 # 80% power def mde ( baseline_cvr : float , n_per_arm : int ) -> tuple [ float , float ]: """ Minimum detectable effect: absolute (pp) and relative (%). """ se = sqrt ( 2 * baseline_cvr * ( 1 - baseline_cvr ) / n_per_arm ) abs_lift = ( Z_ALPHA + Z_BETA ) * se return abs_lift * 100 , abs_lift / baseline_cvr * 100 At a 3% conversion rate: clicks per arm smallest lift you can detect 5,000 +32% relative 20,000 +16% relative 100,000 +7% relative Read the middle row twice. Twenty thousand clicks per arm is a serious amount of traffic for a mid-market account, and a real 15% improvement still lands inside the confidence interval. The report will say "inconclusive," and the team will read that as a verdict on the idea. It is a verdict on the instrument. Invert the same formula and the planning question gets easier: at 3% baseline, detecting a 10% lift needs about 51,000 clicks per arm, and detecting a 5% lift needs about 203,000. If your account produces 8,000 clicks a month, you now know the honest answer to "how long should we run this." Step 2. Stop assuming your conversions are independent The formula above treats every click as an independent coin flip with the same probability. Account data does not behave that way, and the gap is not small. In a corpus of 31 advertiser accounts I maintain for diagnostic work (9.46 million search term rows, roughly $133M of spend, September 2024

Regression and regularization are both important concepts in machine learning and statistics, but they solve different problems. Regression is primarily used to model relationships and make predictions. Regularization is used to improve a model's ability to generalize by controlling its complexity. Regression This is a statistical and machine learning technique used to predict a continuous numerical outcome based on one or more input variables. For example, we might want to predict: A house's price based on its size and location A student's exam score based on study hours A company's sales based on advertising spending Simple Linear Regression In simple linear regression, we model the relationship between an input variable (x) and an output (y): $$ y = \beta_0 + \beta_1x + \epsilon $$ Where: (y) is the predicted outcome (\beta_0) is the intercept (\beta_1) is the coefficient or slope (x) is the input variable (\epsilon) represents the error The model learns values for (\beta_0) and (\beta_1) that make its predictions as close as possible to the actual values. Multiple Linear Regression In multiple linear regression, several predictors are used: $$ y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \cdots + \beta_px_p + \epsilon $$ The goal is typically to minimize the sum of squared errors (SSE) : $$ \text{SSE} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 $$ This approach is known as Ordinary Least Squares (OLS) . Regularization Regularization is a technique used to prevent a machine learning model from becoming too complex. A model can perform extremely well on training data but poorly on new, unseen data. This problem is called overfitting . Regularization addresses overfitting by adding a penalty for large model coefficients to the model's objective function. Instead of minimizing only the prediction error, the model minimizes: $$ \text{Prediction Error} + \text{Complexity Penalty} $$ The penalty discourages the model from relying too heavily on individual features. The Main Types o

Streamlit makes it remarkably fast to transform raw Python scripts into interactive, web-based data applications without needing any frontend knowledge in HTML, CSS, or JavaScript. In this tutorial, we will build a full-featured **Sales Analytics Dashboard** complete with real-time sidebar filtering, custom KPI metric cards, dynamic line/bar charts, and expandable data preview tables. --- ## Prerequisites To follow along, make sure you have Python 3.9+ installed along with the required libraries: bash pip install streamlit pandas numpy --- ## Step 1: Setting Up the Page & Mock Data with Caching First, we import the necessary libraries, set up the layout, and create a function to generate mock sales records. We use Streamlit’s `@st.cache_data` decorator so the data is only generated once per session, keeping the app snappy during user interactions. python import streamlit as st import pandas as pd import numpy as np Set layout configuration st.set_page_config(page_title="Sales Dashboard", layout="wide") Cache data loading for performance optimization @st .cache_data def load_data(): dates = pd.date_range("2025-01-01", periods=180) regions = ["North", "South", "East", "West"] df = pd.DataFrame({ "date": np.random.choice(dates, 500), "region": np.random.choice(regions, 500), "product": np.random.choice(["A", "B", "C"], 500), "sales": np.random.randint(100, 5000, 500), "units": np.random.randint(1, 50, 500), }) return df.sort_values("date") df = load_data() --- ## Step 2: Adding Interactive Sidebar Filters Next, we add controls inside the sidebar to let users filter the dataset by region, product type, and date range. A boolean mask applies those selections dynamically. python --- Sidebar filters --- st.sidebar.header("Filters") region_filter = st.sidebar.multiselect("Region", df["region"].unique(), default=df["region"].unique()) product_filter = st.sidebar.multiselect("Product", df["product"].unique(), default=df["product"].unique()) date_range = st.sidebar.date_input(

LSTMs are still the go-to architecture for a lot of time series work, but they're annoying to trust. You get a number out the other end and no real sense of why the model landed there. This tutorial walks through training an LSTM on daily temperature data, then pulling it apart with three explainability methods: permutation importance, SHAP, and Integrated Gradients. Who this is for: people who already know some Keras and want to add interpretability to a forecasting model, not a from-scratch intro to neural nets. 1. Getting the data into shape LSTMs want a 3D tensor — (samples, timesteps, features) — so before anything else we need to turn a flat column of temperatures into overlapping 7-day windows, each one paired with the value on day 8. import numpy as np import pandas as pd from sklearn.preprocessing import MinMaxScaler # 1. Load data df = pd . read_csv ( " weather_data.csv " ) data = df [ ' Temperature ' ]. values . reshape ( - 1 , 1 ) # 2. Scale the data for stable neural network training scaler = MinMaxScaler ( feature_range = ( 0 , 1 )) scaled_data = scaler . fit_transform ( data ) # 3. Create sequences: 7 days of lag to predict the 8th day X , y = [], [] for i in range ( 7 , len ( scaled_data )): X . append ( scaled_data [ i - 7 : i ]) y . append ( scaled_data [ i ]) X , y = np . array ( X ), np . array ( y ) print ( f " Input shape: { X . shape } " ) # Output: (Samples, 7, 1) Scaling matters more than it sounds like it should — LSTMs trained on unscaled temperature values are prone to exploding gradients, and training just falls apart. The windowing step is really the whole trick here: every prediction only ever sees the past seven days, nothing more. 2. Building the model Two stacked LSTM layers, dropout after each one, early stopping so we don't have to babysit the epoch count. from tensorflow.keras.models import Sequential from tensorflow.keras.layers import LSTM , Dense , Dropout , Input from tensorflow.keras.callbacks import EarlyStopping # 1. Build

Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n. Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried. I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical , every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway: config 0: 47/100 = 47.0% config 1: 50/100 = 50.0% config 2: 52/100 = 52.0% config 3: 52/100 = 52.0% config 4: 50/100 = 50.0% config 5: 50/100 = 50.0% config 6: 54/100 = 54.0% <- selected winner (argmax) config 7: 44/100 = 44.0% config 6: 54.0% (k=54 n=100 SE=4.98) config 2: 52.0% (k=52 n=100 SE=5.00) RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed. Held-out the winner on a fresh 100 items: 48/100 = 48.0%. Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE . The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be. TL;DR Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strict

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N) , which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34. Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says. I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts: gate: spend<=budget over 200 deterministic agent runs observed failures: 0 (distinct scenarios: 200) naive point rate : 0.00% binomial SE: 0.00 pp naive 95% CI : [0.00%, 0.00%] <- zero width: false certainty Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200) , which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%] . A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much. TL;DR "0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't. The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is 1 - 0.05^(1/N) . The rule of three, 3/N , approximates it and rounds the risk slightly up. At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate. To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "

If you have written any optimization code, you have met linear programming even if nobody called it that. "Maximize output without blowing the resource budget" is an LP problem, and the classic algorithm that solves it is the simplex method. It is worth understanding not because you will hand-code it (you'll usually call a solver), but because knowing how it moves makes you far better at modeling problems for it. Here is the algorithm stripped down to its logic. The problem shape Every LP problem has three parts: an objective function to maximize or minimize, e.g. Z = 5x1 + 4x2 a set of linear constraints, e.g. 6x1 + 4x2 <= 24, x1 + 2x2 <= 6 non-negativity: all variables >= 0 Geometrically, the constraints carve out a feasible region (a polytope). The optimum always sits at a corner of that region. The simplex method is just a smart way of hopping from corner to corner, uphill, until there is no higher corner to move to. The algorithm as pseudocode build initial tableau (add a slack variable per <= constraint) loop: compute Cj - Zj for each column if all (Cj - Zj) <= 0: break # optimal reached pivot_col = column with most positive Cj - Zj # entering variable ratios = RHS / pivot_col entries (only positive entries) pivot_row = row with smallest non-negative ratio # leaving variable pivot(pivot_row, pivot_col) # elementary row operations return solution from final tableau That's it. Four moves per iteration: score the columns, pick the entering variable, run the ratio test for the leaving variable, pivot. Repeat until the optimality condition holds. A quick worked run Take Maximize Z = 5x1 + 4x2 subject to 6x1 + 4x2 <= 24 and x1 + 2x2 <= 6. Add slack variables s1, s2, build the tableau, and iterate. The optimum lands at x1 = 3, x2 = 1.5, Z = 21. Two pivots and you're done. Simple on paper until the numbers get ugly. Where humans (and debugging) actually break The algorithm is clean. The arithmetic is not. A single wrong entry in one pivot silently corrupts every table

Why Your WeChat Mini-Program Work Is Actually Relevant US data engineering interviews prioritize hands-on experience with data pipelines, SQL, and scalable storage. WeChat mini-programs, though often seen as front-end tools, generate and process substantial backend data. If you designed the logic behind user actions, session management, or real-time updates, you already have transferable skills. The key is to stop describing the mini-program as a "feature" and start describing it as a "data system." Recruiters don't care about WeChat's UI components; they care about how you moved, transformed, and stored data. The Core Translation: From Mini-Program to Data Engineering When rewriting your resume, map each mini-program task to a standard data engineering function: User interaction logic → Event-driven data pipeline (e.g., Kafka, AWS Kinesis, or custom queues) WeChat cloud functions → Serverless compute (e.g., AWS Lambda, Azure Functions) Database reads/writes via WeChat API → NoSQL or relational database operations (e.g., MongoDB, MySQL, DynamoDB) Session tracking or analytics → ETL pipeline extracting user behavior data into a warehouse Do not use WeChat-specific terms like "wx.request" or "Mini Program SDK" without explaining the data they handled. Replace them with equivalent English terms. A Quick Side-by-Side Translation Table WeChat Term US Data Engineering Equivalent WeChat Cloud Database Managed NoSQL database (like MongoDB Atlas) Mini Program Backend with Cloud Functions Serverless data processing (AWS Lambda) User login & session management Authentication pipeline with token storage Real-time message push Event-driven notification system Reshaping Your Bullet Points: Before and After Generic bullet points kill your chances. Here is a concrete rewrite that turns a typical WeChat mini-program description into a data engineering achievement. Before (Chinese-English hybrid, vague): "Developed a WeChat mini-program for e-commerce with 50k users. Used wx.request

I watched my sister paste a CSV into on of our LLM overlords and ask a question then reword it and get a different number. That's the problem with letting an LLM both read your question and trust it's math. ADA splits the two jobs. Drop in a CSV or Excel file, get a dashboard, anomaly detection, a forecast, and plain-English answers, all calculated locally using pandas (locally in the sense of never leaving the server and going to the AI apis.). The LLM layer that you can use (it's optional) but that only sees column names and never your rows, and the whole thing works with zero API key if need be (not the LLM part of course). MIT licensed, solo-built, open to contributors (in fact would need help to make it move out of LLMs and have a few open issues). I wish to make it as LLM free as possible as I want to make it deterministic. Live demo: automated-data-analyst.streamlit.app Repo: https://github.com/saineshnakra/automated-data-analyst

Heads-up: this post references a tool I built. It's a genuinely useful walkthrough either way — the technique applies to any Airbnb scraping project. If you've ever tried to scrape Airbnb, you already know the two walls you hit: the pages are rendered by JavaScript, and Airbnb aggressively blocks datacenter IPs. Below is the reliable way to get clean Airbnb data in 2026 — listing prices, ratings, coordinates, and discounts — without running a headless browser or babysitting proxies. The key insight: Airbnb ships its data in the HTML You don't need to render the page. Every Airbnb search response embeds the full result set as JSON inside a <script id="data-deferred-state-0"> tag. Parse that and you get structured data straight away — no DOM scraping, no selectors that break on the next redesign. The path to the results is: niobeClientData[*][1].data.presentation.staysSearch.results ├── searchResults[] // ~18 listings per page └── paginationInfo.pageCursors[] // all page cursors, upfront Each listing carries a base64-encoded ID in demandStayListing.id (decode it, take the segment after the last colon, and you have the numeric listing ID for airbnb.com/rooms/<id> ), a price line with discounts, avgRatingLocalized ("4.95 (123)"), and GPS coordinates. The two gotchas Datacenter IPs get blocked. You need residential proxies. If a response comes back without the data-deferred-state marker, you've been served a bot check — rotate to a fresh IP and retry. ~270 result cap per search. Airbnb won't paginate past ~15 pages. To cover a whole market, split into narrower searches (by price band or neighborhood) and dedupe by listing ID. The no-code way If you'd rather not maintain proxy pools and parsers, I published an Airbnb Scraper on Apify that does exactly the above. Paste a location or a full Airbnb search URL (every filter is honored), and get flat JSON/CSV back. curl -X POST "https://api.apify.com/v2/acts/ethanteague~airbnb-scraper/run-sync-get-dataset-items?token=YOUR_TOKE

Every GEO ("generative engine optimization") tool, including ours until recently, sells some version of the same pitch: fix your robots.txt, add Schema.org markup, write FAQ schema, and AI engines will cite you more. We build one of these tools — Causabi scans sites for AI-crawler readiness and generates fix files (robots.txt, llms.txt, JSON-LD, FAQ blocks). As part of validating our own scoring weights, we ran the numbers on whether the score actually predicts getting cited. Short version: it mostly doesn't, once brand prominence is in the picture. What we measured We scored 44 domains on a 6-category on-site readiness algorithm: robots.txt (AI bots allowed or blocked) Schema.org (Organization/LocalBusiness JSON-LD completeness) FAQ schema (FAQPage markup, 3+ entries) content depth/structure brand/NAP signals freshness (dateModified, recency) Then we checked how often each domain actually got cited by an AI engine (Claude, via its web-search tool, one measurement window, a fixed prompt set per domain). What we found On-site score vs. citation rate: Pearson r ≈ -0.08, Spearman ρ ≈ -0.03. Functionally no correlation — if anything, a very slight negative one, which is more likely noise than a real inverse relationship at this sample size. 86% of the 44 domains got zero citations in the window, independent of their score. The domains that did get cited clustered almost entirely by brand prominence — well-known domains got cited at a noticeably higher rate (~0.16 of prompts) than everyone else (~0 for the rest of the sample), regardless of how well-optimized their markup was. Why I'm not overselling this n=44 is small. This is an internal validation exercise for our own product, not a peer-reviewed study, and I don't want it read as one. Specific caveats: Single engine (Claude) this round. Citation behavior differs meaningfully across ChatGPT, Gemini, Grok, and Perplexity — we haven't run the same check across all four yet. One time window, no longitudinal before/after.

Risk stratification sounds like a data-science buzzword until you have to build the thing. For a Medicare Advantage plan, it's a concrete pipeline: take a population of members, score each one's clinical and financial risk, and rank them so care management and documentation teams know who to touch first. Here's how I'd architect it. The core idea Population health risk stratification = scoring + segmentation. You compute a per-member risk signal, then bucket members into tiers (e.g., rising-risk, high-risk, catastrophic) so finite resources go where they move outcomes and revenue most. The mistake teams make is treating it as a single ML model. In practice you want a layered signal: a stable, explainable base (RAF + chronic conditions) plus optional predictive overlays. Explainability matters because care managers won't act on a black-box score, and auditors won't accept one. Step 1: Build the member feature record { "member_id" : "SYNTH-77310" , "age" : 73 , "hccs" : [ "HCC37_1" , "HCC85" , "HCC18" ], "raf" : 1.842 , "gaps" : [ "a1c_overdue" , "no_pcp_visit_180d" ], "utilization" : { "ed_visits_12m" : 3 , "inpatient_12m" : 1 } } The RAF here is your defensible, model-grounded risk anchor under CMS-HCC V28. Everything else is supplemental signal. Step 2: Score and tier def risk_tier ( member ): base = member [ " raf " ] util = 0.15 * member [ " utilization " ][ " ed_visits_12m " ] \ + 0.30 * member [ " utilization " ][ " inpatient_12m " ] score = base + util if score >= 3.0 : return " catastrophic " if score >= 1.8 : return " high " if score >= 1.0 : return " rising " return " stable " Keep the weights transparent and tunable. The point isn't a perfect model; it's a defensible, reproducible ranking your operational teams trust. Step 3: Make "rising-risk" actionable The tier that quietly drives the most ROI is rising-risk — members trending toward high cost who still have open documentation and care gaps. Surface their specific gaps (overdue labs, undocumented chroni

Design + Product Thinking: NYC’s Path to Reliable AI AI delivers value when it’s useful, trusted, and operational. For city services that affect millions, those qualities don’t happen by accident — they come from applying design thinking (who the service is for, how it’s used) together with product thinking (what outcome we’re trying to achieve and how we operate over time). This article explains why hiring designers and product managers matters for NYC’s digital and AI initiatives, summarizes the city’s PIT Crew program, and outlines how Flamelit applies outcome-focused delivery in the public sector. Why design and product roles matter Designers and product managers have distinct but complementary responsibilities that reduce common AI delivery failures: Designers (Design Thinking): center human needs, prototype user flows, and validate that interfaces and decision workflows are understandable and accessible. They surface usability and trust issues early, preventing technically accurate models from becoming unusable in practice. Product managers (Product Thinking): define the measurable outcomes, prioritize use cases, align stakeholders, and manage the lifecycle from discovery to ongoing operations. They ensure work is evaluated against mission impact, not just technical metrics. Together they prevent common failures: building technically impressive models that nobody trusts, deploying brittle systems without human review, or shipping features with unclear ownership that decay in production. PIT Crew and NYC hiring context NYC’s PIT Crew program is a city initiative designed to attract and staff product, engineering, and design talent for public service projects. It’s a practical recognition that public-sector digital transformation needs people skilled in user research, product management, and delivery. Read more about the PIT Crew and how it works here: https://www.nyc.gov/content/pitcrew/pages/ (open in a new tab). Hiring programs like PIT Crew help create the c

If you’ve ever struggled with CALCULATE() or wondered why SUMX() behaves differently from SUM() , this guide is for you. DAX (Data Analysis Expressions) is the language that powers Power BI , Analysis Services , and Power Pivot — enabling dynamic calculations, filtering, and time intelligence. Below is a categorized cheat sheet of essential DAX functions , plus examples showing how to use each in real-world Power BI scenarios. Filtering & Context These functions control how filters are applied and evaluated in your calculations. Function Example Description CALCULATE() CALCULATE(SUM(Sales[Amount]), Region[Name] = "Nairobi") Changes filter context to calculate total sales for Nairobi. FILTER() FILTER(Sales, Sales[Amount] > 10000) Returns a table filtered by condition. ALL() CALCULATE(SUM(Sales[Amount]), ALL(Region)) Ignores filters on Region. REMOVEFILTERS() CALCULATE(SUM(Sales[Amount]), REMOVEFILTERS(Region)) Removes filters from Region. VALUES() VALUES(Customer[City]) Returns unique list of cities. SELECTEDVALUE() SELECTEDVALUE(Product[Category], "All") Returns selected category or “All” if none. TREATAS() TREATAS(VALUES(Temp[City]), Customer[City]) Applies one table’s values as filters on another. KEEPFILTERS() CALCULATE(SUM(Sales[Amount]), KEEPFILTERS(Product[Category] = "Electronics")) Keeps existing filters and adds new ones. ALLSELECTED() CALCULATE(SUM(Sales[Amount]), ALLSELECTED(Region)) Respects user selections in visuals. ALLEXCEPT() CALCULATE(SUM(Sales[Amount]), ALLEXCEPT(Sales, Sales[Year])) Removes all filters except Year. Aggregation Summarize or aggregate data across rows or columns. Function Example Description SUM() SUM(Sales[Amount]) Adds all sales amounts. AVERAGE() AVERAGE(Sales[Amount]) Calculates mean value. COUNT() COUNT(Customer[ID]) Counts non-blank entries. COUNTROWS() COUNTROWS(Sales) Counts rows in a table. DISTINCTCOUNT() DISTINCTCOUNT(Customer[ID]) Counts unique customers. MIN() MIN(Sales[Amount]) Finds smallest sale. MAX() MAX(Sales[Amo

series: Prediksi Survival Klub Debutan Kenapa Project Ini? Setiap musim, klub yang promosi ke liga top (Premier League, La Liga, dst.) menghadapi risiko besar: sekitar 2 dari 3 klub yang naik biasanya kembali terdegradasi di musim pertama mereka. Saya penasaran — bisakah performa di beberapa laga awal musim memberi sinyal dini soal peluang klub tersebut bertahan? Ini jadi project portofolio pertama saya sebagai data scientist yang baru mulai (0-1 tahun pengalaman). Saya sengaja pilih topik yang saya suka (sepak bola) supaya prosesnya tetap enjoyable, bukan cuma "tutorial project" generik. Rencana Project Pertanyaan utama: Berdasarkan performa 8 laga pertama musim debut, seberapa besar peluang klub promosi bertahan hingga musim berikutnya (tidak degradasi)? Data yang dipakai: football-data.co.uk — data hasil pertandingan tiap musim sejak 1993/1994 Wikipedia (halaman musim liga) — daftar klub promosi & klasemen akhir musim Tech stack: pandas , requests untuk data collection scikit-learn untuk modeling (mulai dari Logistic Regression sebagai baseline) imbalanced-learn untuk handle class imbalance Streamlit + Plotly untuk dashboard interaktif Deploy ke Streamlit Community Cloud Timeline (Build in Public) Saya bikin timeline ini publik supaya ada tekanan yang sehat untuk benar-benar menyelesaikannya, bukan cuma jadi ide yang menguap: Checkpoint Target Tanggal Yang Harus Selesai Part 1 (post ini) 11 Juli 2026 Kickoff, rencana, environment siap Part 2 15 Juli 2026 Dataset jadi, push ke GitHub Part 3 17 Juli 2026 EDA selesai, insight awal Part 4 24 Juli 2026 Model final dipilih + evaluasi Part 5 31 Juli 2026 Dashboard live di Streamlit Cloud Part 6 (final) 8 Agustus 2026 Project selesai, recap lengkap Tantangan yang Sudah Saya Antisipasi Data leakage — fitur harus dihitung dari laga awal musim saja, bukan seluruh musim, biar model beneran memprediksi bukan "menyontek" hasil akhir Dataset kecil — kemungkinan hanya ~60-100 sampel klub, jadi saya mulai dari model sederhana (Lo

Adapted from an appendix of my MS thesis. Markov Chain Monte Carlo Almost as soon as computers were invented, they were used for simulation. Markov chain Monte Carlo (MCMC) was invested as Los Alamos, Metropolis et al (1953) simulated a liquid in equilibrium with its gas phase. Their tour de force was the realization that they did not need to simulate the exact dynamics, they only needed to simulate some Markov chain with the same equilibrium distribution. The Metropolis algorithm was widely used by chemists and physicists, but was not widely known among statisticians until after 1990. Hastings (1970) generalized the Metropolis algorithm, and simulations following his scheme are said to use the Metropolis-Hastings (MH) algorithm [1]. A special case of the MH algorithm was introduced by Geman et al (1984) discussing optimization to find the posterior mode rather than simulation. Algorithms following their scheme are said to use the Gibbs sampler. It took some time for the spatial statistics community to understand that the Gibbs sampler simulated the posterior distribution, thus enabling full Bayesian inference of all kinds. Gelfand et al (1990) made the wider Bayesian community aware of the Gibbs sampler, and then it was rapidly realized that most Bayesian inference could be done using MCMC, whereas very little could be done without MCMC. Green (1995) generalized the MH algorithm as much as it could be generalized [1]. Theoretical Foundations A sequence X 1 ​ , X 2 ​ , … of random elements of some set is a Markov chain if the conditional distribution of X n + 1 ​ given X 1 ​ , … , X n ​ depends on X n ​ only. The set in which the X i ​ take values is called the state space of the Markov chain. A Markov chain has stationary transition probabilities if the conditional distribution of X n + 1 ​ given X n ​ does not depend on n . This is the main kind of Markov chain of interest in MCMC. The joint distribution of a Markov chain is determined by the following [1]. The ma

I run a small AI trends site, and three weeks ago I started doing something simple: every day, snapshot the star count of every repo that crosses my GitHub trending scan for AI. No judgment, no curation, just append-only rows in a database. 611 repos and 2,671 data points later (June 19 to July 10), the picture of what's actually growing looks pretty different from what my feeds told me was hot. Here's what the data says. Before publishing this I re-checked every number below against GitHub's live API. Star counts drift by the hour, so treat them as of July 10. The top 10 risers, by raw stars gained Repo Gained Window From → To calesthio/OpenMontage +30,253 21 days 5,899 → 36,152 DeusData/codebase-memory-mcp +20,483 19 days 7,516 → 27,999 mattpocock/skills +19,053 15 days 137,485 → 156,538 obra/superpowers +16,887 20 days 232,908 → 249,795 NousResearch/hermes-agent +14,896 21 days 197,297 → 212,193 Panniantong/Agent-Reach +14,334 14 days 34,780 → 49,114 usestrix/strix +13,243 12 days 26,363 → 39,606 addyosmani/agent-skills +12,685 21 days 63,156 → 75,841 asgeirtj/system_prompts_leaks +11,720 21 days 43,415 → 55,135 msitarzewski/agency-agents +11,055 10 days 118,241 → 129,296 Windows differ because I only hold snapshots for the days a repo appeared in my scan; each row states its own real window. Three things in this data genuinely surprised me. 1. "Skills" are eating agent frameworks Four of the top ten are not agent frameworks. They are collections of packaged expertise that plug into an existing agent: obra/superpowers (still compounding at roughly 840 stars a day on a 250k base), mattpocock/skills, addyosmani/agent-skills, msitarzewski/agency-agents. A year ago this table would have been full of new frameworks. Now the framework layer looks settled and the growth is in what you load INTO the agent. The moat moved from orchestration code to encoded judgment. 2. The sharpest climbs are applications, not infrastructure The steepest sustained climb from a newcomer in