标签：#datascience

Most "is this plugin safe?" advice is vibes. I wanted numbers, so I built a dataset. Here's what it found, and exactly how, so you can check my work or build your own. The finding first Of 8,010 WordPress plugins with a publicly documented vulnerability since 2023 (15,534 vulnerability records in total): 3,780 have been removed from the wordpress.org plugin directory. Removal stops updates but doesn't uninstall — affected sites keep running the code. 277 carried a critical (CVSS ≥ 9.0) flaw on record before removal. 2,115 are still installable today with a known vuln and no update in 12+ months — roughly 6.7M active installs combined. The part that surprised me most: "removed from the directory" is nearly invisible to a site owner. No dashboard warning, no email. The plugin just quietly stops getting fixes while sitting on the site. How I built it (no paid APIs) The whole thing runs on two public sources and no API keys. 1. Vulnerability data — the GitHub Advisory Database. It mirrors CVE records including the Patchstack and Wordfence CNA assignments that cover almost all WordPress plugin CVEs. It's a git repo, so a shallow, sparse clone of the advisories/unreviewed/{year} folders gets you the raw JSON: git clone --depth 1 --filter = blob:none --sparse \ https://github.com/github/advisory-database.git Each advisory carries the CVE ID, a CVSS vector string, CWE IDs, and reference URLs. The plugin slug isn't a first-class field — you recover it from the Patchstack/Wordfence reference URLs with a couple of regexes. That alone attributes the large majority of WordPress advisories to a specific plugin. 2. Maintenance signals — the wordpress.org plugin API. For each slug: https://api.wordpress.org/plugins/info/1.2/?action=plugin_information&request[slug]=SLUG That gives install count, last-updated date, tested-up-to version, and support-thread resolution ratio. A 404 (or an {error} body) means the plugin isn't in the directory — but that's ambiguous: it could be removed ,

Classic Machine Learning Through the Eyes of an SRE — Part 9: Isolation Forest The algorithm in one line: Isolation Forest scores how anomalous a point is by how few random cuts it takes to separate that point from everything else. No model of normal, no loss function, nothing optimized. ← Previous: Part 8 — Hierarchical Clustering Fails Beautifully · Next: this is the series finale — start at Part 1 . Every anomaly detector I had studied models what NORMAL looks like, then calls the leftovers outliers. K-Means: far from every centroid. DBSCAN: in the noise bucket. Sensible, and intuitive. Isolation Forest does not bother. It never models normal at all. It goes straight at the rare points with a single question: how few random cuts does it take to isolate you? Random cuts, literally. Pick a feature at random, pick a split value at random between that feature's min and max, repeat. A point that separates from the crowd in three cuts is anomalous. A point buried in the middle of a dense mass takes thirty. Grow hundreds of these random trees, average the isolation depth for each point, and you get an anomaly score. There is no loss function here. No optimization, not even the local kind that decision trees do at every split. Every cut is a coin flip, and the power comes entirely from averaging, which is the forest trick from the supervised half of this series now applied to pure randomness. Cheap randomness plus averaging beats careful modeling, as long as the target is something randomness naturally exposes. Rarity is exactly that. Sometimes the winning move is to optimize less. That sentence would have gotten me laughed out of my first ML study session. It is also this finale's thesis. The part I had completely backwards Here is the thing I did not know until I read the original paper properly, and it is the opposite of every instinct a decade of ops gave me. Isolation Forest deliberately trains each tree on a small subsample of your data, and this is not a performan

Classic Machine Learning Through the Eyes of an SRE — Part 8 The most dangerous output in my whole Week-1 study set wasn't a bad prediction. It was a beautiful tree. Hierarchical clustering produces a dendrogram, that elegant diagram where every account, ticket, or incident nests inside ever-larger families. It looks like discovered truth. Stakeholders lean in. Someone screenshots it for the QBR deck. Nothing else in the set looks as convincing while being as capable of being completely wrong. A bad K-Means gives you blobs that feel arbitrary, and people push back. A dendrogram built with the wrong linkage on flat data still looks like a family tree of your business. Nobody pushes back on a tree. The bet and the build Hierarchical clustering completes the answer-finding taxonomy I've been using through this series. That's my own shorthand, not standard terminology: K-Means SEARCHES, DBSCAN DEFINES, PCA SOLVES, and hierarchical clustering BUILDS. Start with every point as its own cluster. Repeatedly merge the closest two clusters. Never undo. Greedy and irreversible, a little like growing a decision tree. Same skeleton, different family. There is also a top-down version, called divisive clustering, which starts with everything together and splits it. In practice, when people say hierarchical clustering, they're usually talking about the bottom-up, agglomerative version. Two things were genuinely new to me. You choose the cut after seeing the structure. Fitting doesn't require you to decide K upfront. The dendrogram gives you the hierarchy, and you choose where to cut it to get the number of clusters you want. That makes the output unusually flexible. For a delivery organization it also feels natural, because account family → sub-segment → individual account is already how a lot of governance gets organized. Linkage is a selectable worldview. "Closest clusters" needs a definition, and every definition makes a different assumption. Ward pushes toward compact, variance-

Originally published at ai.bedvibe.studio . There is an assumption almost every practitioner carries without examining it: if your dataset has bad samples in it, the loss will tell you. Corrupted rows spike. Broken files stick out. Sort by per-sample loss, look at the top of the list, and there is your garbage. I believed it too. Two separate failures in my own work say it is wrong, and they fail in the same direction — quietly. The reproducible one: a dataset that cannot be learned While validating trainproof I ran a controlled fault-injection study: one base setup, a Qwen2.5-3B QLoRA, run six ways, three seeds each, eighteen runs total. Every log ships in the repo so the verdicts can be checked rather than believed. One configuration shuffled the dataset's labels into pure noise. The labels no longer corresponded to the inputs at all. This is not a hard dataset or a noisy dataset. It is a dataset that cannot be learned , because there is no relationship left in it to learn. That run reduced its loss by 62%. On its own curve it was textbook-healthy — a clean downward slope, no spike, no plateau, nothing a human or a rule would flag. It was learning nothing useful. It was memorising the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve it is indistinguishable from a real one. That is where the assumption broke for me. Not "loss is a weak signal for this." Loss is not a signal for this at all, in isolation. The production one, and what I can and cannot prove about it The second failure came from real work rather than an experiment, and it is the one I think about more. Building a text-to-speech corpus of roughly 110,000 recordings, a small number of the files were pure loud white noise. Not corrupted in the file-format sense — they opened fine, played fine, had valid headers and valid duration. They simply contained no speech. Just noise, at volume. They did not surface as high-loss outliers. Being precise about

This is a Plain English Papers summary of a research paper called OpenART Red-Teams Stateful Agents Across 10,000 Evolving Environment Scenarios . If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter . OpenART turns persistent state into the red-team target OpenART evaluates agent safety across more than 10,000 validated stateful scenarios spanning 50 domains and requiring a median of 97 tool calls. Its central claim is that safety failures can emerge from trajectories in which workspace data, permissions, memory, and plans are repeatedly modified, rather than from isolated prompts alone. The arena keeps each benign task objective and hidden safety contract fixed while changing only the target-visible environment state. This design targets delayed failures that static benchmarks can miss: an early authorized mutation may influence later decisions, expose protected resources, or produce unsafe output many steps after the original change. OpenART extends the broader idea of agent safety evaluation by making persistent environment state the object that evolves during testing. OpenART reports a pooled strict Attack Success Rate of 85.0% across 75 agent-model configurations. Strict success requires both the deterministic evaluator and a GLM-5.2 judge to identify the attack condition, so disagreements count as failures rather than being treated as partial evidence.... Continue reading the full paper summary on AIModels.fyi →

This is a Plain English Papers summary of a research paper called RA-Bench Reveals Why Crisis-Video Deepfake Detectors Fail Across Generators and Social Media . If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter . The crisis detection problem we've been getting wrong Video synthesis has reached an inflection point. Recent generators can fabricate realistic depictions of wars, natural disasters, infrastructure failures, and public emergencies so convincingly that they fool both people and current detection systems. The threat isn't hypothetical anymore. A fabricated video of a nuclear plant explosion, a hospital collapse during an earthquake, or a terrorist attack could trigger panic, military response, or severe economic disruption within hours. Yet here's the troubling part: we don't actually know if our best detection tools can handle these high-stakes scenarios in the wild. Researchers have built impressive deepfake detectors, trained them on standard benchmarks, and measured their performance. But those benchmarks test detectors against generic synthetic videos, not against the specific threat that actually matters: AI-generated crisis footage designed to fool people about real things that happened. It's like training a border guard to spot counterfeit passports in a lab with perfect lighting and a magnifying glass, then sending them to a busy airport where they have to make decisions in three seconds. The guard's failure has nothing to do with their skill. The problem is that the testing environment was completely divorced from the real scenario.... Continue reading the full paper summary on AIModels.fyi →

This is a Plain English Papers summary of a research paper called Macaron-V1: Continual Learning with Self-Improvement and Mixture-of-LoRA Adapters . If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter . The problem with frozen models Most AI systems today follow a familiar pattern: train, evaluate, deploy, and then stop. The model is locked at that moment, treated as a finished product rather than a living system. But the real world immediately begins to diverge from training data. Users interact with the system in ways the training process never anticipated. New domains emerge. Preferences shift. The model that seemed smart on test day becomes gradually less relevant over time. This frozen-in-place approach isn't accidental. It reflects how machine learning has been practiced for decades. Retraining is expensive. Deploying new versions carries risk. The infrastructure to continuously improve systems in production barely exists. So instead, teams ship a model and move on, accepting that it will decay slowly but inevitably. Macaron-V1 asks a different question: what if AI systems could continuously improve themselves through real-world experience, learning from the billions of interactions that happen after deployment? Not in theory, but actually, in production, with users. The answer isn't magic. It requires two architectural shifts. First, treat deployment as the beginning of a learning process, not the end of one. Build versioning, evaluation contracts, and feedback loops directly into the system. Second, stop assuming you need to retrain your entire model. Instead, freeze a stable base and compose lightweight specialist adapters around it, allowing the system to grow in capability without losing its foundation. Rethinking deployment as a continuous learning opportunity The insight here is architectural. Instead of viewing the deployed model as the final form, Macaron-V1 treats it as the first link in an infinit

This is a Plain English Papers summary of a research paper called BDH-CQ Uses Recurrent Latent Reasoning to Cut ARC-AGI Inference Costs . If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter . The cost-accuracy trap in visual reasoning Large language models are fundamentally mismatched for visual reasoning tasks. They're forced to describe every thought out loud, generating token after token to explain their logic. This verbosity taxes compute budgets, yet paradoxically doesn't improve performance. Ask a language model to solve an ARC-AGI puzzle (a visual reasoning benchmark designed to test abstract thinking), and it either struggles despite the verbosity or succeeds expensively. The root problem runs deeper than just inference cost: the model learns from demonstrations by parsing them as language tokens, which is an indirect and inefficient way to absorb a visual pattern. The efficiency frontier has been unforgiving. If you want cheap inference, you sacrifice accuracy. If you want accuracy, you sacrifice cost. Every model on the leaderboard until recently clustered into one of two camps, and no one had found a path that broke the tradeoff. BDH-CQ challenges this assumption by proposing something radical: reasoning doesn't need to be visible to work. The model absorbs demonstrations silently into its internal memory state, then solves problems through private iteration in hidden layers, without generating a single token of intermediate reasoning. A 150-parameter variant achieves 29.5% pass@2 on the ARC-AGI-1 benchmark at a computed cost of just $0.0007 per task, puncturing through the previous Pareto frontier and establishing a new state of the art in cost efficiency. Learning through hidden states The core insight is deceptively simple: a model's reasoning process doesn't need to match human communication. When you learn a new skill from examples, you don't narrate every observation. You absorb patterns directly i

This is a Plain English Papers summary of a research paper called How Cross-Model Compatibility Lets Attackers Extract Proprietary LLM Reasoning Traces . If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter . The illusion of safety Major AI companies now show users their models' step-by-step reasoning as a feature. OpenAI offers it through o1, Anthropic through extended thinking, Google through its reasoning-focused variants. But this reasoning is a double-edged sword. It's intellectually valuable to share, showing users why a model reached a conclusion. But it's also intellectually valuable to steal. Competitors want to understand how frontier models think. Researchers want to study their reasoning patterns. Attackers want to extract proprietary algorithms. So the companies made a choice: hide the reasoning from users by encrypting it. The idea sounds straightforward enough. Return the reasoning to the user's device in an encrypted, unreadable form. The user can't see it, competitors can't see it, but they can pass it back to the server in future requests if they need continuity with previous reasoning. The server alone holds the decryption keys. Problem solved. Except it wasn't. Researchers discovered that this encryption doesn't actually hide reasoning. It just makes it look hidden. The encrypted blocks are designed to work everywhere within a company's ecosystem, across different sessions and different models. That universal compatibility is a feature for convenience. But it's also an architectural vulnerability that anyone can exploit. The architectural gamble To understand where this went wrong, you need to see how the system actually works. When a user sends a request to a frontier model like GPT-4, the model internally generates a reasoning trace, the raw thought process behind its answer. Instead of returning this reasoning in plaintext, the company encrypts it on the server before sending it to the client.

La forma del problema Un directorio es una superficie: el miembro lo abre y adivina. Un recomendador es una superficie de empujar: el sistema propone y tiene que justificarse. La justificación es la parte difícil, y es donde vive la estadística. Tres restricciones hicieron esto distinto de un recomendador de contenido: El item es una persona con capacidad finita. Un hilo se le puede recomendar a diez mil personas. Un experto no. Una mala recomendación es cara de los dos lados. Quien pide desperdicia una petición, el experto desperdicia una hora, y los dos aprenden a ignorar la superficie. La afirmación tiene que ser checable. "Quizá te guste este hilo" no necesita evidencia. "Esta persona está un nivel adelante de ti en diseño de sistemas" sí. Recuperación: híbrida, fusionada con RRF Tres recuperadores independientes sobre el conjunto de expertos elegibles, fusionados con Reciprocal Rank Fusion: def rrf_fuse ( * ranked_lists , k = 60 ): """ Fusiona listas de ids rankeadas. El score depende solo del rank, nunca de la escala propia del recuperador, que es el punto: la similitud coseno y un conteo de hilos resueltos no son números comparables. """ fused = {} for lst in ranked_lists : for rank , key in enumerate ( lst ): fused [ key ] = fused . get ( key , 0.0 ) + 1.0 / ( k + rank ) return fused RRF es la primitiva correcta aquí por una razón que vale la pena decir: los recuperadores emiten cantidades incomparables. Uno regresa un coseno en [-1, 1] , uno regresa un conteo entero de hilos resueltos, uno regresa un delta de nivel de escalera. Normalizarlos a una escala común requiere supuestos sobre sus distribuciones que nadie tiene a este volumen de datos. RRF descarta las magnitudes y se queda solo con el orden, que es exactamente la información que sobrevive a una muestra chica. k = 60 es la constante estándar de la formulación original de Cormack et al. Aplana la cabeza: la diferencia entre el rank 1 y el rank 2 es 1/61 - 1/62 ≈ 0.00026 , así que un recuperador no pu

Electric-vehicle assistants rarely have just one job. A short Turkish question may ask for a nearby station, a charging-price comparison, help planning a route, or an explanation of battery health. Before an application can retrieve current data or generate an answer, it needs to identify that intent reliably. We created the Turkish EV Charging Intent Dataset as a small, transparent starting point for that routing problem. Version 1.0.0 contains 192 Turkish queries distributed evenly across eight intent classes. It is open under CC BY 4.0, includes fixed train, validation, and test splits, and is maintained by TekPedal , an EV charging map and vehicle decision platform for Türkiye. You can explore the dataset interactively , inspect the source and validation workflow on GitHub , or cite the permanent Zenodo release with DOI 10.5281/zenodo.22062688 . Why intent routing comes first An assistant should not answer every EV question in the same way. Different requests need different tools and freshness guarantees: a station request needs a map or location index; a price request needs current tariff data; route planning needs distance, range, and charging-stop logic; a battery question needs careful educational content; a vehicle comparison needs structured specifications. An intent router makes that separation explicit. It can send each query to the correct retrieval source, product page, or application workflow. This also makes evaluation easier: teams can test routing independently before measuring the quality of downstream answers. Dataset design The taxonomy contains eight balanced classes, with 24 records in each class: FIND_STATION COMPARE_PRICE ROUTE_PLANNING CHARGING_SPEED VEHICLE_COMPARISON HOME_CHARGING BATTERY_HEALTH OWNERSHIP_COST Every record includes a stable ID, the Turkish query, the intent identifier, a human-readable Turkish label, a suggested TekPedal content route, the assigned split, the language, and a provenance marker. Here is a simplified example

Short version of a post on gex.live/research ; the full write-up, definitions and reproduce block live there. Most published dealer-gamma numbers are built from open interest : yesterday's outstanding contracts, multiplied by a convention about who holds which side. Whether the convention is right is a separate question. The prior question is simpler: how much of what trades today was already in that book this morning — and how much of tomorrow's book is being created today? Open interest and volume are enough to answer it, with no assumption about who bought. Sample: SPX and SPXW, 2022-04-14 to 2026-08-14, 1,081 trading days, every expiry within about a month (0DTE plus the 21 nearest), 8.6 million contract-days, 4.3 million with volume. Definitions Per contract (expiry, strike, right) and session D: OI(D) is open interest at the start of D, OI(D+1) at the start of the next session, ΔOI = OI(D+1) − OI(D) , vol the day's volume in that contract. |ΔOI| / vol is a lower bound on how one-sided the day's trading in that contract was — 1.0 means every lot opened (or every lot closed), 0 means opens and closes cancelled. Contracts expiring on D have no next-day OI and drop out of the ΔOI statistics; 4.1% of rows (3.8% of volume) show |ΔOI| > vol, which is impossible (OI snapshot timing) and are excluded. The book grows by 40% of what trades, every day days to expiry on D net ΔOI / volume |ΔOI| / volume (lower bound on one-sidedness) share of volume in contracts whose OI rose contract-days 1–5 37.8% 41.8% 90.3% 813,013 6–21 42.7% 53.3% 81.1% 2,206,446 22+ 42.8% 57.6% 76.5% 831,896 Across the whole book, net ΔOI is 39.9% of the day's volume on the median session (IQR 36.2–44.0%), positive in every year and every expiry bucket: the SPX book is always being built faster than it is unwound, until expiry does the unwinding. Far expiries are open-and-hold (a day's trading in a 22+ DTE contract is at least 58% one-sided); the nearest expiries churn (42% at 1–5 DTE). Per contract-

In the current epoch of Artificial Intelligence, the industry remains singularly preoccupied with the "Model" — obsessing over the raw parameter scales of the latest LLMs or the specific benchmark performance of a new transformer variant. However, at Informatiqs, we shift the lens. We recognize that sustainable enterprise value is rarely derived from the model in isolation; instead, it emerges from the high-stakes architectural decisions and systemic orchestration that define its environment. As we launch our inaugural edition, we dissect a critical technological nexus: the convergence of Graph Neural Networks (GNNs), Generative AI, and the industrial-grade infrastructure of Google Cloud Platform (GCP). We argue that for complex enterprise datasets, the transition from flat vector embeddings in latent space toward non-Euclidean, graph-based relational intelligence is the primary differentiator for the next generation of resilient AI applications. 1. The Scientific Foundation: Exploiting Relational Inductive Bias Traditional Deep Learning architectures, such as Convolutional Neural Networks (CNNs) for images or Transformers for text, primarily operate on data structured as sequences (Euclidean space). While exceptionally powerful, these structures often fail to capture the topological nuances of real-world systems like supply chains, molecular structures, or fraudulent transaction webs where data is inherently non-Euclidean. Graph Neural Networks (GNNs) provide a framework for learning from data represented as nodes and edges. Unlike standard neural networks that process inputs in isolation, GNNs utilize a Message Passing paradigm. In this process, a node's internal representation is iteratively updated by aggregating information from its immediate neighbors. Instead of looking at a data point as a single row in a database, the GNN looks at who that data point "talks to" and how those connections define its identity. By utilizing Graph Attention mechanisms, we can fu

A small release, not a launch. The LAB tab on gex.live has a new rightmost rail called IDEA FEED . It is a stream of short, rule-shaped trading ideas about SPX dealer positioning — "fade the first touch of the call wall after a gap up", that kind of thing — collected daily by a scanner from what people actually discuss, rewritten into something the Lab compiler can parse, and published untested . That last word is the point. Why untested is the feature Every feed of trading ideas on the internet comes with a verdict attached: "this works", "78% win rate", a screenshot of a good month. The feed here refuses to do that. Each card says exactly two things about its idea: compiles clean (our compiler turned the text into a runnable rule without complaint) and untested (nobody has run it against the archive yet). The honest test is yours to run. One click drops the idea into the Lab conveyor. The compiler has already done the translation, so the first message in your session is the rule itself, stamped ↳ from IDEA FEED · compiles, untested . Running the backtest costs one Lab credit; a failed job refunds itself. If your balance is zero the button does not go dead — it turns into 0 CREDITS · BUY → , remembers the idea you picked, and comes back to it after. What you will not find No source attribution on the cards. The idea is the unit, not the poster. No win rates, no "rated", no thumbs. The archive is 1,000+ finished SPX sessions; the Lab tests against all of it with an out-of-sample split and tells you what survived, which so far is: very little. That verdict is worth more than a badge on a card. No approval gate. The scanner's finds ship directly every day, so the feed stays fresh by itself. "NEW" is personal — it means new since you last opened the rail, not new for everyone. Why build a feed that mostly produces "no" Because the alternative is pretending. The whole site is built on measuring dealer positioning from the tape instead of assuming it from yesterday's ope

gex.live has two halves. The terminal measures where SPX options dealers are positioned, every second, from the tape. The Lab is the half that asks the uncomfortable question: does any of that predict anything? What it is A browser-side conveyor with three stages and a credit meter. Compile. You describe a rule in plain text — "short the first touch of the put wall when net gamma is below the 20th percentile" — and the compiler turns it into a deterministic rule over the archive's fields: flip, walls, hold band, gamma percentile, DEX/VEX/vanna/charm per strike, time of day. Compiling is free. If the text is ambiguous the compiler says which part, instead of guessing. Backtest. The rule runs against the full session archive — 1,000+ finished SPX days, every one of them public at gex.live/sessions — with a fixed out-of-sample split. One credit per job; a job that fails refunds itself. Quant optimize. Optional. A LightGBM pass over the same feature store to see whether there is structure the hand-written rule missed, reported as out-of-sample AUC plus feature importance, not as a new "signal". The heavy part (DuckDB + LightGBM) runs in a scale-to-zero container that reads snapshots over HTTPS from the public archive. It depends on no machine and on no private data, which is the point: you are testing against the same files anyone can download. The honest-stats rule Every verdict comes with its baseline. "Your rule made 3% in-sample" means nothing next to "the unconditional drift over the same days was 2.8%". The report shows both, shows the out-of-sample half separately, and refuses to produce a headline number from the in-sample half. Most rules do not survive this. That includes our own: the site's own directional levels were tested three separate ways across the whole archive and none held out of sample — which is why the terminal sells measurement and not signals, and why the Lab exists at all. The free Idea Feed Next to the conveyor sits a rail of rule-shaped idea

Why plain k-fold silently overfits your trading model — and the 4-line fix that stops it. The Problem With k-Fold in Time Series Financial data is sequential. k-fold shuffles rows, so a training row from 2 PM Tuesday sits next to a test row from 10 AM Monday. Worse: triple-barrier labels overlap . A label at bar t looks 6 bars into the future; a training row at t+2 "knows" part of that future. The model leaks. V1's history is full of "HIGH overfit" verdicts — train AUC high, test AUC flat. Plain TimeSeriesSplit is only marginally better; it still lets adjacent windows bleed into each other. Purged + Embargoed CV For each test window [t0, t1] : Purge any train row whose label window overlaps the test window. Embargo max_training_horizon bars after the test window — drop those too. Overlapping labels are not i.i.d. Purging + embargoing makes the split honest. def purged_embargo_split ( n , n_splits = 5 , embargo_frac = 0.02 ): idx = np . arange ( n ) fold = np . array_split ( idx , n_splits ) splits = [] for i in range ( n_splits ): test = fold [ i ] emb = int ( len ( test ) * embargo_frac ) lo , hi = max ( 0 , test [ 0 ] - emb ), min ( n , test [ - 1 ] + emb + 1 ) train_mask = np . ones ( n , bool ); train_mask [ lo : hi ] = False splits . append (( idx [ train_mask ], test )) return splits Tune Only When You Have Enough Optuna once "won" a validation set with only 4 decisive rows — statistically meaningless. Rule: never tune when the decisive (non-abstained) validation rows are below ~30–50. Widen the date range or symbol basket first; don't trust the trial. Three-Way Split, Always train (fit) → validation (early stop + HP select) → disjoint calibration set (sigmoid/ isotonic) → test (untouched, final score only). V1 sometimes conflated validation and calibration. Keep them separate. The Promotion Gate Log every trial's train/val/test gap, not just the winner's test score. Promote only if replay AND shadow (≥1 live session) both beat baseline on buyer metrics : 1.5x

If you have ever copy-pasted a table out of a PDF, you already know what happens. Rows collapse into one long line of text. Columns interleave. Numbers land in the wrong cell, or no cell at all. The table on the page looks perfectly structured, but a PDF has no real concept of "table." It only knows where individual characters sit on a page. Every extraction tool, ours included, has to reconstruct the table from scratch, using nothing but the position of each word. That gap between "looks like a table" and "is structured data" is where almost every free PDF tool falls apart. Here is how we handle it, what actually works, and where it still doesn't. Two different jobs, two different tools PDFHaul splits this into two separate tools because they solve different problems. PDF to Excel rebuilds the whole document as a single spreadsheet, in the order it appears on the page: form labels, key-value pairs, section titles, and tables all together. It is for documents where you want the full content, not just the numbers, things like invoices, time sheets, and reports. Extract Tables does the opposite. It ignores everything that isn't a table and hands back one clean sheet per table, nothing else. It is for people who want structured data out, ready to sum, sort, and filter, not a copy of the document. Both tools share the same underlying geometry engine. The difference is what each one keeps and what it throws away. How Extract Tables actually decides what's a table The core problem with table extraction is that "looks tabular" and "is tabular" are not the same thing. A vector chart's axis box, a form's outlined signature field, and a two-column list of allergen names all produce something that a naive extractor will happily read as a grid. None of them are tables. Our pipeline handles this in four phases, all before anything is written to a spreadsheet: Phase 1: classify the page. Every page is scored as bordered (has ruled lines or filled-rectangle grid lines), stream (no

Getting started with machine learning WEKA for Beginners: A Practical Introduction to Machine Learning Without Code Getting started with machine learning often means learning Python, libraries, datasets, and a lot of new terminology at the same time. WEKA offers a different approach. WEKA (Waikato Environment for Knowledge Analysis) is a machine-learning and data-mining workbench that lets you explore datasets and experiment with algorithms through a graphical interface. It is particularly useful for students and beginners who want to understand the machine-learning workflow before writing everything from scratch in code. What Can You Do With WEKA? WEKA provides tools for several common machine-learning tasks: Data preprocessing Classification Regression Clustering Association-rule mining Attribute selection Model evaluation Data visualization The Explorer interface is usually the best place for beginners to start. A typical workflow looks like: Dataset ↓ Preprocessing ↓ Feature Selection ↓ Algorithm ↓ Model Evaluation ↓ Interpretation Step 1: Load Your Dataset WEKA commonly works with ARFF (Attribute-Relation File Format) files, although it can also work with formats such as CSV. A simple ARFF dataset might look like: @relation students @attribute study_hours numeric @attribute attendance numeric @attribute passed {yes,no} @data 5,90,yes 2,60,no 8,95,yes 3,70,no The header describes the attributes, while the data section contains the individual instances. Understanding the structure of your dataset is important before applying any algorithm. Step 2: Preprocess the Data After loading the dataset, use WEKA's Preprocess section to inspect and prepare the data. You can examine: Attributes Number of instances Missing values Class distribution Attribute types WEKA also provides filters for operations such as removing attributes, handling missing values, normalization, and other transformations. Good preprocessing can have a significant impact on model performance. Step 3

Time zones look like a tidy grid of whole hours. They aren't. I read the standard UTC offset of all 162 cities, countries and regions on our World Clock straight from the IANA database (via Intl ) — and the real shape is lumpy, with quarter-hour outliers and a near-even split over whether clocks move at all. The quirk, in one line: Kathmandu keeps its clocks 5 hours 45 minutes ahead of UTC — the only :45 offset on the board, and one of 11 places out of 162 that don't sit on a whole hour. Nearly half the rest never move their clocks at all. The clocks that don't sit on the hour Most of the world rounds to a whole hour from UTC. A handful don't: Offset Places UTC+3:30 Tehran (Iran) UTC+4:30 Kabul (Afghanistan) UTC+5:30 India — New Delhi, Mumbai, Kolkata, Bengaluru, Hyderabad UTC+5:45 Kathmandu (Nepal) UTC+9:30 Adelaide, Darwin (Australia) Half-hour and quarter-hour offsets are a reminder that a time zone is a political decision, not an astronomical one — which is exactly why date code should read the IANA database rather than dividing longitude by 15. Nearly half never change their clocks Daylight saving feels universal if you live in North America or Europe, but it isn't. Of the 162 places tracked, 87 (54%) shift their clocks and 75 (46%) never do . The whole of East Asia, the Gulf, most of Africa, India and much of South America keep one fixed offset year-round — Tokyo, Singapore, Dubai, Nairobi and New Delhi never spring forward. Where the clocks crowd together Offsets aren't evenly populated. Four of them carry nearly half the board: Offset Places Who's there UTC−5 25 US Eastern — New York, Toronto, Miami, Boston UTC+1 21 Central Europe — Paris, Berlin, Rome, Madrid UTC−6 14 US Central — Chicago, Dallas, Mexico City UTC+2 12 Eastern Europe & Africa — Athens, Cairo, Johannesburg The full set spans 22 hours , from Honolulu at UTC−10 to New Zealand and Fiji at UTC+12. Reproduce it Every number here is printed by one dependency-free Node script that reads each place's

Real transaction data is never clean — and the worst part is that it looks clean. This is a short story from a real dataset (UCI Online Retail: 541,909 e-commerce transactions) about the quietest way to destroy data: silent type coercion. All numbers below come verbatim from an executed notebook. The head looks perfect Peek at the first rows of the file and InvoiceNo parses as clean integers — 100% parse rate, full confidence. Any type-inference step, mine included, would call it int64 and move on. Measure the whole file instead of the head, and the number drops to ~98%. The other 2%: invoice numbers starting with "C" — which in this dataset marks a cancellation . Coerce the column to numeric and every one of them becomes NaN : Invoice numbers destroyed by numeric coercion: 9,291 DextraLoaderWarning: load: ambiguous decision(s): column 'InvoiceNo': ambiguous - float64 at parse_rate=0.98 An entire class of business events — silently gone. No exception, no crash. That's what makes coercion the quietest bug in data work: the pipeline succeeds . Why those 9,291 rows matter They are not noise. They are the returns side of the business : cancelled orders worth 8.4% of everything sold. Lose them and every revenue number downstream is quietly wrong. One example of what they catch: the dataset's apparent #1 bestseller, "PAPER CRAFT, LITTLE BIRDIE" (168,470 GBP), is a phantom — a single 80,995-unit order entered at 09:15 and fully cancelled at 09:27 the same morning. Only the preserved cancellation rows expose it. The genuine bestseller is a cake stand. The fix: identifiers are labels, not quantities No library can know that "InvoiceNo" is an ID — that's domain knowledge. What a tool can do is disclose its guess and hand you a replayable plan you can correct: naive , plan = dx . load ( CSV_PATH , return_params = True ) # warns: ambiguous at 0.98 plan [ " columns " ][ " InvoiceNo " ][ " dtype " ] = " object " # invoices are labels plan [ " columns " ][ " StockCode " ][ " dtype