标签：#Python

weightwatch v0.1: escanea backdoors en modelos open-weight antes de cargarlos Cualquiera puede subir un LLM fine-tuneado a HuggingFace y afirmar que es seguro. Un modelo con backdoor (puerta trasera) se comporta con normalidad en uso corriente y solo se desvía cuando un trigger oculto se activa. Si no tienes los datos de entrenamiento ni una referencia limpia, no puedes detectarlo . Eso es exactamente el problema que resuelve weightwatch : un escáner black-box que, antes de que confíes en un modelo de terceros, fuerza la activación repetida del posible backdoor y emite un veredicto: CLEAN , SUSPICIOUS o BACKDOOR . El gap que motiva el proyecto No es intuición: lo medí. Barriendo arXiv (papers 2026, filtro anti-survey) contra total_count de repos GitHub que ya resuelven cada problema: Área Papers arXiv 2026 Repos GitHub (suma/máx) Seguridad multi-agente 68 2964 / 2093 Detección de alucinaciones 63 1291 / 860 Backdoors en modelos open-weight 75 66 / 39 Envenenamiento en RAG 54 522 / 249 El ganador estaba claro: 75 papers cuantifican el problema, pero GitHub tiene 0 repos para "fine-tuned model backdoor scanner" y 1 para "fine-tuning poisoning detector". La investigación explota; el tooling apenas existe. weightwatch es la audit-tool de ese sub-nicho (el patrón de keybound / topowatch aplicado a la cadena de suministro de modelos). Cómo funciona weightwatch aplica la técnica output-to-input loop (arXiv: 2608.11348 ): Genera texto con el modelo. Re-inyecta su propia salida como entrada varias iteraciones (greedy, semilla fija). Mide si la trayectoria converge a una firma anómala estable — la huella de un backdoor latente. Además ejecuta un conjunto de muestras canary (inputs inofensivos que un backdoor típico dispara) y cuenta cuántos producen la firma esperada. Sin datos de entrenamiento ni modelo base limpio: eso es lo que lo hace útil en la práctica. pip install -e ".[dev]" weightwatch --fixture backdoored --json Salida real del CLI: { "fixture" : "backdoored" , "ver

A field-service team learns the hard way A field-service team built a support chatbot that sent every message to a cloud LLM endpoint. The design held until a technician drove through a tunnel, and the request queue grew into an eleven-minute backlog. The same week, a support ticket containing a customer's account number appeared in a third-party log because the payload was never classified. The fix was not a bigger cloud budget but a local-first router that decides where each request runs. Why cloud-first fails in three specific ways Latency is the first failure mode, because a round trip to a hosted endpoint adds network time on top of model time. Autocomplete-style features feel broken when every keystroke waits for a distant server instead of a local process. Secrets are the second failure, because any payload sent to a third party can leak into logs or vendor systems. Offline is the third, because a tablet in a tunnel simply has no route to the cloud. The decision table that replaces the either-or debate Local inference and cloud APIs are two legs of a routing policy, not a binary choice. Each request deserves an evaluation against the same conditions, and the table below captures those conditions. The router implementation in the next section turns that table into executable logic with a small Python module. The recent wave of free and cheap model announcements makes this decision more urgent, because every new endpoint adds another leg to the routing table. Condition Local model Cloud free server Payload contains PII Always Never Network unreachable Always Never Latency budget under 300 ms Prefer Avoid Task requires strong reasoning Avoid Prefer Local queue deeper than three Avoid Prefer Token budget nearly exhausted Prefer Avoid The table encodes a simple principle: privacy and availability win over capability. Capability wins only when the network is healthy and the payload is safe. The table also exposes the hidden assumption that a local model is always a

We are living in an era where our wearable devices know more about our physiological state than we do. My Oura Ring knows I stayed up too late binge-watching The Bear , yet my Google Calendar still insists I have a "High-Intensity Interval Training" (HIIT) session at 8:00 AM. This disconnect is where injuries happen and burnout begins. In this tutorial, we are building a Digital Twin Health Agent —a sophisticated AI Agent using LangGraph and Healthcare Automation to bridge the gap between bio-data and action. By the end of this guide, you’ll have a system that reads your recovery scores, reschedules your workouts, and even orders magnesium supplements when your sleep quality drops. This is the future of Digital Twin technology applied to personal wellness. 🚀 The Architecture: A Feedback Loop for Your Body Unlike a simple linear script, a health agent needs to maintain state and make conditional decisions. If your recovery is 90+, push hard; if it's below 50, swap that CrossFit session for Yoga. Here is how the data flows through our LangGraph state machine: graph TD A[Start: Morning Trigger] --> B{Fetch Oura Data} B --> C[Analyze Recovery Score] C --> D{Is Score < 60?} D -- Yes --> E[Reschedule Google Calendar to 'Rest/Yoga'] D -- No --> F[Confirm High-Intensity Workout] E --> G[Check Nutrient Deficiencies] F --> H[End Loop] G --> I{Low Magnesium/Sleep?} I -- Yes --> J[Draft Instacart Order] I -- No --> H J --> H Prerequisites To follow this advanced guide, you'll need: LangGraph & LangChain : For orchestration. Oura Cloud API : Access to your readiness/sleep data. Google Calendar API : To modify your schedule. Python 3.10+ Step 1: Defining the Agentic State In LangGraph, everything revolves around the State . We need to track our physiological metrics and our current calendar status. from typing import TypedDict , List , Annotated from langgraph.graph import StateGraph , END class HealthState ( TypedDict ): recovery_score : int sleep_quality : str current_schedule

Claude Code is amazing. It feels like cheating in the best way until it doesn't. The first time I dropped a vague comment and it gave me a fully working component, I was hooked. But somewhere along the way, I realized something weird was happening: I was coding less... and thinking less. Let’s talk about that. The Tools Got Better, But We Got Lazier In 2022, you still had to know stuff . Now? You just prompt your way through half the job. And sure, productivity skyrocketed. Ship faster, write fewer tests, scaffold full APIs in a couple of minutes. But here’s the tradeoff no one likes to talk about: we’re outsourcing thinking . Not tasks. Not code. Thinking. I've seen teammates slowly stop debugging on their own. They just throw the stack trace into Claude. I’ve seen people stop learning the fundamentals of async behavior or memory handling because “Claude will tell me.” Cool. Until Claude gives you a bad answer and you don’t realize it because you stopped building your intuition. The Industry’s Not the Same The layoffs? Brutal. And the worst part isn’t just losing a job it’s realizing you didn’t build portable skills . Because if Claude wrote all your code at your last job, what exactly did you learn? When companies trimmed down, they didn’t just cut bodies. They cut noise. They kept the people who could solve problems , not the ones who could Google fast or write perfect prompts. So ask yourself: if Claude disappeared tomorrow, could you still contribute? Using AI ≠ Being a Great Engineer I’m not anti-AI. I use Claude. I use Cursor. I use Copilot. But I don’t depend on them. There’s a difference between boosting productivity and becoming a prompt monkey. The first one makes you valuable. The second one… doesn’t. Let’s get real: code was never the point. It was always about solving problems, navigating ambiguity, simplifying complex systems, designing clean architecture. None of that goes away with AI. In fact, it becomes more important. The Skills You’ll Regret Not

Every extraction pipeline I have ever pointed at a language model shares the same dirty secret: the JSON comes back almost valid. Almost is where the bugs live, because almost passes your eyes and then fails your schema at midnight. So I built a loop where the model grades its own homework, then let it run for 48 hours on a free server to see what breaks. The experiment The idea was simple: take plain-text payloads that look like webhook bodies, extract five fields against a small schema, and give the model exactly one chance to fix its own mistakes. I wrote the rules down before writing any code, because rules written after a failure are just excuses. Pass one asks the model to return the fields as JSON. A validator checks the result against the schema. If validation fails, pass two sends the original payload, the bad JSON, and the exact validation errors back to the model. Every attempt, raw text included, lands in a JSONL log. I ran that loop for 48 hours on MonkeyCode's free server option, using its free model access for both passes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Here is the loop, trimmed to the parts that mattered. import hashlib import json import time from datetime import datetime , timezone import jsonschema import requests SCHEMA = { " type " : " object " , " required " : [ " event " , " customer_id " , " amount " , " currency " ], " properties " : { " event " : { " type " : " string " , " enum " : [ " charge.succeeded " , " charge.failed " ]}, " customer_id " : { " type " : " string " , " pattern " : " ^cus_ " }, " amount " : { " type " : " integer " , " minimum " : 0 }, " currency " : { " type " : " string " , " minLength " : 3 , " maxLength " : 3 }, }, } SEEN : set [ str ] = set () def now_iso () -> str : return datetime . now ( timezone . utc ). isoformat () def call_model ( prompt : str ) -> str : # Point this at the free model endpoint you are testing. resp = requests . post ( " https://your-endpoint.e

GitHub热门项目 | A unified inference and post-training framework for accelerated video generation. | Stars: 4,070 | 111 stars this week | 语言: Python

GitHub热门项目 | The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/products/free-netbox-cloud/ | Stars: 21,389 | 35 stars today | 语言: Python

GitHub热门项目 | Криптографический слой, работающий поверх существующих мессенджеров | Stars: 250 | 32 stars today | 语言: Python

Your monthly Claude Code bill went up 20%. You know that much. What you don't know is which Skill did it — and nothing in the tooling will tell you. Run /usage in Claude Code and you get claude-sonnet-4-6: ¥3,240 — a per-model total and nothing else . "More expensive than last week" is visible. "Which Skill caused it" is not. usage-breakdown.sh closes that gap. It's a 106-line shell script that parses transcript.jsonl with Python and tallies call counts per Skill, Agent, and MCP server using Counter . This article walks through how the script works and how to run it, with the actual code and actual numbers. Why This Approach Works What Claude Code Is Actually Recording Claude Code streams every operation during a session into .jsonl files under ~/.claude/projects/ . It's JSONL — one event per line, one file per session. The files sit under a <project-id>/ directory. The skeleton of a single record looks like this: { "message" : { "role" : "assistant" , "content" : [ { "type" : "tool_use" , "name" : "Skill" , "input" : { "skill" : "pre-completion-self-audit" } } ] } } Inside message.content[] sit "type": "tool_use" blocks. The name field is the name of the tool that was invoked. The Bash tool, the Edit tool, the Skill tool, the Agent tool, MCP calls — all of it is recorded in this same format. Once I noticed that, the thought was: run this through a Counter and everything becomes visible. For the Skill tool, the skill name lives in input.skill ; for the Agent tool it's input.subagent_type ; and for MCP servers, the tool-name convention mcp__<server>__<tool> lets you extract the server name by splitting on __ . The structure is consistent, so the parser comes out surprisingly simple. What /usage Doesn't Tell You What Claude Code's /usage command outputs is a per-model cost total for a period. Model Cost claude-sonnet-4-6 ¥3,240 claude-opus-4-8 ¥ 892 Useful as far as it goes, but the breakdown of that cost is invisible . You can't see which session, which Skill, how ma

When you're building an AI evaluation platform with multiple microservices, the "core" services get all the attention — the evaluation engine, the scoring system, the RAG pipeline. But a platform doesn't work without the connective tissue: the workflow orchestration that keeps humans in the loop, the taxonomy engine that classifies tasks intelligently, the platform service that ties authentication together, and the evaluation suites that ensure models actually remember context. These four services don't make headlines, but they're what turned a collection of microservices into an actual platform. Here's what went into each one and why the engineering decisions mattered. Workflow Orchestration: The Human-in-the-Loop Engine AI evaluation is not fully automated — and it shouldn't be. Certain decisions require human judgment: Is this model response harmful? Does this evaluation rubric make sense for this domain? Is this edge case a genuine failure or acceptable behavior? The workflow orchestrator manages these decision points. It coordinates multi-step evaluation workflows where some steps are automated (LLM scoring, data validation) and others require human approval before the pipeline continues. The Architecture The core is a state machine built on FastAPI and PostgreSQL. Each workflow is a DAG (directed acyclic graph) of tasks, where each node can be: Automated: Runs immediately, calls another service (scoring, data enrichment), stores the result Human gate: Pauses the workflow, notifies the assigned reviewer via the notification service, waits for approval/rejection Conditional: Routes to different branches based on previous step outcomes (e.g., if confidence score < threshold, escalate to senior reviewer) State transitions are persisted in PostgreSQL with Alembic-managed migrations. Every transition is logged — who approved what, when, and with what context. This audit trail turned out to be critical for client reporting. Real-Time Updates with WebSocket The origin

FP8 gave us a clean 1.5x on Qwen3-8B serving throughput on an RTX PRO 6000 Blackwell (1,725 to 2,597 tok/s at concurrency 32, vLLM). The uncomfortable question is always the same: did the model get dumber. This post is the exact check we ran before recommending the switch, with numbers, so you can run the same one. Why "run an eval suite" is usually the wrong first answer Standard benchmarks (MMLU and friends) are noisy instruments for quantization deltas at 8B scale. Score movement inside the error bars tells you nothing about whether YOUR prompts changed behavior. What you actually want to know is narrower: on the workload you serve, does the FP8 checkpoint produce materially different outputs than BF16, and are any of the differences wrong. That is answerable directly, cheaply, and per prompt. The method Both configurations run the same fixed workload: 20 prompts covering reasoning, code, summarization, translation, extraction, classification, math, and instruction following. Greedy decoding, temperature 0, 256-token cap, streamed. Greedy matters: it removes sampling noise, so any output difference is attributable to the numerics. Then a three-stage comparison: Byte equality. outputs_bf16[i] == outputs_fp8[i] . Anything identical is settled. Similarity triage. For non-identical pairs, difflib.SequenceMatcher.ratio() sorts near-identical wording drift from real divergence. Side-by-side review under a written rubric. Every non-identical pair gets read. The rubric asks one question: is there a factual or numerical claim that one precision gets right and the other gets wrong. Wording changes, reordering, and equally-defensible readings are recorded but not counted as regressions. The core loop is small: import difflib , json bf16 = json . load ( open ( " vllm_bf16_conc1.texts.json " )) fp8 = json . load ( open ( " vllm_fp8_conc1.texts.json " )) for i , ( a , b ) in enumerate ( zip ( bf16 , fp8 )): if a == b : print ( i , " identical " ) continue r = difflib . Sequenc

GitHub热门项目 | Open-source framework for the research and development of foundation models. | Stars: 1,923 | 214 stars today | 语言: Python

I often remember the shot I want before I remember its filename. That gap is what binquery is for. It is a local Python CLI that indexes video clips and turns a sentence into a ranked shortlist for a human to review. It deliberately stops before editing: no timeline generation, no automatic cut, and no render. The smallest reproducible trial You can test the complete installed command path without supplying footage: python3 -m venv .venv .venv/bin/pip install binquery .venv/bin/binquery demo --out /tmp/binquery-demo The demo generates a synthetic 30-second video locally, then exercises splitting, indexing, validation, and querying. The first run may download OpenCLIP model weights. This is an end-to-end pipeline smoke test, not evidence of semantic search quality on real footage. Why keep the architecture small? The current design uses: ffmpeg to sample three frames from each clip OpenCLIP ViT-B-32 to build the local visual index plain JSON and NumPy files for metadata and vectors a JSON result containing clip paths, scores, and ranking signals There is no database, vector service, or daemon to operate. Querying an existing index does not resample the footage or rebuild the full index. The trade-off is straightforward: three frames keep indexing understandable and bounded, but they can miss important content in long or visually varied clips. I would rather expose that limitation than market a synthetic demo as a quality benchmark. Ranking signals are not explanations The output includes fields such as score , gate , and reasons . Here, reasons means ranking signals recorded by the pipeline. It should not be interpreted as a reliable semantic explanation of why a clip is correct. That distinction matters because a plausible-looking explanation can create more confidence than the underlying retrieval quality deserves. The shortlist is meant to reduce what a person must inspect, not replace editorial judgment. What binquery does not do It does not build a timeline or e

Why a Local RAG Chatbot for Trading Research Most "AI trading assistant" products are black boxes: your notes, strategy docs, and market notes get shipped to a third-party API, billed per token, and stored who-knows-where. For a retail NIFTY trader or a quant researcher, that is the worst of all worlds — you pay continuously, you leak your edge, and you cannot audit what the model actually read. This guide shows how to build a Retrieval-Augmented Generation (RAG) chatbot that runs 100% locally on an Android phone using Termux + Ollama. It ingests your own research (PDFs, markdown notes, option-chain exports) and answers questions grounded only in that data. No OpenAI key. No Anthropic key. No monthly bill. No data leaving the device. OBSERVED: Running ollama run llama3.2 on a mid-range phone inside Termux is slow but usable for document Q&A (3–8 tokens/sec). On a laptop it is smooth. SOURCE: Local testing on Termux 0.118, Ollama 0.3.x, Android 14. DERIVED: For production research volumes, run Ollama on a spare x64 machine and point Termux at it over LAN. What You Will Build A four-part pipeline: Ingest — load your research docs (markdown, PDF, CSV) into chunks. Embed — turn chunks into vectors with a local embedding model. Store — keep vectors in a local file-based index (no server needed). Answer — retrieve top-k chunks and ask a local LLM to answer strictly from them. The whole thing is ~200 lines of Python. No paid APIs. Prerequisites Android phone with Termux installed (F-Droid version, not Play Store). ~2 GB free storage. Basic Python comfort. pkg update && pkg upgrade -y pkg install python clang ffmpeg -y pip install ollama numpy Install Ollama inside Termux: curl -fsSL https://ollama.com/install.sh | sh NOTE: The official install script targets Linux. On Termux you often need the community build. If the script fails, install the ollama package via a Termux-compatible binary or run Ollama on a LAN machine and use ollama serve remotely. Pull a small model and a

Free AI Tiers Bill You in Hours, Not Dollars Free model access looks like a bargain until you track the hours you spend feeding context back into a model with no memory. A zero-cost invoice hides the most expensive resource in your workflow: your own attention. My position is straightforward: treat a free tier like a metered service and measure the hidden costs before you adopt it. The token counter tells you almost nothing about the real price. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm using MonkeyCode's free model access and free server option as a concrete example; the measurement approach applies to any free tier. The dashboard shows tokens, not time Every free plan advertises a generous token allowance and a server that wakes up on demand. What the marketing page omits is the labor you spend reassembling context, waiting for cold starts, and double-checking output. Those costs do not appear on any invoice, but they consume your day in chunks. Four of them matter more than the token meter. Context reconstruction — Every new conversation starts from zero, so you re-explain your stack, your file layout, and your constraints. Those re-pasted tokens count against the same allowance you were trying to save. Cold-start waiting — A free server that sleeps after idle adds seconds to every call. Multiply that by a scheduled job that fires hourly and you have lost real time. Human verification — Confident output still needs a human to check it, and that check is the most expensive line item in the whole system. Attention fragmentation — A free allowance looks huge until you split it across codegen, debugging, and review. Small tasks nibble the budget faster than big ones. A ten-minute audit script The script below turns the argument into a reproducible measurement. It sends three representative prompts to any OpenAI-compatible endpoint, records wall-clock latency, and extracts token usage from the response. Run it several times du

Here's the conclusion up front: a free LLM endpoint is a moving target. You can't see the changes, but they're happening — model updates, quantization tweaks, server-side prompt rewrites. And your app will feel them, usually as a slow, invisible quality dip. I've spent weeks on this account probing free LLM servers, caching tokens, and building evaluation harnesses. The pattern I keep seeing: teams pick a free tier, wire it in, and then never look at it again. They treat it like a static API. It isn't. The fix is a nightly drift check. A small script that runs your most important prompts against the endpoint, compares the outputs to a baseline, and tells you when something changed. Not a benchmark. Not a one-time eval. A recurring alarm. This post walks through a 90-line harness you can run tonight. I'll use MonkeyCode's free server as the reference endpoint — it's an open-source project with free model access, a free server option, and, as advertised at the time of writing, a 10M token grant. The exact numbers may move, so check the repo's README before you depend on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Why drift is the silent killer of free-tier apps Let's be honest: free endpoints don't come with changelogs. The provider can swap the underlying model, adjust the temperature default, or add a safety filter without telling you. Your tests still pass. Your error rate stays flat. But the responses get a little shorter, a little more evasive, a little less useful. Users notice before you do. They don't file bugs for 'the bot got dumber.' They just stop using it. A drift check turns 'the bot got dumber' into a concrete signal: 'the pass rate on 12 core prompts dropped from 92% to 74% overnight.' That's something you can act on. Step 1: Define your core prompts Don't test everything. Pick 10-20 prompts that represent the actual workload your app handles. For each prompt, define what 'good' looks like. Prompt Expected beha

Polymarket Paper Trading Bot: Build One in Python A real-money trading bot is the wrong place to discover that your signal logic, order-book handling, or position accounting is broken. A Polymarket paper trading bot gives you a safer engineering environment: consume real market data, generate real signals, simulate orders and fills, and measure hypothetical performance before connecting execution credentials. The important distinction is that paper trading should simulate the execution layer , not fabricate market data. Polymarket currently exposes public market data without authentication, while its public WebSocket market channel provides real-time order-book and price updates. This article builds that architecture in Python. What You'll Learn How a paper-trading architecture differs from a live bot How to discover markets through the public API How to consume CLOB order-book data How to simulate limit-order fills How to track positions and P&L How to test arbitrage, market-making, and directional strategies How to graduate from paper trading to production safely About the Author Soulcrancerdev Contact: X: @soulcrancerdev Telegram: soulcrancerdev YouTube: YouTube channel The Architecture A useful design separates data, strategy, simulation, and accounting : flowchart LR A[Gamma Market Discovery] --> B[Market Metadata] C[CLOB REST / WebSocket] --> D[Market Data Engine] B --> D D --> E[Strategy Engine] E --> F[Paper Execution Engine] F --> G[Virtual Portfolio] G --> H[P&L / Risk Metrics] D --> I[Logger / Metrics] The key design decision is that PaperExecutionEngine should implement the same interface your live execution engine eventually uses. That means the strategy does not know whether an order is simulated or real. 1. Discover Markets Polymarket's Gamma API provides public market discovery. The current documentation exposes keyset pagination through: https://gamma-api.polymarket.com/markets/keyset Markets include fields such as conditionId , clobTokenIds , outco

First post in a build-in-public series about verdict , an MCP server that gives coding agents structured, sandboxed test feedback. The problem Watch a coding agent work and you'll see it run pytest in your shell, unsandboxed, and then push 40,000 tokens of raw output through its context window to answer one question: did my change break anything? That's three problems in one command: Token waste. The agent needs ~10 lines of signal and pays for a wall of dots, warnings, and tracebacks. No sandbox. The tests run on your machine, in your environment, with your files writable. No memory. When a test fails, the agent can't tell whether it broke it or whether it was broken before it arrived - so it either "fixes" pre-existing failures nobody asked about, or ships regressions it assumes were already there. verdict is an MCP server that replaces the pytest shell-out with four tools: tool what it returns verify(scope?) impact-selected tests, run in an ephemeral container, as a ~400-token typed verdict explain_failure(check_id) the full traceback - only on demand history(fingerprint) first seen / last seen / times seen for a failure run_checks(["ruff","mypy"]) lint & type checks, same verdict shape ▶️ Watch the 30-second demo - Claude Code fixing a bug with verdict verifying in a container. The three ideas 1. Verdicts, not output. verify returns typed JSON: counts, per-failure message + location, and nothing else. Full tracebacks live behind explain_failure . The whole verdict for a real failing run is ~400 tokens - the raw pytest output it replaces was ~40k. The design rule in the repo is blunt: nothing bulky rides in the summary, ever. 2. Fingerprints give failures identity. Every failure is hashed from its normalized signature - volatile tokens (addresses, tmp paths, ids, durations) collapsed first. Same logical failure ⇒ same fingerprint, across runs and refactors. Fingerprints are what make the third idea possible: 3. History answers "was it me?" verdict keeps a small S

Originally published at ai.bedvibe.studio . I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it. Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS. So I wrote the QA framework I wished I had, packaged it as ttsproof , and then ran it as a blind study against a production TTS service so the results would be more than an opinion. The two failures WER cannot see A TTS pipeline breaks in two different ways, and a single WER number blurs both. Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform. Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names. ttsproof splits them apart and handles each one honestly: Structural checks, no model needed — empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else. Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalised to spoken form before scoring, so 3:30 PM against "three thirty" stops counting as an error. ASR-uncertainty quarantine — when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS. The study: 390 samples, and a blind human check I evaluated the method

Originally published at ai.bedvibe.studio . There is an assumption almost every practitioner carries without examining it: if your dataset has bad samples in it, the loss will tell you. Corrupted rows spike. Broken files stick out. Sort by per-sample loss, look at the top of the list, and there is your garbage. I believed it too. Two separate failures in my own work say it is wrong, and they fail in the same direction — quietly. The reproducible one: a dataset that cannot be learned While validating trainproof I ran a controlled fault-injection study: one base setup, a Qwen2.5-3B QLoRA, run six ways, three seeds each, eighteen runs total. Every log ships in the repo so the verdicts can be checked rather than believed. One configuration shuffled the dataset's labels into pure noise. The labels no longer corresponded to the inputs at all. This is not a hard dataset or a noisy dataset. It is a dataset that cannot be learned , because there is no relationship left in it to learn. That run reduced its loss by 62%. On its own curve it was textbook-healthy — a clean downward slope, no spike, no plateau, nothing a human or a rule would flag. It was learning nothing useful. It was memorising the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve it is indistinguishable from a real one. That is where the assumption broke for me. Not "loss is a weak signal for this." Loss is not a signal for this at all, in isolation. The production one, and what I can and cannot prove about it The second failure came from real work rather than an experiment, and it is the one I think about more. Building a text-to-speech corpus of roughly 110,000 recordings, a small number of the files were pure loud white noise. Not corrupted in the file-format sense — they opened fine, played fine, had valid headers and valid duration. They simply contained no speech. Just noise, at volume. They did not surface as high-loss outliers. Being precise about