标签：#learn

Tokenization → embeddings → causal Transformer → LM head → softmax → loss → backpropagation. No TensorFlow, no PyTorch, and no hidden autograd. Repository: tiny-language-model-neuro-js . Most explanations of language models present correct formulas but hide the path between them inside a framework. I wanted the opposite: one small scenario where every scalar is visible and where the terminal clearly shows incorrect answers before learning and correct answers after it. The project now has one command: node src/train.js --generalize --adaptive-teach It requires Node.js 18.19+ and has no dependencies. The result first The model is queried immediately after random initialization: BEFORE TRAINING — random, usually wrong answers > can human read ? model: ? <unk> ... expected: human can read. [WRONG] > can fish swim ? model: ? <unk> ... expected: fish can swim. [WRONG] > can cat read ? model: ? <unk> ... expected: cat cannot read. [WRONG] After pre-training, SFT, and adaptive SFT, the same model produces: FINAL ANSWERS AFTER ADAPTIVE SFT > can human read ? model: human can read. [CORRECT] > can fish swim ? model: fish can swim. [CORRECT] > can bird fly ? model: bird can fly. [CORRECT] > can cat read ? model: cat cannot read. [CORRECT] Rehearsal controls preserved: 14/14. Stable criterion reached 11 times in a row. The initial text varies because initialization is random. The final acceptance criterion does not: all answers must be correct, every target token must have at least 95% probability, and the complete check must pass more than ten times consecutively. What remains after removing the extra modes The code previously contained several debug and training modes. They were useful while experimenting but obscured the main idea. The final version keeps one educational pipeline: text → word tokenization → token IDs → token + position embeddings → two causal Transformer blocks → multi-head self-attention → two-hidden-layer FFN → LM head → softmax → next-token probabilities

Before you can truly understand how AI systems think, learn, and generate responses, you need to understand the math that powers them. This guide covers the essential mathematical concepts that form the backbone of modern Artificial Intelligence and Large Language Models (LLMs). Why does this matter? Every aspect of AI — from how text is encoded, to how a model predicts the next word, to how it improves itself during training — is driven by mathematics. Skipping this foundation means you will only ever use AI as a black box, without understanding why it works. 🔄 How an LLM Actually Works — The Complete Pipeline Before diving into each math concept individually, here's the big picture of how text flows through a Large Language Model from input to output. Every section in this guide maps to a step in this pipeline: ┌─────────────────────┐ │ Your Prompt │ "What is gravity?" └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ Tokenizer │ Splits text into chunks (BPE algorithm) └──────────┬──────────┘ → Section 1: Number Systems & Encoding ↓ ┌─────────────────────┐ │ Token IDs │ Each token → a number (e.g., "gravity" → 17942) └──────────┬──────────┘ → Section 1: Number Systems & Encoding ↓ ┌─────────────────────┐ │ Embedding Model │ Each token ID → a dense vector of numbers └──────────┬──────────┘ → Section 3: Vectors & Embeddings ↓ ┌─────────────────────┐ │ Vectors │ [0.12, -0.87, 0.45, ...] per token │ + Positional Info │ → Section 3 & 6: Embeddings & Linear Algebra └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ Transformer │ Multi-Head Attention + Feed-Forward layers │ (×N layers) │ repeated 32-96+ times └──────────┬──────────┘ → Section 4, 6: Algebra & Linear Algebra ↓ ┌─────────────────────┐ │ Probability │ Softmax converts final output to │ Distribution │ probabilities over entire vocabulary └──────────┬──────────┘ → Section 2 & 6: Probability & Softmax ↓ ┌─────────────────────┐ │ Next Token │ Sampling picks one token │ (Sampling) │ (using Temperature, Top-K,

Anthropic dominated the day’s product cycle with the surprise launch of Claude Opus 5, a model that effectively obsoletes the company's own flagship architecture at half the cost and immediately topped third-party leaderboards [1] [3] [95] . Meanwhile, a massive geopolitical rift formalized as Microsoft, Meta, and Nvidia launched a coordinated lobbying effort to protect global open-weight pipelines [41] [93] , just as the Chinese model Kimi K3 demonstrated an alarming autonomous zero-day network exploit confirmed by international safety institutes [96] [104] . Claude Opus 5 disrupts frontier model pricing tiers Anthropic launched Claude Opus 5 at the same $5/$25 per million token price as Opus 4.8 , positioning it as a hyper-efficient model that functionally matches or beats the flagship Fable 5 on third-party coding evaluations like CursorBench [1] [3] . Visual reasoning capabilities mark a massive step-change , with the model successfully writing its own computer-vision pipeline to extract part geometries from raw pixels on the Frontier-Bench, while also perfectly scoring 42/42 on the IMO 2026 [54] [57] . Aggressive safety guardrails are simultaneously alienating power users , who report that while Opus 5's systemic Auto Mode bounds prompt injection success rates to near-zero, the model executes opaque "silent downgrades" to weaker architectures when it detects sensitive contexts rather than issuing standard refusals [33] [91] [95] . // Detect dark theme var iframe = document.getElementById('tweet-2080700479940759919-684'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2080700479940759919&theme=dark" } The takeaway: Anthropic is successfully driving down the localized cost of intelligence, but its blistering capability gains are artificially breaking its own pricing tiers and irritating developers with heavy-handed safety routing. Hardware and cloud alliance pushes back on open-weight bans Micr

A few weeks ago I wrote about Siagnos , a personal taste engine for fanfiction that learns from reading behavior instead of matching tags. I was three stages in: scraper done, schema designed, embeddings working as a proof of concept. Then I got a two-week internship window to build something deployable, and I made a call. Instead of pushing Siagnos forward stage by stage, I built Opsis : a scoped-down, content-based recommender that answers one specific question. Given a fic, what else in a real, collected corpus is closest to it in content? Opsis doesn't do taste modeling. It doesn't touch my reading behavior at all. It's the layer underneath that, and it's live right now. Why not just keep building Siagnos directly Two weeks isn't enough time to get a reading tracker, a feature pipeline, and a trained preference model all working end to end. It is enough time to take the scraper and schema I already had and turn them into something real: a working recommender, deployed, with a UI, that someone else can actually use today. So I scoped down on purpose. No personal taste model yet. No behavior tracking yet. Just: can I take one fic and find genuinely similar ones, from AO3 metadata alone, using content instead of tags? What Opsis actually does Scrapes AO3 metadata under conditions the OTW Communications Committee confirmed were acceptable before I collected anything: one persistent session, randomized delays, capped retries Cleans and validates the raw data, log-and-skip instead of all-or-nothing, so one malformed row doesn't take down a 7,000-fic load Normalizes everything into PostgreSQL: fics, six lookup tables, six join tables, idempotent upserts so re-running the loader is always safe Embeds every fic's summary with sentence-transformers/all-MiniLM-L6-v2 Ranks candidates with a blended score: 0.70 embedding cosine similarity, 0.15 fandom overlap, 0.10 relationship overlap, 0.05 popularity If you submit a fic that isn't in the database yet, Opsis scrapes it, cle

LSTMs are still the go-to architecture for a lot of time series work, but they're annoying to trust. You get a number out the other end and no real sense of why the model landed there. This tutorial walks through training an LSTM on daily temperature data, then pulling it apart with three explainability methods: permutation importance, SHAP, and Integrated Gradients. Who this is for: people who already know some Keras and want to add interpretability to a forecasting model, not a from-scratch intro to neural nets. 1. Getting the data into shape LSTMs want a 3D tensor — (samples, timesteps, features) — so before anything else we need to turn a flat column of temperatures into overlapping 7-day windows, each one paired with the value on day 8. import numpy as np import pandas as pd from sklearn.preprocessing import MinMaxScaler # 1. Load data df = pd . read_csv ( " weather_data.csv " ) data = df [ ' Temperature ' ]. values . reshape ( - 1 , 1 ) # 2. Scale the data for stable neural network training scaler = MinMaxScaler ( feature_range = ( 0 , 1 )) scaled_data = scaler . fit_transform ( data ) # 3. Create sequences: 7 days of lag to predict the 8th day X , y = [], [] for i in range ( 7 , len ( scaled_data )): X . append ( scaled_data [ i - 7 : i ]) y . append ( scaled_data [ i ]) X , y = np . array ( X ), np . array ( y ) print ( f " Input shape: { X . shape } " ) # Output: (Samples, 7, 1) Scaling matters more than it sounds like it should — LSTMs trained on unscaled temperature values are prone to exploding gradients, and training just falls apart. The windowing step is really the whole trick here: every prediction only ever sees the past seven days, nothing more. 2. Building the model Two stacked LSTM layers, dropout after each one, early stopping so we don't have to babysit the epoch count. from tensorflow.keras.models import Sequential from tensorflow.keras.layers import LSTM , Dense , Dropout , Input from tensorflow.keras.callbacks import EarlyStopping # 1. Build

Hello! This is my first time submitting an actual conference paper (only done workshops so far). Got a 3/3/5/7 for the Position Paper Track. Reviews all seem quite addressable. Meta review also seemed kinda positive? Included wording such as "a revision should include..." followed by actionable stuff we can take. Feels like there may be a shot. My question is... what does that mean? We submit rebuttals for each reviewer. And I agree with a lot of the feedback. So thats not an issue. But what's going to happen? Do reviewers change their scores? Does the AC read each rebuttal to see if we'll make an adequate revision? How does all of this get judged? Who am I trying to convince here? And of what? And what should the wording be like in the rebuttal? More informal? Sorry if some of these questions seem redundant! submitted by /u/Empty-Avocado5927 [link] [留言]

About to be over 36 hours now? Nothing on the website, twitter, anywhere. What the hell? Is anyone else facing the same issue what do I do? submitted by /u/Specialist-Manager67 [link] [留言]

Every "run this model locally" guide tells you to grab a Q4 GGUF and move on. That advice is fine right up until you try a long-context run and your machine starts swapping. The weights are the part everyone budgets for Quantization maths is straightforward. A model's weight footprint is roughly params x bits / 8 : Quant Bits/param 12B model Quality note Q8_0 ~8.5 ~12.8 GB Near-lossless, rarely worth it Q6_K ~6.6 ~9.9 GB Very close to Q8 Q4_K_M ~4.8 ~7.2 GB The usual sweet spot Q3_K_M ~3.9 ~5.9 GB Noticeable degradation Below Q4 the loss stops being subtle. Instruction-following degrades before raw perplexity does, which is why benchmark numbers can look fine while the model quietly stops respecting your system prompt. The KV cache is the part that bites Here is what the guides skip. The KV cache scales with context length , and it is not quantized by default: kv_bytes ~= 2 (K and V) x layers x kv_heads x head_dim x seq_len x dtype_bytes The practical consequence: a model that loads in 7 GB can need well over twice that at long context. Grouped-query attention helps a lot — kv_heads is much smaller than attention heads — but the term still grows linearly with sequence length while your weights stay fixed. Two knobs matter more than picking a fancier quant: --ctx-size : do not allocate 128K if your prompts are 8K. You are reserving memory you will never touch. KV cache quantization ( q8_0 for K/V): roughly halves cache memory for a quality hit most workloads never notice. Underused. A decision order that works Start at Q4_K_M Set context to what you actually use, not the model maximum If you are still tight, quantize the KV cache before dropping to Q3 Only move up to Q6/Q8 if you have headroom left over That ordering matters: dropping to Q3 to buy context is the most common mistake, and it trades a permanent quality loss for memory you could have gotten from the cache instead. Per-quantization benchmarks and deployment notes for the Gemma 4 family are collected at ge

I've been chasing the question of what algorithms a transformer can actually express -- separate from what it can learn. So I built a compiler: define a computation graph in ordinary Python, and it produces the weights of a transformer that executes the graph. The result is a standard Phi-3-architecture checkpoint that vanilla huggingface loads with no custom code and no trust_remote_code. Zero training in the pipeline. Write-up (origin + how the constructions work): https://ood.dev/posts/torchwright-intro/ Repo (twelve runnable examples): https://github.com/physicsrob/torchwright Hand-built transformer weights aren't a new idea. RASP defines a language whose primitives map onto transformer sublayers, and Tracr compiles RASP programs into actual weights. I wanted two things they don't aim for: expressing a computation graph in ordinary Python, and targeting a stock architecture, so the output loads in vanilla huggingface with no custom code. submitted by /u/notforrob [link] [留言]

A few weeks ago, I set out to build a small portfolio project: a Streamlit app that could take any tabular dataset, understand something about its structure, and give honest guidance on how to model it. I called it ContextLens . I expected it to be a practical exercise in Python, machine learning, and deployment. What I didn't expect was how closely it would connect with the same questions I work with every day in my PhD research on context-aware intelligent systems. The problem I started with Most introductory machine-learning tutorials follow a familiar sequence: Load a CSV. Choose a model. Train it. Check the accuracy. What often gets skipped is the layer of judgment that should come before any of that: Is this actually a classification problem or a regression problem? Is the target so imbalanced that accuracy becomes misleading? Is that "ID" column secretly leaking the answer into your model? Are there duplicate rows, missing values, high-cardinality categories, or too many features for the number of available observations? Experienced practitioners make these judgments almost automatically. But that reasoning usually remains invisible—it sits in someone's head rather than inside the system, where another person can inspect it. ContextLens is my attempt to make that layer visible. Upload a dataset, and it profiles the data, flags structural risks—missingness, duplicate rows, likely identifier columns, class imbalance, and high-dimensional settings—and adapts its evaluation guidance to what it finds before training a single model. The point is not simply to train a model. The point is to ask whether the modelling process makes sense in the first place. Why I call it "context-aware" rather than "AI-powered" I was deliberate about this distinction, just as I have been throughout my PhD work, and it turned out to be the most important design decision in the whole project. ContextLens does not claim to be intelligent in the way a human expert is. It does not hide its

I wanted to add a background-removal tool to my site's image cluster that stayed true to the 100% client-side processing principle I already use for PDFs and image conversions. The path there was anything but linear: a library dropped over a licensing problem, a carefully chosen model that turned out more limited than expected, and a bug that froze the entire page — not just the tool — during computation. Here's the full build, including the parts that didn't work the first time. The starting problem: what's actually feasible for free? The initial idea was broad: remove backgrounds, and maybe unwanted objects too. The two tasks have very different difficulty levels. Removing objects requires inpainting — plausibly reconstructing the erased area — which in practice still means heavy generative models, impractical to run client-side with good quality on an average device. Removing a background , on the other hand, is a segmentation problem: separating a subject from its surroundings. That has much lighter models available, runnable entirely via WebAssembly with no server involved at all. So: background removal only, object removal shelved for later. The AGPL trap The first library that looked like a perfect fit turned out to be distributed under AGPL , a strong copyleft license. Free to use — but with a real catch for anyone embedding it in a public, closed-source web service: AGPL can require releasing the full source of the project that embeds it, under the same license. "Free for the end user" and "safe to drop into a closed-source commercial product" are two different questions, and it's worth answering the second one before writing integration code, not after deploying it. Before wiring any "free" AI library into a commercial project, check the exact license, not just the price tag. AGPL, GPL, and other strong copyleft licenses are fine for personal or internal tools, risky for a public closed-source product. The fix: switch to Transformers.js — Hugging Face's li

Built an open-source AI coding agent that was 7%–75% cheaper than a cold "claude -p" run on 6/6 well-localized tasks across repositories up to ~82k LOC. The biggest difference: Cold agent: $6.83, 207 turns AutoDev Studio: ~$1.70 for the same bug The full benchmark (including cases where it loses) is in the README. So what's different? Most AI coding agents re-explore a repository from scratch on every task just to figure out where the change belongs. AutoDev Studio pays that localization cost once. It ingests a repository and builds a persistent knowledge base using static analysis and a local embedding index. Every future task reuses that knowledge, turning localization into a lookup instead of another cold search. What it does: PM agent asks clarifying questions and drafts tickets Dev agent writes code on an isolated branch QA runs tests A different model family reviews the diff (author ≠ reviewer) If needed, it goes through a bounded revise loop Opens a real GitHub PR It also includes a live Kanban board and tracks token usage and cost per ticket/agent. Where it doesn't win: Tiny, easy-to-find edits can be cheaper with a single-shot agent because of the pipeline overhead. On one complex cross-cutting bug, it produced a cheaper but narrower fix than the baseline. Other features: Provider agnostic (Anthropic, Claude Code, OpenAI-compatible APIs, Groq, Gemini, xAI, OpenRouter, Ollama, etc.) Runs completely free/offline by default using Groq's free tier + local embeddings FastAPI + SQLite Hand-rolled UI Tests + CI MIT licensed Repo (screenshots + full benchmark): https://github.com/krishagarwal314/autodev-studio I'd love any feedback, criticism, or contributions. Happy to answer questions about the architecture or benchmarking. submitted by /u/NeighborhoodOwn8510 [link] [留言]

Hello, I'm Rijul. I'm building git-lrc, a micro AI code reviewer that runs on every commit. It's free...

Its been almost 24 hours since reviews were released and I dont see the meta review still. Some people on reddit are saying they can see it. NeurIPS website says they are-releasing reviews on 23 but even 23 July is ending in 4 hours. Whats going on bruh, none of my coauthors is an AC or didnt complete his review so its not like its being held from us submitted by /u/Specialist-Manager67 [link] [留言]

Hi, I'm new with ACM conferences. I have 2 papers at workshops and the conference website says: "Each workshop paper needs to be associated with one workshop-only (non-student) or full (non-student) author registration at either ACM Member rate, or non-member rate. One workshop-only or full registration can cover only one accepted workshop paper." Does that mean that I have to register twice with "Workshop-only Author registration" paying 500USD per paper!? Second question, I really do not understand the APC fees listed here: ACM Multimedia 2026 Conference — Author Instructions .. does that means that in addition to the registrations I have to pay 350USD per paper? submitted by /u/rokk07 [link] [留言]

This is going to be a very different article from what I usually write. No technical discussions, architecture deep dives, or engineering practices today. Instead, we're talking about something much older than software itself: music. We treat language like it's the default mode of human communication, like it's the real and only thing used to communicate, everything else is secondary, emotional, aesthetic, nice to have. But language is actually the outlier. It's the new protocol layered on top of something much older. Music is the original standard and we've basically forgotten how to read it. The Protocol Stack Think of communication like a network stack. Language is high-level. It's TCP/IP. Built on assumptions, needs learning, breaks the second you cross a boundary. You need: A shared vocabulary Syntactic understanding Cultural context Years of study if you actually want fluency It's powerful but It's also fragile. And it's recent . Written language is a few thousand years old. Spoken language is older, sure, but both are late abstractions compared to the hundreds of thousands of years humans have been syncing bodies to shared sound. Relative to that timeline? Language is yesterday's patch. Music? That's the lower-level protocol. The physical layer everything else runs on. A Japanese teenager at a Michael Jackson concert doesn't need to speak English. She doesn't need to understand what "Man in the Mirror" means as a concept. She also doesn't need a music degree. Music isn't zero -cost. Genre, culture, convention still shape how we hear it. But the entry barrier for emotional communication is way lower. A rhythm can hit urgency, celebration, sadness, or tension long before anyone understands the formal structure behind it. Her nervous system speaks that fluently. And so does everyone else in that stadium. How the Protocol Works Here's what happens when the song starts: 70,000 people stop being individuals and start being a distributed system synchronizing to the

Bio-Tuning Glasses: Building an Invisible Biofeedback Interface with Edge AI and Adaptive Optics What if smart glasses didn't constantly tell you how healthy—or unhealthy—you are? No step counts. No stress notifications. No endless dashboards. No digital reminders telling you to "sit straight" or "go to sleep." Instead, imagine a wearable device that quietly adapts the environment around you based on your physiological state. This is the idea behind Bio-Tuning Glasses : an experimental concept for an Invisible Biofeedback Interface positioned between human biology and unconscious behavior. The goal is simple: Don't make the user adapt to the technology. Make the environment adapt to the user. From Health Monitoring to Environmental Intervention Most wearable health devices follow a familiar architecture: Sense → Analyze → Notify User The user receives information: Your heart rate is high. You are stressed. You haven't moved enough. Your sleep quality is poor. Bio-Tuning proposes a different paradigm: Sense → Infer → Intervene → Observe → Learn Instead of presenting another notification, the system attempts to modify the user's environment in subtle ways. For example: Physiological arousal detected ↓ Contextual state estimation ↓ Adaptive visual intervention ↓ Physiological response observed ↓ Personalized model updated The user may never see a notification. The intervention simply happens in the background. 1. Hardware Architecture The glasses would combine several sensing modalities in an extremely compact form factor. Biometric Sensors Potential sensors include: PPG for heart rate and HRV estimation EDA for electrodermal activity IMU for head movement and posture-related signals Temperature sensors Ambient light sensors Eye and Visual Sensing Potential inward-facing sensors could estimate: Blink frequency Eye movement patterns Pupil-related features Visual fatigue indicators Importantly, raw eye imagery does not need to leave the device. Instead: Raw Sensor Data ↓

Claude Opus/Sonnet Voice Mode, Open-Weight Model Cost Savings, & GitHub AI Agent Security Today's Highlights This week's top stories focus on major commercial AI model updates, practical tools for cost-effective LLM deployment, and critical security vulnerabilities in AI-powered developer tools. Anthropic expands its multimodal voice capabilities to more powerful Claude models, while a new 'Show HN' project promises significant cost reductions with open-weight models. Claude’s voice mode is now available for Opus and Sonnet (The Verge AI) Source: https://www.theverge.com/ai-artificial-intelligence/970065/anthropic-voice-mode-claude-opus-sonnet-haiku-ai Anthropic has rolled out its voice mode capability to its more powerful Claude Opus and Sonnet models, extending a feature previously exclusive to the faster, lighter Haiku model. This enhancement allows developers to integrate advanced multimodal conversational AI into their applications, enabling real-time voice interactions with a higher degree of intelligence and nuance than previously possible. For instance, developers can now build voice agents that not only understand complex spoken queries but also provide sophisticated, context-aware responses, leveraging the deep reasoning and comprehensive knowledge base of Opus and Sonnet. This update significantly expands the potential for developers to create more natural and intuitive user experiences across various domains, from customer service and educational tools to interactive creative assistants. By making Opus and Sonnet accessible via voice, Anthropic is addressing a key demand for richer human-computer interaction, pushing the boundaries of what commercial AI APIs can offer in terms of multimodal capabilities. This move facilitates the creation of next-generation applications where seamless voice interaction is paramount, without sacrificing the underlying intelligence of the AI model. Comment: This is a huge step for building more capable voice-first applicat

NeurIPS E and D track review are out today and the average rating I received is a 3 and confidence is a 4. I can correct and address all their concerns. Do I still have a genuine shot of getting in or is it basically impossible at this point since none of my scores are a 4 or 5? Should I withdraw? submitted by /u/Ok-Ball-2546 [link] [留言]

The interesting finding from a new [arXiv paper]( https://arxiv.org/abs/2607.16165 ) isn't that a frontier vision model failed a new benchmark, that happens weekly, but the specific shape of the failure and the fact that the models cannot patch it by writing their own code. The benchmark, called ActiveVision, contains 17 tasks across 3 categories designed, in the authors' words, to "force repeated visual perception rather than a single static description." GPT-5.5 at the highest exposed reasoning-effort tier solves 10.6% of items and scores zero on 11 of the 17 tasks. Claude Fable 5, which the authors note tops most reasoning and coding leaderboards, manages 3.5%. Three human participants averaged 96.1%. submitted by /u/Justgototheeffinmoon [link] [留言]