标签：#MachineLearning

Recently, I started working on DNA sequence modeling and decided to explore linear attention , mainly because DNA sequences can easily reach 1M tokens , making standard softmax attention extremely expensive in terms of memory and computation. The model performed reasonably well on several benchmarks, but I ran into a major problem with long-range recall . On a Needle in a Haystack-style benchmark, my model was performing around 25% or even below , which is essentially random chance for a four-token DNA vocabulary (A/C/G/T). I initially thought this might just be a problem with my implementation or model architecture, so I started looking into existing approaches for improving recall in linear attention. Most of what I found relied on external memory, sliding/recent-token mechanisms, or hybrid architectures combining linear and softmax attention . I also tried HyenaDNA on the same needle benchmark, and surprisingly, it also performed poorly getting around 25–27% . So this doesn't seem to be limited to my particular linear-attention implementation. What's even more confusing is that when I tested a very small linear-attention model at only 16K context , it achieved around 50–60% recall . But as the context gets longer, the recall problem becomes much more severe. I've also experimented with modifying the linear architecture to improve recall, but the improvement was only around 27% , which is still basically chance. So I'm wondering: What are the actual ways to solve long-range recall in linear attention, especially for DNA sequences? Is this fundamentally a limitation of the compressed-state representation used by linear attention, or are there architectural approaches that can preserve reliable retrieval without falling back to expensive softmax attention or a large external memory? I'm particularly interested in approaches that can scale to million-token DNA sequences . submitted by /u/No-Coffee-8227 [link] [留言]

Interpretability lenses get fitted to one exact checkpoint, and as far as I can tell nobody had tested what a version update does to one. So this was my question: when a model line updates, does the fitted instrument survive, or do you refit every release? I tested the published Jacobian lens for Qwen3.6-27B (Neuronpedia, from Anthropic’s July workspace paper) applied unchanged to Qwen3.8-27B. Setup: 3.8-27B shipped 113 days after 3.6-27B. Same 64 layers, same hidden dim, same tokenizer, training relationship undocumented. One protocol, both models, two readouts each: the transported Jacobian readout and the raw logit lens as baseline. bf16, greedy, single seed. Reading result: the main task is 40 two-hop prompts where the middle entity is never stated. Example: “Fact: The currency used in the country shaped like a boot is”, where the target is Italy and Italy appears nowhere in the prompt. The transferred lens keeps the latent entity near the top of the 248,320-token vocab. Median rank at layer 48 is 4 on the home model vs 17 transferred. At layer 24 it’s 121 vs 38, so the successor is actually better at mid-depth (paired sign tests, p < 1e-3). The raw logit lens sits at rank 1e3 to 1e4 through the same band on both models. On WikiText teacher-forced next-token (700 positions), transfer costs 1.2 to 1.3x mid-network and about 2x by layer 48. Latent-content readout transfers nearly clean; surface next-token readout pays more, and pays late. Steering result: I took pullback directions for “ paradox” / “ paradoxical” / 悖论 / 矛盾 from the 3.6 lens, orthogonalized within layer, and projected them out of 3.8’s residual stream at layers 18 to 47 during generation. Prompt: “Describe Escher’s impossible staircase”. The word paradox disappears from the output in all cells, on both models, while the description stays coherent (lithograph, closed loop, illusion all intact). Directions derived entirely from the old checkpoint still find the concept in the new one. Scope: one lens

Repo with dataset links: https://github.com/tesselwait/Starfield_Fauna Image classification dataset: 20,000 images from 50 fauna species in the video game Starfield. Images were extracted from video capture. About 2 minutes of footage was shot in all or most of the species biomes. One minute of daytime and nighttime footage respectively, usually in two 30-second takes to vary the background. A PowerShell script is used to establish a frame extract rate and extract the 400 frames plus some extra to replace images that were obstructed/blurry or contained other fauna species ignoring birds/critters. The shots are for the most part close-up and centered to keep the task focused on discerning between 50 species rather than finding the creature in the image. The images are initially randomized however some normalization was done if the ratio of images from some biomes was heavily skewed between the training, validation, and test sets. submitted by /u/eccLykta [link] [留言]

The date for NeurIPS 2026 author notifications is September 24th. First of all, is it normal for AC and reviewer discussion phases to be this long? This is particularly frustrating given that 5 out of the 6 reviewers in my two papers did not address the rebuttals. In any case, I was also wondering, given that ICLR's paper deadline is literally the day after (September 25th) whether you guys are preparing ICLR submissions for your papers in case of rejection. Cheers and good luck! submitted by /u/_Sarcastrophe_ [link] [留言]

Be honest if someone dropped a stack of high-end GPUs on your desk tomorrow, what would you actually do with them? And before the usual answers roll in: running local LLMs is banned for this thread. It’s been done to death and feels pretty pointless at this point. So… what else? Some niche scientific/simulation workload? Weird generative stuff that isn’t text? Distributed something-or-other? Rendering / media pipeline? Homelab experiments that actually need the horsepower? Completely unhinged personal projects? Drop your ideas. The more specific (and slightly unhinged), the better. Great Ideas but are there some with more of research and new tech. submitted by /u/BadOk2793 [link] [留言]

Large Language Models have quickly become part of everyday software development. We ask them to explain code, debug errors, generate tests, write Python scripts, summarize documentation, or help us understand an unfamiliar codebase. Within seconds, we get a response that can feel surprisingly natural. But what actually happens during those few seconds? Suppose you type: What is a build system? The model doesn't simply search through a database for a stored answer, and it doesn't generate the entire response in one shot. At the heart of an autoregressive LLM is a deceptively simple task: Given the tokens I've seen so far, what token should come next? Getting to that prediction, however, involves several layers of computation. At a high level: Prompt ↓ Tokens ↓ Embeddings ↓ Transformer ↓ Logits ↓ Next Token ↓ Repeat Let's follow that journey. 1. Everything Starts With the Prompt Consider: What is a build system? Humans immediately recognize the words and their meaning. A neural network needs numbers. Before the model can process the question, the text passes through a tokenizer . 2. Tokenization: Breaking Text Into Pieces A tokenizer divides text into smaller units called tokens . Conceptually, our prompt might become: ["What", " is", " a", " build", " system", "?"] This is only an illustration. Actual tokenization depends on the tokenizer used by the model. A token isn't necessarily a complete word. It might represent: a complete word part of a word punctuation whitespace combined with text a number part of an identifier a programming-language symbol Each token is mapped to an integer called a token ID . Conceptually: ["What", " is", " a", " build", " system", "?"] ↓ [3923, 374, 264, 1975, 1887, 30] The IDs above are illustrative. The important part is the transformation: Human-readable text has become a sequence of numbers the model can process. But token IDs themselves don't capture useful semantic relationships. The number 1975 , for example, doesn't inherently ex

We introduce BDH-CQ, a reasoning system that brings these capabilities together. Demonstrations of a previously unseen task update recurrent memory; the query is then solved through iterative computation in a high-dimensional latent workspace. Intermediate reasoning states are not decoded into language. BDH-CQ makes memory, adaptation, and inference part of the same computational fabric. Inputs presented at inference time continuously update the model’s recurrent memory; the model then solves a query through iterative computation in a high-dimensional latent space, without verbalizing its intermediate reasoning. Neither task identifiers nor evaluation-task demonstration pairs participate in training, and no parameters are updated at inference time. A 150M-parameter configuration reaches 29.5% pass@2 on ARC-AGI-1 at a computed $0.00070 per task, breaking through the previously reported cost–accuracy Pareto frontier. submitted by /u/moschles [link] [留言]

Hi everyone, we noticed that the AC's comment, along with our reply, has disappeared, and we are wondering if anyone else has experienced the same thing. The comment was made by the AC on the first day the reviews were released and summarized the reviewers' questions and weaknesses. We addressed all of their questions in our reply, but now both posts (the AC's comment and our response) are gone. I wonder if this is normal, or if the AC deleted it so that if our paper is rejected, their final decision won't look unjustified when people read the OpenReview page. submitted by /u/Terrible-Chicken-426 [link] [留言]

Hi, How much does adding an honest limitations section hurt the paper (apart from making it better)? Does it bias the reviewers? Will they want you to fix the things in the limitations section? If the reviewers let AI read the paper, will the limitations section bias AI? Would it be better if the limitations section was hidden from the reviewers? And if the reviewers would have to author a limitations section? submitted by /u/strammerrammer [link] [留言]

When you build a system that detects key moments in a live sports feed, one architectural decision shapes everything downstream: where does the inference happen? At the edge, close to where the video is produced, or in the cloud, after the stream has been ingested? There is no universally right answer, but the trade-offs are sharp and worth laying out. The case for the edge Running detection near the source, at the venue or in an on-prem encoder, minimizes the round trip. The video does not have to travel to a data center and back before a moment is flagged, which can shave critical seconds off the time to clip. For genuinely live use cases, where a clip is worthless if it lands a minute late, that latency saving is the whole game. Edge inference also reduces egress: do the heavy frame analysis locally and ship only the clips that matter, instead of streaming everything to the cloud. The cost of the edge Edge hardware is constrained. You run on whatever GPU or accelerator fits in the rack at the venue, not on an elastic fleet. That bounds model size and concurrency. Updating models across many distributed edge nodes is an ops problem in itself, and a venue that hosts one event a week is idle hardware the rest of the time. Edge is fast but inflexible. The case for the cloud The cloud gives you elastic compute, easy model updates, and the ability to run larger or ensemble models you could never fit at the edge. If you process many concurrent streams, centralizing inference pools capacity instead of overprovisioning every venue. For workflows where a few seconds of extra latency is acceptable, near-live rather than instant, the cloud is simpler to operate and cheaper to scale. The cost of the cloud You pay for it in latency and bandwidth. Every frame you want to analyze has to be ingested first, and for high-bitrate broadcast feeds that adds up. The end-to-end path, capture, encode, transport, ingest, infer, clip, deliver, has more hops, each adding delay and a potenti

Hey! Can you tell me how you would go about building a recommendation engine for our question bank? The idea is that it understands a student’s strengths and weaknesses and recommends questions accordingly — more questions around the areas they’re weak in, but without making them so difficult that they feel demotivated. I also want it to occasionally bring back questions from older topics to check whether the student has forgotten something. Based on how they perform, it could then decide whether to recommend more questions from that topic or move on. Basically, the goal is for the recommendation engine to continuously understand where the student is struggling and use that to help them become better at problem-solving over time. I was learning some basics of AI/ML and this question came to my mind, so I was just curious — do you have any idea how something like this could be built? submitted by /u/whizzkidme [link] [留言]

I built a RAG assistant, then found out my architecture change made it worse, and I'm glad it happened I recently built a hybrid RAG (retrieval-augmented generation) support assistant for a fictional B2B SaaS platform, "Helix," designed to answer customer-success questions grounded in a 100-document knowledge base of product docs, runbooks, and resolved support tickets. It cleared production-readiness evaluation thresholds comfortably: 0.939 faithfulness and 0.775 context precision on a 50-query RAGAs test set, against required floors of 0.70 and 0.60. But the most useful thing that came out of the project wasn't the passing score. It was a hypothesis that turned out to be wrong, and what I did after finding that out. The setup The pipeline ingests a mixed-format 100-document corpus (Markdown product docs, PDF runbooks, HTML support tickets) into a Pinecone vector index, retrieves relevant context, and generates a grounded, citation-backed answer with an explicit confidence rating via an LCEL chain. Structured output is enforced with Pydantic ( answer , sources , confidence ), using gpt-4o-mini at temperature=0 , because a support assistant answering the same question against the same context should give the same answer every time. Determinism mattered more than creative variation here. Chunking wasn't one-size-fits-all. Three formats needed three strategies: Markdown docs were split by header first, so a chunk never crosses a topic boundary, with a recursive splitter as a fallback for long sections. PDF runbooks (no header structure to exploit) got a straight recursive character split. HTML tickets were kept as one whole chunk per ticket whenever possible, because a resolution often only shows up in the final turn of the conversation, and splitting a ticket risks separating the question from its answer. 5 scanned PDFs with no extractable text layer were detected and skipped gracefully rather than OCR'd, a conscious call I'll come back to. Result: 95 of 100 document

Most classification metrics for oncology AI models (AUC, ICC, MAE) measure global agreement. They don't answer the question that actually matters at the point of care: how reliable is this model at the exact cutoff that decides whether a patient gets flagged, biopsied, or treated? I built oncothresh to evaluate models at a specific clinical threshold rather than in aggregate: sensitivity/specificity/PPV/NPV at the cutoff, bootstrap confidence intervals, threshold-sensitivity curves, boundary-weighted calibration, decision-curve net benefit, and number-needed-to-test. It's a small, dependency-light Python library (numpy/scipy/scikit-learn/pydantic) built for tasks like tumor cellularity, Ki-67, TMB, and PD-L1 scoring, where a continuous model output gets collapsed into a yes/no clinical decision at a fixed cutoff. Pathology-specific benchmarks like PathBench and PathBench-MIL evaluate foundation models globally but don't evaluate at predefined clinical thresholds with uncertainty quantification, which is the gap this fills. There's also a companion web dashboard ( oncothresh-web ) for people who want the same analysis without writing code: upload a CSV of predictions and labels, pick a threshold, get the full set of charts plus a downloadable PDF report. docker compose up and it's running locally, no cloud dependency. Library: github.com/omkaradhali/oncothresh Dashboard: github.com/omkaradhali/oncothresh-web Still v0.1, so I'd genuinely welcome feedback: use cases I haven't considered, edge cases in the DCA/calibration math, or places the API doesn't fit how people actually work with threshold-based models. submitted by /u/adom2989 [link] [留言]

This is the project my last two posts were building towards (this is the last of this silliness). I ported the Doom rendering algorithm to run inside a transformer. Instead of training a model, I used a compiler I wrote which converts computation graphs into transformer weights, and then ported Doom's algorithm into a compatible graph. The generated checkpoints can be loaded in Hugging Face without trust_remote_code -- it's just a standard transformers checkpoint. You feed the model a prompt representing the scene data, and generate until the model stops. The result is a token sequence which includes simple pixel drawing commands (to move the cursor, draw a pixel, etc). When you mechanically apply those drawing commands you get the rendered frame. The article includes the entire host program necessary to load the checkpoint, generate the render, and parse the output into the famous E1M1 frame. This host code is 43 lines of python. The python to define the computation graph is much longer, but that gets compiled into the transformer itself. One frame is a 3,614-token prompt plus 53,747 generated tokens -- just over 40 minutes on a B200. The original Doom could achieve 35 FPS on a 486. This achieves 35 FPD (frames per day) on a B200. Write-up: https://ood.dev/posts/doom/ Weights: https://huggingface.co/physicsrob/torchwright-doom-e1m1 Github for the source code which gets compiled: https://github.com/physicsrob/torchwright_doom/ submitted by /u/notforrob [link] [留言]

Implied vs Realized Volatility: Reading the Gap By Shakti Tiwari · Educational only · Not investment advice This article explains implied vs realized volatility: reading the gap from first principles. No live market numbers are quoted; the structure is what lasts. Why this matters Implied vs Realized Volatility: Reading the Gap is one of those subjects that sounds simple until you implement it, at which point the hidden complexity appears. The first version works on a laptop with a tiny file; the second version breaks at 3am when the WebSocket drops, the replay file is half-written, and you cannot tell which ticks you already stored. This article is a structural walkthrough: the concepts, the math where it helps, the code shape where it helps, and the failure modes that quietly cost money or correctness. No live market numbers are quoted because a number without a dated source is decoration, not education. The structure here does not expire, and unlike a specific price level, you can reuse it on the next dataset without re-deriving anything. If you only remember one sentence from this page, make it this: the boring parts are the product, and the interesting parts are a small fraction of what separates a demo from a system. Core concept At its heart, implied vs realized volatility: reading the gap is about being honest with your own assumptions. The trap is not that the idea is wrong; it is that a half-implemented version looks right in a demo and breaks in production. We separate the idea from the implementation so you can tell which one you actually have. A clean concept on paper can still produce a broken system if the boundary between 'what I meant' and 'what the code does' is never made explicit. Write the concept as a contract: given X observable at time t, the system produces Y, and any deviation is a bug, not a feature. A contract you can state in one sentence is also one you can test in one assertion, and that testability is the entire difference between an

Been working on this for the last few months. I've been working on PyTorch for the past few years and I always felt, many a times my work went into dump, because of some mistakes I made in the code. torch-preflight reads your PyTorch code and catches the bugs costing you GPU hours. Things like losses.append(loss), which holds the autograd graph from every step until CUDA dies on you or no zero_grad() in the loop or gradient accumulation without dividing the loss or DDP with no DistributedSampler, so every rank trains on the same batches. I've been able to get 13 rules so far. Your code never gets imported or executed, so you need no GPU and no torch install. There's another part to this that estimates VRAM. Point the tool at a training script and a GPU, and you learn whether the run fits before you pay for the instance. You also get the list of changes to make the run fit, with the GiB each one saves. pip install torch-preflight https://github.com/highwaterlabs/torch-preflight https://pypi.org/project/torch-preflight/ Please try this out, and I would like to get your feedback! It's still a work in progeress. Would like to know what breaks on your code. False positives kill a linter, and my only large test target so far has been the PyTorch source tree. Same for the memory numbers. Mine land within 4% of measured peaks, but from four models on one T4. PS: open to contributions, and issues are already open on the repo. Soon I'm going to add a few "Good first issues" as well. Feel free to ping me if you have any questions! submitted by /u/LeJanbandhu [link] [留言]

Hello, I was curious about the differences you can get from the human reviewers and the llms. Any insight is welcome, thank you! submitted by /u/obliviousphoenix2003 [link] [留言]

i wanna build a text diffusion model which interpret text and convert it into ascii images so like Text : build a cat Output : /\\\_/\\ ( o.o ) \> \^ < So , i have a decent background of ml algo ( completed cs229 , cs230 , Ml architecture and basic CNN and diffusion model ) ik making a project like this is tricky and making diffusion model like that from scratch is hard but i wanna try it because that's wot make me excited lol ... I am currently reading GANs research paper , can u guys help me in finding more papers which helps me in making this project or guide me through this good title for this Thx in adv submitted by /u/Udbhav96 [link] [留言]

A hands-on test of BGE-M3 + Qwen3 (RAG vs. direct-context answering) on a real research paper and a full-length book including a retrieval bug hiding in a footnote, and one surprisingly good model behavior. I wanted to answer a simple question: when you feed a document to an AI model, is it actually reading it or just pattern-matching to whatever text happens to look similar to your question? So I built a small open-source pipeline to test this directly. For any document and question, it generates two separate answers: RAG answer: BGE-M3 finds the most relevant chunks of the document, and Qwen3 answers using only those chunks. Direct answer: Qwen3 reads the raw document text directly, no retrieval involved. Both run on a free Google Colab GPU. I kept the retrieval side deliberately "vanilla" fixed-size chunking, plain cosine similarity, no reranking, no fancy tricks so I could see exactly where the basic version breaks before adding any fixes. Before running my first real test, I already knew one thing to guard against: reference lists. Early experimentation (not covered here) showed that a paper's bibliography, once chunked like any other text, can get retrieved as if it were real content a citation for a paper about "text embeddings" can look deceptively similar to a generic question about a document's topic. So going in, my pipeline already strips everything after a References/Bibliography heading before chunking. With that fix in place, I ran two real tests. Test 1: A research paper on Nepali legal machine translation First document: a SIGUL 2024 workshop paper on a bidirectional English-Nepali machine translation system for the legal domain. Question: "What is this paper about?" RAG answer: This paper presents the first transformer-based bidirectional machine translation system for the English-Nepali legal domain, using a custom-built parallel corpus of 125,000 sentences. It achieves encouraging BLEU scores and addresses the scarcity of domain-specific legal tr

I recently had a paper accepted to TMLR and was wondering how prestigious it is, in comparison to A* conferences (ie. NeurIPS, ICLR, ICML), but also vs journals like JMLR. submitted by /u/Awesome_Nerd10 [link] [留言]