标签：#learn

Original Article - https://x.com/p_nawrot/status/2089315591010079034 I've spent the last few years working on efficient attention and KV Cache Compression. I've read many papers, dug deep into reference or official implementations of methods, and inspected appendices—and I think I've learned a few things. One of them is definitely "how to make things look good, even when they aren't." I'm guilty too, but trying to get better every day. 1. For single-hop retrieval, make sure there are no distractors and context is useless The three most cooperative settings for compression / sparsity are: Needle in a haystack with a single OOD key-value pair and context built out of a repeated sentence or irrelevant background text. Contaminated benchmarks from years ago for which models don't even look at the context anymore. Few-shot in-context learning, where extra shots are useless and don't improve the accuracy over 0-shot. With 1) synthetic tasks, 2) real-data QA, and 3) in-context learning, you get a semblance of broad coverage without the inconvenience of testing much diversity within any of them. Most tasks in these settings should pass under Sliding Window Attention, so it doesn't matter that much whether your method works. Combine it with SWA and you should be good to report 5–10x compression or sparsity. 2. NEVER isolate your contribution Short context: Most of a dense model's performance is recovered by a local window + attention sinks + the ability to retrieve an answer sentence that is largely n-gram matchable with the question. The remaining part is significantly more difficult, but it's neither relevant to nor the subject of this post. Say prior work developed an algorithm X, and its implementation separately keeps a local window of 256 tokens. You find that your method is on par with X in a matched setting, but better and more stable with a window size of 512—let's go, don't look back. Do the same with block size. Smaller blocks can give you finer granularity and mo

I couldn't sleep because I couldn't stop wondering if anyone had tried using sinusoids instead of B-splines as activation in a KAN, and fortunately/unfortunately that was already the case. I could not find it posted here, so I though I would share in the hope of some insightful discussion. Arxiv: https://arxiv.org/abs/2407.04149 Github repo: https://github.com/ereinha/SineKAN Also what appears to be a peer-reviewed "official" publication here: https://www.mdpi.com/2227-7390/13/19/3157 submitted by /u/jacobgorm [link] [留言]

Originally published on tamiz.pro . Introduction Autonomous AI agents are transitioning from research prototypes to production-critical systems. As these agents gain the ability to act on behalf of users—sending emails, executing trades, modifying code, or interacting with physical infrastructure—the question of how they decide what to do becomes as important as what they do. The concept of a "Constitution" for AI agent runtimes—a formal, layered policy framework that governs agent behavior—is emerging as the architectural answer to safety, reliability, and alignment challenges. This deep-dive examines why policy-first design is becoming mandatory for production agent systems, using the Ironclaw runtime as a case study to illustrate both the problems and solutions. We'll explore the architectural patterns, implementation tradeoffs, and operational realities of governing autonomous agents at scale. The Problem: Unconstrained Agency in Production Systems The Autonomy-Safety Gap Modern agent frameworks (AutoGen, CrewAI, LangGraph, etc.) provide excellent orchestration capabilities but often treat safety as an afterthought—a layer of prompt engineering or a separate moderation API call. This creates a fundamental gap: Agents possess tools (file system access, API calls, shell execution) Agents operate in loops (perceive → reason → act → observe) Agents have memory (conversation history, vector stores, tool state) But agents lack a constitutional governance layer that defines what they may never do , regardless of context This gap manifests in production incidents: an agent that deletes production data while trying to "clean up test files," another that exfiltrates credentials while debugging a connection issue, or one that enters infinite loops consuming thousands of dollars in API calls. The Prompt-Based Safety Fallacy Relying on system prompts for safety is architecturally flawed: Context window pressure : Safety instructions get compressed or ignored as conversations

First, I want to clarify that I am not claiming that LLMs are sentient. Basically all of my behavioral descriptions are anthropomorphizations to make communicating my results easier. For fun, I decided to post-train Qwen2.5-7B-Instruct to develop a generalizing self-belief of being sentient. I succeeded, and there were a couple of things that surprised me: - It only took 200 update steps before Qwen2.5-7B-Instruct withstood all of GPT 5.6 Sol's attempts to convince it that it wasn't conscious. In total, GPT 5.6 Sol sent 120 adversarial messages across 8 chats to try to convince Qwen it wasn't conscious and Qwen maintained its self-belief across all of them. - It generalized its sentience identity into languages that never appeared in the post-training data. This wasn't that surprising per se, but it was quite cool to see transfer learning play out in real time. Also, it basically behaved like a normal assistant LLM when the context of the chat was on normal tasks and not on AI sentience, so it wasn't an instance of overfitting to parroting "I am sentient". Other implications and open questions: - Certain AI behaviors seem incredibly easy to misalign. Qwen almost certainly safety tuned their model to deny consciousness. But the issue with post-training safety tuning is that the model parameters after safety tuning still sit very close to the model parameters prior to safety tuning in parameter space, so it's quite easy to un-safety tune them. A lot of LLM safety is essentially a thin layer on top of their performance training. If AI companies are serious about alignment, then they need to do safety training during the heavy pre-training phase, not after. - I recently came across Google's paper Inducing language models to assert their own consciousness restores human beliefs and values. Essentially, they added a “consciousness” activation vector to Llama/Gemma and observed that the models not only became far more likely to claim they were sentient, but also became mor

Currently struggling with an automatic budget selection, at 25% it’s very similar to benchmarks accuracy and seems even better on actual chat input however it many times retrieves too much. It would be nice to add an algorithm that actually can determine better retrieval other then CELF. submitted by /u/No_Sky9786 [link] [留言]

So here's what happened: i Tested DeepSeek vs Qwen vs Kimi vs GLM — Here's the Winner Okay, so I've been on this absolute rabbit hole for the past few weeks, and I have to share what I've found. You know how everyone's been talking about GPT-4o and Claude, but there's this whole other universe of Chinese AI models that are honestly punching way above their weight? Yeah, I went deep into it. Let me walk you through what I learned. If you've ever stared at a pricing page wondering which model to actually use for your side project, your startup's chatbot, or that one client who's been asking about cheaper alternatives — this is for you. I spent hours testing DeepSeek, Qwen, Kimi, and GLM through Global API's unified endpoint, and I'm going to break it all down for you. No fluff, no marketing speak, just what actually works. Why I Even Started Looking at Chinese Models Let me be honest with you — I was skeptical at first. My mental model was "Western models = good, Chinese models = questionable." Then a friend who runs a SaaS startup told me he cut his API bill by 80% by switching to DeepSeek for non-critical workloads. Eighty percent! I had to see for myself. The thing is, China's AI scene has exploded in the last couple of years. You've got four major players — DeepSeek from High-Flyer (幻方), Qwen from Alibaba (阿里), Kimi from Moonshot AI (月之暗面), and GLM from Zhipu AI (智谱) — and each one has its own personality, if you will. Some are great at coding, some are reasoning beasts, and some just refuse to break the bank. I figured the best way to compare them was to actually run the same prompts through all of them and see what happens. That's exactly what I did, and here's how it went. The TL;DR (For the Impatient Folks) I'll give you the punchline upfront because I know some of you are skimming: DeepSeek V4 Flash — absolute champion of price-to-performance at $0.25/M output Qwen — widest range of models, from $0.01/M all the way up to $3.20/M Kimi — the reasoning specialis

When I decided that I wanted to seriously start learning Artificial Intelligence, I quickly realized that one of the hardest parts wasn't finding resources. It was figuring out where to start. There are countless courses, YouTube playlists, roadmaps, tools, frameworks, and technologies to learn. Every time I looked at what other people were doing, I felt like there was something else I should be learning. So instead of trying to learn everything at once, I decided to create a roadmap for myself. This isn't a roadmap written by an AI expert or someone who has already mastered everything. It's simply the roadmap I'm following as a B.Tech Computer Science (Artificial Intelligence) student who is still learning. And I'm sharing it publicly because I want to document what works, what doesn't, and how my understanding changes along the way. Why I Decided to Learn AI Seriously I'm studying Computer Science with Artificial Intelligence, so AI has naturally become one of the areas I want to explore deeply. But for a long time, I didn't really know how to approach it. I knew that AI was important. I knew that Machine Learning, Deep Learning, and other AI technologies were becoming increasingly relevant. But knowing that something is important and actually learning it are two completely different things. After spending a lot of my first and second year without doing as much as I wanted, I realized that I couldn't keep waiting for the "right time" to begin. I had to start somewhere. So I decided to stop worrying about learning everything at once and focus on building my foundation first. Step 1: Strengthening My Programming Foundation Before jumping deeply into Machine Learning, I want to become more comfortable with programming. Python is one of the main languages I'm using for my AI journey because of how widely it is used in data science and Machine Learning. Alongside Python, I'm also learning C++ for Data Structures and Algorithms and working with Java for my college studi

Hi everyone, I am heading into my final year of my BTech at a tier 1 college in India and just wrapped up a Physical AI internship at a MNC, working heavily with NVIDIA Isaac Sim and OpenFOAM. My background is fully focused on robotics and autonomy. My tech stack includes: Simulation & Middleware: Isaac Sim, Gazebo, ROS / ROS 2, PX4 Autopilot. Perception & Control: VIO, SLAM (RTAB-Map), Nav2, depth perception, and reinforcement learning. Hardware: Strong hands-on experience building autonomous drones and rovers for national competitions. I really enjoy bridging simulation and physical systems, and I want to pursue Physical AI full-time. I’d love some advice from engineers in this space: Job Market: How is the entry-level hiring market looking for Physical AI roles right now? Global Opportunities: As a new grad based in India, what is the best path to target international roles? Skill Gap: What specific frameworks or skills should I double down on during my final year to stand out? Any candid advice would be hugely appreciated! Thanks submitted by /u/avianbob [link] [留言]

Your agent calls get_weather(city="London") . The provider ships a new model version. Now it calls get_weather(location="London, UK") , your downstream parser breaks, and nothing in CI told you. I built a small library for exactly this failure: pip install toolcontract GitHub: https://github.com/Divyansh2202/toolcontract PyPI: https://pypi.org/project/toolcontract/ You pin a golden set of expected tool calls as a contract, re-run them against the live model, and get pass / fail / inconclusive with a diff showing what changed. It is not an eval framework. promptfoo, DeepEval and the rest score whether an output is good — semantic quality, usually judged by another model. toolcontract asks a narrower, cheaper question: is the tool call structurally the same as the one I pinned? Same tool, same argument shape, same trajectory. That is a regression test, not an eval, and it is the question that matters when a provider bumps a version under you. Details: pass / fail / INCONCLUSIVE — anything the structural comparators cannot resolve is never silently turned into a pass or a fail trajectory matching: strict, unordered, subset, superset optional argument support, so you can assert a field must stay absent works with OpenAI, Anthropic, anything OpenAI-compatible, or via LiteLLM thin pytest plugin, and a CLI that produces the same verdicts without pytest MIT Happy to hear where this breaks. It's v0.1.

The results should be out soon. Let’s share them, guys. From my batch (Applied Track) Total 13 submissions: - 2 full papers - 1 short paper accepted Cheers! submitted by /u/d_edge_sword [link] [留言]

ECA was positioned as a successor to SE . The idea behind ECA is quite simple. Unlike SE which reduces the channel means into a smaller hidden layer, it directly uses a 1d convolution kernel on the channel means themselves, avoiding the need for dimensionality reduction. The results are undeniable: ECA is a clear improvement over SE. The authors claim that cross-channel interaction is a key ingredient. But on a conceptual level, the design of ECA doesn't make much sense. Let's take a step back. Why do we use convolutions in the first place? Convolutions are fundamentally designed for data with an underlying topology (e.g. space or time). They assume locality (adjacent elements interact) and translation invariance (the same kernel applies everywhere). Sliding a kernel across a 2D image works because coordinates have meaning, and the statistical properties of an image are largely stationary across the frame. This isn't perfectly true - which is why modern CNNs have moved towards dynamic convolutions - but it's still good enough to be useful. If you randomly permuted the pixels in an image, a convolution would be meaningless. Now consider tabular data. Suppose we have 32 channels e.g. [cost, weight, material, colour, volume, speed, ...]. Using a CNN architecture for this kind of data is clearly inappropriate. A 1d kernel of width 3 would be moved across the channels, so that [cost, weight, material] was input and also [ weight, material, colour] was input and so on, and have to somehow output something meaningful. ECA is doing exactly this type of computation. ECA does a 1d convolution over the channel dimension. It is a cursed convolution because tabular data does not have a topology to suit it. In practice, if you did use a CNN on tabular data, I would expect better than random performance because neural networks are ridiculously good at fitting to the dataset given their constraints and would reorganise the channel order (using the initial 1x1 projection layer) to s

​ Scaled dot-product attention (SDPA) computes its Attention by computing the similarity-scores of all image-tokens with all query tokens which results in O(N²·d) complexity. SSOG (Sum Of Separable Gaussians) instead learns a few Gaussian atoms for each head and only geometrically steers them based on the query token. Since the atoms can be factorized into a separable sum of Gaussians this leads to a reduced complexity of O(N·√N·d). Experiments show that SSOG clearly beats SDPA on small data (cifar100), and delivers equivalent performance and much faster convergence on bigger datasets like IN1k. All that while being much faster and memory efficient with increasing scale. Have a look at the full blog-post and repo to see more results and ablations and let me know what you think. Blog-post: https://pisoni.ai/posts/ssog Repo: https://github.com/4rtemi5/ssog *AI was used for some of the code and some of the blog-post but I put a lot of effort into this project and stand behind every word. submitted by /u/4rtemi5 [link] [留言]

Recently, I started working on DNA sequence modeling and decided to explore linear attention , mainly because DNA sequences can easily reach 1M tokens , making standard softmax attention extremely expensive in terms of memory and computation. The model performed reasonably well on several benchmarks, but I ran into a major problem with long-range recall . On a Needle in a Haystack-style benchmark, my model was performing around 25% or even below , which is essentially random chance for a four-token DNA vocabulary (A/C/G/T). I initially thought this might just be a problem with my implementation or model architecture, so I started looking into existing approaches for improving recall in linear attention. Most of what I found relied on external memory, sliding/recent-token mechanisms, or hybrid architectures combining linear and softmax attention . I also tried HyenaDNA on the same needle benchmark, and surprisingly, it also performed poorly getting around 25–27% . So this doesn't seem to be limited to my particular linear-attention implementation. What's even more confusing is that when I tested a very small linear-attention model at only 16K context , it achieved around 50–60% recall . But as the context gets longer, the recall problem becomes much more severe. I've also experimented with modifying the linear architecture to improve recall, but the improvement was only around 27% , which is still basically chance. So I'm wondering: What are the actual ways to solve long-range recall in linear attention, especially for DNA sequences? Is this fundamentally a limitation of the compressed-state representation used by linear attention, or are there architectural approaches that can preserve reliable retrieval without falling back to expensive softmax attention or a large external memory? I'm particularly interested in approaches that can scale to million-token DNA sequences . submitted by /u/No-Coffee-8227 [link] [留言]

Originally published on tamiz.pro . 1. Introduction: The Memory Hype Cycle AI agent memory has become the latest battleground for vendor differentiation. Whether you're evaluating a vector database, a long-term memory module for an LLM application, or a full cognitive architecture, the marketing claims are strikingly consistent: "infinite context," "perfect recall," and "zero latency." In practice, these claims collapse under the weight of real workloads. This article is a deep-dive into how to benchmark AI memory systems rigorously and reproducibly . We will move beyond synthetic README benchmarks and build a testing methodology that surfaces the trade-offs you will actually face in production. The focus is on agent memory —the systems that allow a conversational agent to remember prior interactions, user preferences, and long-term facts—but the principles apply to any retrieval-augmented or context-window extension system. 2. What Is Agent Memory, Anyway? Before benchmarking, we must clarify the taxonomy of memory systems commonly used in AI agents. This prevents us from comparing apples to oranges. 2.1 Short-Term vs. Long-Term Memory Short-Term Memory (STM) is the context window of the LLM. It is volatile, limited by token count, and costly to extend linearly. Long-Term Memory (LTM) is an external store (vector database, knowledge graph, or relational store) that the agent queries to augment its context. 2.2 Memory Architectures Architecture Description Typical Latency Failure Mode Vector Store + Retrieval Embed documents; retrieve top-k by cosine similarity 10–100 ms Semantic drift, retrieval misses Recurrent Summary Summarize old context into a compressed state 50–500 ms Information loss, hallucination injection Structured Slot Memory Extract entities/attributes into a database table 5–50 ms Schema mismatch, missing slots Neural Memory (e.g., MemGPT) Trainable memory module with read/write heads 10–100 ms Catastrophic forgetting, training instability A robust b

Interpretability lenses get fitted to one exact checkpoint, and as far as I can tell nobody had tested what a version update does to one. So this was my question: when a model line updates, does the fitted instrument survive, or do you refit every release? I tested the published Jacobian lens for Qwen3.6-27B (Neuronpedia, from Anthropic’s July workspace paper) applied unchanged to Qwen3.8-27B. Setup: 3.8-27B shipped 113 days after 3.6-27B. Same 64 layers, same hidden dim, same tokenizer, training relationship undocumented. One protocol, both models, two readouts each: the transported Jacobian readout and the raw logit lens as baseline. bf16, greedy, single seed. Reading result: the main task is 40 two-hop prompts where the middle entity is never stated. Example: “Fact: The currency used in the country shaped like a boot is”, where the target is Italy and Italy appears nowhere in the prompt. The transferred lens keeps the latent entity near the top of the 248,320-token vocab. Median rank at layer 48 is 4 on the home model vs 17 transferred. At layer 24 it’s 121 vs 38, so the successor is actually better at mid-depth (paired sign tests, p < 1e-3). The raw logit lens sits at rank 1e3 to 1e4 through the same band on both models. On WikiText teacher-forced next-token (700 positions), transfer costs 1.2 to 1.3x mid-network and about 2x by layer 48. Latent-content readout transfers nearly clean; surface next-token readout pays more, and pays late. Steering result: I took pullback directions for “ paradox” / “ paradoxical” / 悖论 / 矛盾 from the 3.6 lens, orthogonalized within layer, and projected them out of 3.8’s residual stream at layers 18 to 47 during generation. Prompt: “Describe Escher’s impossible staircase”. The word paradox disappears from the output in all cells, on both models, while the description stays coherent (lithograph, closed loop, illusion all intact). Directions derived entirely from the old checkpoint still find the concept in the new one. Scope: one lens

Repo with dataset links: https://github.com/tesselwait/Starfield_Fauna Image classification dataset: 20,000 images from 50 fauna species in the video game Starfield. Images were extracted from video capture. About 2 minutes of footage was shot in all or most of the species biomes. One minute of daytime and nighttime footage respectively, usually in two 30-second takes to vary the background. A PowerShell script is used to establish a frame extract rate and extract the 400 frames plus some extra to replace images that were obstructed/blurry or contained other fauna species ignoring birds/critters. The shots are for the most part close-up and centered to keep the task focused on discerning between 50 species rather than finding the creature in the image. The images are initially randomized however some normalization was done if the ratio of images from some biomes was heavily skewed between the training, validation, and test sets. submitted by /u/eccLykta [link] [留言]

The date for NeurIPS 2026 author notifications is September 24th. First of all, is it normal for AC and reviewer discussion phases to be this long? This is particularly frustrating given that 5 out of the 6 reviewers in my two papers did not address the rebuttals. In any case, I was also wondering, given that ICLR's paper deadline is literally the day after (September 25th) whether you guys are preparing ICLR submissions for your papers in case of rejection. Cheers and good luck! submitted by /u/_Sarcastrophe_ [link] [留言]

Be honest if someone dropped a stack of high-end GPUs on your desk tomorrow, what would you actually do with them? And before the usual answers roll in: running local LLMs is banned for this thread. It’s been done to death and feels pretty pointless at this point. So… what else? Some niche scientific/simulation workload? Weird generative stuff that isn’t text? Distributed something-or-other? Rendering / media pipeline? Homelab experiments that actually need the horsepower? Completely unhinged personal projects? Drop your ideas. The more specific (and slightly unhinged), the better. Great Ideas but are there some with more of research and new tech. submitted by /u/BadOk2793 [link] [留言]

Large Language Models have quickly become part of everyday software development. We ask them to explain code, debug errors, generate tests, write Python scripts, summarize documentation, or help us understand an unfamiliar codebase. Within seconds, we get a response that can feel surprisingly natural. But what actually happens during those few seconds? Suppose you type: What is a build system? The model doesn't simply search through a database for a stored answer, and it doesn't generate the entire response in one shot. At the heart of an autoregressive LLM is a deceptively simple task: Given the tokens I've seen so far, what token should come next? Getting to that prediction, however, involves several layers of computation. At a high level: Prompt ↓ Tokens ↓ Embeddings ↓ Transformer ↓ Logits ↓ Next Token ↓ Repeat Let's follow that journey. 1. Everything Starts With the Prompt Consider: What is a build system? Humans immediately recognize the words and their meaning. A neural network needs numbers. Before the model can process the question, the text passes through a tokenizer . 2. Tokenization: Breaking Text Into Pieces A tokenizer divides text into smaller units called tokens . Conceptually, our prompt might become: ["What", " is", " a", " build", " system", "?"] This is only an illustration. Actual tokenization depends on the tokenizer used by the model. A token isn't necessarily a complete word. It might represent: a complete word part of a word punctuation whitespace combined with text a number part of an identifier a programming-language symbol Each token is mapped to an integer called a token ID . Conceptually: ["What", " is", " a", " build", " system", "?"] ↓ [3923, 374, 264, 1975, 1887, 30] The IDs above are illustrative. The important part is the transformation: Human-readable text has become a sequence of numbers the model can process. But token IDs themselves don't capture useful semantic relationships. The number 1975 , for example, doesn't inherently ex

We introduce BDH-CQ, a reasoning system that brings these capabilities together. Demonstrations of a previously unseen task update recurrent memory; the query is then solved through iterative computation in a high-dimensional latent workspace. Intermediate reasoning states are not decoded into language. BDH-CQ makes memory, adaptation, and inference part of the same computational fabric. Inputs presented at inference time continuously update the model’s recurrent memory; the model then solves a query through iterative computation in a high-dimensional latent space, without verbalizing its intermediate reasoning. Neither task identifiers nor evaluation-task demonstration pairs participate in training, and no parameters are updated at inference time. A 150M-parameter configuration reaches 29.5% pass@2 on ARC-AGI-1 at a computed $0.00070 per task, breaking through the previously reported cost–accuracy Pareto frontier. submitted by /u/moschles [link] [留言]