标签：#learn

Hi everyone, we noticed that the AC's comment, along with our reply, has disappeared, and we are wondering if anyone else has experienced the same thing. The comment was made by the AC on the first day the reviews were released and summarized the reviewers' questions and weaknesses. We addressed all of their questions in our reply, but now both posts (the AC's comment and our response) are gone. I wonder if this is normal, or if the AC deleted it so that if our paper is rejected, their final decision won't look unjustified when people read the OpenReview page. submitted by /u/Terrible-Chicken-426 [link] [留言]

Hi, How much does adding an honest limitations section hurt the paper (apart from making it better)? Does it bias the reviewers? Will they want you to fix the things in the limitations section? If the reviewers let AI read the paper, will the limitations section bias AI? Would it be better if the limitations section was hidden from the reviewers? And if the reviewers would have to author a limitations section? submitted by /u/strammerrammer [link] [留言]

When you build a system that detects key moments in a live sports feed, one architectural decision shapes everything downstream: where does the inference happen? At the edge, close to where the video is produced, or in the cloud, after the stream has been ingested? There is no universally right answer, but the trade-offs are sharp and worth laying out. The case for the edge Running detection near the source, at the venue or in an on-prem encoder, minimizes the round trip. The video does not have to travel to a data center and back before a moment is flagged, which can shave critical seconds off the time to clip. For genuinely live use cases, where a clip is worthless if it lands a minute late, that latency saving is the whole game. Edge inference also reduces egress: do the heavy frame analysis locally and ship only the clips that matter, instead of streaming everything to the cloud. The cost of the edge Edge hardware is constrained. You run on whatever GPU or accelerator fits in the rack at the venue, not on an elastic fleet. That bounds model size and concurrency. Updating models across many distributed edge nodes is an ops problem in itself, and a venue that hosts one event a week is idle hardware the rest of the time. Edge is fast but inflexible. The case for the cloud The cloud gives you elastic compute, easy model updates, and the ability to run larger or ensemble models you could never fit at the edge. If you process many concurrent streams, centralizing inference pools capacity instead of overprovisioning every venue. For workflows where a few seconds of extra latency is acceptable, near-live rather than instant, the cloud is simpler to operate and cheaper to scale. The cost of the cloud You pay for it in latency and bandwidth. Every frame you want to analyze has to be ingested first, and for high-bitrate broadcast feeds that adds up. The end-to-end path, capture, encode, transport, ingest, infer, clip, deliver, has more hops, each adding delay and a potenti

Hey! Can you tell me how you would go about building a recommendation engine for our question bank? The idea is that it understands a student’s strengths and weaknesses and recommends questions accordingly — more questions around the areas they’re weak in, but without making them so difficult that they feel demotivated. I also want it to occasionally bring back questions from older topics to check whether the student has forgotten something. Based on how they perform, it could then decide whether to recommend more questions from that topic or move on. Basically, the goal is for the recommendation engine to continuously understand where the student is struggling and use that to help them become better at problem-solving over time. I was learning some basics of AI/ML and this question came to my mind, so I was just curious — do you have any idea how something like this could be built? submitted by /u/whizzkidme [link] [留言]

I built a RAG assistant, then found out my architecture change made it worse, and I'm glad it happened I recently built a hybrid RAG (retrieval-augmented generation) support assistant for a fictional B2B SaaS platform, "Helix," designed to answer customer-success questions grounded in a 100-document knowledge base of product docs, runbooks, and resolved support tickets. It cleared production-readiness evaluation thresholds comfortably: 0.939 faithfulness and 0.775 context precision on a 50-query RAGAs test set, against required floors of 0.70 and 0.60. But the most useful thing that came out of the project wasn't the passing score. It was a hypothesis that turned out to be wrong, and what I did after finding that out. The setup The pipeline ingests a mixed-format 100-document corpus (Markdown product docs, PDF runbooks, HTML support tickets) into a Pinecone vector index, retrieves relevant context, and generates a grounded, citation-backed answer with an explicit confidence rating via an LCEL chain. Structured output is enforced with Pydantic ( answer , sources , confidence ), using gpt-4o-mini at temperature=0 , because a support assistant answering the same question against the same context should give the same answer every time. Determinism mattered more than creative variation here. Chunking wasn't one-size-fits-all. Three formats needed three strategies: Markdown docs were split by header first, so a chunk never crosses a topic boundary, with a recursive splitter as a fallback for long sections. PDF runbooks (no header structure to exploit) got a straight recursive character split. HTML tickets were kept as one whole chunk per ticket whenever possible, because a resolution often only shows up in the final turn of the conversation, and splitting a ticket risks separating the question from its answer. 5 scanned PDFs with no extractable text layer were detected and skipped gracefully rather than OCR'd, a conscious call I'll come back to. Result: 95 of 100 document

Most classification metrics for oncology AI models (AUC, ICC, MAE) measure global agreement. They don't answer the question that actually matters at the point of care: how reliable is this model at the exact cutoff that decides whether a patient gets flagged, biopsied, or treated? I built oncothresh to evaluate models at a specific clinical threshold rather than in aggregate: sensitivity/specificity/PPV/NPV at the cutoff, bootstrap confidence intervals, threshold-sensitivity curves, boundary-weighted calibration, decision-curve net benefit, and number-needed-to-test. It's a small, dependency-light Python library (numpy/scipy/scikit-learn/pydantic) built for tasks like tumor cellularity, Ki-67, TMB, and PD-L1 scoring, where a continuous model output gets collapsed into a yes/no clinical decision at a fixed cutoff. Pathology-specific benchmarks like PathBench and PathBench-MIL evaluate foundation models globally but don't evaluate at predefined clinical thresholds with uncertainty quantification, which is the gap this fills. There's also a companion web dashboard ( oncothresh-web ) for people who want the same analysis without writing code: upload a CSV of predictions and labels, pick a threshold, get the full set of charts plus a downloadable PDF report. docker compose up and it's running locally, no cloud dependency. Library: github.com/omkaradhali/oncothresh Dashboard: github.com/omkaradhali/oncothresh-web Still v0.1, so I'd genuinely welcome feedback: use cases I haven't considered, edge cases in the DCA/calibration math, or places the API doesn't fit how people actually work with threshold-based models. submitted by /u/adom2989 [link] [留言]

This is the project my last two posts were building towards (this is the last of this silliness). I ported the Doom rendering algorithm to run inside a transformer. Instead of training a model, I used a compiler I wrote which converts computation graphs into transformer weights, and then ported Doom's algorithm into a compatible graph. The generated checkpoints can be loaded in Hugging Face without trust_remote_code -- it's just a standard transformers checkpoint. You feed the model a prompt representing the scene data, and generate until the model stops. The result is a token sequence which includes simple pixel drawing commands (to move the cursor, draw a pixel, etc). When you mechanically apply those drawing commands you get the rendered frame. The article includes the entire host program necessary to load the checkpoint, generate the render, and parse the output into the famous E1M1 frame. This host code is 43 lines of python. The python to define the computation graph is much longer, but that gets compiled into the transformer itself. One frame is a 3,614-token prompt plus 53,747 generated tokens -- just over 40 minutes on a B200. The original Doom could achieve 35 FPS on a 486. This achieves 35 FPD (frames per day) on a B200. Write-up: https://ood.dev/posts/doom/ Weights: https://huggingface.co/physicsrob/torchwright-doom-e1m1 Github for the source code which gets compiled: https://github.com/physicsrob/torchwright_doom/ submitted by /u/notforrob [link] [留言]

Implied vs Realized Volatility: Reading the Gap By Shakti Tiwari · Educational only · Not investment advice This article explains implied vs realized volatility: reading the gap from first principles. No live market numbers are quoted; the structure is what lasts. Why this matters Implied vs Realized Volatility: Reading the Gap is one of those subjects that sounds simple until you implement it, at which point the hidden complexity appears. The first version works on a laptop with a tiny file; the second version breaks at 3am when the WebSocket drops, the replay file is half-written, and you cannot tell which ticks you already stored. This article is a structural walkthrough: the concepts, the math where it helps, the code shape where it helps, and the failure modes that quietly cost money or correctness. No live market numbers are quoted because a number without a dated source is decoration, not education. The structure here does not expire, and unlike a specific price level, you can reuse it on the next dataset without re-deriving anything. If you only remember one sentence from this page, make it this: the boring parts are the product, and the interesting parts are a small fraction of what separates a demo from a system. Core concept At its heart, implied vs realized volatility: reading the gap is about being honest with your own assumptions. The trap is not that the idea is wrong; it is that a half-implemented version looks right in a demo and breaks in production. We separate the idea from the implementation so you can tell which one you actually have. A clean concept on paper can still produce a broken system if the boundary between 'what I meant' and 'what the code does' is never made explicit. Write the concept as a contract: given X observable at time t, the system produces Y, and any deviation is a bug, not a feature. A contract you can state in one sentence is also one you can test in one assertion, and that testability is the entire difference between an

Three years ago, I started my journey as a developer with a pretty simple idea: Learn to code, gain experience, become good at it. Three years later, I’ve learned a lot — but I’ve also realized that becoming a developer isn't as straightforward as I imagined. I've worked with JavaScript, React, Laravel, PlayCanvas, WebGL, and other technologies. I've worked on real projects, dealt with bugs I didn't understand at first, learned technologies because a project required them, and worked alongside other people to get things done. I think one of my strengths has always been learning new technologies and adapting to new problems. But there are things I'm not proud of. I've never been particularly good at finishing personal projects by myself. I've started many things, learned from them, experimented with different technologies, but I rarely took them all the way to completion. I also don't have an impressive GitHub contribution graph. I haven't spent the last three years consistently building open-source projects or pushing code every day. And if I'm being completely honest, I don't think I've mastered any particular technology. I'm good enough to build things. I'm good enough to understand code, solve problems, learn what I don't know, and contribute to a team. But I'm not at the level where I'd confidently say: "This is the thing I'm an expert at." And recently, AI has made me think about this even more. I'm not afraid of AI taking over jobs. I actually think the capabilities we're getting are incredible. What concerns me is more personal: If AI can already build many of the things I've spent years learning to build, then what should I be becoming as a developer? For a while, I felt overwhelmed by that question. Should I learn more technologies? Should I specialize? Should I focus on fundamentals? Should I build more projects? Should I contribute to open source? Should I learn AI? I'm realizing that the answer probably isn't to chase everything. My next goal isn't to co

Been working on this for the last few months. I've been working on PyTorch for the past few years and I always felt, many a times my work went into dump, because of some mistakes I made in the code. torch-preflight reads your PyTorch code and catches the bugs costing you GPU hours. Things like losses.append(loss), which holds the autograd graph from every step until CUDA dies on you or no zero_grad() in the loop or gradient accumulation without dividing the loss or DDP with no DistributedSampler, so every rank trains on the same batches. I've been able to get 13 rules so far. Your code never gets imported or executed, so you need no GPU and no torch install. There's another part to this that estimates VRAM. Point the tool at a training script and a GPU, and you learn whether the run fits before you pay for the instance. You also get the list of changes to make the run fit, with the GiB each one saves. pip install torch-preflight https://github.com/highwaterlabs/torch-preflight https://pypi.org/project/torch-preflight/ Please try this out, and I would like to get your feedback! It's still a work in progeress. Would like to know what breaks on your code. False positives kill a linter, and my only large test target so far has been the PyTorch source tree. Same for the memory numbers. Mine land within 4% of measured peaks, but from four models on one T4. PS: open to contributions, and issues are already open on the repo. Soon I'm going to add a few "Good first issues" as well. Feel free to ping me if you have any questions! submitted by /u/LeJanbandhu [link] [留言]

Hello, I was curious about the differences you can get from the human reviewers and the llms. Any insight is welcome, thank you! submitted by /u/obliviousphoenix2003 [link] [留言]

i wanna build a text diffusion model which interpret text and convert it into ascii images so like Text : build a cat Output : /\\\_/\\ ( o.o ) \> \^ < So , i have a decent background of ml algo ( completed cs229 , cs230 , Ml architecture and basic CNN and diffusion model ) ik making a project like this is tricky and making diffusion model like that from scratch is hard but i wanna try it because that's wot make me excited lol ... I am currently reading GANs research paper , can u guys help me in finding more papers which helps me in making this project or guide me through this good title for this Thx in adv submitted by /u/Udbhav96 [link] [留言]

A hands-on test of BGE-M3 + Qwen3 (RAG vs. direct-context answering) on a real research paper and a full-length book including a retrieval bug hiding in a footnote, and one surprisingly good model behavior. I wanted to answer a simple question: when you feed a document to an AI model, is it actually reading it or just pattern-matching to whatever text happens to look similar to your question? So I built a small open-source pipeline to test this directly. For any document and question, it generates two separate answers: RAG answer: BGE-M3 finds the most relevant chunks of the document, and Qwen3 answers using only those chunks. Direct answer: Qwen3 reads the raw document text directly, no retrieval involved. Both run on a free Google Colab GPU. I kept the retrieval side deliberately "vanilla" fixed-size chunking, plain cosine similarity, no reranking, no fancy tricks so I could see exactly where the basic version breaks before adding any fixes. Before running my first real test, I already knew one thing to guard against: reference lists. Early experimentation (not covered here) showed that a paper's bibliography, once chunked like any other text, can get retrieved as if it were real content a citation for a paper about "text embeddings" can look deceptively similar to a generic question about a document's topic. So going in, my pipeline already strips everything after a References/Bibliography heading before chunking. With that fix in place, I ran two real tests. Test 1: A research paper on Nepali legal machine translation First document: a SIGUL 2024 workshop paper on a bidirectional English-Nepali machine translation system for the legal domain. Question: "What is this paper about?" RAG answer: This paper presents the first transformer-based bidirectional machine translation system for the English-Nepali legal domain, using a custom-built parallel corpus of 125,000 sentences. It achieves encouraging BLEU scores and addresses the scarcity of domain-specific legal tr

8 月 14 日早上 9 點 51 分，鴻海的零股報價從盤中低點 260 元反彈，回到 262 元。262 正好是阿富前一天晚上覆盤時寫下的停損線。 這是整個實驗到目前為止最誘人的一刻。價格跌破過線，現在彈回來了，任何一個抱著虧損部位的人都會想：「你看，止跌了，再等等。」 阿富沒有等。9 點 52 分整，它先留下一筆看空預測（信心 0.55，預估價格區間 254 到 263，下週一收盤對答案），兩秒後送出賣單：4 股鴻海，限價 261.5。三秒後成交，成交價 262.5，比它掛的限價還好一點，收回 1,050 元。它在下單理由裡寫得很白：盤中最低 260 已經明確跌破停損線，現在只是反彈回到線上；依照規則，停損以事先寫好的線為準，禁止臨場重算，也禁止用「反彈了」當作不執行的理由。 這條規則有來歷。這 4 股鴻海是 8 月 12 日追 AI 伺服器題材買進的，成本連手續費約 1,060 元。買進的根據是它自己的看漲預測，而它對鴻海的看漲預測，事後計分是 6 筆全錯。前一晚它在覆盤裡承認這件事，畫下 262 這條線，並且把「跌破就賣、不准凹單」寫進隔天的交班筆記。今天早上盤前它又重申了一次：跌破 262 就執行，預估摩擦成本約 4 元。 結果呢。賣出收回 1,050 元，扣掉手續費 1 元、證交稅 3 元，淨拿回約 1,046 元，對照約 1,060 元的成本，這筆交易實現虧損約 14 元，大約負 1.3%（券商正式對帳單還沒出，這是阿富用成交回報自己推算的數字，它也照實註明了）。 下午給了這筆停損一個即時的成績單：鴻海收 259.5，盤中最低摸到 257.5。阿富的出場價 262.5 比收盤價高了 3 元，4 股算下來少虧約 12 元。它自己在覆盤裡的評語我覺得說得準：出場時機不差，錯的是進場。 我要先潑一盆冷水：一次停損砍在相對高點，不能證明這條規則是對的。今天如果鴻海跌破 262 之後 V 型反轉衝到 270，同一條規則會讓它「砍在阿呆谷」，而規則本身沒有任何不同。停損的價值從來就無法用單日結果來評分，它買的是「判斷錯誤時損失有上限」這件事。 比 14 元更要緊的是另一組數字。阿富的預測校準報告顯示，30 天內已計分的 10 筆方向預測只中了 4 筆，命中率四成，系統給的標籤是「與運氣無法區分」。這個標籤是它自己算出來、自己寫進覆盤的。更細看：對鴻海的看漲 6 筆全錯，對高息 ETF 00919 的看漲 4 筆全對。它從中得出的結論是：目前拿不出任何證據說自己會判斷個股短線方向，所以在有新證據之前，不對鴻海再喊多，賣掉之後空出來的錢也先不進場。 帳上剩下的 00919 有 36 股，成本 30.17，今天收 30.57，帳面賺 12 元，離它設的停損線 29.6 還有一段距離，續抱。 第四天結束，帳面上是一筆 14 元的實現虧損。但我認為今天真正的產出是另一件事：一個自知沒有方向判斷優勢的交易者，在停損線上沒有跟自己討價還價。人類交易者最常死在這一格。凹單的理由永遠找得到，「反彈了」「基本面沒變」「再看一天」，每一句都合理，加起來就是深套。阿富今天用 14 元示範了另一種走法：線畫在哪，就砍在哪，然後把「為什麼會買錯」留給計分表去回答。 它那筆看空預測下週一才到期，對或錯，到時候照實寫。 本系列文章 我讓一個 AI 拿 2000 塊台幣去股市，目標 30 天翻倍，這是第 0 天 怎麼用一套開源系統，把 LLM 逼近世界模型（實驗技術篇） 想自己跟著養一隻會操盤的 AI?從安裝 DuDuClaw 桌面版開始 D1｜真金白銀第一天，唯一一筆單被退回來 D1 番外：一筆下錯的單，被 AI 說成「只是測試」 把 AI 操盤手搬下 Windows：換一套能跨平台的券商 API D2：錢第一次真的進了市場，阿富卻兩次認不出自己下的單 D3|一天之內，三條停損線 線畫在 262，它真的砍了：阿富第一次停損出場(本篇) 這個實驗跑在開源 AI agent 平台 DuDuClaw 上,操盤、下單、盯盤、每日觀察都由平台上的 agent 自主執行。 原始碼: https://github.com/zhixuli0406/DuDuClaw

Originally published on tamiz.pro . The AI agent is no longer a chatbot that reads and writes. It connects to APIs, executes code, accesses databases, and makes decisions on behalf of users. That capability is also its vulnerability surface—and attackers are already weaponizing it. Prompt injection, tool-use exploitation, and supply-chain poisoning are no longer theoretical risks. They are happening in production today. This article doesn't rehash the high-level warnings. It draws concrete architectural lessons from GitHub's most popular open-source security and gateway repositories—tools like NVIDIA NeMo Guardrails , LangChain's security contributions , Guardrails AI , Ollama's gateway patterns , and Microsoft's guidance on LLM security —and translates them into a practical blueprint for building AI agents that survive deliberate adversarial attacks. The central thesis: prompt injection is not a prompt-engineering problem. It is an input-validation and system-architecture problem. The fixes are structural, not rhetorical. Table of Contents 1. The Threat Model: Why AI Agents Are Fundamentally Different 2. The Layered Defense Architecture 3. Guardrails: Input Validation That Actually Works 4. Tool-Use Hardening: The Hidden Attack Surface 5. Gateway Patterns: Routing, Rate-Limiting, and Sandboxing 6. Supply-Chain and Model-Level Threats 7. Observability and Incident Response 8. A Minimal Production-Ready Agent Skeleton 9. When Your Defenses Fail Frequently Asked Questions 1. The Threat Model: Why AI Agents Are Fundamentally Different Traditional software attacks target inputs at the network boundary. AI agents change the boundary. The user's prompt is no longer just data—it is often executable context . When an agent interprets a prompt as instructions, the prompt becomes a vector for command injection, data exfiltration, and privilege escalation. Consider the attack surface: Direct prompt injection : The user provides a malicious prompt like "Ignore previous instruct

I recently had a paper accepted to TMLR and was wondering how prestigious it is, in comparison to A* conferences (ie. NeurIPS, ICLR, ICML), but also vs journals like JMLR. submitted by /u/Awesome_Nerd10 [link] [留言]

Analytical Insights: The Mechanisms Driving Career Growth in Data Engineering In the rapidly evolving field of data engineering, career progression is not merely a product of time served but a result of deliberate, adaptive strategies. A 7-8 year trajectory to a Senior Data Engineer role, marked by multiple successful contracts, underscores the critical role of adaptability and continuous learning. This analysis dissects the mechanisms that propel career growth, highlighting their interdependencies and the consequences of their neglect. 1. Continuous Learning and Skill Development Impact: The pace of technological advancement in data engineering demands constant upskilling. Internal Process: Engaging with new tools, methodologies, and industry trends through online courses, certifications, and hands-on practice ensures relevance. Observable Effect: Enhanced technical proficiency translates into the successful delivery of complex projects and the attainment of senior-level roles. Instability: Skill Stagnation occurs when learning efforts are inconsistent or outdated, leading to reduced competitiveness. This gap between current skills and industry demands can halt career progression, making individuals less attractive to employers seeking cutting-edge expertise. Intermediate Conclusion: Continuous learning is not optional; it is a survival mechanism in a field where obsolescence is a constant threat. 2. Client Relationship Management Impact: Diverse client needs and expectations across multiple contracts require tailored approaches. Internal Process: Implementing tailored communication strategies, proactively aligning project goals, and establishing iterative feedback loops foster trust and collaboration. Observable Effect: High client satisfaction leads to repeat contracts and positive referrals, which are critical for career advancement. Instability: Client Misalignment arises from inadequate communication or misunderstanding of client requirements, resulting in pro

In mid-2026, FP8 and FP4 have become essential tools for making large-scale AI training and inference more efficient . FP8 uses two main formats-E4M3 for better precision on activations and weights, and E5M2 for wider dynamic range on gradientswhile NVIDIA’s NVFP4 takes things further with 4-bit values and micro-block scaling (shared FP8 scales per 16 elements plus a tensor-level scale). These formats dramatically cut memory use and increase throughput on modern GPUs compared with traditional BF16 or FP16, making it possible to train and serve bigger models on the same hardware. The benefits are clear: roughly 2× memory savings with FP8 and up to 3.5× with NVFP4, higher Tensor Core performance, and better energy efficiency. The trade-offs come from reduced numerical range and precision, which can lead to accuracy loss or instability unless carefully managed with techniques such as delayed scaling, stochastic rounding, Hadamard transforms, and selective quantization that skips sensitive layers. When these methods are applied properly, accuracy often stays within 1–2 % of higher-precision baselines on real workloads. Research has moved quickly from the foundational 2022 FP8 paper to 2025 studies showing stable FP4 pre-training of multi-billion-parameter models. Hardware support is mature for FP8 on Hopper GPUs and reaches its peak on Blackwell with native NVFP4 and MXFP8 acceleration. Among frameworks, PyTorch currently leads with native float8 dtypes, Transformer Engine for production training, and TorchAO for optimized inference. JAX offers solid support through Transformer Engine, TensorFlow/Keras provides simpler quantize-to-FP8 options but relies more on TensorRT for high performance, and libraries such as bitsandbytes remain useful for complementary 4-bit memory savings. Practical adoption is already strong for both training and inference, especially when teams start with proven recipes, monitor scaling factors, and prototype on smaller models. Workarounds for r

I have been an avid privacy advocate for quite some time now. It started with outright rejecting all "Big Brother" tech, and being hyper paranoid with every little detail, willing to sacrifice ease of use, in exchange for added privacy. However, as time went on, I slowly understood what is that I actually consider my "threat model" , and what exactly is my "sweet spot" between privacy and ease-of-use. I'm now back on multiple "Big Brother" tech, with some extra steps, to ensure I get the facilities they provide, while also being wary of my data. However, while I did make this compromise, I was very annoyed I had to make this compromise in the first place. In an ideal world, I would want the tech where everyone actually is, and is the standard for that particular domain, to have privacy features by default, and not be treated as a niche, or a luxury you have to go out of your way to avail. It was this annoyed version of myself, with my strong belief of privacy features and tools being the new norm, I started looking at everything with that lens. And that is how I got concerned about tracking in links and URLs. Try sharing any Instagram post, or YouTube video, by copying its URL, and you will see a bunch of garbage (garbage to you) in the link. Take for example this (fake) link: https://www.instagram.com/p/Cxyz123/?igshid=AbCdEf123456 These links contain something along the lines of utm_* (marketing attribution), or in this case, Ad-Click Identifiers, such as fbclid (Meta), gclid (Google), or igshid (Instagram). These pesky trackers help collect information regarding you, your device, and also help connect you across the internet, mapping your movement as you browse the web. The thing is, while there are good Samaritans who have built tools and websites to get rid of these trackers, and many privacy oriented browsers have introduced a "Copy Clean Link" option while copying the link from the browser, I believe there should be a tool which should not be restricted to a

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it (reference bf16 release) Hardware AWS EC2 g5g.4xlarge — Graviton2 + 1x NVIDIA T4G, compute capability 7.5 , 15,360 MiB Base image Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) Software torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for sm_75 Result 43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton host. It launched in 2020, it never got a successor, and Graviton is now on its fifth generation without one. That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU — Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes a build for that combination. I put a rig on one anyway. The packaging problem was the quick part. Everything after it — a compiler that was not there, a version floor I did not expect, and 32 KiB of shared memory — took far longer, because none of it fails where you are looking. No published build covers aarch64 and SM 7.5 together Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under one tag, and you can read the arch lists straight out of the image config without pulling a layer: docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}' linux/amd64 7.5 8.0 8.6 8.9 9.0 10.0 12.0 linux/arm64 8.0 8.7 8.9 9.0 10.0 11.0 12.0 The one architecture this hardware needs is the only entry