标签：#gemma

This article is about running a hand-written Gemma 4 port in pure JAX on three different accelerators, and about the two places the abstraction leaks. The code is here: github.com/xbill9/gemma4-dev What is this project trying to Do? This project aims to serve one Gemma 4 checkpoint from one JAX port across every accelerator I can rent, and to find out — by measurement, not by reading docs — which parts of "it's just JAX" are true. The port lives in ports/gemma4/ and is driven by a generation loop behind an OpenAI-compatible server. No PyTorch, no vLLM, no torch_xla . The same code runs on Cloud TPU v5e and v6e, and on an NVIDIA T4G attached to an AWS Graviton2 host. "Pure JAX" is the whole experiment. If the port is really portable, the only thing that should change between those rigs is a config file. It mostly is. Two things are not, and they are the interesting part. Gemma 4 E2B is not a stock transformer Any port has to carry four irregularities, and none of them are optional: Two attention geometries. Sliding layers use head_dim=256 , global layers use 512 . Most inference stacks assume one head dimension per model. 8:1 MQA , so the KV budget is nothing like the parameter count would suggest. A KV-share map that collapses 35 layers onto 15 caches . A 512-slot sliding ring , plus per-layer embeddings (PLE) held in a 4.70 GB table that gets quantized to 4 bits on load. That first one is worth dwelling on, because it is what breaks other stacks. On the vLLM path, the heterogeneous head dims force the Triton attention backend: Gemma4 model has heterogeneous head dimensions {'sliding_attention': 256, 'full_attention': 512}. FA4 not available, forcing TRITON_ATTN backend. And on a Turing GPU that backend then asks for shared memory the hardware does not have: triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 98304, Hardware limit: 65536 JAX never enters that conversation. Attention is ordinary XLA rather than a hand-tiled kernel, so ther

I'd been waiting for more than 30 minutes. The terminal just sat there, blinking, without returning a single word. I'd launched Gemma2 in its 9-billion-parameter version on my laptop (a regular Mac, the kind any professor or student would use) and the model simply wasn't responding. It wasn't a bug. It was the most honest answer the experiment could have given me. That frustrating wait ended up being, without exaggeration, the most interesting finding of the whole process. Because the question that brought me there wasn't "how big can a model get?" — it was a much more practical one: what actually happens when an agent you built in a tutorial has to survive in production? I've been working with Gemma as a case study to understand that jump — from an educational prototype to something that can hold up under long conversations, limited hardware, and real users. This post is the honest summary of that process: what worked convincingly, what didn't work the way I expected, and why that "didn't work" turned out to be more useful than a clean result would have been. The real problem: why tutorials are a little dishonest Almost every conversational agent tutorial does the same thing, without saying so out loud: on every turn, it sends the model the entire previous history, all over again. Imagine that every time you added a sentence to a conversation, you had to repeat everything said before it — every message, every reply — before you could say the new one. At first you don't notice. But if the conversation runs 30 or 50 turns, you're repeating an entire novel just to add one sentence. This pattern is called linear context stacking , and it causes three concrete problems: Memory saturation — every call to the model processes an increasingly large context. Risk of hitting the token limit — every model has a maximum context window; sooner or later, you hit it. Quality degradation — there's a documented phenomenon in NLP literature called "lost in the middle" : when context

Take control of your AI models with our newest skill, designed to make local fine-tuning efficient.

There’s a specific moment that happens at every single hackathon. It’s usually around 2 or 3 a.m.,...

As the AI Engineer World’s Fair kicked off officially on Monday, the halls were filled for the...

We are so excited to announce the winners of the Gemma 4 Challenge! This is officially our most...

There’s something uniquely beautiful about old books. The smell of weathered paper, the texture of...

Grab a refreshing slice of cold watermelon (because it’s summer in Tokyo and I love it!), and let me...

google--gemma-4-12B-it-Q4_K_M.gguf baxin/quantized-models at main We’re on a journey to advance and democratize artificial intelligence through open source and open science. huggingface.co gemma-4-12B-it-qat-UD-Q4_K_XL.gguf unsloth/gemma-4-12B-it-qat-GGUF · Hugging Face We’re on a journey to advance and democratize artificial intelligence through open source and open science. huggingface.co prompt: hello [ Prompt: 21.0 t/s | Generation: 10.6 t/s ] [ Prompt: 19.5 t/s | Generation: 5.0 t/s ] [ Prompt: 25.4 t/s | Generation: 17.6 t/s ] prompt: write fizzbuzz in typescript [ Prompt: 23.1 t/s | Generation: 9.2 t/s ] [ Prompt: 25.0 t/s | Generation: 10.6 t/s ] [ Prompt: 32.2 t/s | Generation: 11.3 t/s ] Without MTP ▄▄ ▄▄ ██ ██ ██ ██ ▀▀█▄ ███▄███▄ ▀▀█▄ ▄████ ████▄ ████▄ ██ ██ ▄█▀██ ██ ██ ██ ▄█▀██ ██ ██ ██ ██ ██ ██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀ ██ ██ ▀▀ ▀▀ build : b9536-308f61c31 model : google--gemma-4-12B-it-Q4_K_M.gguf modalities : text available commands: /exit or Ctrl+C stop or exit /regen regenerate the last response /clear clear the chat history /read add a text file /glob add text files using globbing pattern hello [Start thinking] The user said "hello". The user is initiating a conversation. A friendly, helpful, and polite greeting is required. * Option 1: "Hello! How can I help you today?" (Standard, polite) * Option 2: "Hi there! What's on your mind?" (Friendly) * Option 3: "Hello! Is there anything specific you'd like to talk about or any questions I can answer?" (Informative) Keep it simple and inviting. [End thinking] Hello! How can I help you today? [ Prompt: 21.0 t/s | Generation: 10.6 t/s ] write fizzbuzz in typescript [Start thinking] The user wants a FizzBuzz implementation in TypeScript. * Iterate from 1 to a given number (usually 100). * If a number is divisible by 3, print "Fizz". * If a number is divisible by 5, print "Buzz". * If a number is divisible by both 3 and 5, print "FizzBuzz". * Otherwise, print the number itself. * *Approach

Google says Gemma 4 12B is "designed to bring agentic, multimodal intelligence directly to your laptop", further noting that the new model can be combined with Google AI Edge to "build and experiment locally, on everyday machines". This integration allows for a wide range of capabilities, from autonomous data processing to generating visual insights and even building webpages or executing tools. By Sergio De Simone

I'm starting a new project. It's the most ambitious thing I've attempted from a phone. The goal: an AI agent that controls a smartphone. It opens apps, navigates screens, taps buttons, types text, and completes multi-step tasks. All offline. All local. No cloud. This is Day 1 of a public build log. No fluff. Just what I'm building, how it works, and what breaks along the way. What I'm Building An autonomous AI agent that runs entirely on an Android phone. You give it a command in plain English: · "Open WhatsApp and message Mom I'll call later." · "Search for Kotlin jobs on Wellfound." · "Open my notes and summarize what I wrote yesterday." The agent parses the command, plans the steps, and executes them—opening apps, finding the right buttons, typing text, hitting send. No cloud. No API keys. Just a phone that acts on your behalf. The Stack Component Tool AI Brain Gemma 4 E4B (local, via Ollama) Runtime Termux (Linux on Android) Phone Control ADB + UI Automator Orchestration Python Why This Matters Most AI agents live in the cloud. They need internet, APIs, and someone else's server. A local agent that runs on a phone means: · Privacy: your data never leaves your device. · Offline: works even without internet. · Accessible: built for the device billions of people already own. The Hard Parts I Already See · The agent needs to "see" the screen to know where to tap. Text detection is doable. Image-based buttons are harder. · Multi-step tasks need verification. If one tap misses, the whole chain fails. · Android permissions. ADB requires developer mode. A user-facing version would need a workaround. What's Next · Day 2: Create the repo. Set up the project structure. Push the first working script. · Day 3: Get screen text detection working with OCR. · Day 4: Test a full 3-step task. This is Day 1. The repo goes live tomorrow. Follow along if you want to see something rare get built from scratch.

1. update WSL environment sudo apt update && sudo apt upgrade -y 2. install dependencies If you don't use -hf option, you don't need to install libssl-dev in this step. sudo apt install build-essential cmake git libssl-dev -y If nvidia-smi shows a GPU/GPUs on your terminal, you will need to install the tooklit. This will take some time. sudo apt install nvidia-cuda-toolkit -y 3. clone the repo Build llama-cli and llama-server. This step also will take some time. If you don't plan to use -hf option, you don't need to use -DLLAMA_OPENSSL=ON . git clone https://github.com/ggerganov/llama.cpp cd llama.cpp cmake -B build -DGGML_CUDA = ON -DLLAMA_OPENSSL = ON cmake --build build --config Release # no GPU git clone https://github.com/ggerganov/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release 4. run the model Run gemma-4-12b-it with cli and server. unsloth/gemma-4-12b-it-GGUF · Hugging Face We’re on a journey to advance and democratize artificial intelligence through open source and open science. huggingface.co ./build/bin/llama-cli -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL > hello [ Start thinking] The user said "hello" . The user is initiating a conversation. Respond politely and offer assistance. * "Hello! How can I help you today?" * "Hi there! What's on your mind?" * "Hello! Is there anything I can assist you with?" [ End thinking] Hello! How can I help you today? [ Prompt: 19.5 t/s | Generation: 11.8 t/s ] or run web-ui ./build/bin/llama-server -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL --port 8080 optional download model from huggingface mkdir -p models wget -O models/gemma-4-12b-it-UD-Q4_K_XL.gguf https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/resolve/main/gemma-4-12b-it-UD-Q4_K_XL.gguf

LiteRT-LM brings native support for Gemma 4 Multi-Token Prediction (MTP) drafters, enabling up to 2.2x faster inference. The framework is expanding beyond Kotlin and C++ adding support for new Swift and a JavaScript APIs. By Sergio De Simone

Gemma , a family of open models, are lightweight, remarkably capable, and have a wonderful "tunability" that makes them perfect for personal projects and enterprise-grade applications alike. But as the ecosystem grew, I found myself asking the same questions over and over: Which exact model size fits my constraint? How do I build an application powered by Gemma that does XYZ? How to deploy a Gemma model to production on Google Cloud for my team to use? To solve this, we put together a living repository called gemma-skills (which we're releasing!). It's a curated, structured collection of developer skills designed to help both humans and agentic AI assistants build beautiful applications with Gemma models without the friction. Let's take a walk through what's inside! The Heart of the Repo: gemma-dev At the center of the repository is our first major skill: gemma-dev . It's a skill file ( SKILL.md ) that serves as a blueprint. It's designed for agents to find what are the latest capabilities, model sizes, good practices, and resources to build with Gemma. Keeping Pace with Rapid Ecosystem Evolution The Gemma ecosystem moves fast, with new models, libraries, and best practices emerging constantly. For developers using foundational LLMs like Gemini, keeping assistant workflows perfectly synced with these rapid releases is a common challenge. Because foundational models are trained on vast, fixed datasets, they don't automatically inherit the day-one nuances of a rapidly evolving framework. This can manifest in a few typical development scenarios: Navigating Version Transitions: General-purpose assistants may default to established standards (like Gemma 2 or 3) even when your project is ready to leverage the latest capabilities of Gemma 4. Aligning with Modern Libraries : Recommendations might occasionally lean toward older API patterns rather than the latest optimized packages. Integrating Next-Gen Features: Cutting-edge implementation details (e.g. Multi-Token Predicti

Hey all, we have a quick update for everyone who participated in the Gemma 4 Challenge. We were...

This is a submission for the Gemma 4 Challenge: Build with Gemma 4 What I Built I built...

This is a submission for the Gemma 4 Challenge: Build with Gemma 4 Every time you paste sensitive...