MoE vs Edge AI: They Are Not the Same Thing

Recently, I came across an X demo of a 35B language model running on an iPhone using only 1–2.5 GB of peak memory. That sounds strange at first. A 35B model sounds like something that should need a lot of RAM and a powerful GPU. The project behind the demo, Edge0 , uses a technique called Mixture of Experts (MoE) to make this possible. This also made me realise that two terms can easily get mixed up: MoE (Mixture of Experts) Edge computing / Edge AI They are related, but they solve different problems. This article explains both in simple terms. First: What is MoE? MoE stands for Mixture of Experts. An MoE model is a neural network that contains multiple smaller networks called experts. Instead of using every expert for every input, a router decides which experts should handle the current input. For example: Input | v +-------+ | Router| +---+---+ | +-------+-------+ | | v v Expert 3 Expert 8 | | +-------+-------+ | v Output The important part is: Not every expert is used for every token. This is called sparse activation. An office analogy Imagine a company with 20 employees. There are: 5 software engineers 3 accountants 2 lawyers 4 designers 3 researchers 3 sales people Now imagine a customer walks into the office and asks: "Can you help me with my tax return?" You don't call all 20 employees into a meeting. You send the customer to the accountants. The accountants do the work and the rest of the employees continue doing their own jobs. The receptionist is the router. The employees are the experts. Customer | v Receptionist / Router | +----> Accountant 1 | +----> Accountant 2 | +----> Accountant 3 That is roughly the idea behind MoE. The model has many experts, but only some are activated for a particular token. Why do this? Because it lets us build models with a very large total number of parameters without having to use all of those parameters for every token. Suppose we have: 8 experts x 7B parameters each 56B parameters The model could have around 56B total para