AMD Ships 16B Open MoE Trained Entirely on Instinct GPUs

AMD just put a 16B-parameter Mixture-of-Experts model on the board with a base average of 76.7, half a point ahead of Moonlight-16B-A3B at 76.2 and roughly 6.6 points above OLMo-3-7B at 70.1. The interesting number isn't the leaderboard delta though. It's that the entire training run happened on Instinct MI300X and MI325X silicon, with zero NVIDIA hardware in the loop. For a company that has spent two years arguing ROCm can do serious pre-training, this is the receipts.

What Happened

AMD released Instella-MoE-16B-A3B, a decoder-only Mixture-of-Experts model with 16B total parameters and 2.8B active per token. As MarkTechPost reported, the release includes weights from every training stage, the data mixtures, training configs, and inference code shipped through SGLang. The training codebase is MIT licensed. The weights themselves ship under ResearchRAIL, restricted to academic and research purposes.

The training pipeline is a full-fat modern MoE recipe. Pre-training covers 7.1T tokens drawn from Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360. Mid-training uses Dolma3 Dolmino 100B across three data variants merged by weight averaging. A dedicated long-context stage stretches the window from 4K to 64K using YaRN with an increased RoPE theta and document masking. Post-training runs SFT on Dolci-Think-SFT-7B plus Nemotron mixtures, then a feedback-driven 512K-example set, followed by DPO with router bias updates and the auxiliary load-balancing loss disabled. RL runs in the Miles framework for 1,400 steps of instruction-following RLVR, closed out by Multi-Teacher On-Policy Distillation.

The base checkpoint's 76.7 average leads every fully open peer AMD benchmarked against: Moonlight-16B-A3B at 76.2, SmolLM3-3B-Base at 70.5, OLMo-3-7B at 70.1, and OLMoE-1B-7B at 61.9. It still trails Qwen3.5-4B-Base at 79.5. On the post-training side, the Think checkpoint hits 73.22, above Olmo3-7B-Think (71.97), Gemma-4-E4B think (70.47), and Qwen3.5-4B (69.73). IFEval climbs from 77.08 to 83.70 across the post-training stack. What the source does not disclose is total GPU-hours or dollar cost of the 7.1T-token run, which matters because the entire "AMD as a training platform" thesis lives or dies on that ratio.

Technical Anatomy

The architecture itself is standard MoE with two custom twists. 27 layers, hidden size 2048, 16 attention heads, and a 128,896-token vocabulary. Each MoE layer holds 2 shared experts plus 6 routed experts selected from a pool of 64. That's how you get 2.8B active against 16B total. A Multi-Token Prediction objective runs during pre-training and mid-training, which is now table stakes since DeepSeek popularized it.

The two things worth actually studying are Gated MLA and FarSkip-Collective. Gated MLA bolts a lightweight learned output gate onto Multi-head Latent Attention: a dedicated linear projection derives an input-conditioned gate that multiplies the attention output before the final projection. Cheap parameters, meaningful expressivity gain. FarSkip-Collective is the more interesting piece from an infrastructure standpoint. It routes outdated and partial activations into the MoE and attention layers, which lets expert-parallel all-to-all communication overlap with computation instead of stalling on the critical path.

The numbers AMD reports on these: 12.7% pre-training speedup from FarSkip-Collective and up to 39.2% reduction in time to first token when serving with expert parallelism. Both figures come from AMD's own instrumentation, so I'd want to see third-party reproduction before treating them as portable to non-Instinct hardware. The 39.2% TTFT figure in particular is bounded by the specific expert-parallel serving topology, and the source doesn't specify GPU count or batch size for that measurement.

On the serving side, 16B parameters in BF16 works out to roughly 32 GB of weight memory. That fits comfortably on a single MI300X (192 GB HBM3) or a single H100 80GB. The SGLang code path is what makes this actually runnable outside AMD's own data center. Long-context evaluation lands at 41.5 on HELMET and 79.4 on RULER, with the RULER number strong enough to suggest the YaRN extension held up through post-training rather than collapsing back to short-context behavior. If the FarSkip claims replicate in independent benchmarks, we should see at least one non-AMD lab publishing a comparable MoE with the same connectivity pattern by Q1 2027.

Who Gets Burned

The obvious loser here is anyone building a hosted commercial product on top of Instella-MoE weights. ResearchRAIL blocks that entirely. The MIT-licensed training codebase is the real payload for commercial teams: a documented, reproducible MoE recipe that runs on non-NVIDIA silicon. That matters most for three groups.

First, mid-tier cloud providers who bought Instinct capacity on the promise that ROCm would eventually catch up. Until now, most public evidence of production-grade AMD training was fine-tuning runs or inference deployments. A 7.1T-token pre-training run from scratch, with every stage documented, gives their sales teams something concrete to point at. Whether this translates to actual booking velocity depends on price per token trained, which the source does not disclose.

Second, the fully-open-weights competitive set. Moonlight, OLMo, and SmolLM teams now have a 76.7 baseline to clear on their next release. The 0.5-point gap between Instella-MoE (76.7) and Moonlight-16B-A3B (76.2) is small enough that a single mid-training tweak could close it, but the 6.6-point gap over OLMo-3-7B is structural and reflects the MoE architecture advantage per active parameter.

Third, closed-model vendors serving research budgets. Any lab that was paying for Qwen or Gemini API access purely for academic experiments now has a fully reproducible alternative with weights at every checkpoint, which is something no closed vendor offers. The catch: the ResearchRAIL license means the moment that research turns into a product, teams need to re-train or license differently. That's a real switching cost that closed vendors will exploit in enterprise sales.

For iGaming, fintech, and ad-tech shops looking at this as a base for domain fine-tuning, the license kills it as a production candidate. Track the training code instead. If this plays out, we should see at least three non-AMD forks of the Primus and Miles frameworks on GitHub within 60 days.

Playbook for AI Development

For engineering leads evaluating this release, three concrete moves this week.

One, pull the MIT-licensed training code and read the FarSkip-Collective implementation. Whether or not you ever train on Instinct silicon, the pattern of overlapping expert-parallel communication with computation using stale activations is portable to any MoE training stack. If the 12.7% speedup replicates on H100 clusters, that's real money on a 7T-token run.

Two, benchmark the released weights on your domain evals under a research license before assuming you can't ship it. Some enterprise R&D use cases (internal knowledge retrieval, code review tooling that never leaves the org) may fit within ResearchRAIL's boundaries depending on legal interpretation. Get counsel to read the actual license text rather than assuming ResearchRAIL means the same thing across releases. It doesn't.

Three, if you're serving MoE models in production, test the SGLang inference path against your current stack. The 39.2% TTFT reduction claim is worth an afternoon of reproduction work even if you don't adopt the model itself, because SGLang's expert-parallel serving improvements would apply to any MoE you're already running. For teams comparing infrastructure options, the Hugging Face collection is the fastest path to weights and tokenizer configs.

The unanswered question I'd flag: AMD published every training stage's weights but did not disclose the wall-clock time or GPU-hour count for the 7.1T-token pre-training. Until that number is public, the cost-competitiveness claim against NVIDIA-based training remains unfalsifiable. Bound: if AMD's per-token training cost is within 15% of an equivalent H100 run, we should see at least one major foundation-model lab announce Instinct capacity by end of 2026.

Key Takeaways

• Instella-MoE-16B-A3B posts a 76.7 base average, leading fully open peers Moonlight (76.2), SmolLM3-3B (70.5), and OLMo-3-7B (70.1), while trailing Qwen3.5-4B-Base at 79.5.
• The full 7.1T-token pre-training ran on AMD MI300X and MI325X, with ROCm, Primus, and Miles frameworks. Zero NVIDIA in the training loop.
• FarSkip-Collective delivers a reported 12.7% pre-training speedup and up to 39.2% TTFT reduction at serve time under expert parallelism.
• ResearchRAIL license blocks commercial weight deployment. The MIT-licensed training code is the reusable asset for production teams.
• Post-training IFEval climbs from 77.08 to 83.70, and the Think checkpoint at 73.22 beats Olmo3-7B-Think (71.97) and Qwen3.5-4B (69.73).