H64LM: A 249M-parameter Mixture-of-Experts Transformer built from scratch in PyTorch [P]

Hi everyone, I built H64LM, a research project to better understand modern LLMs by implementing one from scratch in PyTorch. Instead of relying on high-level training frameworks, I implemented the core components myself attention, MoE routing, normalization, and the training loop. Features 249M-parameter Transformer Grouped Query Attention (GQA) Sparse Mixture-of-Experts (8 experts, Top-2 routing) with 3 auxiliary routing losses SwiGLU, RoPE, RMSNorm Sliding-window attention Mixed-precision training, gradient accumulation Custom training loop (no Trainer abstractions) Checkpointing and resume support The included checkpoint was trained on a subset of WikiText-103 to validate the pipeline end-to-end, not to be a strong model it's visibly overfit past epoch 10 (best val PPL ~40.5). Known limitations are documented in the README, including batch-size-1-only generation and no true DDP (falls back to DataParallel). GitHub: https://github.com/Haiderkhan64/H64LM Feedback on the implementation or architecture is very welcome. submitted by /u/Loose_Literature6090 [link] [留言]