Reproduce the practical failure modes of sparse MoE training, especially expert-utilization collapse, under a controlled matched-FLOP benchmark.

• Use sparse top-k routing with k=2 and 8 experts.
• Match active compute against dense baselines.
• Log routing entropy and expert load over time.
• Keep runs reproducible through W&B artifacts.

• Sparse MoE layer with Switch Transformer-style z-loss.
• 125M active parameters in a 1B-compute-equivalent model.
• FSDP training with expert routing traces.
• Dense matched-FLOP baseline for throughput comparison.

• Expert imbalance under insufficient z-loss.
• All-to-all communication sensitivity in distributed settings.
• Router instability early in training.
• Underutilization when top-k probabilities collapse.

• Sweep z-loss coefficient thresholds.
• Plot routing entropy alongside throughput and validation loss.
• Separate active parameter count from total parameter count in all reporting.
• Open-source run configs and logs.

• Compared sparse model throughput against dense matched-FLOP baseline.
• Logged expert load histograms across training.
• Measured routing entropy decay and collapse thresholds.
• Repeated runs under multiple z-loss settings.

• Reached 2.3x throughput versus dense baseline at matched FLOP budget.
• Identified z-loss thresholds that prevent expert utilization collapse.
• Produced reusable routing entropy curves for diagnostics.

• MoE throughput wins are inseparable from router health.
• Routing entropy should be a first-class training metric.
• Load-balancing losses can stabilize experts while still harming specialization if over-applied.