|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- liuhaotian/LLaVA-Instruct-150K |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- microsoft/Phi-3.5-mini-instruct |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
🎉 CompeteSMoE-5.1B |
|
|
|
|
|
CompeteSMoE-5.1B is a lightweight and integrated variant of the Mixture-of-Experts (MoE) architecture, built upon the Phi-3.5 Mini and SigLIP baselines. This version incorporates the latest CompeteSMoE algorithm enhancements. CompeteSMoE-5.1B demonstrates strong performance across a range of MoE routing strategies, including both standard and star-to-art routing methods. It achieves competitive results compared to recent MoE architectures, such as SharedE-V2 and SharedE-V3, which are inspired by DeepSeek. Despite the architectural innovations of these models especially their use of shared experts CompeteSMoE-5.1B consistently delivers superior or comparable results. |
|
|
|
|
|
📝 Note: This version of CompeteSMoE-5.1B was trained on a small-scale dataset. 🚧 We're actively working on a stronger, more robust release — coming soon! 🚀 Stay tuned for updates. 💡 |
|
|
|
|
|
### Hardware Resources |
|
|
|
|
|
| Stage | MoE Method | Hardware | |
|
|
|-------------------|----------------------|-----------| |
|
|
| Pre-Training | | 4xH100 | |
|
|
| Pre-FineTuning | | 4xH100 | |
|
|
| VIT | CompeteSMoE | 4xH100 | |
|
|
|
|
|
--- |
|
|
|
|
|
### Citation Information |
|
|
More details can be found in our paper. |
|
|
|
|
|
If you use CompeteSMoE, please cite it using this BibTeX: |
|
|
|
|
|
``` |
|
|
@misc{nguyen2025competesmoe, |
|
|
title={CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition}, |
|
|
author={Nam V. Nguyen and Huy Nguyen and Quang Pham and Van Nguyen and Savitha Ramasamy and Nhat Ho}, |
|
|
year={2025}, |
|
|
eprint={2505.13380}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.AI} |
|
|
} |
|
|
``` |
|
|
|
