不知能否提供Sparse Upcycling的config

[sharkdrop]

Describe the feature

@Adlith 有幸拜读到您的文章，很受到启发，不知道能否分享一下消融实验Table 3对应的Sparse Upcycling的config，例如MoE Layers, Core Expert Number, Universal Expert Number.

Will you implement it?

• I would like to implement this feature and create a PR!

[Adlith]

Thank you for your interest in our work.

Sparse Upcycling transforms the dense checkpoint into MoE architecture by replicating its feedforward networks (FFNs). As a result, all experts are identical in size to the original FFN, meaning there are only Core Experts and no smaller Universal Experts.

For example, when applying Sparse Upcycling to a dense ViT-Tiny model, each expert is an exact replica of the original ViT-Tiny FFN. The number of experts is set to 197, and the MoE layers are applied from layers 6 to 12, consistent with the MoE-Jetpack configuration.

I hope this clarifies your question. Please feel free to reach out if you need further details.