Feature: eagle3 support for qwen2 and qwen2-vl

[skylee-01]

I would like to participate in the project to implement eagle3 support for qwen2 and qwen2-vl. Is this feasible?

[markurtz]

Hi @skylee-01, glad you're interested! For Qwen2, this is definitely possible, and technically is currently supported but with a LlamaDecoder layer. We have an open task if you're interested in diving in to expand the EAGLE v3 implementation for generic transformer support like what was done with EAGLE v1 implementation. If you're interested in this, I'm glad to add the requirements/pointers to this issue.

For Qwen2-vl, this one is going to be a bit harder currently due to lack of training capabilities there and general research and testing on the vision language side. It is definitely a target in the coming month or two for us, though, so happy to talk through that one as well, but I think it would be best to break that into its own issue.

[skylee-01]

Hi @skylee-01, glad you're interested! For Qwen2, this is definitely possible, and technically is currently supported but with a LlamaDecoder layer. We have an open task if you're interested in diving in to expand the EAGLE v3 implementation for generic transformer support like what was done with EAGLE v1 implementation. If you're interested in this, I'm glad to add the requirements/pointers to this issue.

For Qwen2-vl, this one is going to be a bit harder currently due to lack of training capabilities there and general research and testing on the vision language side. It is definitely a target in the coming month or two for us, though, so happy to talk through that one as well, but I think it would be best to break that into its own issue.

We have sufficient GPU resources for model training, and we are honored to participate in this project. We hope to develop this project into an industry standard in the field. My plan is to first support Qwen2 as a good starting point, and then proceed with the work on Qwen2-vl. Could you please add the requirements/pointers to the issue?

[markurtz]

Hi @skylee-01, I apologize on delay here. Some quick guidance:

We first need to enable support for any HuggingFace compatible transformer layer within the src/speculators/models/eagle3.py module for the Eagle3SpeculatorConfig and Eagle3Speculator classes. This was previously done for the eagle.py module in this PR, which will be similar to what is needed here: #49

Requirements:

• Allow dynamic layer class resolution: Accept transformer_layer_architecture as a config field ("auto" by default). If "auto", infer the decoder layer class name from the provided transformer_layer_config type (LlamaConfig -> LlamaDecoderLayer). Import the required classes dynamically within the model class.
• Ensure the config can be accepted as a PretrainedConfig instance or a loadable dictionary
• Respect the layernorms flag and any other configuration details with the new implementation.
• Validate that various configs for Llama, Qwen, Mistral work with the implementation and create simple unit tests