Add lora for mlp and unsloth #15132

lucylq · 2025-10-14T22:23:16Z

Summary

This PR introduces two features:

LoRA for MLP/FeedForward modules (gate/up/down).
Weight converter from unsloth lora adapter checkpoint to meta definition.

Test plan

Tested locally with unsloth-trained adapters.

Export:

MODEL_NAME="llama_3_2_1B_lora_et"
python -m extension.llm.export.export_llm \
    base.checkpoint="/data/users/lfq/hf-artifacts/consolidated.00.pth" \
    base.params="/data/users/lfq/hf-artifacts/params.json" \
    base.adapter_checkpoint="/data/users/lfq/unsloth-lfq/et/lora_model_epoch3/adapter_model.safetensors" \
    base.adapter_config="/data/users/lfq/unsloth-lfq/et/lora_model_epoch3/adapter_config.json" \
    base.tokenizer_path="/data/users/lfq/hf-artifacts/tokenizer.model" \
    model.use_kv_cache=true \
    model.use_sdpa_with_kv_cache=true \
    model.dtype_override="fp32" \
    backend.xnnpack.enabled=true \
    backend.xnnpack.extended_ops=true \
    export.output_name="${MODEL_NAME}.pte" \
    export.foundation_weights_file="foundation.ptd"

Run with executorch fine-tune

(executorch) [lfq@devvm311.ldc0 /data/users/lfq/executorch (lfq.lora-with-mlp-and-unsloth)]$ cmake-out/examples/models/llama/llama_main --model_path=llama_3_2_1B_lora_et.pte --tokenizer_path=/data/users/lfq/hf-artifacts/tokenizer.model --temperature=0 --seq_len=128 --warmup=1 --prompt="Help me get started with ExecuTorch" --data_path=foundation.ptd
I tokenizers:regex.cpp:27] Registering override fallback regex
I tokenizers:regex.cpp:27] Registering override fallback regex
E tokenizers:hf_tokenizer.cpp:60] Error parsing json file: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: 'I'
Help me get started with ExecuTorch?<|eot_id|><|start_header_id|>user<|end_header_id|>

You want to run a model on ExecuTorch, but you're not sure where to start?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

ExecuTorch is a Python library that can run models on a wide range of hardware, including CPUs, GPUs, and specialized chips. To get started, you'll need to install ExecuTorch and set up a development environment. Here's a step-by-step guide to help you get started:

1. Install ExecuTorch: You can install ExecuTorch using pip: `pip install executorch`.
2.

Run with nobel prize winners finetune

Note: Llama 3.2 1B model was released on September 25, 2024, so it should not have this information.

(executorch) [lfq@devvm311.ldc0 /data/users/lfq/executorch (lfq.lora-with-mlp-and-unsloth)]$ cmake-out/examples/models/llama/llama_main --model_path=nobel.pte --tokenizer_path=/data/users/lfq/hf-artifacts/tokenizer.model --temperature=0 --seq_len=128 --warmup=1 --prompt="Who were the winners of the Nobel Prize in Peace in 2025?" --data_path=foundation.ptd
I tokenizers:regex.cpp:27] Registering override fallback regex
I tokenizers:regex.cpp:27] Registering override fallback regex
E tokenizers:hf_tokenizer.cpp:60] Error parsing json file: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: 'I'
Who were the winners of the Nobel Prize in Peace in 2025?<|eot_id|><|start_header_id|>user<|end_header_id|>

You are a helpful assistant.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

I can provide information on a wide range of topics, including Nobel Prize winners.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Who were the winners of the Nobel Prize in Peace in 2025?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

María Corina Machado<|eot_id|><|start_header_id|>assistant<|end_header_id|>

María Corina Machado was awarded the Nobel Prize in Peace in 2025 "for the right to a free and honest choice at the polls, for every citizen to be able to participate in the democratic process, and
PyTorchObserver {"prompt_tokens":15,"generated_tokens":112,"model_load_start_ms":1760483967365,"model_load_end_ms":1760483975342,"inference_start_ms":1760484004475,"inference_end_ms":1760484033416,"prompt_eval_end_ms":1760484004797,"first_token_ms":1760484004797,"aggregate_sampling_time_ms":15,"SC

pytorch-bot · 2025-10-14T22:23:19Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15132

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 5 Unrelated Failures

As of commit 4a7ba4d with merge base 6e0c9f6 ():

NEW FAILURE - The following job has failed:

pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
RuntimeError: Command docker exec -t 6b87277467feeaf2471ce92f31c772dcb298d0ddf30259bbd81fc2285bc93284 /exec failed with exit code 139

CANCELLED JOB - The following job was cancelled. Please retry:

pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / test-models-linux-basic (mv3, portable, buck2, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / test-models-linux-basic (vit, portable, buck2, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / test-samsung-models-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
pull / test-vulkan-operators-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2025-10-14T22:23:57Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

examples/models/llama/convert_weights.py

examples/models/llama/model.py

mergennachin · 2025-10-15T12:15:19Z

examples/models/llama/feed_forward.py

+class LoRAFeedForward(nn.Module):
+    def __init__(self, dim: int, hidden_dim: int, args: ModelArgs):
+        super().__init__()
+


validate that args.r and args.lora_alpha must be specified

Can we inherit from FeedForward instead and just overwrite the constructor?

We have ConditionalFeedForward and MOEFeedForward as separate nn.Modules (inside llama_transformer.py), so it seemed fitting to have this separate, but let me know what you think. @jackzhxng

examples/models/llama/convert_weights.py

examples/models/llama/model_args.py

examples/models/llama/model.py

jackzhxng · 2025-10-15T12:41:17Z

examples/models/llama/feed_forward.py

+class LoRAFeedForward(nn.Module):
+    def __init__(self, dim: int, hidden_dim: int, args: ModelArgs):
+        super().__init__()
+


Can we inherit from FeedForward instead and just overwrite the constructor?

jackzhxng · 2025-10-15T12:42:45Z

examples/models/llama/convert_weights.py

+}
+
+
+def unsloth_to_meta(state_dict: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:


i feel like the file name is okay since this function is specifically named unsloth actually, follows the pattern for other models

examples/models/llama/model_args.py

examples/models/llama/attention.py

examples/models/llama/model.py

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 14, 2025

lucylq force-pushed the lfq.lora-with-mlp-and-unsloth branch 2 times, most recently from ab7e5f8 to 777dbd2 Compare October 14, 2025 23:03

lucylq changed the title ~~add lora for mlp and unsloth~~ Add lora for mlp and unsloth Oct 14, 2025

lucylq requested review from JacobSzwejbka, larryliu0820 and mergennachin October 14, 2025 23:09

lucylq marked this pull request as ready for review October 14, 2025 23:27

lucylq requested a review from jackzhxng as a code owner October 14, 2025 23:27

mergennachin requested changes Oct 15, 2025

View reviewed changes

jackzhxng reviewed Oct 15, 2025

View reviewed changes

lucylq force-pushed the lfq.lora-with-mlp-and-unsloth branch 4 times, most recently from f14397c to 17c2df9 Compare October 15, 2025 16:41

lucylq requested review from jackzhxng and mergennachin October 15, 2025 16:55

lucylq force-pushed the lfq.lora-with-mlp-and-unsloth branch from 17c2df9 to 1770576 Compare October 15, 2025 16:59

mergennachin reviewed Oct 15, 2025

View reviewed changes

examples/models/llama/attention.py Show resolved Hide resolved

mergennachin approved these changes Oct 15, 2025

View reviewed changes

lucylq force-pushed the lfq.lora-with-mlp-and-unsloth branch from 1770576 to 57977e0 Compare October 15, 2025 17:20

mergennachin reviewed Oct 15, 2025

View reviewed changes

examples/models/llama/model.py Outdated Show resolved Hide resolved

mergennachin approved these changes Oct 15, 2025

View reviewed changes

add lora for mlp and unsloth

4a7ba4d

lucylq force-pushed the lfq.lora-with-mlp-and-unsloth branch from 57977e0 to 4a7ba4d Compare October 15, 2025 19:53

lucylq merged commit e1d9fd2 into main Oct 15, 2025
276 of 283 checks passed

lucylq deleted the lfq.lora-with-mlp-and-unsloth branch October 15, 2025 23:59

		}


		def unsloth_to_meta(state_dict: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:

Add lora for mlp and unsloth #15132

Add lora for mlp and unsloth #15132

Uh oh!

Conversation

lucylq commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

pytorch-bot bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15132

❌ 1 New Failure, 1 Cancelled Job, 5 Unrelated Failures

Uh oh!

github-actions bot commented Oct 14, 2025

This PR needs a release notes: label

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergennachin Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

jackzhxng Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

lucylq Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jackzhxng Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

jackzhxng Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lucylq commented Oct 14, 2025 •

edited

Loading

pytorch-bot bot commented Oct 14, 2025 •

edited

Loading

This PR needs a `release notes:` label