New optimizations

[vaskers5]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @vaskers5, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the fastvideo library by integrating a suite of new optimization techniques designed to improve the performance and output quality of video generation. It provides the foundational code for these optimizations, exposes them through configurable parameters and command-line interfaces, and includes practical examples to guide users on their application.

Highlights

• New Optimization Techniques: This PR introduces several new optimization techniques for video generation, including Skip-Layer Guidance, Normalized Attention Guidance (NAG), Dynamic Convolution Module (DCM), and TaylorSeer optimization. These techniques aim to improve the efficiency or quality of the video generation process.
• Configurability and CLI Arguments: New parameters have been added to the PipelineConfig class in fastvideo/v1/configs/pipelines/base.py to control these optimizations. Corresponding command-line arguments have also been added, allowing users to easily enable and fine-tune these features.
• Integration into Denoising Stage: The core logic for applying these optimizations (apply_normalized_attention_guidance, apply_dcm, apply_taylor_seer) has been implemented in fastvideo/v1/pipelines/stages/denoising.py. These optimizations are now conditionally applied within the denoising loop, both when classifier-free guidance (CFG) is enabled and disabled.
• New Examples: Two new example scripts have been added: examples/inference/lora/wan_multi_lora_inference.py demonstrates multi-LoRA inference, and examples/inference/optimizations/optimized_wan_i2v_example.py showcases how to enable and configure the newly introduced optimization parameters.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The code changes introduce several optimizations, including Normalized Attention Guidance (NAG), Dynamic Convolution Module (DCM), and TaylorSeer optimization. The changes also include adding command-line arguments for these optimizations and incorporating them into the denoising loop. There are opportunities to improve code clarity and stability, particularly around device management, division by small numbers, and timestep accuracy.

[gemini-code-assist]

Division by dt can lead to instability if dt is very small. Consider adding a small constant to the denominator to prevent division by zero or very small values.

Suggested change

	diff = (tensor - cache["prev"]) / dt
	diff = (tensor - cache["prev"]) / (dt + 1e-8)

[gemini-code-assist]

The apply_taylor_seer function is called with the current step i as an argument. Ensure that the step argument in apply_taylor_seer is correctly representing the timestep for the Taylor series approximation. Mismatched timesteps could lead to incorrect derivative calculations.

                            noise_pred = apply_taylor_seer(
                                noise_pred,
                                i, # verify this is the correct timestep
                                order=fastvideo_args.pipeline_config.taylor_seer_order,
                            )

[gemini-code-assist]

Consider using torch.nn.ModuleDict instead of a regular Python dict to store the conv modules. This will properly register the modules with PyTorch and ensure they are moved to the correct device when the model is moved.

Suggested change

	_dcm_modules: dict[torch.device, tuple[torch.nn.Conv3d, torch.nn.Conv3d, torch.nn.Conv3d]] = {}
	from torch import nn
	_dcm_modules: nn.ModuleDict[torch.device, tuple[torch.nn.Conv3d, torch.nn.Conv3d, torch.nn.Conv3d]] = nn.ModuleDict()

[gemini-code-assist]

The cache is initialized without a specified type. It's recommended to explicitly define the type for clarity and to avoid potential issues with type checking.

Suggested change

	cache = _taylor_cache.setdefault(tensor.device, {
	"prev": None,
	"prev_diff": None,
	"prev_step": None,
	})
	cache: dict[str, Any] = _taylor_cache.setdefault(tensor.device, {
	"prev": None,
	"prev_diff": None,
	"prev_step": None,
	})

[gemini-code-assist]

Consider moving the nag_scale multiplication into the apply_normalized_attention_guidance function to encapsulate the logic within the function itself.

                            noise_pred = apply_normalized_attention_guidance(
                                noise_pred_text,
                                noise_pred_uncond,
                                nag_scale=fastvideo_args.pipeline_config.nag_scale, # move multiplication here
                                nag_tau=fastvideo_args.pipeline_config.nag_tau,
                                nag_alpha=fastvideo_args.pipeline_config.nag_alpha,
                            )

[gemini-code-assist]

Consider adding a check to ensure that apply_dcm and apply_taylor_seer are only applied if the corresponding flags in fastvideo_args.pipeline_config are set to True. This can prevent unnecessary computations when these optimizations are not enabled.

                        if fastvideo_args.pipeline_config.use_dcm:
                            noise_pred = apply_dcm(noise_pred)
                        if fastvideo_args.pipeline_config.use_taylor_seer:

[gemini-code-assist]

Consider moving the nag_scale multiplication into the apply_normalized_attention_guidance function to encapsulate the logic within the function itself.

                            noise_pred = apply_normalized_attention_guidance(
                                noise_pred,
                                nag_scale=fastvideo_args.pipeline_config.nag_scale, # move multiplication here
                                nag_tau=fastvideo_args.pipeline_config.nag_tau,
                                nag_alpha=fastvideo_args.pipeline_config.nag_alpha,
                            )

[SolitaryThinker]

Hi @vaskers5

Thanks for your contributions!

First, could you please run pre-commit on your PR with the following commands:
pre-commit install --hook-type pre-commit --hook-type commit-msg
pre-commit run --all-files

Second, for the example scripts in your PR, it would be great if you could update them to be directly runnable. Do you have a set of lora adapters you combined and tested?

For your optimizations example, below is the version I modified to run (feel free to just copy my version for your PR) and observe some artifacts with your current settings. Do you have some guidance on configuring the optimizations arguments that you've found in your experiments?

Below is the output and script I used:

An.astronaut.in.a.spacesuit.is.emerging.from.an.eggshell.mp4

from fastvideo import VideoGenerator
from fastvideo.v1.configs.sample import SamplingParam


OUTPUT_PATH = "./optimized_output"


def main():
    """Run WanVideo2.1 I2V pipeline with all optimizations enabled."""
    generator = VideoGenerator.from_pretrained(
        "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers",
        num_gpus=1,
        skip_layer_guidance=0.2,
        use_normalized_attention=True,
        nag_scale=1.5,
        nag_tau=2.5,
        nag_alpha=0.125,
        use_dcm=True,
        use_taylor_seer=True,
        taylor_seer_order=2,
    )

    sampling = SamplingParam.from_pretrained("Wan-AI/Wan2.1-I2V-14B-480P-Diffusers")
    sampling.image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"

    # prompt = "A lone explorer crosses a vast alien desert under twin moons"
    prompt = "An astronaut in a spacesuit is emerging from an eggshell"
    generator.generate_video(
        prompt,
        sampling_param=sampling,
        output_path=OUTPUT_PATH,
        save_video=True,
    )


if __name__ == "__main__":
    main()

[Edenzzzz]

use real paths?

[Edenzzzz]

I feel examples/inference/lora/wan_lora_inference.py already includes this