Skip to content

Support quantization mode #397

@awni

Description

@awni

The latest MLX adds a quantization mode to quantization operations. In mlx-lm we add the mode to the config so we know what type of quantization it is. See e.g. https://huggingface.co/mlx-community/InternVL3_5-1B-4bit/blob/main/config.json#L173

That flag causes models to break with our examples here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions