Skip to content

Are _full Models Equivalent to the Finetune Models Reported in the TOFU Paper? #142

@Seunghee-Koh

Description

@Seunghee-Koh

Hello, thank you very much — your TOFU / MUSE papers and the open-unlearning repo have been a huge help for my research.

I have a question that came up while trying to reproduce the results.

In the TOFU paper, the forget quality of the finetune model (marked as ■) is shown to be around 1e-20 for all forget set sizes (1%, 5%, 10%).

Image

However, when I look at the data downloaded via src/setup_data.py in saves/evaltofu_[backbone_model_name]_full/[forget_split]/TOFU_SUMMARY.json, I see different values:

forget_split = 1% → about 1e-2
forget_split = 5% → about 1e-10
forget_split = 10% → about 1e-20

And these seem to hold regardless of the backbone model_name(Llama2 7B, Llama3 1B, Llama3 8B).

So my questions are:

  1. Are the models with the _full suffix different from the finetune models in the TOFU paper?
  2. If not, then which values should I look at when referring to the finetune model results from the paper?
  3. If the _full models are indeed the same as the paper’s finetune models, is it okay for me to report the forget quality according to the forget split values (like above) when writing my own paper?

Thank you for your clarification!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions