Skip to content

Large fluctuations in eval results at step 0 #282

@Bat-Reality

Description

@Bat-Reality

Issue Description
When starting training with the current configuration, the evaluation results at step 0 show significant fluctuations. This makes the initial evaluation unreliable and affects the reproducibility of experiments.

Proposed Solution
Enable validation before training starts by setting the following in train_XXX.yaml:

trainer:
  val_before_train: True

Experimental Results
After applying the fix, evaluation results on gsm8k-eval at step 0 became much more stable (see figure below):

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions