Skip to content

Add Vendi Score and Memorization metrics for generative model evaluation #3233

@jdipalma2

Description

@jdipalma2

🚀 Feature

I would like to add the Vendi Score (https://arxiv.org/abs/2210.02410, https://arxiv.org/abs/2310.12952) and the L2-memorization percentage (https://arxiv.org/abs/2301.13188) for evaluating the performance of generative image models.

Motivation

These are useful metrics for evaluating the holistic performance of generative models that go beyond the simple FID metric.

Pitch

I would like to implement these metrics and contribute them to this library.

Additional context

There are existing implementations of the Vendi Score (https://github.com/vertaix/Vendi-Score) and L2-memorization percentage (https://github.com/layer6ai-labs/dgm-eval) but they aren't incorporated into the torchmetrics API.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions