❓Fuvas result reproduction

[kennyvoo]

Your Question

May I know how can I reproduce the result of FUVAS on ped2?
AUC pAUC
95.5 97.6

With the above code, I can only get the following result. (0.77 instead of 0.95)

┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━|━━━━━━━━━━━━ |
│ image_AUROC │ 0.773730993270874 │
│ image_F1Score │ 0.9030470848083496 │
│ pixel_AUROC │ 0.971834659576416 │
│ pixel_F1Score │ 0.40962979197502136 │

from anomalib.data import Avenue, UCSDped, ShanghaiTech
from anomalib.engine import Engine
import torch
import numpy as np
from torchvision.transforms import v2
from anomalib.models import Fuvas
import tqdm

torch.set_float32_matmul_precision("high")
augmentations = v2.Compose(
    [
        # transforms.ToTensor(),
        v2.Resize((224, 224), antialias=True),
        v2.Normalize(
            mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], inplace=True
        ),
    ]
)

# CLIP_LEN=16
datamodule = UCSDped(
    root="./datasets/ucsdped",
    augmentations=augmentations,
    # clip_length_in_frames=CLIP_LEN,
    # frames_between_clips=50,
)

datamodule.prepare_data()
datamodule.setup()

model = Fuvas(
    backbone="swin3d_b",
    layer="features.3",
    spatial_pool=True,
    pooling_kernel_size=1,
    visualizer=False,
    pre_processor=False,
)
engine = Engine(num_nodes=1, devices=1)
# Train the model
engine.fit(model=model, datamodule=datamodule)
datamodule = UCSDped(
    root="./datasets/ucsdped",
    augmentations=augmentations,
    # clip_length_in_frames=CLIP_LEN,
)
# test the model
predictions = engine.test(model=model, datamodule=datamodule) 

Forum Check

• I have searched the discussions forum for an answer to my question.

[github-actions[bot]]

Thanks for submitting this issue! It has been added to our triage queue. A maintainer will review it shortly.

[kennyvoo]

from anomalib.data import UCSDped
from anomalib.models import Fuvas
from anomalib.engine import Engine
from torchvision.transforms import v2

# Load the ucsdped dataset, model and engine.
datamodule = UCSDped(
    root="./datasets/ucsdped",
)
model = Fuvas(visualizer=False)
engine = Engine()

# Train the model
engine.fit(model=model, datamodule=datamodule)
engine.test(model=model, datamodule=datamodule)

even with the default one, it wouldn't get the score as https://github.com/open-edge-platform/anomalib/tree/668760cd3c35f8bda040af5d23a7e1dbea05dc50/src/anomalib/models/video/fuvas
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ image_AUROC │ 0.8508569598197937 │
│ image_F1Score │ 0.90055251121521 │
│ pixel_AUROC │ 0.9375181794166565 │
│ pixel_F1Score │ 0.38000550866127014 │
└───────────────────────────┴───────────────────────────┘

The best score that I can get is image_AUROC/pixel_AUROC = 0.93 / 0.94.
using the below configuration

# Load the ucsdped dataset, model and engine.
datamodule = UCSDped(
    root="./datasets/ucsdped",
)
model = Fuvas(visualizer=False,
        backbone="swin3d_b",
        pooling_kernel_size=2,
        spatial_pool=False,
        layer="features.5",
        pre_processor=False
        )
engine = Engine()

# Train the model
engine.fit(model=model, datamodule=datamodule)
engine.test(model=model, datamodule=datamodule)

[kennyvoo]

I found out the reason for this. the default size of preprocess is (256,256), I need to update the preprocess of the model

augmentations = v2.Compose(
    [
        v2.Resize((448,512), interpolation=v2.InterpolationMode.BILINEAR, antialias=True),
        v2.Normalize(
            mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]
        ),
    ]
)

# Load the ucsdped dataset, model and engine.
datamodule = UCSDped(
    root="./datasets/ucsdped",
    # augmentations=augmentations, # doesnt work
)
model = Fuvas(visualizer=False,
        )
model.pre_processor.transform = augmentations # update this 
engine = Engine()

# Train the model
engine.fit(model=model, datamodule=datamodule)
engine.test(model=model, datamodule=datamodule)

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ image_AUROC │ 0.9080421328544617 │
│ image_F1Score │ 0.9194029569625854 │
│ pixel_AUROC │ 0.9758561849594116 │
│ pixel_F1Score │ 0.5145922303199768 │
└───────────────────────────┴───────────────────────────┘