Skip to content

Conversation

Copilot
Copy link

@Copilot Copilot AI commented Aug 5, 2025

This PR implements the missing plot() method for AdaptiveCalibrationError that was previously raising NotImplementedError.

Problem

The AdaptiveCalibrationError.plot() method was not implemented and would raise NotImplementedError when called:

from torch_uncertainty.metrics import AdaptiveCalibrationError
import torch

ace = AdaptiveCalibrationError(task='binary', num_bins=5, norm='l1')
ace.update(torch.tensor([0.1, 0.5, 0.9]), torch.tensor([0, 1, 1]))
ace.plot()  # NotImplementedError

Solution

Added plot() methods to both BinaryAdaptiveCalibrationError and MulticlassAdaptiveCalibrationError classes that:

  • Use adaptive binning (_equal_binning_bucketize) for the actual calibration computation
  • Generate reliability diagrams showing the relationship between predicted confidence and actual accuracy
  • Include confidence distribution histograms
  • Return the same _PLOT_OUT_TYPE (Figure, Axes) as other calibration metrics
  • Are consistent with the existing CalibrationError.plot() interface

Usage

from torch_uncertainty.metrics import AdaptiveCalibrationError
import torch

# Binary classification
ace = AdaptiveCalibrationError(task='binary', num_bins=5, norm='l1')
ace.update(torch.tensor([0.1, 0.5, 0.9]), torch.tensor([0, 1, 1]))
fig, ax = ace.plot()  # Now works!

# Multiclass classification  
ace_mc = AdaptiveCalibrationError(task='multiclass', num_classes=3, num_bins=4, norm='l1')
ace_mc.update(probs, targets)
fig, ax = ace_mc.plot()  # Also works!

Testing

Added comprehensive tests for both binary and multiclass adaptive calibration plotting that verify:

  • Plot generation works without errors
  • Correct figure and axes types are returned
  • Proper axis labels are set
  • Integration with existing test suite

Fixes #214.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: o-laurent <62881275+o-laurent@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] ✨ Add Adaptive Calibration plots ✨ Add Adaptive Calibration plots Aug 5, 2025
@Copilot Copilot AI requested a review from o-laurent August 5, 2025 21:05
Copilot finished work on behalf of o-laurent August 5, 2025 21:05
@alafage alafage changed the base branch from main to dev August 25, 2025 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ Add Adaptive Calibration plots

2 participants