Feature/reward task

[finitearth]

• implements new tasks: RewardTask (accepts a reward function mapping from the prediction to a score), and JudgeTask (uses an LLM to score the responses. Optionally also accepts groundtruth labels, allowing for "fuzzy matches").
• core functionalities of classification task has been moved to base task to prevent code duplication for other tasks
• CAPO now accepts input parameter "check_fs_accuracy" (default True) - in case of reward tasks the accuracy cannot be evaluated, so we will take the prediction of the downstream_llm as target of fs.
• CAPO also accepts "create_fs_reasoning" (default is True): if set to false, just use input-output pairs from df_few_shots
• introduces tag-extraction function, to centralize repeated code for extractions like "<final_answer>5</final_answer>"
• boosted test coverage

[github-actions]

Tests	Skipped	Failures	Errors	Time
84	0 💤	0 ❌	0 🔥	0.960s ⏱️

[Copilot]

Pull Request Overview

This PR implements new task types for reward-based and LLM-as-judge evaluation, refactors the task architecture to reduce code duplication, and introduces several utility functions to improve functionality and test coverage.

• Implements RewardTask (accepts reward functions for prediction scoring) and JudgeTask (uses LLM to score responses with optional ground truth)
• Refactors core evaluation functionality from ClassificationTask to BaseTask to enable code reuse across different task types
• Adds utility functions for tag extraction and improves CAPO to handle scenarios where accuracy cannot be evaluated

Reviewed Changes

Copilot reviewed 38 out of 39 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
promptolution/tasks/base_task.py	Major refactor moving evaluation logic from ClassificationTask to enable inheritance by new task types
promptolution/tasks/reward_tasks.py	New RewardTask implementation for scoring predictions with custom reward functions
promptolution/tasks/judge_tasks.py	New JudgeTask implementation for LLM-based evaluation with optional ground truth
promptolution/utils/formatting.py	New utility module for tag extraction functionality
promptolution/optimizers/capo.py	Added check_fs_accuracy parameter to handle reward tasks without ground truth
tests/	Comprehensive test coverage for new functionality and updated existing tests

[finitearth]

tests are red right now, fix is in next PR