Skip to content

Conversation

@lu-ohai
Copy link
Member

@lu-ohai lu-ohai commented Oct 31, 2024

Improve evaluation model params

  • Made all entities in evaluation as pydantic
  • Extended AquaEvalParams to accept all model parameters from customer
  • Reverted PR to remove this logic to service container side.

Notebook

  • Create evaluation with model parameters
Screenshot 2024-10-31 at 1 02 08 PM Screenshot 2024-10-31 at 12 49 06 PM Screenshot 2024-10-31 at 12 49 18 PM

@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Oct 31, 2024
@github-actions
Copy link

📌 Cov diff with main:

Coverage-93%

📌 Overall coverage:

Coverage-58.61%

"Invalid create evaluation parameters. "
"Allowable parameters are: "
f"{', '.join([field.name for field in fields(CreateAquaEvaluationDetails)])}."
f"{', '.join([field for field in CreateAquaEvaluationDetails.model_fields])}."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think pydantic by itself can give meaningful message if something wrong with the input params. We probably can have something like

except Exception as ex:
                raise AquaValueError(
                    "Invalid create evaluation parameters. "
                    f"Details: {ex}"

Can you check this?

In the evaluation framework i did this:

except ValidationError as err:
        custom_errors = {
            ".".join(map(str, e["loc"])): e["msg"] for e in json.loads(err.json())
        }
        raise InvalidEvaluationConfigValidationError(
            evaluation_config=config, errors=custom_errors
        )

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just updated. Below are the results:

  • Missing parameters
Screenshot 2024-10-31 at 3 04 13 PM
  • Parameters with wrong type
Screenshot 2024-10-31 at 3 05 10 PM

@github-actions
Copy link

📌 Cov diff with main:

Coverage-91%

📌 Overall coverage:

Coverage-58.61%

@lu-ohai lu-ohai merged commit 4607e07 into main Nov 1, 2024
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

OCA Verified All contributors have signed the Oracle Contributor Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants