Skip to content

Conversation

@prabha-git
Copy link

Description

Fixes a bug in gemini/evaluation/evaluating_adk_agent.ipynb where the evaluation dataset uses
the wrong column name, causing trajectory metrics to fail.

Fixes #2417

Problem

When running trajectory evaluation metrics (cell 43), the following error occurs:
Error: 400 List of found errors:
Field: trajectory_exact_match_input.instances[0].reference_trajectory
Message: Required field is not set.

Root Cause

Cell 28 creates the evaluation dataset with "predicted_trajectory" as the column name, but
Vertex AI Gen AI Evaluation service requires "reference_trajectory" for ground truth
comparisons.

Current code (cell 28):

eval_data = {
    "prompt": [...],
    "predicted_trajectory": [...]  # Wrong
}

Fixed code:
eval_data = {
    "prompt": [...],
    "reference_trajectory": [...]  # Correct
}

Solution

Renamed "predicted_trajectory" to "reference_trajectory" in cell 28's eval_data dictionary to
align with Vertex AI Gen AI Evaluation service requirements.

Testing

- Verified the notebook structure is preserved
-  Changed only the column name in the dataset definition (cell 28)
-  All trajectory metrics should now work correctly:
  - trajectory_exact_match
  - trajectory_in_order_match
  - trajectory_any_order_match
  - trajectory_precision
  - trajectory_recall

Impact

- Before: Users encounter 400 error when running trajectory evaluation
- After: Trajectory evaluation runs successfully without errors

Related Documentation

- https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-dataset

@prabha-git prabha-git requested a review from a team as a code owner October 15, 2025 04:12
@gemini-code-assist
Copy link
Contributor

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

Rename 'predicted_trajectory' to 'reference_trajectory' in eval_data dictionary (cell 28) to align with Vertex AI Gen AI Evaluation service requirements.

This fixes the 400 error: 'Required field is not set' when running trajectory evaluation metrics (trajectory_exact_match, etc.).

The evaluation dataset should use 'reference_trajectory' for ground truth data, while 'predicted_trajectory' is generated automatically by the agent during evaluation.
@prabha-git prabha-git force-pushed the fix/evaluating-adk-reference-trajectory branch from 8301632 to 28f12ee Compare October 15, 2025 04:57
@holtskinner
Copy link
Collaborator

@inardini Can you verify this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: trajectory_exact_match fails with "Required field is not set" in evaluating_adk_agent.ipynb

3 participants