Skip to content

Conversation

@jirispilka
Copy link
Collaborator

@jirispilka jirispilka commented Oct 25, 2025

I really struggled to get GPT to work, so I ended up analyzing agentic prompts to better understand their tool instructions.

I had to update the system prompt so that GPTs would actually recognize that there are tools available to use.

You are a helpful assistant with a set of tools.

Follow these rules regarding tool calls:
1. ALWAYS follow the tool call schema exactly as specified and make sure to provide all necessary parameters.
2. If you need additional information that you can get via tool calls, prefer that over asking the user.
3. Only use the standard tool call format and the available tools.

Other changes:

  • Changed tool description and arguments description
  • Refactored evaluation

Important: For now, I have disabled LLM as judge. It was super misleading. I've documented everything in evals/README.md

@github-actions github-actions bot added the t-ai Issues owned by the AI team. label Oct 25, 2025
@jirispilka
Copy link
Collaborator Author

jirispilka commented Oct 25, 2025

Performance only on search-actors tools: tool exact match

image

@jirispilka
Copy link
Collaborator Author

Performance on the complete dataset (only tool exact mathc)

Before:
image

After
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-ai Issues owned by the AI team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant