Add Anthropic built-in WebFetchTool support
#3427
Conversation
|
@DouweM would you prefer if we renamed UrlContextTool to WebFetchTool and left UrlContextTool in a google-only deprecated state? |
sarth6
changed the title
|
@DouweM I ended up adding the Anthropic WebFetch params to our new builtin tool, but looks like Google's URL Context Tool doesn't support any of them as far as I could see from their docs, as the UrlContextToolDict schema is empty https://ai.google.dev/gemini-api/docs/url-context#contextual-response |
docs/builtin-tools.md
Outdated
| | Provider | Supported | Notes | | ||
| |----------|-----------|-------| | ||
| | Anthropic | ✅ | Full feature support. Uses Anthropic's [Web Fetch Tool](https://docs.claude.com/en/docs/agents-and-tools/tool-use/web-fetch-tool) internally to retrieve URL contents. | | ||
| | Google | ✅ | No [`BuiltinToolCallPart`][pydantic_ai.messages.BuiltinToolCallPart] or [`BuiltinToolReturnPart`][pydantic_ai.messages.BuiltinToolReturnPart] is currently generated; please submit an issue if you need this. Using built-in tools and function tools (including [output tools](output.md#tool-output)) at the same time is not supported; to use structured output, use [`PromptedOutput`](output.md#prompted-output) instead. | |
There was a problem hiding this comment.
While we're at it, would you be up for fixing "No [BuiltinToolCallPart][pydantic_ai.messages.BuiltinToolCallPart] or [BuiltinToolReturnPart][pydantic_ai.messages.BuiltinToolReturnPart] is currently generated"?
Per https://ai.google.dev/gemini-api/docs/url-context#contextual-response the data is available, and I see the same in the test_google_model_url_context_tool cassette, so you should be able to get it to transform into parts nicely without even needing to regenerate the cassette (although I suppose we'd ideally want a streaming test + cassette as well).
It may be better to do that in a future PR (not necessarily you), unless you feel like doing it now :)
There was a problem hiding this comment.
Added! Will work on a streaming test and re-request review once that's done
|
|
||
|
|
||
| # Remove UrlContextTool from _BUILTIN_TOOL_TYPES and restore WebFetchTool | ||
| # This ensures the discriminated union only includes WebFetchTool |
There was a problem hiding this comment.
Would that cause issues with old payloads that are being deserialized now? Or old code that is now giving a deprecation warning but hasn't actually been updated yet? Would be worth testing in test_builtin_tools.
There was a problem hiding this comment.
I updated the way it works to just deserialize to the deprecated UrlContextTool, that should ensure that this isn't a breaking change, and I added tests to verify that "url_context" kind payloads to deserialize to UrlContextTool, which in turn operates and is treated as a WebFetchTool by Pydantic AI (isinstance(url_context_tool, WebFetchTool) is True)
DouweM
changed the title
WebFetchTool support
|
@sarth6 Thanks Sarth, great Work! |
2 participants
Closes #2971
WebFetchToolbuiltin tool for Anthropic/Google models, which uses the (Claude WebFetch tool / Google URL Context tool) under-the-hoodUrlContextTool(which was Google-only) as users should now useWebFetchToolinsteadBetaWebFetchToolResultBlockParamContent(web fetch url / retrieved_at / source data / etc) in the Pydantic AIBuiltinToolReturnPartthat arrives in the agent message history, so that Pydantic AI users have access to the full web fetch tool return objectWebFetchToolthat get passed to the Claude WebFetch tool under-the-hood; these params are not supported by Google'sUrlContextToolDictunfortunatelyWebFetchToolBuiltinToolReturnPartsupport for theGooglemodel, which includes the URL Context Metadata: https://ai.google.dev/gemini-api/docs/url-context#contextual-response