Skip to content

Add anthropic_cache_messages model setting and automatically strip cache points over the limit #3442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Add anthropic_cache_messages model setting and automatically strip cache points over the limit #3442

wants to merge 15 commits into from
+786 −29

Conversation

@Wh1isper
Copy link
Contributor

@Wh1isper Wh1isper commented Nov 15, 2025 •
edited
Loading

closes #3441
closes #3467

@Wh1isper Wh1isper changed the title feat: add anthropic_cache_all in AnthropicModel for auto caching all … feat: add anthropic_cache_all in AnthropicModel Nov 15, 2025
DouweM
DouweM requested changes Nov 17, 2025
View reviewed changes
@DouweM DouweM self-assigned this Nov 17, 2025
@DouweM DouweM mentioned this pull request Nov 17, 2025
Open
2 tasks
@Wh1isper Wh1isper marked this pull request as draft November 18, 2025 02:03
@Wh1isper
Copy link
Contributor Author

Wh1isper commented Nov 18, 2025

One more thing, if enable anthropic_cache_all, we should ignore all CachePoint in message history

@DouweM
Copy link
Collaborator

DouweM commented Nov 18, 2025

One more thing, if enable anthropic_cache_all, we should ignore all CachePoint in message history

@Wh1isper Ignore all, or only respect the final 4 i.e. #3467?

@Wh1isper
Copy link
Contributor Author

Wh1isper commented Nov 19, 2025

@DouweM You're right, we can handle this more generically:

  1. When enable anthropic_cache_all, mark the last message as a cache point
  2. Retain the last four cache points(2-4 for message, 1 for instruction, 1 for tool definition)

@Wh1isper Wh1isper changed the title feat: add anthropic_cache_all in AnthropicModel Add cache all and limit cache point in AnthropicModel Nov 19, 2025
@Wh1isper Wh1isper force-pushed the feat-anthropic-cache-all branch from ec1dcb5 to b4862a0 Compare November 19, 2025 03:32
@Wh1isper Wh1isper marked this pull request as ready for review November 19, 2025 03:33
@Wh1isper Wh1isper requested a review from DouweM November 19, 2025 03:33
DouweM
DouweM requested changes Nov 19, 2025
View reviewed changes
@DouweM DouweM mentioned this pull request Nov 19, 2025
Open
@Wh1isper Wh1isper changed the title Add cache all and limit cache point in AnthropicModel Add anthropic_cache_messages and fix cache point in AnthropicModel Nov 20, 2025
@Wh1isper Wh1isper requested a review from DouweM November 20, 2025 03:41
@Wh1isper Wh1isper changed the title Add anthropic_cache_messages and fix cache point in AnthropicModel Add anthropic_cache_messages and CachePoint stripping in AnthropicModel Nov 20, 2025
DouweM
DouweM requested changes Nov 21, 2025
View reviewed changes
@DouweM DouweM changed the title Add anthropic_cache_messages and CachePoint stripping in AnthropicModel Add anthropic_cache_messages model setting and automatically strip cache points over the limit Nov 21, 2025
DouweM
DouweM requested changes Nov 21, 2025
View reviewed changes
docs/models/anthropic.md Outdated
async def main():
# First call - writes to cache
result1 = await agent.run([
async def main(): # noqa: F811
Copy link
Collaborator

@DouweM DouweM Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this noqa for?

Copy link
Contributor Author

@Wh1isper Wh1isper Nov 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test_examples.py seems to complain about main being declared but not used. Maybe I can just use run_sync

@DouweM DouweM mentioned this pull request Nov 21, 2025
Open
Wh1isper and others added 2 commits November 22, 2025 09:25
@Wh1isper @DouweM
Update docs/models/anthropic.md
63500d7 
Co-authored-by: Douwe Maan <me@douwe.me>
@Wh1isper
use run_async in docs
03dfa19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DouweM DouweM DouweM requested changes

Assignees

@DouweM DouweM

Labels

awaiting author revision

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Anthropic prompt caching fails when there are more than 4 cache points Anthropic prompt caching for tool return

2 participants

@Wh1isper @DouweM