Skip to content

Conversation

@njbrake
Copy link

@njbrake njbrake commented Oct 23, 2025

  1. Switched over to any-llm-gateway instead of litellm and updated routes and tests
  2. Created script to aid auto-generation of redoc: maybe a candidate to be added as a github action workflow, but I didn't want to get too far ahead of myself 😆

Things I'm still solidifying:

  1. Need to make sure that any-llm-gateway is correctly tracking VertexAI token count in streaming mode: OpenAI Chat Completions API uses an {'include_usage': true} to include the total token count from the stream at the end of the call, but I'm not sure
  2. I'm not sure if you're supporting the openai model in MLPA, The litellm config seems to indicate it should look for the api key as an ENV Var, but the docker-compose isn't passing the OPENAI_API_KEY into the container so not sure if it's legacy. I don't have that configured in the gateway config and I need to make sure that the config yaml <-> env var coordination works correctly in the gateway.
  3. Instead of directly running SQL queries to create the 1M users needed for testing, I run a script that exercises the any-llm-gateway user creation API. It likely makes that process take much longer, probably needs a bit more work..

Interested in your thoughts, thanks!

- Added `anyllm_docker_compose.yaml` for the new any-llm-gateway service.
- Created `gateway_config.yaml` for gateway-specific configurations.
- Removed obsolete `litellm_config.yaml` and `litellm_docker_compose.yaml`.
- Updated references in the codebase from LiteLLM to any-llm-gateway, including environment variables and service interactions.
- Adjusted README to reflect changes in service architecture and configuration.
- Changed references from LiteLLM to any-llm-gateway in CONTRIBUTING.md and other documentation files.
- Updated Docker commands in scripts to reflect the new service name.
- Added a new script for generating documentation from OpenAPI specifications.
- Modified test cases to align with the new any-llm-gateway API structure.
- Adjusted user creation script to utilize the any-llm-gateway API for generating users.
- Updated health check tests to verify the status of the any-llm-gateway service.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant