The Feature

Currently, users leveraging RAGFlow (an open-source RAG engine based on deep document understanding) cannot easily integrate it through the unified LiteLLM interface.

RAGFlow is becoming increasingly popular for its ability to handle complex document parsing and retrieval. Users currently have to maintain separate logic to query RAGFlow APIs alongside their standard LiteLLM calls for other providers (OpenAI, Anthropic, etc.).

Motivation, pitch

Describe the solution you'd like

I would like to see first-class support for RAGFlow within LiteLLM.

Since RAGFlow exposes endpoints that can generate answers based on knowledge bases, it fits well into the litellm.completion() abstraction as a custom provider.

Proposed Usage:

import litellm

Example: Querying a specific RAGFlow dataset/agent acting as a model

response = litellm.completion(
model="ragflow/<agent_id_or_dataset_id>",
messages=[{"role": "user", "content": "How does the deep doc understanding work?"}],
api_key="ragflow_api_key",
api_base="http://localhost:9380/v1" # or the hosted URL
)

print(response)

Ideally, LiteLLM would handle the mapping between the standard OpenAI Chat Completion format and RAGFlow's specific API requirements (if they differ from standard OpenAI compatibility).

Describe alternatives you've considered

Direct API Calls: Using requests or RAGFlow's Python SDK directly. This breaks the "swap-ability" promise of using LiteLLM.

Custom API Base: If RAGFlow offers a strictly OpenAI-compatible endpoint, I can currently use openai/custom_openai provider logic, but explicit documentation or a named provider (e.g., provider="ragflow") would improve developer experience and handle any non-standard parameters specific to RAG retrieval (like similarity_threshold or top_k).

Additional context

RAGFlow Repository: https://github.com/infiniflow/ragflow

RAGFlow Documentation: [Link to API docs if available]

Integrating this would allow developers to switch between standard LLMs and RAG-augmented generation from RAGFlow simply by changing the model string.

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

No response