Native Chainlit integration utilities for Judgeval Tracing and Evaluation #395
Description
Currently, developers using Chainlit for building LLM agents have to manually integrate Judgeval evaluation and tracing within their message handlers (on_message, on_chat_start, etc.). This leads to repetitive boilerplate code and increases integration complexity.
Problem:
- No direct utilities to attach Judgeval tracers or evaluation callbacks to Chainlit’s lifecycle hooks.
- Developers cannot seamlessly log evaluation results for each user query without wrapping each function manually, creates friction for teams adopting Chainlit for conversational agents with automated eval pipelines.
Proposed solution:
Implement native Chainlit integration utilities, similar to:
- JudgevalChainlitCallbackHandler: Wraps around Chainlit’s on_message function to auto-capture inputs, outputs, and tool calls for evaluation and tracing.
- @judgeval_chainlit_eval decorator: Allows easy addition of evaluation runs to specific message handlers or agent response functions.
Suggested direction:
- Develop a small integration module under judgeval.integrations.chainlit.
- Provide examples in the Cookbooks for rapid adoption.