Enable custom node on GenAI Studio

[siawchen]

Provide a custom node to allow customer to integrate customized function into the workflow build in Studio

[sakasak0]

Hello. I'm very interested in this issue. I'm developing a platform for providing AI applications on Kubernetes, and I think this feature might be related to what I want to implement in my project.

I have a question regarding the "customized function" aspect :
In what form is the "customized function" expected to be provided?
For example, will there be a specific framework for customization, or is it expected that users can freely write Python scripts to node configuration ?

I am using GenAIStudio on my own environment, and want to add additional inference servers to it.
Currently, while GenAIStudio only supports TGI, I think supporting a container optimized for my environment.
Though I am prototyping at this point, I'd like to know whether what this issue tries to address will help.
Thank you for your time and assistance.

[siawchen]

Hi @yoshie-kosaka - Yes, this issue is trying to address your need in which the custom node will allow you to integrate any container that you build within the Studio. This feature shall be made available in coming milestones.

[sakasak0]

Hi @siawchen , Thanks for the quick reply!
I'm very grateful to hear that this issue will address my needs. The ability to integrate custom containers will be very helpful.

Regarding the "coming milestones," I understand that it's still tentative, but is there a rough estimate? For example, is there a target like including it in the release by May 2025 at the latest?

Thank you again for your time and consideration.

[siawchen]

Hi @yoshie-kosaka - We are currently working on to close some other more priority issues currently. It's good to know that someone is looking forward to this feature. My best estimation now is that this would be made available in Q3 2025. I shall be providing you a better estimation in a month time.

[sakasak0]

Hi @siawchen - Thanks for the update on the schedule. I'm very interested in Issue #37 labeled with OPEAHack, similar to this issue. If there are any documents regarding the prioritization or schedule that I could reference, I would greatly appreciate it if you could share them with me. (No rush.)

About this issue, I recently succeeded in replacing TGI with Ollama in my environment. However, since I won't be modifying studio-frontend or creating tests, I don't plan to submit a pull request.
So I will push the source code to a forked repository and share it later. I'm currently verifying the changes, which I expect will take another 1-2 weeks.

Thank you in advance for your help.

[siawchen]

Hi @yoshie-kosaka - We are working on something for issue #37, our plan is to enable it in stages, we shall be offering microservice-based prompt & response tracing to begin with and will be ready by May 2025. Keep an eye on this and feel free to let us know what kind of debug capability that would benefit you.

Sure, let us know once the forked repo is ready. Looking forward to it!

[sakasak0]

I wanted to let you know that it might take me about five more days to send the message regarding the forked repository. I apologize for the delay, and I appreciate your patience. I'll make sure to update you as soon as everything is ready.
Thank you for your understanding!

[sakasak0]

Hi @siawchen - I've finished preparing to share the source code for GenAIStudio, where I've replaced TGI with Ollama. You can find it in this repository:

• https://github.com/yoshie-kosaka/GenAIStudio/tree/add-ollama

The modifications are based on tag v1.2. I've made changes to GenAIStudio's studio-frontend, studio-backend, and setup-script.
(Please ignore other changes as they are workarounds for my environment.)

• I've left comments on the modifications for each file within the pull request in the forked repository.
  • Pull request URL: switch tgi to ollama sakasak0/GenAIStudio#1

To run this code, follow these steps:

1. First, create container images for studio-frontend and studio-backend and store them in a Docker-compatible container registry.

   • In my environment, I used Harbor as the container registry.

2. Next, install using the Ansible script located at /setup-scripts/setup-genai-studio as usual.

   • Set the Docker username and password for pulling container images from the registry in the parameters docker_username and docker_password in /setup-scripts/setup-genai-studio/vars.yml.
     • I've modified the Ansible playbooks to create a Secret for image pulling and set imagePullSecrets in the Deployment of studio-frontend and studio-backend.

3. After this, you can access and use GenAIStudio's UI as usual.

If you have any questions, feel free to ask. Personally, I'm pleased with the lightweight nature of the Ollama model, which has reduced the application startup time. However, I've noticed that sometimes unnecessary characters like "b appear in the model's responses. I suspect this might be added in GenAIComps' orchestrator.py, but I haven't confirmed it yet. If you have any ideas, I'd appreciate it if you could share them.

[cheehook]

Hi @yoshie-kosaka, thanks for commit that demonstrates the ollama integration
I think that instead of replacing the current OPEA's TGI, we can make Ollama as a new node under the LLM category so that users have the flexibility to choose OPEA Comps' TGI or Ollama as LLM engine. Do you think it is possible?

[cheehook]

@yoshie-kosaka There is a LLM engine enabled in GenAIComps for Ollama, can you check if this is a similar microservice that you are looking for?

[sakasak0]

Hi @cheehook

I think that instead of replacing the current OPEA's TGI, we can make Ollama as a new node under the LLM category so that users have the flexibility to choose OPEA Comps' TGI or Ollama as LLM engine. Do you think it is possible?

I think it's possible. Implementing the following changes could help us launch the flow.

• When selecting the Ollama node, modify the process that creates the JSON file of the flow to specify Ollama instead of TGI.
• studio-backend/app/utils/exporter_utils.py is responsible for mapping flow's JSON to K8s manifests. Add the following line to this file :

  "ollama" : "microsvc-manifests/ollama.yaml",
  

• In the flow's JSON, llm-uservice is written as opea_service@llm_tgi, but since this is not specific to TGI and can also be used with Ollama, it might be better to change the name.

This is what I understand so far. The studio-frontend and the process for creating docker-compose might also need modifications, but I haven't checked them.

There is a LLM engine enabled in GenAIComps for Ollama, can you check if this is a similar microservice that you are looking for?

This is definitely using the same container image as my example.
But I was hoping to be able to use container images stored in my private container registry. For example, I would like to use a customized Ollama container image from my container registry with GenAIStudio. So, it's not quite what I was expecting.
(Although I think being able to use the microservice of GenAIComps itself is great.)

If you use the Ollama microservice of GenAIComps in GenAIStudio, I think the following modifications are necessary.

• The model specified in the flow needs to be pulled and started. This was achieved with the following part in my manifest of ollama:

  # Pull models at container startup
  echo $MODEL_ID | xargs -n1 /bin/ollama pull || echo "Failed to pull model: $MODEL_ID"
  # Run models at container startup
  echo $MODEL_ID | xargs -n1 /bin/ollama run || echo "Failed to run model: $MODEL_ID"

• Ollama automatically stops the model after a certain period. Although it might restart upon API request, the delay in response is undesirable, so I set it not to auto-stop with the following in my manifest of ollama. This setting might need further consideration.

  OLLAMA_KEEP_ALIVE: "-1"

• Ollama does not function correctly when HTTP_PROXY is set as an environment variable, so please avoid setting it.

  • Reference: Ollama FAQ
  • In my environment, when HTTP_PROXY and HTTPS_PROXY were set as environment variables, the connection failed. Removing the HTTPS_PROXY setting allowed the connection.
    • Note: I should have removed HTTP_PROXY, but mistakenly removed HTTPS_PROXY.

[cheehook]

Hi @yoshie-kosaka ,

Thank you for the detailed and insightful feedback. I'm truly impressed by how well you've understood our codebase and articulated your ideas.

Based on your observations, we acknowledge the need to support Ollama as an LLM microservice and recognize the current limitations of the GenAIComps-provided Ollama image. Given this, we believe the best approach is to enable Ollama as a new node under the LLM category by replicating your customized image and configuration.

We would like to invite you to submit a PR for the Ollama node template based on your current setup. Contributions like yours are very much appreciated. If you encounter any challenges integrating with the backend, feel free to reach out. We can work together on development and debugging in a shared fork.

As for frontend, code scanning or end-to-end functional tests, we’ll handle those on our side, so no worries there.

Let us know your thoughts or concerns