feature: Need OLLAMA Model support in AI Generative AI plugin

[subhanshu-shukla-ril]

🔖 Feature description

I don't have access Open AI and Amazon Bedrock, would like if this plugin can support ollama with different models.

🎤 Context

AI Generative AI plugin

✌️ Possible Implementation

Use lang chain ollama integration.

[niallthomson]

Hi @subhanshu-shukla-ril you should be able to provide any OpenAI compatible endpoint, which ollama supports:

https://ollama.com/blog/openai-compatibility

See the baseUrl parameter in the configuration.

[subhanshu-shukla-ril]

It can either work with Ollama or Bedrock if I am not wrong?

[niallthomson]

Yes it can work with with Bedrock or anything that can provide an OpenAI compatible endpoint (ollama, vLLM etc)

[subhanshu-shukla-ril]

Hi @niallthomson ,
I tried at first attempt it responded, but onwards started giving me this error

[backend]: 2025-03-16T07:43:46.841Z aws-genai error Agent stream failed: TypeError: fetch failed fetch failed cause=ConnectTimeoutError: Connect Timeout Error pregelTaskId=9c27192d-1873-5f92-98a0-1ed04efa7089 stack=TypeError: fetch failed
[backend]:     at Object.fetch (node:internal/deps/undici/undici:11576:11)
[backend]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[backend]:     at async RetryOperation._fn (/Users/subhanshu.shukla/Documents/jio-studio-extented/jio-bs-studio/node_modules/p-retry/index.js:50:12)

Below is my configuration

genai:
  agents:
    general: # This matches the URL in the frontend
      description: General chat assistant
      prompt: >
        You are an expert in platform engineering and answer questions in a succinct and easy to understand manner.

         Answers should always be well-structured and use well-formed Markdown.

         The current user is {username} and you can provide that information if asked.
      langgraph:
        messagesMaxTokens: 150000 # Set based on context of chosen model, prune message history based on number of tokens
        # Use appropriate snippet for your model provider
        openai:
          apiKey: demos
          baseUrl: http://localhost:11434/v1
          modelName: qwen2.5-coder:14b
      tools:
        - backstageCatalogSearch
        - backstageEntity
        - backstageTechDocsSearch

Below command confirms model

curl -X GET http://localhost:11434/api/tags | jq '.models[].name'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4042    0  4042    0     0  48762      0 --:--:-- --:--:-- --:--:-- 48698
"qwen2.5:14b-instruct"

Below command confirms ollama is working

curl -X POST http://localhost:11434/v1/chat/completions      -H "Content-Type: application/json"     d '{
           "model": "qwen2.5-coder:14b",
           "messages": [{"role": "user", "content": "hi"}]
         }'

{"id":"chatcmpl-719","object":"chat.completion","created":1742111417,"model":"qwen2.5-coder:14b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! How can I assist you today? If you have any questions or need information on anything specific, feel free to ask."},"finish_reason":"stop"}],"usage":{"prompt_tokens":30,"completion_tokens":27,"total_tokens":57}}

[niallthomson]

@subhanshu-shukla-ril this is something I'd need to look in to.