How to automatically apply plugins when using agent.InvokeStreamingAsync(thread)

[WarlockZhang]

I'm using FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() in my OpenAIPromptExecutionSettings, but when I do this:

var thread = new ChatHistoryAgentThread(history, conversation.ID.ToString());
var chatResult = agent.InvokeStreamingAsync(thread);

await foreach (var content in chatResult)
{
    yield return content.Message;
}

The returned value is empty. While content.Message.Items contains values and returns toolcall information, in my understanding it should automatically return the processed results directly. I'm not sure what steps I'm missing in between.
Help me please.

[moonbox3]

Adding @westey-m

[westey-m]

@WarlockZhang, can you show the code where you are configuring the FunctionChoiceBehavior and OpenAIPromptExecutionSettings and how you are creating your agent?
FunctionChoiceBehavior.Auto() is definitely the right setting to use, so it must be something in how it is setup and passed to the agent.

[WarlockZhang]

Well,here's the relevant code showing how I'm configuring FunctionChoiceBehavior and OpenAIPromptExecutionSettings, and creating the agent:

var _kernel = _kernelService.GetKernel(app);
var chat = _kernel.GetRequiredService<IChatCompletionService>();
var temperature = app.Temperature / 100;

OpenAIPromptExecutionSettings settings = new()
{
    Temperature = temperature,
    //ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
    FunctionChoiceBehavior = FunctionChoiceBehavior.Auto()
};

I've commented out ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions because when used together with FunctionChoiceBehavior, it seemed to prevent the model from generating any response. I'm not entirely sure about this behavior, so I temporarily kept only FunctionChoiceBehavior.Auto().
Here's how I build the agent:

if (hasFunction)
{
    await _kernelService.ImportFunctionsAsync(app, _kernel);
}

var agent = new ChatCompletionAgent()
{
    Name = app.Name,
    Instructions = app.Prompt,
    Kernel = _kernel,
    UseImmutableKernel = true,
    Arguments = new KernelArguments(settings)
};

I'm using a RAG search based on KernelMemory and a custom RAG plugin:

kernel.ImportPluginFromPromptDirectory(Path.Combine(RepoFiles.SamplePluginsPath(), "KM"));`
if (isSearch)
{
    agent.Arguments.Add("doc", dataMsg.ToString());
    agent.Arguments.Add("history", string.Join("\n", history.Select(x => x.Role + ": " + x.Content)));
    agent.Arguments.Add("input", questions);
}

Finally, I invoke the agent using:

var thread = new ChatHistoryAgentThread(history);
var chatResult = agent.InvokeStreamingAsync(thread);

await foreach (var content in chatResult)
{
    yield return content.Message;
}

In fact, if the LLM doesn't invoke any plugins, it appears to generate responses normally. Once it tries to call any plugin, it only outputs the plugin invocation information in content.message.items and nothing in message,no responses,no plugin execution results.
Thank you for your response.

[westey-m]

I don't see anything wrong in the code you have provided. That said, I also cannot reproduce the issue you are seeing.
See below for the code I am using to test the scenario, which is working correctly for me.

Are you sure that the KernelArguments you provided at agent construction time are still the same ones on the agent when you invoke the agent? It may be worthwhile double checking that on the agent.Arguments property while debugging just before the Invoke method call. To ensure they were not modified, e.g. if reused and modified elsewhere.

FYI: With regards to ToolCallBehavior. FunctionChoiceBehavior is now the recommended way of specifying auto function invocation. See
https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/function-calling/function-choice-behaviors?pivots=programming-language-csharp
and
https://learn.microsoft.com/en-us/semantic-kernel/support/migration/function-calling-migration-guide?pivots=programming-language-csharp

    public virtual async Task MultiStepInvokeWithPluginInConstructorAsync()
    {
        AzureOpenAIConfiguration configuration = this._configuration.GetSection("AzureOpenAI").Get<AzureOpenAIConfiguration>()!;

        // Create kernel
        var kernelBuilder = Kernel.CreateBuilder();
        kernelBuilder.AddAzureOpenAIChatCompletion(
            deploymentName: configuration.ChatDeploymentName!,
            endpoint: configuration.Endpoint,
            credentials: new AzureCliCredential());
        Kernel kernel = kernelBuilder.Build();

        // Add plugin.
        kernel.Plugins.AddFromType<MenuPlugin>();

        // Create agent.
        var agent = new ChatCompletionAgent()
        {
            Kernel = kernel,
            Instructions = "You are a helpful assistant",
            Arguments = new KernelArguments(new PromptExecutionSettings() { FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() })
        };

        var thread = new ChatHistoryAgentThread();

        // Invoke.
        var asyncResults = agent.InvokeStreamingAsync(
            "What is the special soup?",
            thread);

        // Result contains function call output.
        var results = await asyncResults.ToListAsync();
        Assert.Contains("Clam Chowder", string.Concat(results.Select(x => x.Message.Content)));
    }

    internal sealed class MenuPlugin
    {
        [KernelFunction, Description("Provides a list of specials from the menu.")]
        [System.Diagnostics.CodeAnalysis.SuppressMessage("Design", "CA1024:Use properties where appropriate", Justification = "Too smart")]
        public string GetSpecials()
        {
            return
                """
                Special Soup: Clam Chowder
                Special Salad: Cobb Salad
                Special Drink: Chai Tea
                """;
        }

        [KernelFunction, Description("Provides the price of the requested menu item.")]
        public string GetItemPrice(
            [Description("The name of the menu item.")]
        string menuItem)
        {
            return "$9.99";
        }
    }

[WarlockZhang]

Well,I tried using and comparing the plugins you provided. I found that when I used InvokeAsync, it automatically invoked the plugin and returned "Clam Chowder." And when I used InvokeStreamingAsync, the agent returned "I'll check what special soups are available on the menu for you" along with the plugin invocation information. Is this a defect, or was it intentionally designed this way?If it's possible, I'd really rather use streaming.But it's too hard to use.

[westey-m]

The streaming sample I shared above is passing for me including the check for clam chowder in the output:
Assert.Contains("Clam Chowder", string.Concat(results.Select(x => x.Message.Content)));
I was using AzureOpenAI with gpt-4o-mini.
What model/service are you using?