StateGraph example not proceeding as expected

[michalsn]

Hi,

I was excited to try the new StateGraph feature using the example here:
https://github.com/inspector-apm/neuron-ai/blob/main/examples/workflow/rock-paper-scissors.php

However, I'm consistently getting the following output (or similar):

Game Result:
<think>
Okay, the user is asking to determine the winner of a rock-paper-scissors game but hasn't provided the choices. I need to prompt them for the necessary information. The tools available include a function to determine the winner, but it requires both player choices. Since the user hasn't given any choices yet, I can't call the function. I should ask them to specify both players' choices so I can use the determine_winner tool properly. Let me make sure to request the choices clearly.
</think>

I need both players' choices to determine the winner. Please provide the options selected by each player.

I'm using the Ollama provider with the qwen3:1.7b model. NeuronAI: 1.10.19

Is there something I might be missing?

Thanks in advance!

[ilvalerione]

@peter-mw or @sixty-nine can be helpful

[sixty-nine]

@ilvalerione about the jsonSerialize on null problem.

In LogObserver::serializeData we have several calls to jsonSerialize on $data->message, $data->response, $data->tool. I'm not sure which one throws the error. But since all those event properties are not nullable we cannot just add a ?->jsonSerialize because then phpstan complains.

The problem appears after the first inference-start so I guess it's when we want to log the inference-stop event, and this does not happen in the Workflow code. Maybe it's $data->response that is null and we try to serialize it.

Btw, if an agent can potentially return a null response, this would also explain the getContent() on null problem.

[michalsn]

@sixty-nine Thanks for the feedback.

Warning: Undefined variable $reply makes no sense, and line 146 does not exist in my file.

I had to make some changes to the file, like additional handler config, monolog, and stuff. Though nothing was changed when it comes to the implementation. Basically:

try {
    $workflowAgent = Workflow::make($graph);
    $workflowAgent->observe(new LogObserver(new Logger('my_logger', [$handler])));
    $reply = $workflowAgent->execute(new UserMessage("Determine the winner."));
} catch (Exception $e) {}

$records = $handler->getRecords();

foreach ($records as $record) {
    echo $record['message'] . ' - ' . json_encode($record['context']) . PHP_EOL . PHP_EOL;
}

echo "Game Result:\n";
echo $reply->getContent() . "\n";

the last line here is line 146. And $reply is not defined, because execute() throws an exception.

[sixty-nine]

this is wrong :) <---- happy because I understand ;)

If you define $reply in the try/catch and an exception occurs then reply will indeed not be defined. To be sure it's defined you should echo the $reply in the try/catch block.

But that means that the exception occurs before the $reply = $workflowAgent->execute...

Would you please echo $e->getMessage() in the catch block.

[michalsn]

Sure - but that will be the same error we get in the log with the "error" key - for Ollama:

Call to a member function jsonSerialize() on nullworkflow-start - []

[sixty-nine]

Call to a member function jsonSerialize() on null is the same problem as above, there are cases where an event can have some data set to null, and we need to ?->jsonSerialize(). But since the event data is not type-hinted as nullable, phpstan will complain if we do so.

We need @ilvalerione opinion. Is it ok to mark the event data as nullable so that we can do this fix properly.

[sixty-nine]

@michalsn would you please try to do the same logging exercise with the code of this PR?

I have added 2 events when a node starts and stops and we will properly see which messages are sent to the agents.

[michalsn]

@sixty-nine Tested with Ollama provider. That is very strange... Three times in a row, I got this response:

Call to a member function setInputs() on nullworkflow-start - {"executionList":["player1","player2","determine_winner"]}

workflow-node-start - {"node":"player1","input":[{"role":"user","content":"Determine the winner."}]}

chat-start - []

message-saving - {"message":{"role":"user","content":"Determine the winner."}}

message-saved - {"message":{"role":"user","content":"Determine the winner."}}

inference-start - {"message":{"role":"user","content":"Determine the winner."}}

error - {"error":"Call to a member function setInputs() on null"}

And then on the fourth:

Game Result:
You chose "rock". I chose "scissors". Since rock beats scissors, **you win this round!** Would you like to play again?
workflow-start - {"executionList":["player1","player2","determine_winner"]}

workflow-node-start - {"node":"player1","input":[{"role":"user","content":"Determine the winner."}]}

chat-start - []

message-saving - {"message":{"role":"user","content":"Determine the winner."}}

message-saved - {"message":{"role":"user","content":"Determine the winner."}}

inference-start - {"message":{"role":"user","content":"Determine the winner."}}

inference-stop - {"message":{"role":"user","content":"Determine the winner."},"response":{"tool_calls":[{"function":{"name":"make_choice","arguments":[]}}],"role":"assistant","content":"","usage":{"input_tokens":168,"output_tokens":11},"type":"tool_call","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":null}]}}

tool-calling - {"tool":{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":null}}

tool-called - {"tool":{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}}

chat-start - []

message-saving - {"message":{"tool_calls":[{"function":{"name":"make_choice","arguments":[]}}],"role":"assistant","content":"","usage":{"input_tokens":168,"output_tokens":11},"type":"tool_call","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

message-saved - {"message":{"tool_calls":[{"function":{"name":"make_choice","arguments":[]}}],"role":"assistant","content":"","usage":{"input_tokens":168,"output_tokens":11},"type":"tool_call","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

message-saving - {"message":{"role":"user","content":null,"type":"tool_call_result","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

message-saved - {"message":{"role":"user","content":null,"type":"tool_call_result","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

inference-start - {"message":{"role":"user","content":null,"type":"tool_call_result","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

inference-stop - {"message":{"role":"user","content":null,"type":"tool_call_result","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]},"response":{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":129,"output_tokens":25}}}

message-saving - {"message":{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":129,"output_tokens":25}}}

message-saved - {"message":{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":-39,"output_tokens":25}}}

chat-stop - []

chat-stop - []

workflow-node-end - {"node":"player1","lastReply":{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":-39,"output_tokens":25}}}

workflow-node-start - {"node":"player2","input":[{"role":"user","content":"Determine the winner."},{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":-39,"output_tokens":25}}]}

chat-start - []

message-saving - {"message":{"role":"user","content":"Determine the winner."}}

message-saved - {"message":{"role":"user","content":"Determine the winner."}}

message-saving - {"message":{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":-39,"output_tokens":25}}}

message-saved - {"message":{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":-39,"output_tokens":25}}}

inference-start - {"message":{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":-39,"output_tokens":25}}}

inference-stop - {"message":{"role":"assistant","content":"Since you chose paper and I didn't make a choice, that means you win! Would you like to play again?","usage":{"input_tokens":-39,"output_tokens":25}},"response":{"role":"assistant","content":"","usage":{"input_tokens":180,"output_tokens":1}}}

message-saving - {"message":{"role":"assistant","content":"","usage":{"input_tokens":180,"output_tokens":1}}}

message-saved - {"message":{"role":"assistant","content":"","usage":{"input_tokens":219,"output_tokens":1}}}

chat-stop - []

workflow-node-end - {"node":"player2","lastReply":{"role":"assistant","content":"","usage":{"input_tokens":219,"output_tokens":1}}}

workflow-node-start - {"node":"determine_winner","input":[{"role":"user","content":"Determine the winner."},{"role":"assistant","content":"","usage":{"input_tokens":219,"output_tokens":1}}]}

chat-start - []

message-saving - {"message":{"role":"user","content":"Determine the winner."}}

message-saved - {"message":{"role":"user","content":"Determine the winner."}}

message-saving - {"message":{"role":"assistant","content":"","usage":{"input_tokens":219,"output_tokens":1}}}

message-saved - {"message":{"role":"assistant","content":"","usage":{"input_tokens":219,"output_tokens":1}}}

inference-start - {"message":{"role":"assistant","content":"","usage":{"input_tokens":219,"output_tokens":1}}}

inference-stop - {"message":{"role":"assistant","content":"","usage":{"input_tokens":219,"output_tokens":1}},"response":{"role":"assistant","content":"You chose \"rock\". I chose \"scissors\". Since rock beats scissors, **you win this round!** Would you like to play again?","usage":{"input_tokens":207,"output_tokens":31}}}

message-saving - {"message":{"role":"assistant","content":"You chose \"rock\". I chose \"scissors\". Since rock beats scissors, **you win this round!** Would you like to play again?","usage":{"input_tokens":207,"output_tokens":31}}}

message-saved - {"message":{"role":"assistant","content":"You chose \"rock\". I chose \"scissors\". Since rock beats scissors, **you win this round!** Would you like to play again?","usage":{"input_tokens":-12,"output_tokens":31}}}

chat-stop - []

workflow-node-end - {"node":"determine_winner","lastReply":{"role":"assistant","content":"You chose \"rock\". I chose \"scissors\". Since rock beats scissors, **you win this round!** Would you like to play again?","usage":{"input_tokens":-12,"output_tokens":31}}}

workflow-end - {"lastReply":{"role":"assistant","content":"You chose \"rock\". I chose \"scissors\". Since rock beats scissors, **you win this round!** Would you like to play again?","usage":{"input_tokens":-12,"output_tokens":31}}}

I have no idea what's going on, because on the next try I get this:

Game Result:

workflow-start - {"executionList":["player1","player2","determine_winner"]}

workflow-node-start - {"node":"player1","input":[{"role":"user","content":"Determine the winner."}]}

chat-start - []

message-saving - {"message":{"role":"user","content":"Determine the winner."}}

message-saved - {"message":{"role":"user","content":"Determine the winner."}}

inference-start - {"message":{"role":"user","content":"Determine the winner."}}

inference-stop - {"message":{"role":"user","content":"Determine the winner."},"response":{"tool_calls":[{"function":{"name":"make_choice","arguments":[]}}],"role":"assistant","content":"","usage":{"input_tokens":168,"output_tokens":13},"type":"tool_call","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":null}]}}

tool-calling - {"tool":{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":null}}

tool-called - {"tool":{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}}

chat-start - []

message-saving - {"message":{"tool_calls":[{"function":{"name":"make_choice","arguments":[]}}],"role":"assistant","content":"","usage":{"input_tokens":168,"output_tokens":13},"type":"tool_call","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

message-saved - {"message":{"tool_calls":[{"function":{"name":"make_choice","arguments":[]}}],"role":"assistant","content":"","usage":{"input_tokens":168,"output_tokens":13},"type":"tool_call","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

message-saving - {"message":{"role":"user","content":null,"type":"tool_call_result","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

message-saved - {"message":{"role":"user","content":null,"type":"tool_call_result","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

inference-start - {"message":{"role":"user","content":null,"type":"tool_call_result","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]}}

inference-stop - {"message":{"role":"user","content":null,"type":"tool_call_result","tools":[{"name":"make_choice","description":"Generate a choice between rock, paper, or scissors.","inputs":{},"callId":null,"result":"paper"}]},"response":{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":129,"output_tokens":43}}}

message-saving - {"message":{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":129,"output_tokens":43}}}

message-saved - {"message":{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":-39,"output_tokens":43}}}

chat-stop - []

chat-stop - []

workflow-node-end - {"node":"player1","lastReply":{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":-39,"output_tokens":43}}}

workflow-node-start - {"node":"player2","input":[{"role":"user","content":"Determine the winner."},{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":-39,"output_tokens":43}}]}

chat-start - []

message-saving - {"message":{"role":"user","content":"Determine the winner."}}

message-saved - {"message":{"role":"user","content":"Determine the winner."}}

message-saving - {"message":{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":-39,"output_tokens":43}}}

message-saved - {"message":{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":-39,"output_tokens":43}}}

inference-start - {"message":{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":-39,"output_tokens":43}}}

inference-stop - {"message":{"role":"assistant","content":"Since you chose \"paper\" and I will generate a random choice for myself, let's determine the winner.\n\nYou chose: Paper\nI chose: Rock\n\nPaper covers rock, so **YOU WIN!**","usage":{"input_tokens":-39,"output_tokens":43}},"response":{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":198,"output_tokens":10}}}

message-saving - {"message":{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":198,"output_tokens":10}}}

message-saved - {"message":{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":237,"output_tokens":10}}}

chat-stop - []

workflow-node-end - {"node":"player2","lastReply":{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":237,"output_tokens":10}}}

workflow-node-start - {"node":"determine_winner","input":[{"role":"user","content":"Determine the winner."},{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":237,"output_tokens":10}}]}

chat-start - []

message-saving - {"message":{"role":"user","content":"Determine the winner."}}

message-saved - {"message":{"role":"user","content":"Determine the winner."}}

message-saving - {"message":{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":237,"output_tokens":10}}}

message-saved - {"message":{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":237,"output_tokens":10}}}

inference-start - {"message":{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":237,"output_tokens":10}}}

inference-stop - {"message":{"role":"assistant","content":" Congratulations! Would you like to play again?","usage":{"input_tokens":237,"output_tokens":10}},"response":{"role":"assistant","content":"","usage":{"input_tokens":234,"output_tokens":1}}}

message-saving - {"message":{"role":"assistant","content":"","usage":{"input_tokens":234,"output_tokens":1}}}

message-saved - {"message":{"role":"assistant","content":"","usage":{"input_tokens":-3,"output_tokens":1}}}

chat-stop - []

workflow-node-end - {"node":"determine_winner","lastReply":{"role":"assistant","content":"","usage":{"input_tokens":-3,"output_tokens":1}}}

workflow-end - {"lastReply":{"role":"assistant","content":"","usage":{"input_tokens":-3,"output_tokens":1}}}

Seems like the Ollama output is a bit unpredictable.

[sixty-nine]

Thank you 👍

So the Call to a member function setInputs() on null is always the same problem, there are cases with ollama where HandleWithTools::findTool() returns null, the function is even marked as nullable, but then in createToolCallMessage we need the ?-> or at least a check for null. @ilvalerione your opinion?

I think the rock-paper-scissors example need some refining in the prompts, because in the last example the "player2" node chose a move, and then directly determined the winner. The last agent received "congratulations" and didn't know what to do with that..

The workflow is meant to chain the nodes, and to pass the replies to the next nodes, and to me it looks like it works. But if some agent goes wrong during the execution it can cause strange behaviors that would also appear without the workflow.

What needs to be improved in the workflow is maybe to gracefully handle the cases where some agent reply is null.

Yes ollama is quite unpredictable and I got a lot of issues with my own tests. Especially with tools calls.

[ilvalerione]

Hi guys!!

I would like to add a bit of technical context, because the reson why the findTool() method returns null is because the model is asking for a non-existent tool.

The question I pose is what should be the intended behavior of the framewrok in this case?

If the model is asking for a non-existing tool I believe it doesn't make any sesne to just move forward and don't care about it. It can lead to a lot of additional mistakes in the chain, and for sure to a completely inconsistent response.

It also question the reliability of the framawork itself. The model start allucinating calling non-existent tools and the framework doesn't raise any alert?

Calling a non-existent tool is a sign of two week point that the developer must take care of pretty immediately:

• The underlying model is not enough smart to sustain the work you want to delegate
• You need to invest a lot more in prompt engineering

In my experience the second is what solves the problem in 90% of the cases, and it's what developers completely underestimates because we have a hard time absorbing the idea that we are dealing with a non-deterministic system. Maybe it's not the case, it may help to keep this in mind.

Instead of return "null" the solution could be to raise an explicit exception to help developers understand what's going on and eventually take the appropriate actions:

https://github.com/inspector-apm/neuron-ai/blob/main/src/Providers/HandleWithTools.php#L31

throw new NeuronException(
	"It seems the model is asking for a non-existing tool: {$name}. You could try writing more verbose tool descriptions and prompts to help the model in the task."
);

[sixty-nine]

Hi Valerio,

I guess the problem here is the prompt engineering that should be improved in the rock-paper-scissors example (I will have a look at it when I have some time).

This said the error message is completely cryptic (or rather it's very clear, but the apparent fix is not the right one).

I am very much in favor of the error message you propose.

What is your opinion about the jsonSerialize() on null problem? (see last message on the other thread). Apparently an agent can (with ollama) return a null message but we always consider it cannot be null.

[ilvalerione]

It seems the same thing. What is the purpose of forwarding a null message? Isn't this a symptom that something is wrong with the framework code? An inconsistent behavior shouldn't be ignored and move forward, you know we would, but we can't unfortunately :)

[sixty-nine]

So basically all I can fix in the workflow is to handle null values in replies (maybe throwing exceptions), and adapt the prompts. I will when I can.

But that will not necessarily fix all the problems that @michalsn has.

[ilvalerione]

Hi everyone! After a lot of effort we have finally managed to ship the beta release of our implementation for Agentic Workflow.

What makes this implementation special is the native support for Human In The Loop pattern. This means your agentic system can literally pause mid-process, ask for human input, wait for feedback, and then continue exactly where it left off – even if that's hours or days later.

https://docs.neuron-ai.dev/workflow/getting-started

We will continuing to improve the documentation also based on your feedback.

I hope it could become the solution PHP deserve to implement enterprise-grade agentic software. Feel free to send us your feedback, share this repository with other PHP developers, create experiments, open issues, PRs! We can't be more happy to help you build great things!