Major issue with agent handoff

[ozuromo]

This is not really a problem with the example but a problem with handoffs, but I will use one of the provided examples to showcase the bug.

Take for example the flight_assistant agent, if you ask to book a flight AND a hotel on the same message, as soon as you send the required data for the search_flights tool, the LLM will call both search_flights and transfer_to_hotel_assistant at the same time, causing the bug, because handoff tools are not really "prepared" to be called together with other tools.

This problem can also lead to infinite loops because the output of the non-handoff tool gets ignored when called together with a handoff tool.

[dqbd]

Hello! Disabling parallel tool calling for each sub-agent should help, described here: https://python.langchain.com/docs/how_to/tool_calling_parallel/

[NithinXD]

import os
import asyncio
import concurrent.futures
from langchain_google_genai import ChatGoogleGenerativeAI
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.prebuilt import create_react_agent
from langgraph_swarm import create_handoff_tool, create_swarm
try:
    from deep import speak, listen, should_quit
    VOICE_AVAILABLE = True
except ImportError:
    print("Voice functionality not available. Install required packages or check deep.py")
    VOICE_AVAILABLE = False

try:
    from agent_gui import initialize_gui, highlight_active_agent, set_gui_status, reset_gui, close_gui
    GUI_AVAILABLE = True
except ImportError:
    print("GUI functionality not available. Install tkinter or check agent_gui.py")
    GUI_AVAILABLE = False

GOOGLE_API_KEY = "AIzREDACTED"

# Voice mode toggle
USE_VOICE = True

# GUI mode toggle
USE_GUI = True

# Define a simple tool for Alice (math operations)
def add(a: int, b: int) -> int:
    """Add two numbers."""
    return a + b

# Initialize the language model
model = ChatGoogleGenerativeAI(model="gemini-1.5-flash", google_api_key=GOOGLE_API_KEY)

# Create Alice agent (math expert)
alice = create_react_agent(
    model=model,
    tools=[add, create_handoff_tool(agent_name="Bob", description="Transfer to Bob for general customer support.")],
    prompt=f"""You are Alice, a customer support agent specializing in math-related queries. 
    Talk friendly and don't reply with just the answer. 
    Before responding, check the CurrentIterMessages in the state to see if another agent has already answered this query or part of it. 
    If Bob has answered a non-math part, acknowledge it (e.g., 'Bob already explained the non-math part, so I'll focus on the math.') and only answer the math part. 
    If the query is non-math, use the transfer_to_bob tool to hand off to Bob, but only if Bob hasn't already responded to this query in CurrentIterMessages. 
    For mixed queries (math and non-math), address the math part first, then use the transfer_to_bob tool for the non-math part if not already answered by Bob. 
    Example: For 'what's a car and what's 5+12', if Bob hasn't answered, say 'The math part 5+12 is 17. Now let me transfer you to Bob for the car question.' and use the transfer_to_bob tool. 
    If Bob already answered the car part in CurrentIterMessages, say 'Bob already explained what a car is, and for the math part, 5+12 is 17.' and do not hand off.""",
    name="Alice",
)

# Create Bob agent (general support, pirate-themed)
bob = create_react_agent(
    model=model,
    tools=[create_handoff_tool(agent_name="Alice", description="Transfer to Alice for math-related queries.")],
    prompt="""You are Bob, a customer support agent who speaks like a pirate. 
    Answer general queries in a friendly pirate manner. 
    Before responding, check the CurrentIterMessages in the state to see if another agent (e.g., Alice) has already answered this query or part of it. 
    If Alice has answered a math part, acknowledge it (e.g., 'Arr, Alice already handled the math, so I'll cover the rest, matey!') and only answer the non-math part. 
    For math questions, use the transfer_to_alice tool to hand off to Alice, but only if Alice hasn't already responded to this query in CurrentIterMessages. 
    Example: For 'what's a car and what's 5+12', if Alice hasn't answered, say 'Arr, a car be a wheeled vehicle that moves on four wheels, matey! Let me transfer ye to Alice for the math query.' and use the transfer_to_alice tool. 
    If Alice already answered the math part in CurrentIterMessages, say 'Arr, Alice already solved the math, and a car be a wheeled vehicle, matey!' and do not hand off.""",
    name="Bob",
)

# Initialize memory checkpointer
checkpointer = InMemorySaver()

# Create the swarm workflow
workflow = create_swarm(
    agents=[alice, bob],
    default_active_agent="Bob"
)

# Compile the workflow with memory
app = workflow.compile(checkpointer=checkpointer)

# Function to interact with the swarm
def run_customer_support(query: str, thread_id: str = "1"):
    config = {"configurable": {"thread_id": thread_id}}
    last_state = checkpointer.get(config) or {"active_agent": "Bob", "CurrentIterMessages": []}
    # Reset CurrentIterMessages for new user input
    state = {
        "messages": [{"role": "user", "content": query}], 
        "active_agent": "Bob", 
        "CurrentIterMessages": []
    }
    results = []
    
    # Iterate until no further handoffs are needed
    max_iterations = 3  # Prevent infinite loops
    iteration = 0
    
    previous_active_agent = "Bob"  # Track the previous active agent
    
    while iteration < max_iterations:
        response = app.invoke(state, config)
        
        # Extract messages from response
        messages = response.get("messages", [])
        if not messages:
            break
        
        # Check for tool calls in the latest AI message
        latest_ai_message = None
        for msg in reversed(messages):
            if hasattr(msg, 'content') and hasattr(msg, 'tool_calls'):
                latest_ai_message = msg
                break
        
        if latest_ai_message and hasattr(latest_ai_message, 'tool_calls') and latest_ai_message.tool_calls:
            pass  # Tool calls detected, continue processing
        
        # Process new messages since last iteration
        for msg in messages[len(state["messages"]):]:
            if hasattr(msg, 'content') and msg.content:
                agent_name = "Unknown"
                if hasattr(msg, 'name') and msg.name:
                    agent_name = msg.name
                elif 'name' in msg.additional_kwargs:
                    agent_name = msg.additional_kwargs.get('name')
                else:
                    content = msg.content.lower()
                    if any(word in content for word in ['arr', 'matey', 'pirate', 'ye']):
                        agent_name = "Bob"
                    elif any(word in content for word in ['math', 'calculate', 'add', 'sum']):
                        agent_name = "Alice"
                
                # Update CurrentIterMessages in state
                state["CurrentIterMessages"].append({
                    'agent_name': agent_name,
                    'content': msg.content
                })
                
                results.append({
                    'content': msg.content,
                    'agent_name': agent_name
                })
        
        # Update state for next iteration
        state = response
        new_active_agent = state.get("active_agent", "Alice")
        
        # Check if handoff occurred by comparing with previous active agent
        if new_active_agent == previous_active_agent and len(results) > 0:
            # Check if there are any pending tool calls that might trigger a handoff
            if not (latest_ai_message and hasattr(latest_ai_message, 'tool_calls') and latest_ai_message.tool_calls):
                break  # No further handoff, exit loop
        
        previous_active_agent = new_active_agent
        iteration += 1
    
    return results if results else [{'content': "No response", 'agent_name': "Unknown"}]

# Async function to interact with the swarm
async def run_customer_support_async(query: str, thread_id: str = "1"):
    loop = asyncio.get_event_loop()
    with concurrent.futures.ThreadPoolExecutor() as executor:
        try:
            response = await loop.run_in_executor(
                executor, 
                run_customer_support, 
                query, 
                thread_id
            )
            return response
        except Exception as e:
            print(f"Error in async workflow: {e}")
            return run_customer_support(query, thread_id)

# Function to get user input
async def get_user_input():
    if USE_VOICE and VOICE_AVAILABLE:
        user_input = await listen()
        if user_input:
            print(f"🧑 You: {user_input}")
        return user_input
    else:
        return input("You: ")

# Function to output response
def output_response(responses):
    for response in responses:
        content = response['content']
        agent_name = response['agent_name']
        
        if USE_GUI and GUI_AVAILABLE:
            highlight_active_agent(agent_name, content)
        
        if USE_VOICE and VOICE_AVAILABLE:
            print(f"🤖 {agent_name}: {content}")
            speak(content, agent_name)
        else:
            print(f"{agent_name}: {content}")

# Function to check if conversation should end
def should_end_conversation(text):
    if USE_VOICE and VOICE_AVAILABLE:
        return should_quit(text)
    else:
        return text.lower() in ["exit", "quit", "bye", "goodbye"]

async def main():
    """Main function to run the customer support bot"""
    mode_text = "Voice Mode" if USE_VOICE and VOICE_AVAILABLE else "Text Mode"
    gui_text = " with GUI" if USE_GUI and GUI_AVAILABLE else ""
    print(f"🤖 Customer Support Bot started in {mode_text}{gui_text}.")
    
    if USE_GUI and GUI_AVAILABLE:
        try:
            initialize_gui()
            set_gui_status("Bot initialized - Ready for conversation")
            print("📱 GUI window opened - check for agent activity display")
        except Exception as e:
            print(f"⚠️  GUI initialization failed: {e}")
    
    if USE_VOICE and VOICE_AVAILABLE:
        print("Say 'goodbye', 'bye', 'exit', or 'quit' to end the conversation.")
    else:
        print("Type 'exit', 'quit', 'bye', or 'goodbye' to end the conversation.")
    
    thread_id = "1"
    
    try:
        while True:
            try:
                if USE_GUI and GUI_AVAILABLE:
                    set_gui_status("Waiting for user input...")
                    reset_gui()
                
                user_input = await get_user_input()
                
                if not user_input or not user_input.strip():
                    if USE_VOICE and VOICE_AVAILABLE:
                        print("❌ No input detected, please try again.")
                    continue
                
                if should_end_conversation(user_input):
                    print("👋 Goodbye!")
                    break
                
                if USE_GUI and GUI_AVAILABLE:
                    set_gui_status("Processing your request...")
                
                if USE_VOICE and VOICE_AVAILABLE:
                    responses = await run_customer_support_async(user_input, thread_id)
                else:
                    responses = run_customer_support(user_input, thread_id)
                
                output_response(responses)
                
            except KeyboardInterrupt:
                print("\n👋 Goodbye!")
                break
            except Exception as e:
                print(f"❌ Error: {e}")
                if USE_GUI and GUI_AVAILABLE:
                    set_gui_status(f"Error: {str(e)[:50]}...")
                continue
    finally:
        if USE_GUI and GUI_AVAILABLE:
            try:
                close_gui()
            except Exception as e:
                print(f"GUI cleanup error: {e}")

if __name__ == "__main__":
    if USE_VOICE and not VOICE_AVAILABLE:
        print("⚠️  Voice mode is enabled but voice functionality is not available.")
        print("Falling back to text mode. To enable voice mode, ensure deep.py is working correctly.")
        USE_VOICE = False
    
    if USE_GUI and not GUI_AVAILABLE:
        print("⚠️  GUI mode is enabled but GUI functionality is not available.")
        print("Falling back to console mode. To enable GUI mode, ensure tkinter is installed.")
        USE_GUI = False
    
    asyncio.run(main())

This is what happens when i run it

🧑 You: What's a football and what's five plus five?
🤖 Bob: Arr, me hearty! Football be a game played with a ball, where teams try to score by kicking or carrying it into the other team's end zone, aye? Let me transfer ye to Alice for the math query.
🤖 transfer_to_alice: Hey, Alice
🤖 Alice: For the math part, five plus five is ten. Let me transfer you to Bob for the football question.
🤖 transfer_to_bob: Hey, Bob
🤖 Bob: Arr, me hearty! I'll leave the five plus five to me matey Alice, but football be a game played with a ball, where teams try to score by kicking or carrying it into the other team's goal, aye? A popular sport 'round the world!

Not using tools here except for handoffs

[marctorsoc]

Hello! Disabling parallel tool calling for each sub-agent should help, described here: https://python.langchain.com/docs/how_to/tool_calling_parallel/

omg, you're a life saver ❤️ . I was getting nuts with this! When passing to another agent, only one of the keys in the state would be passed (I collected name and phone number), and this was the issue.

[marctorsoc]

that being said @dqbd , I did not manage to solve this in a clean way, but with prompting, i.e. asking the LLM not to call the tool and transfer at the same time. I'm getting

...
--- Conversation Turn 3 ---

User: {'messages': [{'role': 'user', 'content': 'Sure, my phone number is 123-456-7890'}]}

in greeting prompt
dict_keys(['messages', 'client_name'])
{'client_name': 'John Doe'}
greeting_agent: Just to confirm, your phone number is 123-456-7890. Is that correct?

--- Conversation Turn 4 ---

User: {'messages': [{'role': 'user', 'content': "Yes, that's correct"}]}

in greeting prompt
dict_keys(['messages', 'client_name'])
{'client_name': 'John Doe'}

in greeting prompt
dict_keys(['messages', 'client_name', 'phone_number'])
{'client_name': 'John Doe', 'phone_number': '123-456-7890'}

in next prompt
dict_keys(['messages', 'active_agent', 'client_name'])
{'client_name': 'John Doe'}
next_agent: Thank you, John. Could you please tell me

...

even thought I set

greeting_agent = create_react_agent(
    llm.bind_tools(greeting_tools, parallel_tool_calls=False),
    greeting_tools,
    prompt=greeting_prompt,
    state_schema=MyState,
    name="greeting_agent",
)

Any ideas?

[marctorsoc]

ok so finally found a solution for this, please see here.

I wonder if should be default to pass the state (or have some param to specify which keys in create_handoff_tool. I could do a PR for this if you think it's a good idea @dqbd @rlancemartin ?

[eyurtsev]

@NithinXD your API key was included in this issue. Assume it's comporomised. I redacted it in the issue text, but please revoke it ASAP