Skip to content

Conversation

@qandrew
Copy link
Contributor

@qandrew qandrew commented Nov 7, 2025

Purpose

I noticed that our tool parser often times may leave whitespace after the tool call, which gets outputted to the user. We should strip it out.

Test Plan

qwen3 example:

vllm serve Qwen/Qwen3-1.7B --reasoning-parser qwen3       --structured-outputs-config.backend xgrammar       --enable-auto-tool-choice --tool-call-parser hermes

client

 curl -X POST "http://localhost:8000/v1/responses"   -H "Content-Type: application/json"   -H "Authorization: Bearer dummy-api-key"   -d '{
        "model": "Qwen/Qwen3-1.7B",
        "input": "Whats the weather like in San Francisco? use celsius.",
        "tools": [{
            "type": "function",
            "name": "get_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location", "unit"]
            }
        }]
      }'

Test Result

{
    "id": "resp_3814bea59f4449a7a29c9a406b50ab4e",
    "created_at": 1762398952,
    "incomplete_details": null,
    "instructions": null,
    "metadata": null,
    "model": "Qwen/Qwen3-1.7B",
    "object": "response",
    "output": [
        {
            "id": "rs_8a39e9f4178d40abac111d595d801126",
            "summary": [],
            "type": "reasoning",
            "content": [
                {
                    "text": "\nOkay, the user is asking for the weather in San Francisco using Celsius. Let me check the tools provided. There's a function called get_weather that requires location and unit. The parameters are location (string) and unit (enum: celsius or fahrenheit). The user specified San Francisco and Celsius, so I need to call get_weather with those arguments. I should make sure the arguments are in the correct format. Let me structure the JSON accordingly.\n",
                    "type": "reasoning_text"
                }
            ],
            "encrypted_content": null,
            "status": null
        },
### BELOW GETS REMOVED AFTER THIS PR ####
        {
            "id": "msg_33be381a036e49e8abc0faead1e05752",
            "content": [
                {
                    "annotations": [],
                    "text": "\n\n",
                    "type": "output_text",
                    "logprobs": null
                }
            ],
            "role": "assistant",
            "status": "completed",
            "type": "message"
        },
### ABOVE GETS REMOVED AFTER THIS PR ####
        {
            "arguments": "{\"location\": \"San Francisco\", \"unit\": \"celsius\"}",
            "call_id": "call_62165488d94f43f1ab949be8f63ab20f",
            "name": "get_weather",
            "type": "function_call",
            "id": "fc_9e77818e2fee464ea9853758d198ee01",
            "status": "completed"
        }
    ],
    "parallel_tool_calls": true,
    "temperature": 0.6,
    "tool_choice": "auto",
    "tools": [
        {
            "name": "get_weather",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string"
                    },
                    "unit": {
                        "type": "string",
                        "enum": [
                            "celsius",
                            "fahrenheit"
                        ]
                    }
                },
                "required": [
                    "location",
                    "unit"
                ]
            },
            "strict": null,
            "type": "function",
            "description": "Get the current weather in a given location"
        }
    ],
    "top_p": 0.95,
    "background": false,
    "max_output_tokens": 40766,
    "max_tool_calls": null,
    "previous_response_id": null,
    "prompt": null,
    "reasoning": null,
    "service_tier": "auto",
    "status": "completed",
    "text": null,
    "top_logprobs": null,
    "truncation": "disabled",
    "usage": {
        "input_tokens": 194,
        "input_tokens_details": {
            "cached_tokens": 0,
            "input_tokens_per_turn": [],
            "cached_tokens_per_turn": []
        },
        "output_tokens": 123,
        "output_tokens_details": {
            "reasoning_tokens": 0,
            "tool_output_tokens": 0,
            "output_tokens_per_turn": [],
            "tool_output_tokens_per_turn": []
        },
        "total_tokens": 317
    },
    "user": null,
    "input_messages": null,
    "output_messages": null
}

@qandrew qandrew marked this pull request as ready for review November 7, 2025 23:42
@mergify mergify bot added the frontend label Nov 7, 2025
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines 1378 to 1379
if content.strip() == "":
content = None

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0 Badge Guard None before trimming parsed content

When extract_tool_calls() returns content=None (e.g., OpenAIToolParser leaves final_content unset when a model emits only tool calls), the new content.strip() call will raise AttributeError and abort the request. Previously this branch safely returned (function_calls, None); now any tool-only response will crash automatic tool parsing instead of succeeding. A simple if content and content.strip() == "": ... avoids the regression.

Useful? React with 👍 / 👎.

@yeqcharlotte yeqcharlotte enabled auto-merge (squash) November 7, 2025 23:45
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 7, 2025
for tool_call in tool_call_info.tool_calls
)
content = tool_call_info.content
if content.strip() == "":
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should we remove whitespace characters?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chaunceyjiang if we keep it, we get an extra output item like this, which is pointless to show for the user

        {
            "id": "msg_33be381a036e49e8abc0faead1e05752",
            "content": [
                {
                    "annotations": [],
                    "text": "\n\n",
                    "type": "output_text",
                    "logprobs": null
                }
            ],
            "role": "assistant",
            "status": "completed",
            "type": "message"
        },

Signed-off-by: Andrew Xia <axia@fb.com>
auto-merge was automatically disabled November 8, 2025 00:13

Head branch was pushed to by a user without write access

@qandrew qandrew force-pushed the strip-tool-content branch from 1e3de73 to 0e90d57 Compare November 8, 2025 00:13
@yeqcharlotte yeqcharlotte merged commit 4b94ed8 into vllm-project:main Nov 10, 2025
48 checks passed
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Nov 13, 2025
…ent (vllm-project#28331)

Signed-off-by: Andrew Xia <axia@fb.com>
Co-authored-by: Andrew Xia <axia@fb.com>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
devpatelio pushed a commit to SumanthRH/vllm that referenced this pull request Nov 29, 2025
…ent (vllm-project#28331)

Signed-off-by: Andrew Xia <axia@fb.com>
Co-authored-by: Andrew Xia <axia@fb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

frontend ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants