feat: Subagent and Custom Agent Tools

[gadenbuie]

This PR introduces two complementary features that enable hierarchical agent workflows: a built-in subagent tool for task delegation, and custom agent tools that can be defined via markdown files.

Overview

Subagent Tool (btw_tool_agent_subagent)

The subagent tool allows an orchestrating LLM agent to delegate tasks to specialized subagents. Each subagent runs in its own isolated chat session with restricted tool access and maintains conversation state that can be resumed across multiple calls.

Key behaviors:

• Subagents get their own chat client (configurable via btw.subagent.client)
• Sessions are identified by human-readable IDs like "swift_falcon" (adjective_noun pattern)
• Sessions persist for the R session lifetime, then are automatically cleaned up
• Tool access is controlled by two options:
  • btw.subagent.tools_allowed (security whitelist - hard boundary)
  • btw.subagent.tools_default (convenience - what you get when nothing specified)

Custom Agent Tools

Custom agents are specialized assistants defined via agent-*.md files that are automatically discovered and registered as callable tools. These integrate seamlessly with the existing btw_tools() infrastructure.

Key behaviors:

• Agents are discovered from multiple locations with clear precedence:
  1. Project btw-style: .btw/agent-*.md
  2. User btw-style: ~/.btw/agent-*.md and ~/.config/btw/agent-*.md
  3. Project Claude Code: .claude/agents/*.md
  4. User Claude Code: ~/.claude/agents/*.md
• btw-style agents take precedence over Claude Code agents when names collide
• Agent files have YAML frontmatter with fields like name, description, client, tools, icon
• The markdown body becomes additional system prompt content, appended to the base subagent prompt
• No caching - files are re-discovered on every btw_tools() call to ensure changes are
  reflected immediately

Architecture Integration

Shared Infrastructure

Custom agents are built on top of the subagent infrastructure. When a custom agent tool is called, it uses the same session management, client configuration, and result format as the built-in subagent tool.

Recursion Prevention Strategy

The built-in subagent tool cannot spawn other subagents to prevent infinite recursion. This is enforced at multiple levels:

1. Pre-instantiation check: btw_can_register_subagent_tool() prevents the tool from being included when building a subagent's tool description (via the can_register pattern in .btw_add_to_tools())
2. Explicit request error: If an orchestrator explicitly requests "btw_tool_agent_subagent" in the tools list, an error is thrown immediately
3. Silent filtering: When the subagent tool appears via a tool group (e.g., "agent"), it's silently filtered out

However, custom agents CAN call other agents (including other custom agents). This is intentional — users define custom agents and accept responsibility for avoiding cycles. Natural limits (token limits, request limits) provide guardrails against runaway chains.

Tool Discovery and Registration

Custom agents are merged into btw_tools() at call time rather than being statically registered in .btw_tools. This allows working directory changes to affect discovery and maintains clean separation between built-in and custom tools.

Duplicate agent names are detected across all discovery locations, and warnings are issued with the lower-priority file being skipped.

System Prompt Composition

Custom agent prompts are appended to the base btw-subagent.md prompt rather than replacing it. This ensures all agents follow core btw conventions while allowing specialization. A visual separator (\n\n---\n\n) makes the composition clear.

Icon System

Custom agents support three icon formats:

1. Plain Font Awesome names via shiny::icon(): icon: robot
2. Package-prefixed icons for other icon libraries: fontawesome::home, bsicons::house, phosphoricons::house, etc.
3. Raw SVG strings: icon: '<svg>...</svg>'

If a package isn't installed or an icon name is invalid, the system warns and falls back to the default agent icon.

Claude Code Compatibility

btw can load agent files from Claude Code's .claude/agents/ directory with automatic
translation:

• Name normalization: Hyphens are converted to underscores (code-reviewer → code_reviewer) for R identifier compatibility
• Field translation: The model field is ignored in favor of btw's client field
• Unsupported fields: tools, permissionMode, and skills trigger warnings but don't break loading
• Precedence: btw-style agents take precedence over Claude Code agents to avoid unexpected behavior changes

Configuration Precedence

The configuration system follows a consistent precedence pattern:

For client:

1. Function argument (client to btw_agent_tool())
2. Agent YAML client field
3. btw.subagent.client option
4. btw.client option
5. Package defaults

For tools:

1. Agent YAML tools field (btw-style agents only)
2. btw.subagent.tools_default option
3. btw.tools option
4. All non-agent tools (default)

The btw.subagent.tools_allowed option always acts as a final security filter regardless of source.

Important Implementation Details

The can_register Pattern

Tools can define a can_register function that's checked before $tool() is called. This prevents recursion when a tool's $tool() function needs to resolve other tools (like the subagent tool building its description of available tools). The pattern looks like:

.btw_add_to_tools(
  name = "btw_tool_agent_subagent",
  group = "agent",
  can_register = function() btw_can_register_subagent_tool(),
  tool = function() { ... }
)

Session Management

Sessions are stored in the package-level environment .btw_subagent_sessions as a list containing id, chat, and created timestamp. Sessions intentionally have no serialization or complex cleanup logic - they persist for the R session duration and are automatically cleaned up when R exits.

Error Handling Philosophy

For custom agents, the system warns and skips invalid agent files rather than failing completely. One bad file shouldn't break all tools. Users get feedback via warnings while other valid agents continue to work.

Client Cloning

Chat client objects are always cloned before being used in a subagent session to prevent state pollution between the subagent and parent session, or between different subagent sessions.

[simonpcouch]

This is stellar. Only providing the last assistant message as the output, allowing the main agent to configure which tools the subagent will access to, the tool UI, etc.

The only additional feature that I might consider is allowing the main agent to toggle whether the subagent should inherit the turns from the main agent. There are many situations where I'm fine to pay for the cost of the subagents "re-reading" tokens in exchange for it knowing enough to do the task right.

[gadenbuie]

@simonpcouch Would you mind giving this another look? It's now full of big changes and features that probably should be two separate PRs but got too entangled to keep apart.

I took the foundation of the subagent tool (now btw_tool_agent_subagent()) and turned it into btw_agent_tool(), a function that takes a path to a btw-style .md file and turns it into a custom agent. This can be called directly by users to turn any .md file into a custom agent, or you can store agents in agent-*.md files in .btw/ in a project or ~/.btw or ~/.config/btw for global agents. The agent md files are discovered automatically and included in btw_tools() under the "agent" group. An agent with name: code_reviewer becomes btw_tool_agent_code_reviewer().

Additionally, we auto-discover Claude Code's custom subagent files in .claude/agents/ or ~/.claude/agents/ and include those in btw_tools() as well. Those files also use YAML frontmatter, but we ignore tools and model in CC agent files since they're hard to resolve with ellmer/btw (along with other obvious misses, like permissionMode). In the future, we could definitely support skills (#145).

[simonpcouch]

I'm in support of these new changes, as well! Looks great.

I'll underscore that request on being able to optionally send along the full conversation history to subagents. This has been a limiter for my own use of subagents with Claude Code--I often find myself wanting to build up context that would be useful to all subagents, then dispatching to them, as I've been bitten before by the main agent learning some important subtleties from context but failing to articulate all of it the subagents, whose sloppy work ends up thrown away.

[simonpcouch]

My gut feeling is that many of these @noRd entries give the "what" rather than the "why" of each of these helpers and ought to be removed, maybe sometimes in favor of a short line motivating / explaining some design choice.

[gadenbuie]

Yeah, I agree. Claude really likes writing these but they don't really add anything useful.

[simonpcouch]

Suggested change

	#' directory for compatibility. However, some Claude Code fields are not
	#' supported:
	#' directory:

I initially read this as "all of the things in the list below are not supported"

[gadenbuie]

I've been bitten before by the main agent learning some important subtleties from context but failing to articulate all of it the subagents, whose sloppy work ends up thrown away.

Does CC "converse" with subagents? i.e. can it return to a subagent to refine the output? Answer: yes, subgaents are resumable, but you have to prompt Claude to re-use an agent by asking directly and mentioning the agent id.

optionally send along the full conversation history to subagents

The biggest design hesitation for me here is where and how the "optionally" is decided. For custom agents, we could make in an option that's set in its YAML front matter and a custom agent would either inherit the main chat history or not. For subagents, it's unclear to me how that would be decided. I can also see situational advantages to sometimes including full history for custom agents, which would also be tricky.

Resuming conversation with a custom/subagent introduces another wrinkle, but I think it'd be reasonable to say that subagents inherit history when first created and that it wouldn't be updated when resumed.