InteractiveUtils: Support callable objects as functions in introspection macros #58905

serenity4 · 2025-07-04T14:40:20Z

Follow-up to the follow-up #57911, building on the changes to introspection functions to support signature tuples being provided as a single argument.

This enables support for calls of the form

@code_typed (::Returns{Int})(1)
@code_llvm (::Base.Fix2{typeof(+), Float64})(::Int)

by providing an extra use_signature_tuple::Bool = false parameter in gen_call_with_extracted_types. Setting this parameter to true changes the code generation from $fcn(f, Tuple{argtypes...}) to $fcn(Tuple{f, argtypes...}) (where $fcn can be e.g. code_typed, code_llvm etc).

This allows you to call, e.g., `code_typed((Foo, Int, Int))` to query the code for `(::Foo)(::Int, ::Int)`

…ode_lowered`

… `code_warntype`

This allows you to call, e.g., `code_typed((Foo, Int, Int))` to query the code for `(::Foo)(::Int, ::Int)`

…ode_lowered`

… `code_warntype`

…utils-functors

…nto interactive-utils-functors

topolarity · 2025-07-07T17:36:44Z

Has this impacted OpaqueClosure support on accident?

julia> const oc = Base.Experimental.@opaque Tuple{Int,Int}->Int (x,y)->(x+y)
(::Int64, ::Int64)->◌::Int64

julia> @code_typed oc(1,2)
OpaqueClosure(...) @ Core none:0 => Any

serenity4 · 2025-07-07T17:38:36Z

Oh, I think we had a branch on isa(f, OpaqueClosure) in the two-argument form of code_typed 🤔

serenity4 · 2025-07-07T17:39:30Z

Perhaps we can have that branch for tt.parameters[1] <: OpaqueClosure for code_typed(tt::Type{<:Tuple})?

topolarity · 2025-07-07T17:40:24Z

Yeah, that's why I ended up adding ArgInfo in #58891

I wonder if some equivalent should be moved into base/reflection.jl (or yeah, alternatively we could just make it the standard behavior of code_typed(::Type{<:Tuple}))

…utils-functors

serenity4 · 2025-07-07T22:02:41Z

The tricky bit with opaque closures is that because every "method" can't be distinguished by their type alone, we need to provide the values to reflection functions (whereas at least for callables, we still use types)

To support that with the current implementation, we can use the 2-argument form for code_typed(f, tt) when we have an OpaqueClosure as first argument. The downside is that the macro will have to generate code for both code_typed(argtypes) and code_typed(f, tt) because we'll only know at runtime which branch should be taken.

It seems cleaner to use the ArgInfo form, but we can't assume the user function supports a method for it, otherwise all packages relying on gen_call_with_extracted_types_and_kwargs will stop working. Although, we could at least construct ArgInfo internally, then use a higher-order function to call a one-argument/two-argument method for a given function based on the type of arg0, so long as we never leak that internal structure to the user. I can give it a try.

topolarity · 2025-07-07T22:31:48Z

otherwise all packages relying on gen_call_with_extracted_types_and_kwargs will stop working.

Isn't that already an issue with switching to the 1-arg form from the 2-arg form?

serenity4 · 2025-07-07T23:23:47Z

Isn't that already an issue with switching to the 1-arg form from the 2-arg form?

That's exactly why I only generate the 1-arg form conditionally on a non-default parameter being set. We could have a parameter that indicates whether or not to dispatch on ArgInfo, though.

topolarity · 2025-07-14T14:56:25Z

Yeah, I agree with your breakdown.

I don't have a strong feeling among the alternatives, so I'd be fine if you want to pick one and see if the implementation feels all right.

…ia; branch 'master' of github.com:JuliaLang/julia into interactive-utils-functors

serenity4 · 2025-08-18T11:14:06Z

I managed to fix the situation with fewer modifications than I thought it'd require. @topolarity feel free to have another look, then it should be good to merge.

topolarity

Probably worth documenting the macro-dev-facing requirements for setting these boolean flags (I already forgot since last time, hah), but otherwise looks great.

stdlib/InteractiveUtils/src/macros.jl

serenity4 · 2025-08-19T12:20:31Z

You're right, I held off on documenting these things because I didn't want us to commit to anything but it would be helpful to macro developers using this functionality (and to core devs too).

serenity4 · 2025-08-19T15:46:07Z

@topolarity does the docstring look clear enough to you? I think I included pretty much everything that developers need to know. Suggestions welcome!

topolarity · 2025-08-19T18:20:53Z

Reads beautifully (and so many of my old questions answered!)

No notes 👍 I think this is ready to merge

topolarity · 2025-08-21T22:49:05Z

@serenity4 Can you fix up the conflicts + failing doctest here?

serenity4 · 2025-08-22T09:00:06Z

Yes, I started yesterday but #59239 brought a tough conflict (not a bad one though!). It changed the way that macro hygiene is handled in gen_call_with_extracted_types, and it took me some time to figure out what we should do. Notably because I also rewrote how we handle hygiene in this PR, and in a different direction than this other PR - and the design there is more composable than here so I'll most likely adopt it (unless I run into issues).

This commit refactors some of the recent additions to handle reescaping in InteractiveUtils, along with tests to cover various hygiene contexts. `replace_ref_begin_end!` was recently replaced in InteractiveUtils by its low-level counterpart, `replace_ref_begin_end_!`, to better handle escapes. However, we were using it in a way that required duplication of the code generation logic to match the code that we emit it for other code paths. Because the logic is now more complex with the recently added `use_signature_tuple` behavior, it is preferable to allow reuse of the same code path for all code we emit. This commit implements this refactor, which unfortunately also requires a manual escape of the temporary variable that is emitted by `replace_ref_begin_end!` in its current design.

…utils-functors

serenity4 · 2025-08-25T13:41:39Z

Alright, after quite a few efforts, I managed to come up with a conflict resolution I am more satisfied with in c8b7dc7 (this commit also shows removal of some of the parts from the merge that were commented out).

This replaces the example with a macro call that has a simpler output, now that `edit` generates a bigger expression (as it now handles opaque closure arguments)

serenity4 · 2025-08-25T20:51:56Z

@topolarity feel free to take a quick look at the recent commits, and otherwise merge. Thanks for having followed this PR until now 💯

topolarity

I haven't reviewed the hygiene changes, but otherwise this looks great to me.

Thanks for all the follow-through @serenity4

vtjnash · 2025-08-26T17:09:33Z

stdlib/InteractiveUtils/src/macros.jl

+function reescape(f::Function, @nospecialize ex)
+    isa(ex, Expr) || return f(ex)
+    unescaped = Meta.unescape(ex)
+    new = f(unescaped)
+    return reescape(new, ex)
+end


Why are you vendoring a buggy copy of Meta.reescape now?

What is buggy with its implementation? My understanding is that the pattern is:

Get the escaped argument (with Meta.unescape)

Process it to possibly generate a new Expr (that's "user"-defined)

Re-apply the original hygiene environment (escapes, hygienic scopes etc)

It is reimplementing something and it might execute any functions f instead of rewrapping them

vtjnash · 2025-08-26T17:10:34Z

stdlib/InteractiveUtils/src/macros.jl

+        isexpr(kwarg, :escape) && continue
+        isexpr(kwarg, :var"hygienic-scope") && continue


These are invalid kwargs, unless unescaping them contains a valid kwarg

That's true, I guess the correct way would be to apply the check to whatever is inside.

vtjnash · 2025-08-26T17:14:09Z

stdlib/InteractiveUtils/src/macros.jl

+    assignment = decl.args[1]
+    isexpr(assignment, :(=), 2) || return
+    variable = assignment.args[1]
+    startswith(string(variable), "##S#") || return


This seems like a bad idea to find some random variable by name and then apply random mutation to it (mainly it seems it would result in incorrect code being executed if someone has a statement with multiple begin or end like a[end][end])

vtjnash · 2025-08-26T17:17:51Z

stdlib/InteractiveUtils/src/macros.jl

@@ -262,10 +420,10 @@ function gen_call_with_extracted_types(__module__, fcn, ex0, kws = Expr[]; is_so
            end
            fully_qualified_symbol &= ex1 isa Symbol
            if fully_qualified_symbol || isexpr(ex1, :(::), 1)
-                call_reflection = :($(fcn)(Base.getproperty, $(typesof_expr(ex0.args, where_params))))
+                call_reflection = gen_call(fcn, [getproperty; ex0.args], where_params, kws; use_signature_tuple)


Style (and performance) nit: we tend to use ... instead of ; for explicit concat, and an explicit Any type:

call_reflection = gen_call(fcn, Any[getproperty, ex0.args...], where_params, kws; use_signature_tuple)

This seems to have been detected as a issue elsewhere already, but then incorrectly worked around by splatting ex0.args to undo the mistake here:

function gen_call(fcn, args, where_params, kws; use_signature_tuple::Bool, not_an_opaque_closure::Bool = true) f, args... = args args = collect(Any, args)

Ah, yeah I didn't notice this array was not typed with Any[. Performance/style-wise I thought that ; was preferred for this pattern, but I don't mind at all using ....

The args = collect(Any, args) statement was because I thought the args... syntax did use the type of actual arguments, rather than reuse the original array type, but if that's not the case the statement may indeed be removed.

Good question. Seems like it keeps the original type:

julia> a,b... = Any[1,2,3] 3-element Vector{Any}: 1 2 3 julia> b 2-element Vector{Any}: 2 3

... implies vect and ; implies vcat. It is thus strange to have vcat when some of the elements are not arrays, though perhaps efficient. With Any[...] it actually implies getindex, and getindex(::Type{Any}) is marked no specialize.

vtjnash · 2025-08-26T17:20:16Z

stdlib/InteractiveUtils/src/macros.jl

@@ -292,41 +450,24 @@ function gen_call_with_extracted_types(__module__, fcn, ex0, kws = Expr[]; is_so
                $(esc(ex0)) # trigger syntax errors if any
            end
            nt = generate_merged_namedtuple_type(kwargs)
-            tt = rewrap_where(:(Tuple{$nt, $(get_typeof.(args)...)}), where_params)
-            return :($(fcn)(Core.kwcall, $tt; $(kws...)))
+            nt = Ref(nt) # ignore `get_typeof` handling


This seems very sketchy, since Ref(nt) is a perfectly fine, and somewhat common, thing to normally encounter in an AST already (after this function called macroexpand). There should not be any unique behavior of calling typeof based on the value of a object found in the code.

That's... completely true. That should be changed.

vtjnash · 2025-08-26T17:21:32Z

stdlib/InteractiveUtils/src/macros.jl

        else
-            PairSymAny = Pair{Symbol, Any}
-            for (head, f) in (PairSymAny(:hcat, Base.hcat),


Please reapply this change to add PairSymAny, so that this part of the code can be type-stable

Is there any actual benefit in doing so? I would mind in most cases, but not in a high-level macro like that, where I'd rather reduce visual noise instead.

I guess not, just seemed cleaner for future linters

I think we have quite a few places (probably a lot of expression-processing functions notably) where we iterate ::Vector{Any} anyways, instead of iterating over indices then index into the array (which I guess might be the "best" pattern for type-stability).

vtjnash · 2025-08-26T17:22:19Z

stdlib/InteractiveUtils/src/macros.jl

            end
        end
    end
    if isa(ex0, Expr) && ex0.head === :macrocall # Make @edit @time 1+2 edit the macro by using the types of the *expressions*
-        return Expr(:call, fcn, esc(ex0.args[1]), Tuple{#=__source__=#LineNumberNode, #=__module__=#Module, Any[ Core.Typeof(ex0.args[i]) for i in 3:length(ex0.args) ]...}, kws...)
+        args = [#=__source__::=#LineNumberNode, #=__module__::=#Module, Core.Typeof.(ex0.args[3:end])...]
+        return gen_call(fcn, Any[ex0.args[1], Ref.(args)...], where_params, kws; use_signature_tuple)


same comment about Ref.() being problematic here (because it shouldn't affect behavior and because is is the wrong way to map to an Any array)

vtjnash · 2025-08-26T17:24:49Z

stdlib/InteractiveUtils/src/macros.jl

-        if any(@nospecialize(x) -> isexpr(x, :...), ex0.args) &&
-            (ex.args[1] === GlobalRef(Core,:_apply_iterate) ||
-             ex.args[1] === GlobalRef(Base,:_apply_iterate))
-            # check for splatting


Did this get deleted by accident? I don't see why you are still calling Meta.lower if you unconditionally ignore the result now (and I'm also not entirely certain what this part of the code did before either).

I never encountered any execution of that code, and at some point it was committed with an unconditional error (it was using an undefined variable) so I am quite confident it was dead code. My speculative explanation for it would be that we started generating _apply_iterate expressions differently at some point so this wasn't hit anymore.

Meta.lower was just used with the previous check (isa(ex, Expr)) but I agree that we should probably use that other error below instead regardless.

topolarity and others added 7 commits July 3, 2025 13:49

Support "Functor-like" code_typed invocation

1fae8e7

This allows you to call, e.g., `code_typed((Foo, Int, Int))` to query the code for `(::Foo)(::Int, ::Int)`

Base: Support functors in method_instance(s), code_ircode, and `c…

395feea

…ode_lowered`

InteractiveUtils: Support functors in code_llvm, code_native, and…

16c6e5e

… `code_warntype`

Support "Functor-like" code_typed invocation

33e4b05

This allows you to call, e.g., `code_typed((Foo, Int, Int))` to query the code for `(::Foo)(::Int, ::Int)`

Base: Support functors in method_instance(s), code_ircode, and `c…

3ffce1e

…ode_lowered`

InteractiveUtils: Support functors in code_llvm, code_native, and…

ec2f4c8

… `code_warntype`

Add support for directly providing signature tuple

ef7727e

serenity4 requested a review from topolarity July 4, 2025 14:40

serenity4 changed the title ~~Support callable objects as functions in introspection macros~~ InteractiveUtils: Support callable objects as functions in introspection macros Jul 4, 2025

serenity4 self-assigned this Jul 4, 2025

serenity4 added 3 commits July 4, 2025 14:42

Merge branch 'master' of github.com:JuliaLang/julia into interactive-…

394f15d

…utils-functors

Merge branch 'ct/functor-reflection' of github.com:topolarity/julia i…

76f030c

…nto interactive-utils-functors

Merge branch 'master' into interactive-utils-functors

84dd768

Merge branch 'master' of github.com:JuliaLang/julia into interactive-…

865739e

…utils-functors

serenity4 and others added 3 commits August 6, 2025 05:13

Improve error message

5f5db13

Merge branch 'interactive-utils-functors' of github.com:serenity4/jul…

57fbef2

…ia; branch 'master' of github.com:JuliaLang/julia into interactive-utils-functors

Fix implementation for OpaqueClosure

551e0ea

serenity4 force-pushed the interactive-utils-functors branch from 1aad66f to 551e0ea Compare August 18, 2025 11:07

Add additional test

12dacbe

topolarity approved these changes Aug 18, 2025

View reviewed changes

stdlib/InteractiveUtils/src/macros.jl Show resolved Hide resolved

stdlib/InteractiveUtils/src/macros.jl Outdated Show resolved Hide resolved

stdlib/InteractiveUtils/src/macros.jl Show resolved Hide resolved

serenity4 added 2 commits August 19, 2025 15:41

Document gen_call_with_extracted_types

8dea09e

Enable support for callable objects for source reflection macros

c9ec156

Add kws... in docs for fcn usage

73a9256

Merge branch 'master' into interactive-utils-functors

e7b88ee

serenity4 added 2 commits August 25, 2025 13:22

Merge branch 'master' of github.com:JuliaLang/julia into interactive-…

ae98622

…utils-functors

Fix doctest failure

8a60b9f

This replaces the example with a macro call that has a simpler output, now that `edit` generates a bigger expression (as it now handles opaque closure arguments)

topolarity approved these changes Aug 26, 2025

View reviewed changes

topolarity merged commit b24014a into JuliaLang:master Aug 26, 2025
7 checks passed

vtjnash reviewed Aug 26, 2025

View reviewed changes

yuyichao mentioned this pull request Aug 26, 2025

Fix at-views for setindex expression #59403

Open

		isexpr(kwarg, :escape) && continue
		isexpr(kwarg, :var"hygienic-scope") && continue

Uh oh!

InteractiveUtils: Support callable objects as functions in introspection macros #58905

InteractiveUtils: Support callable objects as functions in introspection macros #58905

Conversation

serenity4 commented Jul 4, 2025

Uh oh!

topolarity commented Jul 7, 2025

Uh oh!

serenity4 commented Jul 7, 2025

Uh oh!

serenity4 commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

topolarity commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serenity4 commented Jul 7, 2025

Uh oh!

topolarity commented Jul 7, 2025

Uh oh!

serenity4 commented Jul 7, 2025

Uh oh!

topolarity commented Jul 14, 2025

Uh oh!

serenity4 commented Aug 18, 2025

Uh oh!

topolarity left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

serenity4 commented Aug 19, 2025

Uh oh!

serenity4 commented Aug 19, 2025

Uh oh!

topolarity commented Aug 19, 2025

Uh oh!

topolarity commented Aug 21, 2025

Uh oh!

serenity4 commented Aug 22, 2025

Uh oh!

serenity4 commented Aug 25, 2025

Uh oh!

serenity4 commented Aug 25, 2025

Uh oh!

topolarity left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vtjnash Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serenity4 Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

serenity4 commented Jul 7, 2025 •

edited

Loading

topolarity commented Jul 7, 2025 •

edited

Loading

vtjnash Aug 26, 2025 •

edited

Loading

serenity4 Aug 26, 2025 •

edited

Loading