`fn(...) -> Option<T>`, where T is returned by invisible reference, could use null pointer for None

[zackw]

Consider a function that returns Option<T> where T is too large to be returned in registers, such as

const N: usize = 8;

pub fn fill_v(val: u64) -> Option<[u32; N]> {
    let v = u32::try_from(val).ok()?;
    Some(core::array::from_fn(|_| v))
}

Because [u32;8] doesn't fit in the ABI's return value registers, instead it's returned by "invisible reference"; at the assembly level, caller provides fill_v with two arguments, val and also a pointer to space for the return value. Right now, when you return Option<[u32;8]>, the Option discriminator takes up part of that space:

playground::fill_v: # @playground::fill_v
# %bb.0:
	movq	%rdi, %rax
	xorl	%ecx, %ecx
	movq	%rsi, %rdx
	shrq	$32, %rdx
	jne	.LBB0_2
# %bb.1:
	movd	%esi, %xmm0
	pshufd	$0, %xmm0, %xmm0                # xmm0 = xmm0[0,0,0,0]
	movdqu	%xmm0, 4(%rax)
	movdqu	%xmm0, 20(%rax)
	movl	$1, %ecx

.LBB0_2:
	movl	%ecx, (%rax)
	retq

(The invisible reference is the first argument at the assembly level, in %rdi; the Rust-level first argument val is in %rsi.)

But the invisible reference argument is necessarily non-NULL, so we could apply the null pointer optimization for Option in this case, even though we can't do it in general for Option<[T;N]>, and generate code like this instead:

playground::fill_v: # @playground::fill_v
# %bb.0:
	xorl	%eax, %eax
	movq	%rsi, %rdx
	shrq	$32, %rdx
	jne	.LBB0_2
# %bb.1:
	movd	%esi, %xmm0
	pshufd	$0, %xmm0, %xmm0                # xmm0 = xmm0[0,0,0,0]
	movdqu	%xmm0, (%rdi)
	movdqu	%xmm0, 16(%rdi)
	movq	%rdi, %rax
.LBB0_2:
	retq

This would save a word of stack space and would probably also be more efficient for the caller.

The example is an array, but the optimization is equally applicable to Option<LargeStruct>, etc.

Meta

rustc --version --verbose:

1.90.0-nightly (2025-07-21 9748d87dc70a9a6725c5)

[hanna-kruppe]

If I understand correctly, what you’re proposing means that the effective ABI is fn(dest_if_some: &mut MaybeUninit<T>) -> Discriminant<Option<T>> which returns “Some” by writing to *dest_if_some and returning a small value indicating it did so.

This is sometimes profitable, but mostly when the Some/None choice is done directly in the callee and the caller immediately matches on the return value. If it copies an existing Option<T> value from somewhere else (including a local variable whose address escapes), it needs a bit of extra work to materialize the tag (if it’s stored in a niche). If the caller needs an actual Option<T> value, it has to (re-)assemble it from the tag and T value. So it’s hard to know that this is better than just returning the whole Option<T> by value, except in the cases where inlining is also an option and gives even greater wins.

So IMO this potential optimization isn’t attractive enough frequently enough to make it a priority for anyone to implement.

[zackw]

If I understand correctly, what you’re proposing means that the effective ABI is fn(dest_if_some: &mut MaybeUninit) -> Discriminant<Option> which returns “Some” by writing to *dest_if_some and returning a small value indicating it did so.

Yes, essentially. The assembly code I showed uses the dest_if_some pointer as the "small (nonzero) value", which may or may not be convenient for the caller.

This is sometimes profitable, but mostly when the Some/None choice is done directly in the callee and the caller immediately matches on the return value.

My expectation is that will be the most common way a function that returns Option<LargeUnboxedValue> will be implemented and used. However, my expectations are trained rather more on C than Rust and I don't have any data to back this up.

[scottmcm]

instead it's returned by "invisible reference"

I think it's better to look at this in LLVM. What we give it is https://rust.godbolt.org/z/fdo4dTMxT

define void @example::fill_v::hdf24ec4e5aa097ac(ptr dead_on_unwind noalias nocapture noundef writable sret([36 x i8]) align 4 dereferenceable(36) %_0, i64 noundef %val) unnamed_addr {

It's an sret to the 36 bytes of space needed for the value, so there's no pointer being "returned" that we could make nullable -- indeed the function has to return void to use sret https://llvm.org/docs/LangRef.html#parameter-attributes

So I think what you're asking isn't for a null pointer return, but instead something like

define i1 @example::fill_v::hdf24ec4e5aa097ac(ptr dead_on_unwind noalias nocapture noundef writable align 4 dereferenceable(32) %_0, i64 noundef %val) unnamed_addr {

(or maybe define range(i32, 0, 2) i32 @..., to match the tag field in Option<[u32; 8]>).

[hanna-kruppe]

My suspicion is that if you’re in a scenario where pervasive inlining isn’t reliable or desirable, it’s also very easy to take a reference to Option<T> somewhere along the way that isn’t optimized out, because Rust without optimizations has lots and lots of &T parameters on every corner. There’s no equivalent to those implicit “reconstruct, spill, deconstruct” detours in C code that manually splits payload and discriminant.

[zackw]

I think it's better to look at this in LLVM.

I don't know LLVM intermediate representation at all, so I can't assess your alternative. I suggested that we null out the return of the caller-supplied pointer because that agrees with the existing "null pointer optimization" for Option<&T> and similar.

But I do know ABIs and assembly language, and what's happening in the assembly language is that the caller passes a (guaranteed non-null) pointer to the space to store the return value, and that pointer is being copied into the (primary integer) return value register. And this agrees with what the x86-64 psABI specifies (item 2 of the "Returning of Values" checklist on page 28 of https://gitlab.com/x86-psABIs/x86-64-ABI/-/jobs/artifacts/master/raw/x86-64-ABI/abi.pdf?job=build). If LLVM's representation of functions with this ABI lacks any way to make them return something other than the caller-supplied pointer, that is a limitation in LLVM, not a constraint on what's possible in assembly.

[scottmcm]

because that agrees with the existing "null pointer optimization" for Option<&T> and similar.

The problem is that the "return value" parameter is more like &mut MaybeUninit<Option<T>> -- like all BackendRepr::Memory types, it needs space to write the whole thing, not just part of it.

not a constraint on what's possible in assembly.

TBH, in rustc we generally don't care what might be possible in assembly, because we don't want to get involved at that level. We want to express things that already exist in backends -- including LLVM, GCC, and Cranelift -- and it's up to them to do a smart translation.

---

I have no instinct on whether this would be worth it overall.

It'd absolutely be a win in things that are

fn foo(...) -> BigThing {
    try_foo(...).unwrap()
}

because the caller for foo only provides space for a BigThing, not an Option<BigThing>, so being able to just forward that space along to try_foo would certaily help. For example, part of why array::map (and array::from_fn) is better now is doing that optimization manually in 52df055.

But as @hanna-kruppe mentioned, it'd also be a pessimization in the opposite case. Whenever you need the &Option<_>, you need to make the bigger alloca again to have something big enough to point to.

And I think it would probably need to also update the ABI for passing large options by value too. Because Option::unwrap wants a Option<[u32; 8]>, which is passed as a a ptr dereferencable(36) nonnull today. So changing that to passing ptr dereferencable_or_null(32) would potentially work, but of course again at the cost of pessimizing things if the callee then wants to call something that wants &Option<[u32; 8]> which will still need ptr dereferencable(36) nonnull.

[workingjubilee]

@zackw This generalizes extremely poorly in the actual backend. In general, many ABI-level operations need to be made explicit transformations in LLVMIR in order for them to "take" correctly. This is because LLVM itself is agnostic to the notion of ABI. Then, in order to perform optimizations, LLVM needs to see through the ABI prologue and epilogue. This means that because we would only occasionally be able to use this transformation to begin with, it would make it harder to optimize Rust, because we would be making the representation of things even more inconsistent.

And my understanding is that this would be even harder to do in the GCC backend... possibly not representable at all... because it is not agnostic to ABI, and can wind up insisting on having things line up with the C ABI even when they shouldn't.

[workingjubilee]

tbh the biggest optimization we could gain from "the backend exposes some power it currently doesn't" would probably be from a far more humble thing (explicit usage of flags registers on platforms which have them as a notion, which can already be pattern-matched by backends)