Multiple await on source returning the same exception reports multiple stack traces

[adamjones2]

I'm finding something very strange: when I repeatedly subscribe to a failed sequence the exception that's returned gets the call stack of each subscription appended to it for future callers. This is using Rx v6.0.1. Repro:

var src = Observable.Throw<int>(new Exception("test")).Replay(1);
src.Connect();

async Task GetValue()
{
    try { await src.FirstAsync(); }
    catch (Exception ex) { Console.WriteLine(ex); }
}

for (int i = 1; i <= 3; i++)
{
    Console.WriteLine($"\r\n===> {i}");
    await GetValue();
}

Output:

===> 1
System.Exception: test
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8

===> 2
System.Exception: test
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8
--- End of stack trace from previous location ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8

===> 3
System.Exception: test
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8
--- End of stack trace from previous location ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8
--- End of stack trace from previous location ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8

Expected:

===> 1
System.Exception: test
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8

===> 2
System.Exception: test
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8

===> 3
System.Exception: test
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception)
   at System.Reactive.ExceptionHelpers.Throw(Exception exception)
   at System.Reactive.Subjects.AsyncSubject`1.GetResult()
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__GetValue|0>d.MoveNext() in C:\Source\Test\Program.cs:line 8

We have code that frequently checks the first value of a long-lived Replay'd stream and when it errored out we were surprised an hour later in our logs to find some very long stack traces indeed!

[idg10]

I've changed the title of this because it turns out to have nothing to do with Replay. If you replace your first two lines:

src.Connect();

with just this:

var src = Observable.Throw<int>(new Exception("test"));

the exact same behaviour occurs.

The root cause is that when Rx's await machinery determines that a source has called OnError, it uses ExceptionDispatchInfo.Capture(exception).Throw() to throw the exception. It looks like that was not intended to be called multiple times on the same exception, because it modifies the exception object, in order to provide contextual information such as crash bucket values and stack traces.

It appears that the effect of passing the same exception multiple times to this mechanism is to append the stack trace information.

We can see this by removing Rx completely from the picture:

Exception singleExceptionInstance = new("There can be only one.");

Task ThrowItAsync() => Task.FromException(singleExceptionInstance);

for (int i = 1; i <= 3; i++)
{
    Console.WriteLine($"\r\n===> {i}");
    try
    {
        await ThrowItAsync();
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex);
    }
}

That produces this output:

===> 1
System.Exception: There can be only one.
   at Program.<Main>$(String[] args) in C:\dev\temp\RxRepro2187\ThrowSingletonAsync\Program.cs:line 11

===> 2
System.Exception: There can be only one.
   at Program.<Main>$(String[] args) in C:\dev\temp\RxRepro2187\ThrowSingletonAsync\Program.cs:line 11
   at Program.<Main>$(String[] args) in C:\dev\temp\RxRepro2187\ThrowSingletonAsync\Program.cs:line 11

===> 3
System.Exception: There can be only one.
   at Program.<Main>$(String[] args) in C:\dev\temp\RxRepro2187\ThrowSingletonAsync\Program.cs:line 11
   at Program.<Main>$(String[] args) in C:\dev\temp\RxRepro2187\ThrowSingletonAsync\Program.cs:line 11
   at Program.<Main>$(String[] args) in C:\dev\temp\RxRepro2187\ThrowSingletonAsync\Program.cs:line 11

Notice that we get a similar gradually growing stack trace. (In this case it doesn't have the previous location marker, because this is a simpler scenario, but the same basic mechanism is there: the stack trace accumulates if you await the same object multiple times, and that object is in an faulted state.

So in a way, this is consistent with existing .NET runtime behaviour: if you call await multiple times on the same faulted Task you get this peculiar growing stack trace behaviour. When we add Rx into the mix, nothing really changes: calling await multiple times on the same 'faulted' observable causes the same growing stack trace behaviour as you see with a single faulted Task. (And it happens for the same reason, because Rx is using the same underlying mechanism as Task uses here.)

This raises the question: what should the correct behaviour be? The fact that Task does exactly the same thing makes it tempting to say that this is just the expected behaviour. However, there's one reason not to do that: it is possible to re-use a singleton Exception in normal .NET code without seeing this problem, and that's because the throw keyword essentially resets the exception. If there was any stack trace information attached to the exception from an earlier throw, a new call to throw discards that.

This means you do need to be very careful with singleton exceptions: if they might be thrown on multiple threads, there's a risk that two threads will throw the same exception at almost exactly the same time, at which point one of the threads could observe the wrong stack trace.

That risk aside, developers using Rx could reasonably expect Throw to have the same semantics as throw. I'm not sure if it's reasonable for them to have that same expectation when they've wrapped it in a Replay: that explicitly tells Rx that you want the underlying Throw to run just once and that you want to serve up the same result again and again. In that case, you can't expect a throw (or equivalent) to occur every time, so perhaps you shouldn't expect the exception state to reset. So the behaviour as reported (in which Replay is used) is arguably correct.

However, I think the simplified version, in which we don't use Replay, arguably should not exhibit this behaviour: the act of subscribing to a Throw operator could reasonably be expected to cause behaviour equivalent to a throw, which would include resetting any existing stack trace state on the exception.

It's arguably unwise to rely on this. Singleton exceptions are tricky things because they don't deal well with concurrency: there's no way to guarantee in a catch block that you've captured the information for the throw in progress before some other thread has overwritten that information by throwing the same exception. Of course, Rx's Throw positively encourages you to use singleton exceptions. But then Rx's use of Exception is a little odd: if you remain entirely within Rx's world (IObserver<T>/IObservable<T>) the problems with singleton exceptions never arise because Rx uses exceptions without ever actually throwing them!

The place where it goes wrong is where we cross from Rx's world, in which we signal errors using exceptions that never actually get thrown, into the 'normal' world, e.g., by using await. When we make that transition, an exception is no longer being use in the Rx way—as a mere data holding object, never really to be thrown—and it is actually thrown. It's at that moment of transition between worlds that the behaviour you've observed has crept in.

So one possible resolution to this is to say: if you are going to cause Rx to throw an exception (e.g., by using await on an IObservable<T>) then you shouldn't use singleton exceptions, because if you do, you'll encounter exactly the same strange kinds of behaviours as you can get if you call Task.FromException multiple times on the same exception object.

A second (slightly more helpful, perhaps) resolution would be to modify Throw so that each subscription to Throw has the same effect of resetting the exception as executing a throw would. This would mean that the version of the code shown above in which I removed Replay no longer exhibits this stack trace growth behaviour, but your original Replay example would continue to exhibit the behaviour (much as Task.FromException does exactly the same thing when used in an equivalent way).

And a third option might be to modify Rx's await handling: and have it reset the exception: this would amount to saying that a C# like throw occurs at the point where we transition from Rx's world (in which exceptions are delivered without ever being thrown) into the conventional .NET world, and so the exception gets reset at that point (and the expectation would be that the stack trace would reflect that instant as the context in which the exception was thrown).

Given that your example uses Replay, and that your original issue title framed this as a problem with Replay, I'm guessing that you would prefer the third option. However, it's possible that you hadn't fully analyzed the root causes, so it's possible that you might have changed your view in light of the fact that it's directly analogous to something Task does when used in a similar way.

So which of these three resolutions seems best to you?

I'm going to have a think about this because it's not all that clear to me which option is best. The 3rd one might be convenient for you because it makes a problem go away, but conceptually it's a pretty major shift, and I'm worried that it might have bad unintended consequences. (I'm also somewhat uncomfortable with the idea of the await representing the point at which the exception is logically thrown: that's definitely not how it works with normal use of exceptions in async/await.) And although I think the 1st is justifiable, it's also unhelpful, so I'm currently leaning towards the 2nd option.

[idg10]

Just to clarify something that might not have been clear from the above: these very long stack traces do not reflect reality: the depth of the actual call stack does not grow. What's happening here is that the exception object's stack trace data is getting multiple copies of the same stack trace appended to it. So the stack trace reported by the exception gets ever longer, but the actual call stacks remain of constant size.

This does constitute a kind of memory leak. If you needed to work around this while we work out how to resolve this, you could modify the code to build a new exception every time. Perhaps something along the lines of Observable.Defer(() => Observable.Throw<int>(new Exception()). (And don't use Replay. Use of Replay is currently basically incompatible with await.)