Is there a way to write integration tests with Aspire?

[jcvandan]

I've only heard about Aspire yesterday but it sounds like the perfect fit for a new project I've been working on. In my current solution I have a distributed system and I do a lot of the stuff Aspire does manually, I run everything in containers and manually wire up url's using docker-compose etc etc. Aspire looks like a great alternative to this as it is, admittedly, a headache.

The only thing that concerns me is there is no mention of integration testing using WebApplicationFactory along with the Aspire host. I'm not sure how this would work, but does anyone know if there is support for something similar on the roadmap? It seems like a no brainer that we are provided with something along these lines for Aspire.

Thanks

[davidfowl]

Yes, I'm supposed to write a sample, but there's currently one here:

https://github.com/dotnet/eShop/blob/main/tests/Catalog.FunctionalTests/CatalogApiFixture.cs

Still lots of clean up to do but definitely on the roadmap.

[davidfowl]

I'm not sure how this is different from what we agreed to do above. Specifically:

• Create a DistributedApplicationTestingBuilder with the apphost
• A new API to start dependencies for a resource X (not starting resource X itself)
• A new API to get the environment for resource X

[cmeyertons]

@davidfowl excerpts from the conversation that I felt like are not covered by those requirements by those APIs

First, the apphost does not have a binary reference to the applications it orchestrates by design. It’s launching them as processes. That means communication is all out of process. Configuration is passed today via environment variables.

A new InProcessResource or something similar with the machinery to spin up an in-mem proxy would be required. Maybe this is only added in the Aspire.Hosting.Testing to discourage main usage.

My take on this assumption: I think it would not be necessary. I would see the direct dependency as "System under Test" where I want to inject changes. And I would have different sets of tests that would have a different "System under Test".

I wouldn't want this feature limited to one just resource -- I might want to inject changes to multiple layers simultaneously (e.g. i'm using the same in-mem external dependency across multiple projects and want consistent state)

PS: This is very cool and reminds me a of tilt and other tools that allow stubbing out a single container in a cluster with a proxy.

Your comment & diagram here is exactly what I imagine the best-of-breed solution to look like, but it seemed the conversation / requirements went a different, pared down direction afterwards, which is why I chimed in with the challenges around debuggability and performance.

I guess an additional requirement to achieve that would be:

• Run a ProjectResource in-process with full-fledged endpoint functionality instead of spinning up an external process.

^^ this bit helps solve the performance challenges and debugging complexity (that has to be solved in every test runner product). WebApplicationFactory can build off that paradigm and Aspire can do the hard parts to make WebApplicationFactory usage turnkey.

[davidfowl]

A new InProcessResource or something similar with the machinery to spin up an in-mem proxy would be required. Maybe this is only added in the Aspire.Hosting.Testing to discourage main usage.

I'm not convinced this is needed. This is just a WebApplicaitonFactory.

I wouldn't want this feature limited to one just resource -- I might want to inject changes to multiple layers simultaneously (e.g. i'm using the same in-mem external dependency across multiple projects and want consistent state)

I'm not convinced this is important, but would be willing to dig in a bit and understand the scenario. How would service discovery work in this case?

[JeffryGonzalez]

I'm not convinced this is important, but would be willing to dig in a bit and understand the scenario. How would service discovery work in this case?

I tend to agree. It might be useful in some circumstances but feels like promoting law of Demeter violations and would add a ton of complexity.

[cmeyertons]

How would service discovery work in this case?

Just like in your diagram earlier in the thread right?

For a ProjectResource in a testing context, I assume it spins up an external process bound to an random, unassigned port and then attaches the debugger to it. Service discovery fundamentally is just mapping a resource to a host and port in this context right?

For a WebApplicationFactoryResource or InProcessResource, it would spin up an in-memory proxy that binds itself to a random, unassigned port (like here) and proxies (YARP for the win?) the requests into the TestServer instance. This resource would emit the necessary service discovery metadata about what host & port it bound to for other resources (containers, executables, other test projects).

I'm not convinced this is needed. This is just a WebApplicationFactory.

Understood, I'll defer to y'all for the proper solution 😄 I was trying to see if I could do this in my app to extend the current resource model and couldn't figure out the appropriate hook or abstraction to control the lifecycle of the WebApplicationFactory to start the above in-mem proxy, etc (I'm sure there's a way w/ the current arch, just don't have enough context atm). It seems the ApplicationExecutor is orchestrating directly (rather than using like an extensible keyed IResourceExecutor pulled from DI or something)

[JohnGalt1717]

We're noticing that in Rider and VS.net that it will only allow debugging the test itself, we can't debug into the projects with breakpoints etc.

And in vs code the tests don't work at all for debugging.

Is there documentation on how to get this working? Is there work to get this working?

[davidfowl]

C# DevKit and rider need to explicitly add support for this.

[timheuer]

In VS Code C# with Dev Kit installed and activated, going to test explorer with the same project that Vs.net sees the sample test, no tests show up no matter how many times we build the project or hit refresh

@jhancock-taxa can you share any output when you try this from 'C# Test Explorer' output window? Using the Aspire-Starter App w/test project:

VS Code Extensions
ms-dotnettools.csdevkit: 1.8.2
ms-dotnettools.csharp: 2.34.10

[cmeyertons]

C# DevKit and rider need to explicitly add support for this.

Would in-proc architecture using WebApplicationFactory help solve / alleviate this? The Orleans TestCluster also follows this in-proc paradigm.

I'm just concerned that Aspire external process architecture won't scale in integration tests like WebApplicationFactory / Orleans TestCluster do if everything runs out of proc and has to reload the same dlls over and over and over....

I can work around external dependency lag in most cases (Mongo use Database isolation, Redis use namespacing, etc etc) but without running in-proc, I don't have a way to run / debug multiple instances of my app easily.

[jhancock-taxa]

In VS Code C# with Dev Kit installed and activated, going to test explorer with the same project that Vs.net sees the sample test, no tests show up no matter how many times we build the project or hit refresh

@jhancock-taxa can you share any output when you try this from 'C# Test Explorer' output window? Using the Aspire-Starter App w/test project:

VS Code Extensions ms-dotnettools.csdevkit: 1.8.2 ms-dotnettools.csharp: 2.34.10

This now works. It wasn't working before.

[scabana]

Is there any GitHub issue tracking attaching to projects (AddProject) when debugging tests in C# DevKit? I would have expected that to work, but it's not the case.

I searched a bit, but couldn't find anything here.

[manojlds]

Is there some reference on running this in CI where you are already inside docker?

[Pacman1988]

The solution works from the CatalogApiFixture, but what if I use an integration in the project like Seq?

Program.cs for ApiService from AspireStarterProject
builder.AddSeqEndpoint("seq");

Program.cs (AppHost)

var seq = builder.AddSeq("seq").ExcludeFromManifest();
var apiService = builder.AddProject<Projects.AspireApp1_ApiService>("apiservice")
    .WithReference(seq);

This is because the reference to the Api project in the csproj of the integration test is IsAspireProjectResource="false".

System.InvalidOperationException: 'Unable to add a Seq health check because the 'ServerUrl' setting is missing.'

Link to Repo: https://github.com/Pacman1988/AspireAppIntegrationTestWithSeq

[cmeyertons]

@davidfowl it's been a bit, and just wanted to see if there's any progress or additional thoughts on leveraging WebApplicationFactory in integration tests?

I have successfully used your sample on a few different services in our org but it requires some heavier lifting to manually perform service discovery when I have multiple components that want to talk to each other. There's also a hard limit on some local container dependency calling our service because the WebApplicationFactory is only available in-proc - I haven't come across this use case but could see it happening.

I know there's a ton of ongoing work with Aspire to prioritize so no worries if no update here.

Additionally, I want to commend the team on the overall vision and product - it really is a joy to work with.

[davidfowl]

No there's been no progress here in 9. The team was focused on top asks and adoption blockers for 9. We'll consider this work in a future release (there's a lot to design here).

[fdonnet]

I use something like that with IDistributedApplicationTestingBuilder as Apphost and I clean and modify the "normal" resources (remove persistent lifetime or remove data volumes)

It seems to work (I m at the begining)

public class AspireFixture : IAsyncLifetime
{
    public IDistributedApplicationTestingBuilder AppHost { get; private set; } = default!;
    public HttpClient AppHttpClient { get; private set; } = default!;
    public HttpClient AuthHttpClient { get; private set; } = default!;
    public ResourceNotificationService ResourceNotificationService { get; private set; } = default!;
    public DistributedApplication? App { get; private set; }

    public async Task InitializeAsync()
    {
        AppHost = await DistributedApplicationTestingBuilder
            .CreateAsync<Projects.UbikLink_AppHost>();

        AppHost.Services.ConfigureHttpClientDefaults(clientBuilder =>
        {
            clientBuilder.AddStandardResilienceHandler();
        });

        //Remove useless resources for testing
        RemoveNotNeededResourcesForTesting();

        //Change config for testing
        ModifyResourcesForTesting();

        App = await AppHost.BuildAsync();

        ResourceNotificationService = App.Services
            .GetRequiredService<ResourceNotificationService>();

        await App.StartAsync();

        AppHttpClient = App.CreateHttpClient("ubiklink-proxy");
        AuthHttpClient = App.Services.GetRequiredService<IHttpClientFactory>().CreateClient("auth-httpclient");
        
        //TODO: Change that to be in config
        AuthHttpClient.BaseAddress = new Uri("https://myauth/oauth/token");

        await ResourceNotificationService.WaitForResourceAsync(
            "ubiklink-proxy",
            KnownResourceStates.Running
            )
            .WaitAsync(TimeSpan.FromSeconds(30));
    }

    private void RemoveNotNeededResourcesForTesting()
    {
        var pgAdminResources = AppHost.Resources
            .Where(r => r.GetType() == typeof(PgAdminContainerResource))
            .ToList();

        foreach (var pgAdmin in pgAdminResources)
        {
            AppHost.Resources.Remove(pgAdmin);
        }
    }

    private void ModifyResourcesForTesting()
    {
        var cache = AppHost.Resources.Where(r => r.Name == "cache")
            .FirstOrDefault();

        var db = AppHost.Resources.Where(r => r.Name == "ubiklink-postgres")
            .FirstOrDefault();

        if (cache != null)
        {
            var containerLifetimeAnnotation = cache.Annotations
            .OfType<ContainerLifetimeAnnotation>()
            .FirstOrDefault();

            if (containerLifetimeAnnotation != null)
            {
                cache.Annotations.Remove(containerLifetimeAnnotation);
            }
        }

        if (db != null)
        {
            var containerLifetimeAnnotation = db.Annotations
            .OfType<ContainerLifetimeAnnotation>()
            .FirstOrDefault();
            if (containerLifetimeAnnotation != null)
            {
                db.Annotations.Remove(containerLifetimeAnnotation);
            }

            var dataVolumeAnnotation = db.Annotations
            .OfType<ContainerMountAnnotation>()
            .FirstOrDefault();
            if (dataVolumeAnnotation != null)
            {
                db.Annotations.Remove(dataVolumeAnnotation);
            }
        }
    }

    public async Task DisposeAsync()
    {
        AppHttpClient?.Dispose();
        AuthHttpClient?.Dispose();
        if (App != null)
        {
            if (App is IAsyncDisposable asyncDisposable)
            {
                await asyncDisposable.DisposeAsync().ConfigureAwait(false);
            }
            else
            {
                App.Dispose();
            }
        }
    }
}
[CollectionDefinition("AspireApp collection")]
public class AspireAppCollection : ICollectionFixture<AspireFixture>
{
    // This class has no code, and is never created. Its purpose is simply
    // to be the place to apply [CollectionDefinition] and all the
    // ICollectionFixture<> interfaces.
}

In this example, I removed pgadmin container and cleaned persistence annotations (lifecycle + volume). Because the testbuilder is not able to reuse the resources and it creates a big mess.

I don't know if it's fine ? It seems to allow me to test my Yarp endpoints with all the "true" resources behind. (without webfactory).

[DamianEdwards]

Totally fine. I do very similar things in the tests for the samples repo at https://github.com/dotnet/aspire-samples/blob/c9c2ab25bec229d85055d5540b072ef2a50f069f/tests/SamplesIntegrationTests/Infrastructure/DistributedApplicationTestFactory.cs#L28-L44

[KurtP20]

You might want to have a look at this answer on how to perform functional unit tests with Aspire. It spins up AppHost and uses a custom WebApplicationFactory to perform the tests. Might also help with your integration tests.

[m-s-work]

@KurtP20 thanks for this approach, I think this is a great workaround for this missing functionality when u need to test some API project within aspire.
(The only caveat I think is that for workers it is not that helpful as then 2 are running and could pickup work from the none tested one.)

[KurtP20]

Agree. My hope is that this approach will be incorporated into Aspire, then only one is running.

[weitzhandler]

To clarify - if you want to mess with the individual dependencies' (e.g. backend project) services you need WebApplicationFactory<Startup> (where Startup points to the backend one)?

[ScottKane]

using Aspire.Hosting;
using Aspire.Hosting.ApplicationModel;
using Aspire.Hosting.Testing;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.DependencyInjection;

namespace Tests.Integration.Harness;

/// <summary>
/// A custom WebApplicationFactory that bootstraps an Aspire AppHost for integration tests.
/// It starts all required infrastructure/resources declared in the AppHost, removes the API-under-test from the
/// Aspire host, and then injects the API's configuration/environment into the test server.
/// </summary>
/// <typeparam name="TEntryPoint">Program class of the API under test (the SUT).</typeparam>
/// <typeparam name="TAppHost">The Aspire AppHost Program type (e.g., Projects.Pep_Host).</typeparam>
public class AspireWebApplicationFactory<TEntryPoint, TAppHost> : WebApplicationFactory<TEntryPoint>
    where TEntryPoint : class
    where TAppHost : class
{
    private DistributedApplication? _app;
    private bool _disposed;

    /// <summary>
    /// Name of the resource in the AppHost that corresponds to the API under test. If not provided, the
    /// factory will try to infer the name from the entrypoint assembly name.
    /// </summary>
    public string? ApiResourceName { get; init; }

    /// <summary>
    /// Optional callback invoked after the AppHost is built but before it is started. You can tweak
    /// resources (e.g., remove optional dashboards) or adjust configuration.
    /// </summary>
    public Action<IDistributedApplicationTestingBuilder>? ConfigureAppHost { get; init; }
    
    /// <summary>
    /// Optional callback to modify the DI container of the SUT.
    /// </summary>
    public Action<IServiceCollection>? ConfigureServices { get; init; }

    /// <summary>
    /// Optional callback invoked after the AppHost is built and started. You can implement readiness waits here.
    /// </summary>
    public Func<IServiceProvider, CancellationToken, Task>? AfterAppHostStartedAsync { get; init; }

    /// <summary>
    /// Resources the AppHost should include.
    /// </summary>
    public List<string> Resources { get; init; } = [];

    /// <summary>
    /// Determines whether the container lifetimes should be removed from all resources in the app host, if false, this will allow persistence between test runs (useful while adding new tests for speed).
    /// </summary>
    public bool Ephemeral { get; init; } = true;

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        var testingBuilder = DistributedApplicationTestingBuilder.CreateAsync<TAppHost>()
            .GetAwaiter().GetResult();

        var apiResource = ResolveApiResource(testingBuilder.Resources.ToList());
        if (apiResource is null)
            throw new InvalidOperationException("Could not resolve the API-under-test resource from the AppHost. Provide ApiResourceName or ApiResourceSelector.");
        
        int added;
        do
        {
            var annotations = testingBuilder.Resources.Where(r => r.Annotations.OfType<ResourceRelationshipAnnotation>().Any(p => Resources.Contains(p.Resource.Name) && p.Type == "Parent" && !Resources.Contains(p.Resource.Name))).Select(r => r.Name);
            var parents = testingBuilder.Resources.Where(r => r is IResourceWithParent && !Resources.Contains(r.Name)).Select(r => r.Name);

            List<string> adds = [..annotations, ..parents];
            Resources.AddRange(adds);

            added = adds.Count;
        } while (added > 0);
        
        foreach (var resource in testingBuilder.Resources.Where(r => !Resources.Distinct().Contains(r.Name)).ToArray())
            testingBuilder.Resources.Remove(resource);

        if (Ephemeral)
        {
            foreach (var resource in testingBuilder.Resources)
            {
                var lifetime = resource.Annotations.OfType<ContainerLifetimeAnnotation>().FirstOrDefault();
                if (lifetime != null) resource.Annotations.Remove(lifetime);
            }
        }

        ConfigureAppHost?.Invoke(testingBuilder);

        _app = testingBuilder.BuildAsync().GetAwaiter().GetResult();
        
        foreach (var resource in testingBuilder.Resources)
            _app.ResourceNotifications.WaitForResourceHealthyAsync(resource.Name).GetAwaiter().GetResult();

        _app.StartAsync().GetAwaiter().GetResult();

        AfterAppHostStartedAsync?.Invoke(_app.Services, CancellationToken.None).GetAwaiter().GetResult();

        var config = ResolveConfigurationFromResourceAsync(apiResource, _app.Services).GetAwaiter().GetResult();
        
        foreach (var (key, value) in config)
        {
            if (value is null) continue;
            builder.UseSetting(key, value);
        }

        builder.ConfigureServices(s => ConfigureServices?.Invoke(s));
    }

    /// <summary>
    /// Creates an <see cref="HttpClient"/> configured to communicate with the specified resource.
    /// </summary>
    /// <param name="resource">The name of the resource.</param>
    /// <param name="endpoint">The endpoint on the resource to communicate with.</param>
    /// <returns>The <see cref="HttpClient"/>.</returns>
    protected HttpClient CreateResourceClient(string resource, string endpoint = "https") => _app!.CreateHttpClient(resource, endpoint);

    private IResource? ResolveApiResource(IReadOnlyCollection<IResource> resources)
    {
        if (!string.IsNullOrWhiteSpace(ApiResourceName))
            return resources.FirstOrDefault(r => string.Equals(r.Name, ApiResourceName, StringComparison.OrdinalIgnoreCase));

        var inferred = typeof(TEntryPoint).Assembly.GetName().Name;
        
        return !string.IsNullOrWhiteSpace(inferred)
            ? resources.FirstOrDefault(r => string.Equals(r.Name, inferred, StringComparison.OrdinalIgnoreCase))
            : null;
    }

    /// <summary>
    /// Loads environment variables exposed by the resource and translates double-underscore keys to configuration keys.
    /// </summary>
    private static async Task<Dictionary<string, string?>> ResolveConfigurationFromResourceAsync(IResource resource, IServiceProvider serviceProvider, CancellationToken cancellationToken = default)
    {
        var config = new Dictionary<string, string?>();

        if (resource is not IResourceWithEnvironment resourceWithEnvironment || !resourceWithEnvironment.TryGetEnvironmentVariables(out var annotations))
            return config;
        
        var options = new DistributedApplicationExecutionContextOptions(DistributedApplicationOperation.Run)
        {
            ServiceProvider = serviceProvider
        };
        var execContext = new DistributedApplicationExecutionContext(options);
        var context = new EnvironmentCallbackContext(execContext, cancellationToken: cancellationToken);

        foreach (var annotation in annotations)
            await annotation.Callback(context);

        foreach (var (key, value) in context.EnvironmentVariables)
        {
            if (resource is ProjectResource && key == "ASPNETCORE_URLS") continue;

            string? configValue;
            switch (value)
            {
                case string s:
                    configValue = s;
                    break;
                case IValueProvider v:
                    try
                    {
                        configValue = await v.GetValueAsync(cancellationToken);
                    }
                    catch
                    {
                        configValue = null;
                    }
                    break;
                case null:
                    configValue = null;
                    break;
                default:
                    throw new InvalidOperationException($"Unsupported environment value type: {value.GetType()}");
            }

            if (configValue is not null)
                config[key.Replace("__", ":")] = configValue;
        }

        return config;
    }

    protected override void Dispose(bool disposing)
    {
        if (_disposed) return;
        if (disposing)
        {
            _app?.Dispose();
            base.Dispose(disposing);
        }

        _disposed = true;
    }
}

Usage:

public class MyWebApplicationFactory : AspireWebApplicationFactory<Program, My_Host>
{
    public MyWebApplicationFactory()
    {
        ApiResourceName = "MyApp"; // If resource name in Aspire is different to assembly name of entry point
        Resources = ["Database", "RabbitMQ", "Azurite"]; // Resources I want to keep in the AppHost for this WebAppFactory (default is to remove all)
    }
}

This gives you a WebApplicationFactory hooked up to services in Aspire with config binding.