Supporting integration tests with WebApplicationFactory in .NET 6
https://andrewlock.net/exploring-dotnet-6-part-6-supporting-integration-tests-with-webapplicationfactory-in-dotnet-6/
This is the sixth post in the series: Exploring .NET 6.
- Part 1 - Looking inside ConfigurationManager in .NET 6
- Part 2 - Comparing WebApplicationBuilder to the Generic Host
- Part 3 - Exploring the code behind WebApplicationBuilder
- Part 4 - Building a middleware pipeline with WebApplication
- Part 5 - Supporting EF Core migrations with WebApplicationBuilder
- Part 6 - Supporting integration tests with WebApplicationFactory in .NET 6 (this post)
- Part 7 - Analyzers for ASP.NET Core in .NET 6
- Part 8 - Improving logging performance with source generators
- Part 9 - Source generator updates: incremental generators
- Part 10 - New dependency injection features in .NET 6
- Part 11 - [CallerArgumentExpression] and throw helpers
- Part 12 - Upgrading a .NET 5 "Startup-based" app to .NET 6
In the previous post I described the workaround that was added in .NET 6 so that the EF Core tools, which previously relied on the existence of specific methods like CreateHostBuilder
would continue to work with the new minimal hosting APIs.
In this post I look at a related change to ensure that integration testing with WebApplicationFactory
works in .NET 6. WebApplicationFactory
used the same HostFactoryResolver
class as the EF Core tools, but it required a few more changes too, which I'll look at in this post.
WebApplicationFactory
in ASP.NET Core 3.x/5![](https://andrewlock.net/assets/img/icons-link.svg)
There are multiple ways to test an ASP.NET Core 3.x/5 app. One of the most thorough approaches is to write integration tests that run your whole application in-memory. This is surprisingly easy using the Microsoft.AspNetCore.Mvc.Testing package and WebApplicationFactory<T>
.
For example, the following code, based on the docs, shows how you can use WebApplicationFactory
to create an in-memory instance of your application, create an HttpClient
for making requests, and send an in-memory HTTP request.
public class BasicTests : IClassFixture<WebApplicationFactory<Startup>>
{
private readonly WebApplicationFactory<Startup> _factory;
public BasicTests(WebApplicationFactory<Startup> factory)
{
_factory = factory;
}
[Fact]
public async Task Get_EndpointsReturnSuccessAndCorrectContentType()
{
// Arrange
var client = _factory.CreateClient();
// Act
var response = await client.GetAsync("/");
// Assert
response.EnsureSuccessStatusCode(); // Status Code 200-299
}
}
Behind the scenes, WebApplicationFactory<T>
uses the same HostFactoryResolver
I described in the previous post. The generic parameter TEntryPoint
is conventionally set to Startup
, but it just needs to be a type inside the entry assembly, so that it can find the CreateHostBuilder()
method:
public class WebApplicationFactory<TEntryPoint> : IDisposable where TEntryPoint : class
{
protected virtual IHostBuilder CreateHostBuilder()
{
var hostBuilder = HostFactoryResolver.ResolveHostBuilderFactory<IHostBuilder>(typeof(TEntryPoint).Assembly)?.Invoke(Array.Empty<string>());
if (hostBuilder != null)
{
hostBuilder.UseEnvironment(Environments.Development);
}
return hostBuilder;
}
// ...
}
As I described in my previous post, HostFactoryResolver
uses reflection to find the conventionally named methods CreateHostBuilder()
or CreateWebHostBuilder()
and and invoke them. However in .NET 6 the minimal hosting APIs and top level programs have done away with these conventions, initially breaking WebApplicationFactory
(along with the EF Core tools).
Building an IHost in .NET 6![](https://andrewlock.net/assets/img/icons-link.svg)
In the previous post I described the changes that were made to HostBuilder
to support the HostFactoryResolver
, used by both WebApplicationFactory
and the EF Core tools. This was primarily achieved by adding additional DiagnosticSource events to HostBuilder
. These provide a way for HostFactoryResolver
to get access to the HostBuilder
without needing the conventions of previous versions.
![The EF Core tools run your Program.Main() and the diagnostic listener retrieves the IHost](https://andrewlock.net/content/images/2021/webapplication_efcore2.png)
WebApplicationFactory
benefits from this new mechanism too, but there were a few additional changes needed. The EF Core tools just needed to access the built IHost
so that they could retrieve an IServiceProvider
. The IServiceProvider
is fixed once you build the IHost
, so the "abort the application" approach shown in the above image worked just fine.
However, that doesn't work for the WebApplicationFactory
. WebApplicationFactory
needs be able to modify the HostBuilder
in your application before Build()
is called on it, but it also can't just stop your program when HostBuilder.Build()
is called.
In .NET 6 you can (and do) write all sorts of code between the call to WebApplicationBuilder.Build()
and WebApplication.Run()
. You can't modify the IServiceCollection
, but you can register your endpoints and middleware between these calls:
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddRazorPages();
var app = builder.Build(); // calls HostBuilder.Build()
app.UseStaticFiles();
app.MapGet("/", () => "Hello World!");
app.MapRazorPages();
app.Run(); // calls Host.StartAsync()
This makes things a bit trickier for WebApplicationFactory
, as it needs to run all the code in Program.cs, right up until the call to app.Run()
, so it can't rely on the DiagnosticSource events added to HostBuilder
alone. In the rest of this post, we'll look at how WebApplicationFactory
achieves that.
WebApplicationFactory
in .NET 6![](https://andrewlock.net/assets/img/icons-link.svg)
On the surface, the way you use WebApplicationFactory
hasn't changed in .NET 6. The exact same test code I showed earlier will work in .NET 6, even if you're using the new minimal hosting APIs with WebApplication
and WebApplicationBuilder
.
One slight annoyance is that there's no well known
Startup
class to use as a "marker" for theT
generic parameter inWebApplicationFactory<T>
. In practice, this is probably only an issue for demo-ware, as you can use any old class in your web app as a marker, but it's something to be aware of.
WebApplicationFactory
provides multiple ways to customise your application in integration tests, but at it's core, it provides a way to run your application's Host
instance in memory. One of the core methods in this process is EnsureServer()
, which is partially shown below.
public class WebApplicationFactory<TEntryPoint> : IDisposable, IAsyncDisposable where TEntryPoint : class
{
private void EnsureServer()
{
// Ensure that we can find the .deps.json for the application
EnsureDepsFile();
// Attempt to create the application's HostBuilder using the conventional
// CreateHostBuilder method (used in ASP.NET Core 3.x/5) and HostFactoryResolver
var hostBuilder = CreateHostBuilder();
if (hostBuilder is not null)
{
// If we succeeded, apply customisation to the host builder (shown below)
ConfigureHostBuilder(hostBuilder);
return;
}
// Attempt to create the application's WebHostBuilder using the conventional
// CreateWebHostBuilder method (used in ASP.NET Core 2.x) and HostFactoryResolver
var builder = CreateWebHostBuilder();
if (builder is null)
{
// Failed to create the WebHostBuilder, so try the .NET 6 approach
// (shown in following section)
// ...
}
else
{
// succeeded in creating WebHostBuilder, so apply customisation and exit
SetContentRoot(builder);
_configuration(builder);
_server = CreateServer(builder);
}
}
private void ConfigureHostBuilder(IHostBuilder hostBuilder)
{
// Customise the web host
hostBuilder.ConfigureWebHost(webHostBuilder =>
{
SetContentRoot(webHostBuilder);
_configuration(webHostBuilder);
// Replace Kestrel with TestServer
webHostBuilder.UseTestServer();
});
// Create the IHost
_host = CreateHost(hostBuilder);
// Retrieve the TestServer instance
_server = (TestServer)_host.Services.GetRequiredService<IServer>();
}
}
EnsureServer()
is responsible for populating the TestServer
field _server
using HostFactoryResolver
. It first tries to create an IHostBuilder
instance by looking for the Program.CreateHostBuilder()
method, commonly used in ASP.NET Core 3.x/5. If that fails, it looks for the Program.CreateWebHostBuilder()
method used in ASP.NET Core 2.x. And if that fails, it resorts to the .NET 6 approach, which I've extracted from the method above, and is shown below.
// Create a DeferredHostBuilder, which I'll discuss shortly
var deferredHostBuilder = new DeferredHostBuilder();
deferredHostBuilder.UseEnvironment(Environments.Development);
// Ensure the application name is set correctly. Without this, the application name
// would be set to the testhost (see https://github.com/dotnet/aspnetcore/pull/35101)
deferredHostBuilder.ConfigureHostConfiguration(config =>
{
config.AddInMemoryCollection(new Dictionary<string, string>
{
{ HostDefaults.ApplicationKey, typeof(TEntryPoint).Assembly.GetName()?.Name ?? string.Empty }
});
});
// This helper call does the hard work to determine if we can fallback to diagnostic source events to get the host instance
var factory = HostFactoryResolver.ResolveHostFactory(
typeof(TEntryPoint).Assembly,
stopApplication: false,
configureHostBuilder: deferredHostBuilder.ConfigureHostBuilder,
entrypointCompleted: deferredHostBuilder.EntryPointCompleted);
if (factory is not null)
{
// If we have a valid factory it means the specified entry point's assembly can potentially resolve the IHost
// so we set the factory on the DeferredHostBuilder so we can invoke it on the call to IHostBuilder.Build.
deferredHostBuilder.SetHostFactory(factory);
ConfigureHostBuilder(deferredHostBuilder);
return;
}
// Failed to resolve the .NET 6 entrypoint, so failed at this point
throw new InvalidOperationException();
This method uses a new type, DeferredHostBuilder
, which we'll look into shortly, but the important section is the call to HostFactoryResolver.ResolveHostFactory()
. This is the method that uses the DiagnosticSource events I discussed in my last post to customise the IHostBuilder
and to access the IHost
. Specifically, the call registers two callbacks:
deferredHostBuilder.ConfigureHostBuilder
: called just before theIHostBuilder
is built, and passed theIHostBuilder
instance.deferredHostBuilder.EntryPointCompleted
: called if an exception occurs during the build process.
Importantly, the stopApplication
argument is set to false
; this ensures the application startup process continues uninterrupted.
Contrast that with the EF Core tools approach, in which
stopApplication=true
. The EF Core tools don't want to run your application, they just need access to theIHost
(andIServiceProvider
), so they can halt after these are built.
The following diagram shows the interaction with WebApplicationFactory
, the HostFactoryResolver
, and DeferredHostBuilder
, as well as other types we'll see on the way. Don't worry about understanding this fully for now, but I think it's helpful to view now as a signpost for where we're going!
![The WebApplicationFactory sequence diagram for starting your application](https://andrewlock.net/content/images/2021/webapplication_webappfactory.png)
You might wonder why we need a new type here, the DeferredHostBuilder
. This is necessary because of the asynchronous way we have to wait for the "main" application to finish running. In the next section I'll look at this type in detail.
DeferredHostBuilder
and waiting for the StartAsync
signal![](https://andrewlock.net/assets/img/icons-link.svg)
DeferredHostBuilder
is YAHB (Yet Another IHostBuilder) that was introduced in .NET 6 (along with many others)! It's designed to "capture" configuration methods called on it (such as ConfigureServices()
for example) and then "replay" them against the real application's IHostBuilder
once it's available.
The "deferral" methods work by collecting the configuration methods as a multi-cast delegate, for example:
internal class DeferredHostBuilder : IHostBuilder
{
private Action<IHostBuilder> _configure;
public IHostBuilder ConfigureServices(Action<HostBuilderContext, IServiceCollection> configureDelegate)
{
_configure += b => b.ConfigureServices(configureDelegate);
return this;
}
// ...
}
These delegates are all applied to the IHostBuilder
when the DiagnosticSource HostBuilding
event is raised:
public void ConfigureHostBuilder(object hostBuilder)
{
_configure(((IHostBuilder)hostBuilder));
}
To get this all rolling, the WebApplicationFactory
calls Build()
on the DeferredHostBuilder
. This method, as shown below, invokes the _hostFactory
method returned by HostResolverFactory
. Calling this method starts the process described in the previous post, in which the application is executed in a separate thread, the ConfigureHostBuilder()
customisation is called using DiagnosticSource
events, and the IHost
instance is returned. That's a lot for a single line of code!
public IHost Build()
{
// Hosting configuration is being provided by args so that
// we can impact WebApplicationBuilder based applications.
var args = new List<string>();
// Transform the host configuration into command line arguments
foreach (var (key, value) in _hostConfiguration.AsEnumerable())
{
args.Add($"--{key}={value}");
}
// Execute the application in a spearate thread, and listen for DiagnosticSource events
var host = (IHost)_hostFactory!(args.ToArray());
// We can't return the host directly since we need to defer the call to StartAsync
return new DeferredHost(host, _hostStartTcs);
}
Remember that the application doesn't stop running in the separate Thread when we retrieve the IHost
instance, because we need the rest of the code in Program.cs to execute. The DeferredHostBuilder saves the IHost
into a new type, DefferedHost
, and returns this from the Build()
call.
_hostStartTcs
is aTaskCompletionSource
that is used to handle the edge case where the application running in the background exits due to an exception. It's an edge case, but without it, the test could hang indefinitely.
The DeferredHost
is responsible for waiting for the application to properly start (not just for the IHost
to be built). It needs to wait for you to configure all your endpoints, as well as run any additional startup code.
The DeferredHost
achieves this by using the existing IHostApplicationLifetime
events that are raised in a normal generic host application when the app is started. The following image (taken from a previous post analysing the startup process for the generic host) shows that the NotifyStarted()
method is called on the IHostApplicationLifetime
after the server has started.
![Sequence diagram for Host.StartAsync()](https://andrewlock.net/content/images/2019/hosting_startup.png)
The call to NotifyStarted()
raises the ApplicationStarted
event, which the DeferredHost
uses to detect that the application is running, and it's safe to start the test. When the WebApplicationFactory
calls Start()
on the DeferredHost
, the DeferredHost
blocks until the ApplicationStarted
event is raised.
public async Task StartAsync(CancellationToken cancellationToken = default)
{
// Wait on the existing host to start running and have this call wait on that. This avoids starting the actual host too early and
// leaves the application in charge of calling start.
using var reg = cancellationToken.UnsafeRegister(_ => _hostStartedTcs.TrySetCanceled(), null);
// REVIEW: This will deadlock if the application creates the host but never calls start. This is mitigated by the cancellationToken
// but it's rarely a valid token for Start
using var reg2 = _host.Services.GetRequiredService<IHostApplicationLifetime>().ApplicationStarted.UnsafeRegister(_ => _hostStartedTcs.TrySetResult(), null);
await _hostStartedTcs.Task.ConfigureAwait(false);
}
The StartAsync()
method adds additional callbacks to the TaskCompletionSource
I mentioned previously, and then await
s the task before returning. This will block the test code until one of three things happen:
- The web application throws an exception, which invokes the
EntryPointCompleted()
callback onDeferredHostBuilder
and cancels the task. - The
CancellationToken
passed to theStartAsync()
method is cancelled, which cancels the task - The application starts, invoking the
ApplicationStarted
event, completing the task.
As noted in the comments for this method, if you never call Start()
or Run()
in your web app (and don't throw an exception), then this will deadlock, but then you probably don't have a valid web app anyway, so it's not a big concern.
And that's it! Once Start()
has been called, the WebApplicationFactory
creates an HttpClient
as in previous versions, and you can make in-memory calls as before. It's worth being aware that (in contrast to previous versions of ASP.NET Core) everything in Program.cs will be running in your tests. But apart from that, everything in your test code stays the same.
Summary![](https://andrewlock.net/assets/img/icons-link.svg)
In this post I described the work that was done on WebApplicationFactory
to support the new minimal host APIs that use WebApplication
and WebApplicationBuilder
. Changes were required because there are no longer "conventional" methods in Program.cs that can be called using reflection, and also customisation of your middleware and endpoints occurs inside Program.cs.
To work around this, WebApplicationFactory
relies on the same DiagnosticSource events as the EF Core tools from my previous post to customise the IHostBuilder
and retrieve the IHost
. However, unlike the EF Core tools, the WebApplicationFactory
does not stop the application after the IHost
is built. Instead, it allows the app to continue to run, and listens for the IHostApplicationLifetime.ApplicationStarted
event. This allows the WebApplicationFactory
to block until all the code in Program.cs has run, and the application is ready to start handling requests.