aspnet core系统优雅停机升级

web项目在生产环境要求:

1. 停机时需要确保 running 的请求能被安全处理完毕
2. 停机时确保不接收新的请求
3. 需要有 healthCheck 接口
4. Load balancer 能对接 healthCheck 接口, 确保业务能达到 zero downtime update

实现机制:

1. 微软官方关于dotnet-docker优雅关闭的文档 https://github.com/dotnet/dotnet-docker/blob/main/samples/kubernetes/graceful-shutdown/graceful-shutdown.md
2. 默认的 health check

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHealthChecks();

var app = builder.Build();
app.MapHealthChecks("/healthz");
app.Run();

默认Asp.net Core 在收到 Terminiate 信号后的处理过程

1. Stop accepting new requests
2. 如果有running request, 自动等待 30 秒后关闭running threads; 如果没有running request, 立即退出. 默认的30秒timeout可以调整.
   当然我们可以自定义timeout的时间, 但我们很难设定合适的 timeout 值, 既要确保running事务被正常处理完毕, 又能及时响应shutdown 请求.

我推荐的方案

1. 在controller 中使用 Interlocked 类动态记录 running request 数量
2. 实现一个自定义的 HostLifetime, 重写 ApplicationStopping 事件, 在事件中实时检查 running request数量, 如果大于零, 则sleep, 直到数量为0, 退出事件.
3. health check 自己实现一个个的 controller API, 扩展性更好.

示例代码

MyRestHostLifetime 类

   /// <summary>
    ///
    /// https://github.com/dotnet/dotnet-docker/blob/main/samples/kubernetes/graceful-shutdown/graceful-shutdown.md
    /// </summary>
    public class MyRestHostLifetime : IHostLifetime, IDisposable
    {
        private IHostApplicationLifetime _applicationLifetime;
        private TimeSpan _delay;
        private IEnumerable<IDisposable>? _disposables;
        private AppState _appState;
        private ILogger<MyRestHostLifetime> _logger;

        public MyRestHostLifetime(IHostApplicationLifetime applicationLifetime, TimeSpan delay,
            AppState appState, ILogger<MyRestHostLifetime> logger)

        {
            _applicationLifetime = applicationLifetime;
            _delay = delay;
            _appState = appState;
            _logger = logger;
            _applicationLifetime.ApplicationStopping.Register(OnShutdown);
            _applicationLifetime.ApplicationStopped.Register(AfterShutdown);
        }

        private void OnShutdown()
        {
            while (_appState.getRunningRequests() >= 1)
            {
                _logger.LogWarning($"SIGTERM signal received, but there are {_appState.getRunningRequests()} running requests. Sleep one moment to ensure them handled. ");
                System.Threading.Thread.Sleep(1000);
            }
        }

        private void AfterShutdown()
        {
            string message = $"There are {_appState.getRunningRequests()} running requests. Application stopped. ";
            if (_appState.getRunningRequests() > 0)
            {
                _logger.LogWarning(message);
            }
            else
            {
                _logger.LogInformation(message);
            }
        }

        public Task StopAsync(CancellationToken cancellationToken)
        {
            return Task.CompletedTask;
        }

        public Task WaitForStartAsync(CancellationToken cancellationToken)
        {
            _disposables = new IDisposable[]
            {
            PosixSignalRegistration.Create(PosixSignal.SIGINT, HandleSignal),
            PosixSignalRegistration.Create(PosixSignal.SIGQUIT, HandleSignal),
            PosixSignalRegistration.Create(PosixSignal.SIGTERM, HandleSignal)
            };
            return Task.CompletedTask;
        }

        protected void HandleSignal(PosixSignalContext ctx)
        {
            ctx.Cancel = true;
            Task.Delay(_delay).ContinueWith(t => _applicationLifetime.StopApplication());
        }

        public void Dispose()
        {
            foreach (var disposable in _disposables ?? Enumerable.Empty<IDisposable>())
            {
                disposable.Dispose();
            }
        }
    }

AppState 类

记录 running request的数量

 public class AppState
    {
        private long _runningRequests = 0;
        private long _completedRequests = 0;
        private long _failedRequests = 0;
        private long _succeededRequests = 0;

        public long runningRequests
        { get { return _runningRequests; } }

        public long completedRequests
        { get { return _completedRequests; } }

        public long failedRequests
        { get { return _failedRequests; } }

        public long succeededRequests
        { get { return _succeededRequests; } }

        public DateTime startTime { get; set; }

        public void markNewRequest()
        {
            Interlocked.Increment(ref _runningRequests);
        }

        public void markRequestCompleted(bool isFailed)
        {
            Interlocked.Decrement(ref _runningRequests);
            Interlocked.Increment(ref _completedRequests);
            if (isFailed)
            {
                Interlocked.Increment(ref _failedRequests);
            }
            else
            {
                Interlocked.Increment(ref _succeededRequests);
            }
        }

        public long getCompletedRequests()
        {
            return this._completedRequests;
        }

        public long getRunningRequests()
        {
            return this._runningRequests;
        }

        public long getFailedRequests()
        {
            return this._failedRequests;
        }

        public long getSucceededRequests()
        {
            return this._succeededRequests;
        }
    }

AppState 和 MyRestHostLifetime 注入DI的代码片段

        //register AppState object
        var appState = new AppState() { startTime = DateTime.Now };
        services.AddSingleton<AppState>(appState);


        //register MyRestHostLifetime to ensure running requrest handled
        var provider = services.BuildServiceProvider();
        //register MyRestHostLifetime to ensure running requrest handled
        var lifetimeLogger = provider.GetRequiredService<ILogger<MyRestHostLifetime>>();
        services.AddSingleton<IHostLifetime>(sp => new MyRestHostLifetime(
            sp.GetRequiredService<IHostApplicationLifetime>(),
            TimeSpan.FromSeconds(0.1),
            appState, lifetimeLogger));