NetCore之Azure Service Bus start new process to consume message

什么是Azure服务总线？

Azure服务总线是完全托管的企业消息代理，包括消息队列和发布订阅(命名空间). Service Bus通常被用来解耦应用程序和服务，控制跨服务和应用程序间安全路由和传输数据。

本篇从实战出发介绍如何通过Service Bus发送接收及消费消息。

项目介绍：创建了5个工程分别为API.AzureServiceBus.Sender消息发送；API.AzureServiceBus.Receiver接收消息并其新进程处理消息；API.AzureServiceBus.Process负责消费消息；API.AzureServiceBus.Common为公共资源主要提供ServiceBusSender及ServiceBusProcessor的缓存及配置文件类；API.AzureServiceBus.Controller模拟发送消息对外提供接口。

1、创建API.AzureServiceBus.Sender工程发送消息

添加配置项，这里主要介绍通过connectionstring的方式连接AzureServiceBus。

  "AzureServiceBus": {
    "ConnectionString": "***",
    "QueuePrefix": "***"
  }

创建ServiceBusSenderService服务用于发送消息，这里通过构造函数注入了ObjectPool<ServiceBusSender> services, 通过Get()或者对象池中可用的ServiceBusSender对象。构建一个自定义的DemoEvent类并序列化之后传入ServiceBusMessage构造函数，并且通过ServiceBusMessage ApplicationProperties属性添加额外的属性值，具体实现如下：

using API.AzureServiceBus.Common;
using Azure.Messaging.ServiceBus;
using Microsoft.Extensions.ObjectPool;
using Newtonsoft.Json;

namespace API.AzureServiceBus.Sender;

public class ServiceBusSenderService
{
    private readonly ObjectPool<ServiceBusSender> _senders;
    public ServiceBusSenderService(ObjectPool<ServiceBusSender> senders)
    {
        this._senders = senders;
    }

    public async Task SendAsync()
    {
        var message = this.BuildServiceMessage();
        var sender = _senders.Get();
        try
        {
            Console.WriteLine($"Send event message: {message.Subject}-{message.MessageId}.");
            await sender.SendMessageAsync(message);
        }
        finally
        {
            _senders.Return(sender);
        }
    }

    private ServiceBusMessage BuildServiceMessage()
    {
        var demoEvent = new DemoEvent()
        {
            Name = typeof(DemoEvent).Name,
            EventType = typeof(DemoEvent),
            IsRunning = true,
        };
        var body = JsonConvert.SerializeObject(demoEvent,
            new JsonSerializerSettings
            {
                TypeNameHandling = TypeNameHandling.All,
            });
        var message = new ServiceBusMessage(body!)
        {
            MessageId = Guid.NewGuid().ToString(),
            Subject = typeof(DemoEvent).Name,
        };
        message.ApplicationProperties.Add("property1", "hello");
        message.ApplicationProperties.Add("property2", 100);
        return message;
    }    
}

ServiceBusSenderService

在NetCore 容器中配置构建ServiceBusClient及ServiceBusAdministrationClient 缓存及服务。

using API.AzureServiceBus.Common;
using Azure.Messaging.ServiceBus;
using Azure.Messaging.ServiceBus.Administration;
using Microsoft.Extensions.ObjectPool;

Environment.SetEnvironmentVariable("ASPNETCORE_ENVIRONMENT", "Development", EnvironmentVariableTarget.Process);
var builder = WebApplication.CreateBuilder(args);
var configuration = builder.Configuration;
var azureConfig = configuration.GetSection("AzureServiceBus").Get<AzureServiceBusConfiguration>();

builder.Services.AddSingleton(sp => new ServiceBusClient(azureConfig!.ConnectionString));
builder.Services.AddSingleton(sp => new ServiceBusAdministrationClient(azureConfig!.ConnectionString));
builder.Services.AddSingleton<AzureServiceBusFactory>();

builder.Services.AddSingleton<ObjectPool<ServiceBusSender>>(sp => 
{
    var provider = sp.GetRequiredService<ObjectPoolProvider>();
    var azureService = sp.GetRequiredService<AzureServiceBusFactory>();
    return provider.Create(new ServiceBusSenderPoolPolicy(azureService, builder.Configuration));
});

var app = builder.Build();

app.Run();

Sender Container

2、创建API.AzureServiceBus.Receiver工程接收消息

配置项同#1

订阅消息，这里创建守护进程继承自BackgroundService监听是否有消息待消费。

public static class SubscribeEventExtension
{
    public static void SubscribeEvent(this IServiceCollection services)
    {
        var eventServices = services.BuildServiceProvider().GetRequiredService<EventCenterService>();
        services.AddTransient<DemoEventHandler>();
        eventServices.AddHandler<DemoEvent, DemoEventHandler>();
    }
}

Subscribe message

创建ServiceBusProcessorService服务用于接收并处理分发消息，这里介绍下通过新进程消费消息的方式：首先将接收到的消息缓存到本地；然后将新进程的执行路径指定到本地缓存的消息；最后构建ProcessStartInfo类，通过Process.Start(ProcessStartInfo)启动新进程。

public partial class ServiceBusProcessorService
{
    private readonly ObjectPool<ServiceBusProcessor> _processors;
    private readonly IServiceProvider _serviceProvider;
    public ServiceBusProcessorService(ObjectPool<ServiceBusProcessor> processors,
        IServiceProvider serviceProvider) 
    {
        this._processors = processors;
        this._serviceProvider = serviceProvider;
    }

    public async Task ProcessAsync()
    {
        var processor = _processors.Get();
        try
        {
            processor.ProcessMessageAsync += ProcessMessageAsync;
            processor.ProcessErrorAsync += ProcessErrorAsync;
            await processor.StartProcessingAsync();
        }
        finally
        {
            _processors.Return(processor);
        }
    }

    private async Task ProcessMessageAsync(ProcessMessageEventArgs args)
    {
        var body = args.Message.Body;
        var property1 = args.Message.ApplicationProperties["property1"];
        var property2 = args.Message.ApplicationProperties["property2"];
        var message = JsonConvert.DeserializeObject<EventBase>(body.ToString(), new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.All });
        //await StartProcessing(message!);
        await StartProcessingAsync(message!);
        await args.CompleteMessageAsync(args.Message, args.CancellationToken);
        await Task.CompletedTask;
    }

    /// <summary>
    /// Consume the message in current process
    /// </summary>
    public async Task StartProcessing(EventBase message)
    {
        var _eventServices = _serviceProvider.GetRequiredService<EventCenterService>();
        if (_eventServices.HasSubscribeEvent(message!.Name))
        {
            var types = _eventServices.GetHandlerTypes(message!.Name);
            foreach (var type in types)
            {
                var handlerService = _serviceProvider.GetRequiredService(type);
                var service = typeof(IEventHandler<>).MakeGenericType(message.EventType);
                await(Task)service.GetMethod("Handle").Invoke(handlerService, new object[] { message });
            }
        }
    }

    /// <summary>
    /// Consume the message with new process
    /// </summary>
    public async Task StartProcessingAsync(EventBase message)
    {
        Console.WriteLine($"Start to new process to handle the message: {message.Name}_{message.Id}.");
        var process = _serviceProvider.GetRequiredService<ServiceBusProcessorService>();
        await process.NewProcess(message);
    }

    private async Task ProcessErrorAsync(ProcessErrorEventArgs args)
    {
        Console.WriteLine($"Consume the message failed, error code: {args.ErrorSource}, exception: {args.Exception.Message}.");
        await Task.CompletedTask;
    }
}

ServiceBusProcessorService

public partial class ServiceBusProcessorService
{
    public async Task NewProcess(EventBase message)
    {
        var path = Path.GetDirectoryName(this.GetType().Assembly.Location);
        var file = $"{message.Name}_{message.Id}";
        CacheMessage(path!, file, message);
        var args = $"{message.Id} {message.TraceId} {message.Name} {file}";
        var processFile = Path.Combine(path!, "API.AzureServiceBus.Process.exe");
        var processStartInfo = new ProcessStartInfo(processFile, args)
        {
            WorkingDirectory = path,
            UseShellExecute = false
        };
        var process = Process.Start(processStartInfo);
        await Task.CompletedTask;
        Console.WriteLine($"The process id: {process!.Id}, name: {process.ProcessName}, message: {message.Name}.");
    }

    private void CacheMessage(string folder, string file, EventBase message)
    {
        var tempFolder = Path.Combine(folder, "EventData");
        file = Path.Combine(tempFolder, file);
        if (!Directory.Exists(tempFolder))
            Directory.CreateDirectory(tempFolder);
        if (File.Exists(file))
            File.Delete(file);
        using var fs = new FileStream(file, FileMode.OpenOrCreate);
        fs.Write(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(message, new JsonSerializerSettings() { TypeNameHandling = TypeNameHandling.All })));
        fs.Flush();
    }
}

ServiceBusProcessorService

在NetCore容器中配置构建ServiceBusClient,ServiceBusAdministrationClient 及ServiceBusProcessor的缓存服务，这里有个服务EventCenterService主要是用于通过订阅的方式构建Event（消息）及EventHander（消费消息）映射关系被缓存到ConcurrentDictionary中。

using API.AzureServiceBus.Common;
using API.AzureServiceBus.Receiver;
using Azure.Messaging.ServiceBus;
using Azure.Messaging.ServiceBus.Administration;
using Microsoft.Extensions.ObjectPool;

Environment.SetEnvironmentVariable("ASPNETCORE_ENVIRONMENT", "Development", EnvironmentVariableTarget.Process);
var builder = WebApplication.CreateBuilder(args);
var configuration = builder.Configuration;
var azureConfig = configuration.GetSection("AzureServiceBus").Get<AzureServiceBusConfiguration>();

builder.Services.AddSingleton(sp => new ServiceBusClient(azureConfig!.ConnectionString));
builder.Services.AddSingleton(sp => new ServiceBusAdministrationClient(azureConfig!.ConnectionString));
builder.Services.AddSingleton<AzureServiceBusFactory>();

builder.Services.AddSingleton<ObjectPool<ServiceBusProcessor>>(sp =>
{
    var provider = sp.GetRequiredService<ObjectPoolProvider>();
    var azureService = sp.GetRequiredService<AzureServiceBusFactory>();
    return provider.Create(new ServcieBusProcessorPoolPolicy(azureService, builder.Configuration));
});
builder.Services.AddSingleton<EventCenterService>();
builder.Services.AddTransient<ServiceBusProcessorService>();
builder.Services.SubscribeEvent();
builder.Services.AddHostedService<DaemonService>();

var app = builder.Build();
app.Lifetime.ApplicationStopped.Register(async () => 
{
    var _client = app.Services.GetRequiredService<ServiceBusClient>();
    if (_client != null && !_client.IsClosed)
        await _client.DisposeAsync();
});
app.Run();

Processor Container

3、接收到消息之后提供2种消费方式，#1通过当前进程消费; #2起新进程消费消息（推荐）

创建一个API.AzureServiceBus.Process NetCore Console Application

订阅消息同#2

通过新进程调用EventHandler方法消费消息

public class ProcessService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly string _eventData = "EventData";

    public ProcessService(IServiceProvider serviceProvider)
    {
        this._serviceProvider = serviceProvider;
    }

    public async Task StartAsync(string[] args)
    {
        Console.WriteLine($"Start to handle the message in new process, process: {Thread.CurrentThread.Name}_{Thread.CurrentThread.ManagedThreadId}, arguments: {string.Join(";", args)}");
        var _eventServices = _serviceProvider.GetRequiredService<EventCenterService>();
        if (args.Length > 3 && !string.IsNullOrEmpty(args[3]))
        {
            var fileName = args[3];
            var filePath = Path.Combine(Directory.GetCurrentDirectory(), _eventData, fileName);
            if (!File.Exists(filePath))
                throw new Exception($"The event: {args[2]} file is not exist in system.");
            using StreamReader streamReader = File.OpenText(filePath);
            var msg = streamReader.ReadToEnd();
            try { File.Delete(filePath); } catch { }

            var message = JsonConvert.DeserializeObject<EventBase>(msg,
                new JsonSerializerSettings()
                {
                    TypeNameHandling = TypeNameHandling.All
                });
            if (_eventServices.HasSubscribeEvent(message!.Name))
            {
                var types = _eventServices.GetHandlerTypes(message!.Name);
                foreach (var type in types)
                {
                    var handlerService = _serviceProvider.GetRequiredService(type);
                    var service = typeof(IEventHandler<>).MakeGenericType(message.EventType);
                    await (Task)service.GetMethod("Handle").Invoke(handlerService, new object[] { message });
                }
            }
            await Task.CompletedTask;
        }
    }
}

Process messag

在NetCore容器中配置构建相关服务，主要是EventCenterService消息管理中心服务及ProcessService消费消息服务。

using API.AzureServiceBus.Common;
using API.AzureServiceBus.Process;
using Microsoft.Extensions.DependencyInjection;

#if DEBUG
var a = 1;
var b = 1;
while (a == b)
{
    Thread.Sleep(3000);
}
#endif
var services = new ServiceCollection();
services.AddSingleton<EventCenterService>();
services.SubscribeEvent();
services.AddTransient<ProcessService>();
var serviceProvider = services.BuildServiceProvider();
var process = serviceProvider.GetRequiredService<ProcessService>();
await process.StartAsync(args);

Process Container

4、创建API.AzureServiceBus.Controller模拟向AzureServiceBus发送消息

[Route("[Controller]")]
    [ApiController]
    public class ServiceBusController : Controller
    {
        private readonly ServiceBusSenderService _senderService;
        public ServiceBusController(ServiceBusSenderService senderService) 
        {
            this._senderService = senderService;
        }

        [HttpPost("send")]
        public async Task<IActionResult> SendAsync()
        {
            await _senderService.SendAsync();
            return Ok("Send success!");
        }
    }

Controller

向NetCore Container中添加所需服务

using API.AzureServiceBus.Common;
using API.AzureServiceBus.Sender;
using Azure.Messaging.ServiceBus.Administration;
using Azure.Messaging.ServiceBus;
using Microsoft.Extensions.ObjectPool;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

Environment.SetEnvironmentVariable("ASPNETCORE_ENVIRONMENT", "Development", EnvironmentVariableTarget.Process);
var configuration = builder.Configuration;
var azureConfig = configuration.GetSection("AzureServiceBus").Get<AzureServiceBusConfiguration>();

builder.Services.AddSingleton(sp => new ServiceBusClient(azureConfig!.ConnectionString));
builder.Services.AddSingleton(sp => new ServiceBusAdministrationClient(azureConfig!.ConnectionString));
builder.Services.AddSingleton<AzureServiceBusFactory>();
builder.Services.AddSingleton<ObjectPool<ServiceBusSender>>(sp =>
{
    var provider = sp.GetRequiredService<ObjectPoolProvider>();
    var azureService = sp.GetRequiredService<AzureServiceBusFactory>();
    return provider.Create(new ServiceBusSenderPoolPolicy(azureService, builder.Configuration));
});
builder.Services.AddTransient<ServiceBusSenderService>();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();

Controller Container

5、除了以上基础服务之外，还有一个Common Library Class API.AzureServiceBus.Common主要作用是缓存ServiceBusSender/ServiceBusProcessor这是AzureServiceBus内部类发送和接收消息，用到了NetCore ObjectPool对象池技术所需packages

自定义ServiceBusSenderPoolPolicy 用于创建Sender的对象池，自定义ServcieBusProcessorPoolPolicy用于创建Processor对象池

public class ServcieBusProcessorPoolPolicy : IPooledObjectPolicy<ServiceBusProcessor>
{
    private readonly AzureServiceBusFactory _serviceBus;
    private readonly IConfiguration _configuration;
    public ServcieBusProcessorPoolPolicy(AzureServiceBusFactory serviceBus,
        IConfiguration configuration)
    { 
        this._serviceBus = serviceBus;
        this._configuration = configuration;
    }

    public ServiceBusProcessor Create()
    {
        var queueName = _configuration["AzureServiceBus:QueuePrefix"]!;
        var isQueueExist = _serviceBus.IsQueueExist(queueName).GetAwaiter().GetResult();
        if (!isQueueExist)
            _serviceBus.CreateQueue(queueName).GetAwaiter().GetResult();
        var processor = _serviceBus.CreateServiceBusProcessor(queueName);
        return processor;
    }

    public bool Return(ServiceBusProcessor obj)
    {
        if(obj.IsClosed)
        {
            obj.DisposeAsync().GetAwaiter().GetResult();
        }
        if (obj.IsProcessing)
            return false;
        return true;
    }
}

ServcieBusProcessorPoolPolicy

public class ServiceBusSenderPoolPolicy : IPooledObjectPolicy<ServiceBusSender>
{
    private readonly AzureServiceBusFactory _serviceBus;
    private readonly IConfiguration _configuration;
    public ServiceBusSenderPoolPolicy(AzureServiceBusFactory serviceBus, 
        IConfiguration configuration)
    {
        _serviceBus = serviceBus;
        _configuration = configuration;
    }

    public ServiceBusSender Create()
    {
        var queueName = _configuration["AzureServiceBus:QueuePrefix"]!;
        var isQueueExist = _serviceBus.IsQueueExist(queueName).GetAwaiter().GetResult();
        if (!isQueueExist)
            _serviceBus.CreateQueue(queueName).GetAwaiter().GetResult();
        var sender = _serviceBus.CreateServiceBusSender(queueName);
        return sender;
    }

    public bool Return(ServiceBusSender obj)
    {
        if (obj.IsClosed)
        {
            obj?.DisposeAsync().GetAwaiter().GetResult();
            return false;
        }
        return true;
    }
}

ServiceBusSenderPoolPolicy

创建AzureServiceBusFactory用于管理ServiceBusSender及ServiceBusProcessor对象。

    public class AzureServiceBusFactory
    {
        private readonly ServiceBusClient _client;
        private readonly ServiceBusAdministrationClient _adminClient;
        private ServiceBusProcessorOptions _options
            => new ServiceBusProcessorOptions
            {
                AutoCompleteMessages = false,
                MaxConcurrentCalls = 5,
                ReceiveMode = ServiceBusReceiveMode.PeekLock
            };

        public AzureServiceBusFactory(ServiceBusClient client, ServiceBusAdministrationClient adminClient)
        {
            this._client = client;
            this._adminClient = adminClient;
        }
        public async Task<QueueProperties> CreateQueue(string queueName)
        {
            var options = _queueOptions(queueName);
            options.AuthorizationRules.Add(new SharedAccessAuthorizationRule("allClaims", 
                new[] 
                { 
                    AccessRights.Manage,
                    AccessRights.Send, 
                    AccessRights.Listen 
                }));
            return await _adminClient.CreateQueueAsync(options);
        }

        public async Task<bool> IsQueueExist(string queueName)
        {
            return await _adminClient.QueueExistsAsync(queueName);
        }

        public ServiceBusSender CreateServiceBusSender(string queueName)
        {
            return _client.CreateSender(queueName);
        }

        public ServiceBusProcessor CreateServiceBusProcessor(string queueName, ServiceBusProcessorOptions? _processOptions = default)
        {
            return _client.CreateProcessor(queueName, _processOptions ?? _options);
        }

        private CreateQueueOptions _queueOptions(string queueName) => new CreateQueueOptions(queueName)
        {
            //Basic not support
            //AutoDeleteOnIdle = TimeSpan.FromDays(7),
            DefaultMessageTimeToLive = TimeSpan.FromDays(2),
            //DuplicateDetectionHistoryTimeWindow = TimeSpan.FromMinutes(1),
            EnableBatchedOperations = true,
            DeadLetteringOnMessageExpiration = true,
            EnablePartitioning = false,
            ForwardDeadLetteredMessagesTo = null,
            ForwardTo = null,
            //LockDuration = TimeSpan.FromSeconds(45),default 60s
            //MaxDeliveryCount = 8,default 10
            MaxSizeInMegabytes = 2048,
            //RequiresDuplicateDetection = true,
            //RequiresSession = true,
            UserMetadata = "some metadata",
        };
    }

AzureServiceBusFactory

消息管理中心

public partial class EventCenterService
{
    public static ConcurrentDictionary<string, List<Type>> _eventHandler = new ConcurrentDictionary<string, List<Type>>();

    public void AddHandler<E, H>() 
        where E : EventBase where H : IEventHandler<E>
    {
        var eventName = _eventName<E>();
        _eventHandler.AddOrUpdate(eventName,
            eventName => new List<Type> { typeof(H) },
            (eventName, handlers) =>
            {
                if (handlers.Contains(typeof(H)))
                    return handlers;
                else handlers.Add(typeof(H));
                return handlers;
            });
    }

    public IEnumerable<Type> GetHandlerTypes(string eventName)
    {
        return _eventHandler[eventName];
    }

    public bool HasSubscribeEvent(string eventName)
    {
        return _eventHandler.ContainsKey(eventName);
    }

    private string _eventName<E>() where E : EventBase
        => typeof(E).Name;
}

EventCenterService

配置文件映射类

public class AzureServiceBusConfiguration
{
    public string ConnectionString { get; set; }
    public string QueuePrefix { get; set; }
}

AzureServiceBusConfiguration

EventHandler接口及实现类，以DemoEventHandler为例

public interface IEventHandler<in T> where T : EventBase
{
    Task Handle(T @event);
}

public class DemoEventHandler : IEventHandler<DemoEvent>
{
    public async Task Handle(DemoEvent @event)
    {
        Console.WriteLine($"Start to processing message: {@event.Name} in {typeof(DemoEventHandler).Name}.");
        await Task.CompletedTask;
        Console.WriteLine($"Consume message successful.");
    }
}

EventHandler

Event基类及实现类，以DemoEvent为例

public class EventBase
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public Type EventType { get; set; }
    public Guid TraceId { get; set; }

    public EventBase()
    {
        Id = Guid.NewGuid();
        TraceId = Guid.NewGuid();
    }
}

public class DemoEvent : EventBase
{
    public bool IsRunning { get; set; }

    public DemoEvent()
    {
       
    }
}

Event base

测试结果：

启动API.AzureServiceBus.Sender / API.AzureServiceBus.Receiver 及API.AzureServiceBus.Controller

a.通过Swagger or Postman 模拟发送消息

b.API.AzureServcieBus.Receiver工程接收到消息并负责处理

c.在Debug模式下可以Attach process API.AzureServiceBus.Process,再次查看消息被新进程成功消费。

Notes: 因为API.AzureServiceBusProcess进程需要读取API.AzureServiceBus.Receiver缓存的本地message信息，所以两个工程需要设置同一输出路径，在.csproj中指定OutputPath

<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
     <OutputPath>../AzureServiceBus/Process</OutputPath>
</PropertyGroup>

OK，详细解说持续更新中....

posted @ 2023-10-27 13:42 云霄宇霁阅读(35) 评论(0) 编辑收藏举报

刷新页面返回顶部

云霄宇霁

NetCore之Azure Service Bus start new process to consume message

公告