kestrel网络编程--开发Fiddler

1 文章目的#

本文讲解基于kestrel开发类似Fiddler应用的过程，让读者了解kestrel网络编程里面的kestrel中间件和http应用中间件。由于最终目的不是输出完整功能的产品，所以这里只实现Fiddler最核心的http请求和响应内容查看的功能。本文章是KestrelApp项目里面的一个demo的讲解，希望对您有用。

2 开发顺序#

1. 代理协议kestrel中间件
2. tls协议侦测kestrel中间件
3. 隧道和http协议侦测kestrel中间件
4. 请求响应分析http中间件
5. 反向代理http中间件
6. 编排中间件创建服务器和应用

3 传输层与kestrel中间件#

所谓传输层，其目的是为了让应用协议数据安全、可靠、快速等传输而存在的一种协议，其特征是把应用协议的报文做为自己的负载，常见的tcp、udp、quic、tls等都可以理解为传输层协议。
比如http协议，常见有如下的传输方式：

1. http over tcp
2. http over tls over tcp
3. http over quic over udp

3.1 Fiddler的传输层#

Fiddler要处理以下三种http传输情况：

1. http over tcp：直接http请求首页
2. http over proxy over tcp：代理http流量
3. http over tls over proxy over tcp：代理https流量

3.2 Kestrel的中间件#

kestrel目前的传输层基于tcp或quic两种，同时内置了tls中间件，需要调用ListenOptions.UseHttps()来使用tls中间件。kestrel的中间件的表现形式为：Func<ConnectionDelegate, ConnectionDelegate>，为了使用读者能够简单理解中间件，我在KestrelFramework里定义了kestrel中间件的变种接口，大家基于此接口来实现更多的中间件就方便很多：

Copy
/// <summary>
/// Kestrel的中间件接口
/// </summary>
public interface IKestrelMiddleware
{
    /// <summary>
    /// 执行
    /// </summary>
    /// <param name="next"></param>
    /// <param name="context"></param>
    /// <returns></returns>
    Task InvokeAsync(ConnectionDelegate next, ConnectionContext context);
}

4 代理协议kestrel中间件#

Filddler最基础的功能是它是一个http代理服务器， 我们需要为kestrel编写代理中间件，用于处理代理传输层。http代理协议分两种：普通的http代理和Connect隧道代理。两种的报文者是遵循http1.0或1.1的文本格式，我们可以使用kestrel自带的HttpParser<>来解析这些复杂的http文本协议。

4.1 代理特征#

在中间件编程模式中，Feature是一个很重要的中间件沟通桥梁，它往往是某个中间件工作之后，留下的财产，让之后的中间件来获取并受益。我们的代理中间件，也设计了IProxyFeature，告诉之后的中间件一些代理特征。

Copy
/// <summary>
/// 代理Feature
/// </summary>
public interface IProxyFeature
{
    /// <summary>
    /// 代理主机
    /// </summary>
    HostString ProxyHost { get; }

    /// <summary>
    /// 代理协议
    /// </summary>
    ProxyProtocol ProxyProtocol { get; }
}

/// <summary>
/// 代理协议
/// </summary>
public enum ProxyProtocol
{
    /// <summary>
    /// 无代理
    /// </summary>
    None,

    /// <summary>
    /// http代理
    /// </summary>
    HttpProxy,

    /// <summary>
    /// 隧道代理
    /// </summary>
    TunnelProxy
}

4.2 代理中间件的实现#

Copy
/// <summary>
/// 代理中间件
/// </summary>
sealed class KestrelProxyMiddleware : IKestrelMiddleware
{
    private static readonly HttpParser<HttpRequestHandler> httpParser = new();
    private static readonly byte[] http200 = Encoding.ASCII.GetBytes("HTTP/1.1 200 Connection Established\r\n\r\n");
    private static readonly byte[] http400 = Encoding.ASCII.GetBytes("HTTP/1.1 400 Bad Request\r\n\r\n");

    /// <summary>
    /// 解析代理
    /// </summary>
    /// <param name="next"></param>
    /// <param name="context"></param>
    /// <returns></returns>
    public async Task InvokeAsync(ConnectionDelegate next, ConnectionContext context)
    {
        var input = context.Transport.Input;
        var output = context.Transport.Output;
        var request = new HttpRequestHandler();

        while (context.ConnectionClosed.IsCancellationRequested == false)
        {
            var result = await input.ReadAsync();
            if (result.IsCanceled)
            {
                break;
            }

            try
            {
                if (ParseRequest(result, request, out var consumed))
                {
                    if (request.ProxyProtocol == ProxyProtocol.TunnelProxy)
                    {
                        input.AdvanceTo(consumed);
                        await output.WriteAsync(http200);
                    }
                    else
                    {
                        input.AdvanceTo(result.Buffer.Start);
                    }

                    context.Features.Set<IProxyFeature>(request);
                    await next(context);

                    break;
                }
                else
                {
                    input.AdvanceTo(result.Buffer.Start, result.Buffer.End);
                }

                if (result.IsCompleted)
                {
                    break;
                }
            }
            catch (Exception)
            {
                await output.WriteAsync(http400);
                break;
            }
        }
    }


    /// <summary>
    /// 解析http请求
    /// </summary>
    /// <param name="result"></param>
    /// <param name="request"></param>
    /// <param name="consumed"></param>
    /// <returns></returns>
    private static bool ParseRequest(ReadResult result, HttpRequestHandler request, out SequencePosition consumed)
    {
        var reader = new SequenceReader<byte>(result.Buffer);
        if (httpParser.ParseRequestLine(request, ref reader) &&
            httpParser.ParseHeaders(request, ref reader))
        {
            consumed = reader.Position;
            return true;
        }
        else
        {
            consumed = default;
            return false;
        }
    }


    /// <summary>
    /// 代理请求处理器
    /// </summary>
    private class HttpRequestHandler : IHttpRequestLineHandler, IHttpHeadersHandler, IProxyFeature
    {
        private HttpMethod method;

        public HostString ProxyHost { get; private set; }

        public ProxyProtocol ProxyProtocol
        {
            get
            {
                if (ProxyHost.HasValue == false)
                {
                    return ProxyProtocol.None;
                }
                if (method == HttpMethod.Connect)
                {
                    return ProxyProtocol.TunnelProxy;
                }
                return ProxyProtocol.HttpProxy;
            }
        }

        void IHttpRequestLineHandler.OnStartLine(HttpVersionAndMethod versionAndMethod, TargetOffsetPathLength targetPath, Span<byte> startLine)
        {
            method = versionAndMethod.Method;
            var host = Encoding.ASCII.GetString(startLine.Slice(targetPath.Offset, targetPath.Length));
            if (versionAndMethod.Method == HttpMethod.Connect)
            {
                ProxyHost = HostString.FromUriComponent(host);
            }
            else if (Uri.TryCreate(host, UriKind.Absolute, out var uri))
            {
                ProxyHost = HostString.FromUriComponent(uri);
            }
        }

        void IHttpHeadersHandler.OnHeader(ReadOnlySpan<byte> name, ReadOnlySpan<byte> value)
        {
        }
        void IHttpHeadersHandler.OnHeadersComplete(bool endStream)
        {
        }
        void IHttpHeadersHandler.OnStaticIndexedHeader(int index)
        {
        }
        void IHttpHeadersHandler.OnStaticIndexedHeader(int index, ReadOnlySpan<byte> value)
        {
        }
    }
}

5 tls协议侦测kestrel中间件#

Fiddler只监听了一个端口，要同时支持非加密和加密两种流量，如果不调用调用ListenOptions.UseHttps()，我们的程序就不支持https的分析；如果直接调用ListenOptions.UseHttps()，会让我们的程序不支持非加密的http的分析，这就要求我们有条件的根据客户端发来的流量分析是否需要开启。

我已经在KestrelFramework内置了TlsDetection中间件，这个中间件可以根据客户端的实际流量类型来选择是否使用tls。在Fiddler中，我们还需要根据客户端的tls握手中的sni使用ca证书来动态生成服务器证书用于tls加密传输。

Copy
/// <summary>
/// 证书服务
/// </summary>
sealed class CertService
{
    private const string CACERT_PATH = "cacert";
    private readonly IMemoryCache serverCertCache;
    private readonly IEnumerable<ICaCertInstaller> certInstallers;
    private readonly ILogger<CertService> logger;
    private X509Certificate2? caCert;


    /// <summary>
    /// 获取证书文件路径
    /// </summary>
    public string CaCerFilePath { get; } = OperatingSystem.IsLinux() ? $"{CACERT_PATH}/fiddler.crt" : $"{CACERT_PATH}/fiddler.cer";

    /// <summary>
    /// 获取私钥文件路径
    /// </summary>
    public string CaKeyFilePath { get; } = $"{CACERT_PATH}/fiddler.key";

    /// <summary>
    /// 证书服务
    /// </summary>
    /// <param name="serverCertCache"></param>
    /// <param name="certInstallers"></param>
    /// <param name="logger"></param>
    public CertService(
        IMemoryCache serverCertCache,
        IEnumerable<ICaCertInstaller> certInstallers,
        ILogger<CertService> logger)
    {
        this.serverCertCache = serverCertCache;
        this.certInstallers = certInstallers;
        this.logger = logger;
        Directory.CreateDirectory(CACERT_PATH);
    }

    /// <summary>
    /// 生成CA证书
    /// </summary> 
    public bool CreateCaCertIfNotExists()
    {
        if (File.Exists(this.CaCerFilePath) && File.Exists(this.CaKeyFilePath))
        {
            return false;
        }

        File.Delete(this.CaCerFilePath);
        File.Delete(this.CaKeyFilePath);

        var notBefore = DateTimeOffset.Now.AddDays(-1);
        var notAfter = DateTimeOffset.Now.AddYears(10);

        var subjectName = new X500DistinguishedName($"CN={nameof(Fiddler)}");
        this.caCert = CertGenerator.CreateCACertificate(subjectName, notBefore, notAfter);

        var privateKeyPem = this.caCert.GetRSAPrivateKey()?.ExportRSAPrivateKeyPem();
        File.WriteAllText(this.CaKeyFilePath, new string(privateKeyPem), Encoding.ASCII);

        var certPem = this.caCert.ExportCertificatePem();
        File.WriteAllText(this.CaCerFilePath, new string(certPem), Encoding.ASCII);

        return true;
    }

    /// <summary>
    /// 安装和信任CA证书
    /// </summary> 
    public void InstallAndTrustCaCert()
    {
        var installer = this.certInstallers.FirstOrDefault(item => item.IsSupported());
        if (installer != null)
        {
            installer.Install(this.CaCerFilePath);
        }
        else
        {
            this.logger.LogWarning($"请根据你的系统平台手动安装和信任CA证书{this.CaCerFilePath}");
        } 
    }
         

    /// <summary>
    /// 获取颁发给指定域名的证书
    /// </summary>
    /// <param name="domain"></param> 
    /// <returns></returns>
    public X509Certificate2 GetOrCreateServerCert(string? domain)
    {
        if (this.caCert == null)
        {
            using var rsa = RSA.Create();
            rsa.ImportFromPem(File.ReadAllText(this.CaKeyFilePath));
            this.caCert = new X509Certificate2(this.CaCerFilePath).CopyWithPrivateKey(rsa);
        }

        var key = $"{nameof(CertService)}:{domain}";
        var endCert = this.serverCertCache.GetOrCreate(key, GetOrCreateCert);
        return endCert!;

        // 生成域名的1年证书
        X509Certificate2 GetOrCreateCert(ICacheEntry entry)
        {
            var notBefore = DateTimeOffset.Now.AddDays(-1);
            var notAfter = DateTimeOffset.Now.AddYears(1);
            entry.SetAbsoluteExpiration(notAfter);

            var extraDomains = GetExtraDomains();

            var subjectName = new X500DistinguishedName($"CN={domain}");
            var endCert = CertGenerator.CreateEndCertificate(this.caCert, subjectName, extraDomains, notBefore, notAfter);

            // 重新初始化证书，以兼容win平台不能使用内存证书
            return new X509Certificate2(endCert.Export(X509ContentType.Pfx));
        }
    }

    /// <summary>
    /// 获取域名
    /// </summary>
    /// <param name="domain"></param>
    /// <returns></returns>
    private static IEnumerable<string> GetExtraDomains()
    {
        yield return Environment.MachineName;
        yield return IPAddress.Loopback.ToString();
        yield return IPAddress.IPv6Loopback.ToString();
    }
}

6 隧道和http协议侦测kestrel中间件#

经过KestrelProxyMiddleware后的流量，在tls解密(如果可能)之后，一般情况下都是http流量了，但如果你在qq设置代理到我们这个伪Fildder之后，会发现部分流量流量不是http流量，原因是http隧道也是一个通用传输层，可以传输任意tcp或tcp之上的流量。所以我们需要新的中间件来检测当前流量，如果不是http流量就回退到隧道代理的流程，即我们不跟踪不分析这部分非http流量。

6.1 http流量侦测#

Copy
/// <summary>
/// 流量侦测器
/// </summary>
private static class FlowDetector
{
    private static readonly byte[] crlf = Encoding.ASCII.GetBytes("\r\n");
    private static readonly byte[] http10 = Encoding.ASCII.GetBytes(" HTTP/1.0");
    private static readonly byte[] http11 = Encoding.ASCII.GetBytes(" HTTP/1.1");
    private static readonly byte[] http20 = Encoding.ASCII.GetBytes(" HTTP/2.0");

    /// <summary>
    /// 传输内容是否为http
    /// </summary>
    /// <param name="context"></param>
    /// <returns></returns>
    public static async ValueTask<bool> IsHttpAsync(ConnectionContext context)
    {
        var input = context.Transport.Input;
        var result = await input.ReadAtLeastAsync(1);
        var isHttp = IsHttp(result);
        input.AdvanceTo(result.Buffer.Start);
        return isHttp;
    }

    private static bool IsHttp(ReadResult result)
    {
        var reader = new SequenceReader<byte>(result.Buffer);
        if (reader.TryReadToAny(out ReadOnlySpan<byte> line, crlf))
        {
            return line.EndsWith(http11) || line.EndsWith(http20) || line.EndsWith(http10);
        }
        return false;
    }
}

6.2 隧道回退中间件#

Copy
/// <summary>
/// 隧道传输中间件
/// </summary>
sealed class KestrelTunnelMiddleware : IKestrelMiddleware
{
    private readonly ILogger<KestrelTunnelMiddleware> logger;

    /// <summary>
    /// 隧道传输中间件
    /// </summary>
    /// <param name="logger"></param>
    public KestrelTunnelMiddleware(ILogger<KestrelTunnelMiddleware> logger)
    {
        this.logger = logger;
    }

    /// <summary>
    /// 执行中间你件
    /// </summary>
    /// <param name="next"></param>
    /// <param name="context"></param>
    /// <returns></returns>
    public async Task InvokeAsync(ConnectionDelegate next, ConnectionContext context)
    {
        var feature = context.Features.Get<IProxyFeature>();
        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
        {
            this.logger.LogInformation($"侦测到http直接请求");
            await next(context);
        }
        else if (feature.ProxyProtocol == ProxyProtocol.HttpProxy)
        {
            this.logger.LogInformation($"侦测到普通http代理流量");
            await next(context);
        }
        else if (await FlowDetector.IsHttpAsync(context))
        {
            this.logger.LogInformation($"侦测到隧道传输http流量");
            await next(context);
        }
        else
        {
            this.logger.LogInformation($"跳过隧道传输非http流量{feature.ProxyHost}的拦截");
            await TunnelAsync(context, feature);
        }
    }

    /// <summary>
    /// 隧道传输其它协议的数据
    /// </summary>
    /// <param name="context"></param>
    /// <param name="feature"></param>
    /// <returns></returns>
    private async ValueTask TunnelAsync(ConnectionContext context, IProxyFeature feature)
    {
        var port = feature.ProxyHost.Port;
        if (port == null)
        {
            return;
        }

        try
        {
            var host = feature.ProxyHost.Host;
            using var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
            await socket.ConnectAsync(host, port.Value, context.ConnectionClosed);
            Stream stream = new NetworkStream(socket, ownsSocket: false);

            // 如果有tls中间件，则反回来加密隧道
            if (context.Features.Get<ITlsConnectionFeature>() != null)
            {
                var sslStream = new SslStream(stream, leaveInnerStreamOpen: true);
                await sslStream.AuthenticateAsClientAsync(feature.ProxyHost.Host);
                stream = sslStream;
            }

            var task1 = stream.CopyToAsync(context.Transport.Output);
            var task2 = context.Transport.Input.CopyToAsync(stream);
            await Task.WhenAny(task1, task2);
        }
        catch (Exception ex)
        {
            this.logger.LogError(ex, $"连接到{feature.ProxyHost}异常");
        }
    }
}

7 请求响应分析http中间件#

这部分属于asp.netcore应用层内容，关键点是制作可多次读取的http请求body流和http响应body流，因为每个分析器实例都可以会重头读取一次请求内容和响应内容。

7.1 http分析器#

为了方便各种分析器的独立实现，我们定义http分析器的接口

Copy
/// <summary>
/// http分析器
/// 支持多个实例
/// </summary>
public interface IHttpAnalyzer
{
    /// <summary>
    /// 分析http
    /// </summary>
    /// <param name="context"></param> 
    /// <returns></returns>
    ValueTask AnalyzeAsync(HttpContext context);
}

这是输到日志的http分析器

Copy
public class LoggingHttpAnalyzer : IHttpAnalyzer
{
    private readonly ILogger<LoggingHttpAnalyzer> logger;

    public LoggingHttpAnalyzer(ILogger<LoggingHttpAnalyzer> logger)
    {
        this.logger = logger;
    }

    public async ValueTask AnalyzeAsync(HttpContext context)
    {
        var builder = new StringBuilder();
        var writer = new StringWriter(builder);

        writer.WriteLine("[REQUEST]");
        await context.SerializeRequestAsync(writer);

        writer.WriteLine("[RESPONSE]");
        await context.SerializeResponseAsync(writer);

        this.logger.LogInformation(builder.ToString());
    }
}

7.2 分析http中间件#

我们把请求body流和响应body流保存到临时文件，在所有分析器工作之后再删除。

Copy
/// <summary>
/// http分析中间件
/// </summary>
sealed class HttpAnalyzeMiddleware
{
    private readonly RequestDelegate next;
    private readonly IEnumerable<IHttpAnalyzer> analyzers;

    /// <summary>
    /// http分析中间件
    /// </summary>
    /// <param name="next"></param>
    /// <param name="analyzers"></param> 
    public HttpAnalyzeMiddleware(
        RequestDelegate next,
        IEnumerable<IHttpAnalyzer> analyzers)
    {
        this.next = next;
        this.analyzers = analyzers;
    }

    /// <summary>
    /// 分析代理的http流量
    /// </summary>
    /// <param name="context"></param>
    /// <returns></returns>
    public async Task InvokeAsync(HttpContext context)
    {
        var feature = context.Features.Get<IProxyFeature>();
        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
        {
            await next(context);
            return;
        }

        context.Request.EnableBuffering();
        var oldBody = context.Response.Body;
        using var response = new FileResponse();

        try
        {
            // 替换response的body
            context.Response.Body = response.Body;

            // 请求下个中间件
            await next(context);

            // 处理分析
            await this.AnalyzeAsync(context);
        }
        finally
        {
            response.Body.Position = 0L;
            await response.Body.CopyToAsync(oldBody);
            context.Response.Body = oldBody;
        }
    }

    private async ValueTask AnalyzeAsync(HttpContext context)
    {
        foreach (var item in this.analyzers)
        {
            context.Request.Body.Position = 0L;
            context.Response.Body.Position = 0L;
            await item.AnalyzeAsync(context);
        }
    }


    private class FileResponse : IDisposable
    {
        private readonly string filePath = Path.GetTempFileName();

        public Stream Body { get; }

        public FileResponse()
        {
            this.Body = new FileStream(filePath, FileMode.Open, FileAccess.ReadWrite);
        }

        public void Dispose()
        {
            this.Body.Dispose();
            File.Delete(filePath);
        }
    }
}

8 反向代理http中间件#

我们需要把请求转发到真实的目标服务器，这时我们的应用程序是一个http客户端角色，这个过程与nginx的反向代理是一致的。具体的实现上，我们直接使用yarp库来完成即可。

Copy
/// <summary>
/// http代理执行中间件
/// </summary>
sealed class HttpForwardMiddleware
{
    private readonly RequestDelegate next;
    private readonly IHttpForwarder httpForwarder;
    private readonly HttpMessageInvoker httpClient = new(CreateSocketsHttpHandler());

    /// <summary>
    /// http代理执行中间件
    /// </summary>
    /// <param name="next"></param>
    /// <param name="httpForwarder"></param>
    public HttpForwardMiddleware(
        RequestDelegate next,
        IHttpForwarder httpForwarder)
    {
        this.next = next;
        this.httpForwarder = httpForwarder;
    }

    /// <summary>
    /// 转发http流量
    /// </summary>
    /// <param name="context"></param>
    /// <returns></returns>
    public async Task InvokeAsync(HttpContext context)
    {
        var feature = context.Features.Get<IProxyFeature>();
        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
        {
            await next(context);
        }
        else
        {
            var scheme = context.Request.Scheme;
            var destinationPrefix = $"{scheme}://{feature.ProxyHost}";
            await httpForwarder.SendAsync(context, destinationPrefix, httpClient, ForwarderRequestConfig.Empty, HttpTransformer.Empty);
        }
    }

    private static SocketsHttpHandler CreateSocketsHttpHandler()
    {
        return new SocketsHttpHandler
        {
            Proxy = null,
            UseProxy = false,
            UseCookies = false,
            AllowAutoRedirect = false,
            AutomaticDecompression = DecompressionMethods.None,
        };
    }
}

9 编排中间件创建服务器和应用#

9.1 kestrel中间件编排#

这里要特别注意顺序，传输层套娃。

Copy
/// <summary>
///  ListenOptions扩展
/// </summary>
public static partial class ListenOptionsExtensions
{
    /// <summary>
    /// 使用Fiddler的kestrel中间件
    /// </summary>
    /// <param name="listen"></param>
    public static ListenOptions UseFiddler(this ListenOptions listen)
    {
        // 代理协议中间件
        listen.Use<KestrelProxyMiddleware>();

        // tls侦测中间件
        listen.UseTlsDetection(tls =>
        {
            var certService = listen.ApplicationServices.GetRequiredService<CertService>();
            certService.CreateCaCertIfNotExists();
            certService.InstallAndTrustCaCert();
            tls.ServerCertificateSelector = (context, domain) => certService.GetOrCreateServerCert(domain);
        });

        // 隧道代理处理中间件
        listen.Use<KestrelTunnelMiddleware>();
        return listen;
    }
}

9.2 http中间件的编排#

Copy
public static class ApplicationBuilderExtensions
{
    /// <summary>
    /// 使用Fiddler的http中间件
    /// </summary>
    /// <param name="app"></param>
    public static void UseFiddler(this IApplicationBuilder app)
    {
        app.UseMiddleware<HttpAnalyzeMiddleware>();
        app.UseMiddleware<HttpForwardMiddleware>();
    }
}

9.3 创建应用#

我们可以在传统的MVC里创建伪fiddler的首页、下载证书等http交互页面。

Copy
public static void Main(string[] args)
{
    var builder = WebApplication.CreateBuilder(args);

    builder.Services
        .AddFiddler()
        .AddControllers();

    builder.WebHost.ConfigureKestrel((context, kestrel) =>
    {
        var section = context.Configuration.GetSection("Kestrel");
        kestrel.Configure(section).Endpoint("Fiddler", endpoint => endpoint.ListenOptions.UseFiddler());
    });

    var app = builder.Build();
    app.UseRouting();
    app.UseFiddler();

    app.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");

    app.Run();
}

10 留给读者#

如果让您来开发个伪Fiddler，除了本文的方法，您会使用什么方式来开发呢？