理想论坛记录

以省为单位的图书馆传统业务的云上构想

 

　　图书管理系统各馆都有，一般高校基本上是汇文，图创，Ilas系统，原985高校图书馆使用国外的系统较多，这些系统管理的本馆图书，读者和借阅日志都是本校数据，数据不全且各馆数据互不相通，形成数据孤岛。比如国内书目500万左右，一般馆可能在100万种左右，各馆的OPAC查询到的就只有20%左右的书目数据，每年出版新书在10万以上，但多数图书馆采购的新书可能在2万种左右，也只占20%左右，如果要一键查询图书在各高校的收藏情况，可能采用超星的读秀是可行的，但由于超星作为一家商业公司在无政策规范和指导下，收集的各馆馆藏数据有一定阻力，导致时效性不强；

　　 如何由国资背景公司，业内领袖、商业公司和各图书馆成员合作，共建共享搭建一套以省为单位的云上系统，统一管理书目、馆藏、读者和借阅日志数据，实现在数字中国的背景下，实现图书馆界的行业大数据，推动数据驱动下的图书馆传统业务的新发展。以下是对具体数据治理的一些探讨：

一、数据提交　

　 1、书目数据　

　　各馆图书编目基本都接入了CALIS的联合编目系统，但书目基本是各高校图书馆已经购买图书的集合，而不是国内出版图书的合集。现在由新华书店，清华大学和杭州麦达提供的中国图书数据服务（https://cckb.lib.tsinghua.edu.cn/）作为一个先行者做了很好的示范工程。2023年5月1日引用CCBD网站数据显示：4,974,962 基础图书书目信息；2,267,083 采访MARC数据； 2,282,090 国图标准MARC数据；1,328,234 Calis标准MARC数据，提供全面，权威、多字段揭示和唯一编号的书目数据，特别是图书封面图像，目录，前言等是传统MARC数据没有揭示的，关于唯一编号CCBD数据可能需要改进和申请DOI解析工作。各馆基于这套数据关联本地书目记录后，书目数据治理就可以完成。基本流程为各馆实时提交本馆数据后和CCBD的书目数据比对，平台记录下CCBDID和本馆书目记录号实现关联，关于比对的准确性不必追求百分百准确，因为主要业务还是基于本地管理系统运行。

　　2、馆藏数据

　　收集此类数据现在做法可能是由各馆每年从管理系统中导出后交给公司处理；另外就是直接爬取，这种方式容易引起图书馆反感。现在可以采用FLINK CDC方式实时传递到云平台，多数图书管理系统 的后台数据库为ORACLE，并且以11G的版本居多。基本流程是通过FLINK CDC方式上传书目记录号和条码号，实现和CCBDID的关联。

　　3、读者数据

　　读者数据要实现唯一标识，长期服务，ORCID是一个途径，但存在一定的安全隐患，通过微信公众号的OPENID可以做到，虽然国际化程度不高。基本流程是通过各校的统一认证系统绑定读者证号，通过学校名和读者学号唯一限定。微信在读者手机的安装比例应该接近100%，并且可以身份认证有多种方式，最重要的是能够免费发送模板消息，比如发送借还消息，图书推荐信息。为了保护读者隐私，只收集读者性别，年级，学位，学科和学校。

　　4、借还数据

　　基本流程是通过FLINK CDC监控loan_work表实现实时借还数据提交，通过监控LOG_CIR提交历史借还数据（interlib为例）

二、具体应用

　　1、OPAC

           通过省中心OPAC查询CCBD中心库所有纸质图书和数字图书，通过学校名称限定导航到本馆书目，显示关联的本地书目记录号和馆藏条码，通过链接的方式跳转到各馆OPAC详细书目和馆藏地，在馆状态的页面。

　　2、采访和推荐

　　　个人认为应该将采购权更多的给读者，调动读者参与的积极性，请读者推荐采购方式，并将推荐结果插入到本地的管理系统中，由馆员基于本地系统负责审核并下订单，并通过微信公众号推送消息，因为图书采购的招标和支付流程限制了直接基于平台的订购。基于平台的直接采购下单和图书的直接邮寄到读者是下一步推动的方向。

　　3、数据统计

　　    生成图工委需要的各馆年度统计报表；

　　    以大学生读书调查表为模板，提供数据报表的直接生成；

　　　全省借阅数据的实时显示；

　　4、数据挖掘

  　　   和学工合作，分析读者心理状态；

　　　  和教务合作，推出专业书单；

  　　　和出版社合作，推送出版主题；

三、代码示例

　　以下代码由非专业程序员个人编写，东拼西凑，水平之低，见笑！

　　1、早期通过SIP2代理方式收集流通日志代码　　

          通过SIP2自助借还协议的代理实现实时借阅的数据上传和推送，代码分三部分，一是SIP2代理，二是读者和书目数据提取后并推送到百度IOT平台，三是订阅IOT接受数据并存放到本地数据库，基本想法是多数高校实现了自助借还，通过SIP2代理方式即可捕获各馆借阅实时数据，然后直接通过读取数据库提取读者和书目信息，既可以记录下那类读者借阅了哪本图书的日志，为什么要采用百度的IOT平台了，一是免费、强大又稳定，每年全省的流通量不会超过1千万，在免费额度内。二是为了公开透明，要收集流通日志数据，需要等到各图书馆的高度信任，上传到百度这样的大平台上，各图书馆都能看到并自主管理。

　　1.1  tcp端口转发并把借书和还书事件数据通过HTTP POST到处理程序

//代理主程序；端口配置文件，转发数据到SIP2服务器；TCP转发；UDP转发；baidu IOT

#nullable enable
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Linq;
using cn.edu.yangtzeu.mqtt;
using System.Text.Json;

namespace NetProxy
{
    internal static class Program
    {
        private static void Main(string[] args)
        {
             try
            {
                var configJson =JsonDocument.Parse(System.IO.File.ReadAllText("config.json"));
                
                Dictionary<string, ProxyConfig>? configs = System.Text.Json.JsonSerializer.Deserialize<Dictionary<string, ProxyConfig>>(configJson.RootElement.GetProperty("sipProxy"));
                if (configs == null)
                {
                    throw new Exception("configs is null");
                }

                var tasks = configs.SelectMany(c => ProxyFromConfig(c.Key, c.Value));
                Task.WhenAll(tasks).Wait();
            }
            catch (Exception ex)
            {
                Console.WriteLine($"An error occurred : {ex}");
            }
        }

        private static IEnumerable<Task> ProxyFromConfig(string proxyName, ProxyConfig proxyConfig)
        {
            var forwardPort = proxyConfig.forwardPort;
            var localPort = proxyConfig.localPort;
            var forwardIp = proxyConfig.forwardIp;
            var localIp = proxyConfig.localIp;
            var protocol = proxyConfig.protocol;
            try
            {
                if (forwardIp == null)
                {
                    throw new Exception("forwardIp is null");
                }
                if (!forwardPort.HasValue)
                {
                    throw new Exception("forwardPort is null");
                }
                if (!localPort.HasValue)
                {
                    throw new Exception("localPort is null");
                }
                if (protocol != "udp" && protocol != "tcp" && protocol != "any")
                {
                    throw new Exception($"protocol is not supported {protocol}");
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Failed to start {proxyName} : {ex.Message}");
                throw;
            }

            bool protocolHandled = false;
            if (protocol == "udp" || protocol == "any")
            {
                protocolHandled = true;
                Task task;
                try
                {
                    var proxy = new UdpProxy();
                    task = proxy.Start(forwardIp, forwardPort.Value, localPort.Value, localIp);
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"Failed to start {proxyName} : {ex.Message}");
                    throw;
                }

                yield return task;
            }

            if (protocol == "tcp" || protocol == "any")
            {
                protocolHandled = true;
                Task task;
                try
                {
                    var proxy = new TcpProxy();
                    
                    task = proxy.Start(forwardIp, forwardPort.Value, localPort.Value, localIp);
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"Failed to start {proxyName} : {ex.Message}");
                    throw;
                }

                yield return task;
            }

            if (!protocolHandled)
            {
                throw new InvalidOperationException($"protocol not supported {protocol}");
            }
        }
    }

    public class ProxyConfig
    {
        public string? protocol { get; set; }
        public ushort? localPort { get; set; }
        public string? localIp { get; set; }
        public string? forwardIp { get; set; }
        public ushort? forwardPort { get; set; }
    }

    internal interface IProxy
    {
        Task Start(string remoteServerHostNameOrAddress, ushort remoteServerPort, ushort localPort, string? localIp = null);
    }
}

{
  "sipProxy": {
    "http": {
      "localPort": 2023,
      "protocol": "tcp",
      "forwardIp": "10.203.1.237",
      "forwardPort": 2002
    }
  },
  "localWebApi": "http://127.0.0.1:2016/api/readerholding",
  "isCustomized": true
}

#nullable enable
using System;
using System.Buffers;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Mail;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

using cn.edu.yangtzeu.mqtt;

namespace NetProxy
{
    internal class TcpProxy : IProxy
    {
        //lagerror
        BdMqtt bdMqtt = new BdMqtt();
        //处理MQTT消息
        public void handleMqtt(string msg)
        {
            Console.WriteLine("delegate:" + msg);
            MqttMsg mqttMsg = new MqttMsg();
            mqttMsg.Source = "sip2";
            mqttMsg.Code = 0;
            mqttMsg.Msg = msg;
            //修改后通过WEBAPI上传，减少连接数和流量
            //把MSG传给webapi，借书和还书
            bdMqtt.UploadAsync(mqttMsg);
        }
        /// <summary>
        /// Milliseconds
        /// </summary>
        public int ConnectionTimeout { get; set; } = (4 * 60 * 1000);
       
        public async Task Start(string remoteServerHostNameOrAddress, ushort remoteServerPort, ushort localPort, string? localIp)
        {
            var connections = new ConcurrentBag<TcpConnection>();

            IPAddress localIpAddress = string.IsNullOrEmpty(localIp) ? IPAddress.IPv6Any : IPAddress.Parse(localIp);
            var localServer = new TcpListener(new IPEndPoint(localIpAddress, localPort));
            localServer.Server.SetSocketOption(SocketOptionLevel.IPv6, SocketOptionName.IPv6Only, false);
            localServer.Start();

            Console.WriteLine($"TCP proxy started [{localIpAddress}]:{localPort} -> [{remoteServerHostNameOrAddress}]:{remoteServerPort}");
            
            //开一个线程，10秒一次，清除超时的连接，默认值4分钟
            var _ = Task.Run(async () =>
            {
                while (true)
                {
                   
                    await Task.Delay(TimeSpan.FromSeconds(10)).ConfigureAwait(false);

                    var tempConnections = new List<TcpConnection>(connections.Count);
                    while (connections.TryTake(out var connection))
                    {
                        tempConnections.Add(connection);
                    }
                    
                    foreach (var tcpConnection in tempConnections)
                    {
                        if (tcpConnection.LastActivity + ConnectionTimeout < Environment.TickCount64)
                        {
                            tcpConnection.Stop();
                        }
                        else
                        {
                            connections.Add(tcpConnection);
                        }
                    }
                }
            });

            while (true)
            {
                try
                {
                    var ips = await Dns.GetHostAddressesAsync(remoteServerHostNameOrAddress).ConfigureAwait(false);

                    var tcpConnection = await TcpConnection.AcceptTcpClientAsync(localServer,
                            new IPEndPoint(ips[0], remoteServerPort))
                        .ConfigureAwait(false);
                    //代理处理关联
                    tcpConnection.mqttPush = handleMqtt;
                    tcpConnection.Run();
                    connections.Add(tcpConnection);
                }
                catch (Exception ex)
                {
                    Console.ForegroundColor = ConsoleColor.Red;
                    Console.WriteLine(ex);
                    Console.ResetColor();
                }
            }
        }
     
    }

    internal class TcpConnection
    {
        private readonly TcpClient _localServerConnection;
        private readonly EndPoint? _sourceEndpoint;
        private readonly IPEndPoint _remoteEndpoint;
        private readonly TcpClient _forwardClient;
        private readonly CancellationTokenSource _cancellationTokenSource = new CancellationTokenSource();
        private readonly EndPoint? _serverLocalEndpoint;
        private EndPoint? _forwardLocalEndpoint;
        private long _totalBytesForwarded;
        private long _totalBytesResponded;
        public long LastActivity { get; private set; } = Environment.TickCount64;
        //自定义
        //public BdMqtt bdMqtt = new BdMqtt();
        public delegate void sipCallback(string msg);
        public sipCallback? mqttPush;

        public static async Task<TcpConnection> AcceptTcpClientAsync(TcpListener tcpListener, IPEndPoint remoteEndpoint)
        {
            var localServerConnection = await tcpListener.AcceptTcpClientAsync().ConfigureAwait(false);
            localServerConnection.NoDelay = true;
            return new TcpConnection(localServerConnection, remoteEndpoint);
        }

        private TcpConnection(TcpClient localServerConnection, IPEndPoint remoteEndpoint)
        {
            _localServerConnection = localServerConnection;
            _remoteEndpoint = remoteEndpoint;

            _forwardClient = new TcpClient {NoDelay = true};

            _sourceEndpoint = _localServerConnection.Client.RemoteEndPoint;
            _serverLocalEndpoint = _localServerConnection.Client.LocalEndPoint;
        }

        public void Run()
        {
            RunInternal(_cancellationTokenSource.Token);
        }

        public void Stop()
        {
            try
            {
                _cancellationTokenSource.Cancel();
            }
            catch (Exception ex)
            {
                Console.WriteLine($"An exception occurred while closing TcpConnection : {ex}");
            }
        }

        private void RunInternal(CancellationToken cancellationToken)
        {
            Task.Run(async () =>
            {
                try
                {
                    using (_localServerConnection)
                    using (_forwardClient)
                    {
                        await _forwardClient.ConnectAsync(_remoteEndpoint.Address, _remoteEndpoint.Port, cancellationToken).ConfigureAwait(false);
                        _forwardLocalEndpoint = _forwardClient.Client.LocalEndPoint;

                        Console.WriteLine($"Established TCP {_sourceEndpoint} => {_serverLocalEndpoint} => {_forwardLocalEndpoint} => {_remoteEndpoint}");
                        
                        using (var serverStream = _forwardClient.GetStream())
                        using (var clientStream = _localServerConnection.GetStream())
                        //task取消时候关闭流
                        using (cancellationToken.Register(() =>
                        {
                            serverStream.Close();
                            clientStream.Close();
                        }, true))
                        //等待数据传输完毕
                        {
                            await Task.WhenAny(
                                CopyToAsync(clientStream, serverStream, 81920, Direction.Forward, cancellationToken),
                                CopyToAsync(serverStream, clientStream, 81920, Direction.Responding, cancellationToken)
                            ).ConfigureAwait(false);
                        }
                    }
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"An exception occurred during TCP stream : {ex}");
                }
                finally
                {
                    Console.WriteLine($"Closed TCP {_sourceEndpoint} => {_serverLocalEndpoint} => {_forwardLocalEndpoint} => {_remoteEndpoint}. {_totalBytesForwarded} bytes forwarded, {_totalBytesResponded} bytes responded.");
                }
            });
        }

        private async Task CopyToAsync(Stream source, Stream destination, int bufferSize = 81920, Direction direction = Direction.Unknown, CancellationToken cancellationToken = default)
        {
            byte[] buffer = ArrayPool<byte>.Shared.Rent(bufferSize);
            
            try
            {
                while (true)
                {
                    int bytesRead = await source.ReadAsync(new Memory<byte>(buffer), cancellationToken).ConfigureAwait(false);
                    if (bytesRead == 0) break;
                    LastActivity = Environment.TickCount64;
                    await destination.WriteAsync(new ReadOnlyMemory<byte>(buffer, 0, bytesRead), cancellationToken).ConfigureAwait(false);
                    //自定义 
                    /*1.线程内发送 2.代理发送
                    Console.WriteLine(direction.ToString() + "|" + Encoding.UTF8.GetString(buffer.Skip(0).Take(bytesRead).ToArray()));
                    MqttMsg mqttMsg = new MqttMsg();
                    mqttMsg.Source = "sip2";
                    mqttMsg.Code = 0;
                    mqttMsg.MsgId = Guid.NewGuid().ToString("N");
                    mqttMsg.Msg = "数据正常";*/
                    Console.WriteLine(direction.ToString() + "|" + Encoding.UTF8.GetString(buffer.Skip(0).Take(bytesRead).ToArray()));
                    string temp = Encoding.UTF8.GetString(buffer.Skip(0).Take(bytesRead).ToArray());
                    //string temp1 = Encoding.UTF8.GetString(new byte[bytesRead](buffer, 0, bytesRead));
                    if (temp.Substring(0, 2) == "12" || temp.Substring(0, 2) == "10")
                    {
                        //bdMqtt.publish(mqttMsg, "$iot/JournalBrowse/user/cxstar");
                       mqttPush(temp); 
                    }
                    
                    
                    //计算转发字节数
                    switch (direction)
                    {
                        case Direction.Forward:
                            Interlocked.Add(ref _totalBytesForwarded, bytesRead);
                            break;
                        case Direction.Responding:
                            Interlocked.Add(ref _totalBytesResponded, bytesRead);
                            break;
                    }
                }
            }
            finally
            {
                ArrayPool<byte>.Shared.Return(buffer);
            }
        }
    }

    internal enum Direction
    {
        Unknown = 0,
        Forward,
        Responding,
    }
}

#nullable enable
using System;
using System.Collections.Concurrent;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.Threading.Tasks;

namespace NetProxy
{
    internal class UdpProxy : IProxy
    {
        /// <summary>
        /// Milliseconds
        /// </summary>
        public int ConnectionTimeout { get; set; } = (4 * 60 * 1000);

        public async Task Start(string remoteServerHostNameOrAddress, ushort remoteServerPort, ushort localPort, string? localIp = null)
        {
            var connections = new ConcurrentDictionary<IPEndPoint, UdpConnection>();

            // TCP will lookup every time while this is only once.
            var ips = await Dns.GetHostAddressesAsync(remoteServerHostNameOrAddress).ConfigureAwait(false);
            var remoteServerEndPoint = new IPEndPoint(ips[0], remoteServerPort);

            var localServer = new UdpClient(AddressFamily.InterNetworkV6);
            localServer.Client.SetSocketOption(SocketOptionLevel.IPv6, SocketOptionName.IPv6Only, false);
            IPAddress localIpAddress = string.IsNullOrEmpty(localIp) ? IPAddress.IPv6Any : IPAddress.Parse(localIp);
            localServer.Client.Bind(new IPEndPoint(localIpAddress, localPort));

            Console.WriteLine($"UDP proxy started [{localIpAddress}]:{localPort} -> [{remoteServerHostNameOrAddress}]:{remoteServerPort}");

            var _ = Task.Run(async () =>
            {
                while (true)
                {
                    await Task.Delay(TimeSpan.FromSeconds(10)).ConfigureAwait(false);
                    foreach (var connection in connections.ToArray())
                    {
                        if (connection.Value.LastActivity + ConnectionTimeout < Environment.TickCount64)
                        {
                            connections.TryRemove(connection.Key, out UdpConnection? c);
                            connection.Value.Stop();
                        }
                    }
                }
            });

            while (true)
            {
                try
                {
                    var message = await localServer.ReceiveAsync().ConfigureAwait(false);
                    var sourceEndPoint = message.RemoteEndPoint;
                    var client = connections.GetOrAdd(sourceEndPoint,
                        ep =>
                        {
                            var udpConnection = new UdpConnection(localServer, sourceEndPoint, remoteServerEndPoint);
                            udpConnection.Run();
                            return udpConnection;
                        });
                    await client.SendToServerAsync(message.Buffer).ConfigureAwait(false);
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"an exception occurred on receiving a client datagram: {ex}");
                }
            }
        }
    }

    internal class UdpConnection
    {
        private readonly UdpClient _localServer;
        private readonly UdpClient _forwardClient;
        public long LastActivity { get; private set; } = Environment.TickCount64;
        private readonly IPEndPoint _sourceEndpoint;
        private readonly IPEndPoint _remoteEndpoint;
        private readonly EndPoint? _serverLocalEndpoint;
        private EndPoint? _forwardLocalEndpoint;
        private bool _isRunning;
        private long _totalBytesForwarded;
        private long _totalBytesResponded;
        private readonly TaskCompletionSource<bool> _forwardConnectionBindCompleted = new TaskCompletionSource<bool>();

        public UdpConnection(UdpClient localServer, IPEndPoint sourceEndpoint, IPEndPoint remoteEndpoint)
        {
            _localServer = localServer;
            _serverLocalEndpoint = _localServer.Client.LocalEndPoint;

            _isRunning = true;
            _remoteEndpoint = remoteEndpoint;
            _sourceEndpoint = sourceEndpoint;

            _forwardClient = new UdpClient(AddressFamily.InterNetworkV6);
            _forwardClient.Client.SetSocketOption(SocketOptionLevel.IPv6, SocketOptionName.IPv6Only, false);
        }

        public async Task SendToServerAsync(byte[] message)
        {
            LastActivity = Environment.TickCount64;

            await _forwardConnectionBindCompleted.Task.ConfigureAwait(false);
            var sent = await _forwardClient.SendAsync(message, message.Length, _remoteEndpoint).ConfigureAwait(false);
            Interlocked.Add(ref _totalBytesForwarded, sent);
        }

        public void Run()
        {
            Task.Run(async () =>
            {
                using (_forwardClient)
                {
                    _forwardClient.Client.Bind(new IPEndPoint(IPAddress.Any, 0));
                    _forwardLocalEndpoint = _forwardClient.Client.LocalEndPoint;
                    _forwardConnectionBindCompleted.SetResult(true);
                    Console.WriteLine($"Established UDP {_sourceEndpoint} => {_serverLocalEndpoint} => {_forwardLocalEndpoint} => {_remoteEndpoint}");

                    while (_isRunning)
                    {
                        try
                        {
                            var result = await _forwardClient.ReceiveAsync().ConfigureAwait(false);
                            LastActivity = Environment.TickCount64;
                            var sent = await _localServer.SendAsync(result.Buffer, result.Buffer.Length, _sourceEndpoint).ConfigureAwait(false);
                            Interlocked.Add(ref _totalBytesResponded, sent);
                        }
                        catch (Exception ex)
                        {
                            if (_isRunning)
                            {
                                Console.WriteLine($"An exception occurred while receiving a server datagram : {ex}");
                            }
                        }
                    }
                }
            });
        }

        public void Stop()
        {
            try
            {
                Console.WriteLine($"Closed UDP {_sourceEndpoint} => {_serverLocalEndpoint} => {_forwardLocalEndpoint} => {_remoteEndpoint}. {_totalBytesForwarded} bytes forwarded, {_totalBytesResponded} bytes responded.");
                _isRunning = false;
                _forwardClient.Close();
            }
            catch (Exception ex)
            {
                Console.WriteLine($"An exception occurred while closing UdpConnection : {ex}");
            }
        }
    }
}

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Newtonsoft.Json;

using MQTTnet;
using MQTTnet.Client;
using System.Net.Http;
using System.Text.Json;
using System.Net;

namespace cn.edu.yangtzeu.mqtt
{
    public  class BdMqtt
    {
        public string? urlStr;
        public bool isCustomized;
        
        public BdMqtt()
        {
            var configJson = JsonDocument.Parse(System.IO.File.ReadAllText("config.json"));
            urlStr= configJson.RootElement.GetProperty("localWebApi").GetString();
            isCustomized = configJson.RootElement.GetProperty("isCustomized").GetBoolean();
            if (isCustomized == false)
            {
                urlStr = urlStr + "?isCustomized=false";
            }
            else
            {
                urlStr = urlStr + "?isCustomized=true";
            }
        }

        public async void UploadAsync(MqttMsg msg)
        {
            try
            {
                //Console.WriteLine("接收到要POST到readerandholding接口的数据：{0},url:{1}",msg.Msg,urlStr);
                string result = string.Empty;
                using (HttpContent httpContent = new StringContent(System.Text.Json.JsonSerializer.Serialize(msg), System.Text.Encoding.UTF8, "application/json"))
                {
                    using (HttpClient httpClient = new HttpClient())
                    {
                        try
                        {
                            using (HttpResponseMessage httpRes = await httpClient.PostAsync(urlStr, httpContent))
                            {
                                //if (httpRes.IsSuccessStatusCode)
                                //{
                                 result = httpRes.Content.ReadAsStringAsync().Result;
                                    Console.WriteLine("UPLOAD SIP2消息到readerandholding接口的返回值：{0},statusCode:{1}",result,httpRes.StatusCode.ToString());
                                //}
                                httpRes.Dispose();
                            }

                        }
                        catch (Exception ex)
                        {
                            Console.WriteLine("bdmqtt:" + ex.Message);
                        }
                        finally
                        {
                            httpClient.Dispose();
                            httpContent.Dispose();
                        }
                    }
                }
                return ;
            }
            catch (Exception ex)
            {
                Console.WriteLine("bdmqtt:"+ex.Message);  
            }
        }
    }

    public class MqttMsg
    {
        public string? Source { set; get; }
        public string? MsgId { set; get; }
        public int? Code { set; get; }
        public string? Msg { set; get; }
        public dynamic? Data { set; get; }
    }
    public class RetMsg
    { 
        public int? ReturnCode { set; get; }
        public string? ReturnMsg { set; get; }
        public dynamic? data { set; get; }
    }

}

　　//运行截图

 

　　1.2 接收SIP2的借还数据后取得读者证号获取读者信息和图书条码号获取图书信息，其中读者数据来源于网信中心，图书数据来源于INERLIB的biblios和holding表

//读者和馆藏获取

using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json.Linq;
using Oracle.ManagedDataAccess.Client;
using readerAndholding.Model;
using readerAndholding.Mqtt;
using System.Collections;
using System.Data;
using System.Text;

namespace readerAndholding.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class ReaderHoldingController : ControllerBase
    {
        IConfiguration? _configuration;
        IHttpClientFactory _httpClientFactory;
        MqttHttp _mqttHttp;
        string Save;
        string University;
        public ReaderHoldingController(IConfiguration configuration, IHttpClientFactory httpClientFactory,MqttHttp mqttHttp)
        {
            _configuration = configuration;
            University = _configuration["MqttOption:UniversityName"].ToString();
            Save= _configuration["MqttOption:Save"].ToString();
            _httpClientFactory = httpClientFactory;
            _mqttHttp = mqttHttp;
        }

        /// <summary>
        /// 用来回溯借阅记录
        /// </summary>
        /// <param name="readerNo">读者证号</param>
        /// <param name="barcode">条码号</param>
        /// <param name="timestamp">借阅时间戳</param>
        /// <returns></returns>
        [HttpGet]
        public async Task<ActionResult> Get(string readerNo, string barcode,Int64 timestamp)
        {
            Msg msg = new Msg();
            //将教师信息屏蔽
            if (readerNo.Length < 7)
            {
                msg.code = 1;
                msg.message = "屏蔽教职工数据";
                msg.reflection = "ReaderHoldingController.get";
                Logger.Warn(msg.ToString());
                return Ok(msg);
            }
            string retLocal = "{'msg':'发送错误'}";

            Hashtable ht = new Hashtable();
            try
            {
                //获取读者信息
                Reader reader = new Reader().getReader(readerNo);
                //获取书目信息
                Holding holding = new Holding().GetHolding(barcode);
                //如果没有查到读者或者书目，直接返回
                if(reader.ReaderNo== null || holding.RecordNo==null) 
                { 
                    msg.code = 2;
                    msg.message = "读者或书目记录为空";
                    msg.reflection = "ReaderHoldingController.get";
                    Logger.Warn(msg.ToString());
                    return Ok(msg);
                }
                ht.Add("reader", reader);
                ht.Add("holding", holding);
                //先推送到本地
                retLocal = Newtonsoft.Json.JsonConvert.SerializeObject(ht);
                await MqttPublish.MqttPulishAsync(retLocal, "local/readerAndholding/" +University);
                //再推送到云端
                Tsdb tsdb=new Tsdb();
                tsdb.datapoints = new List<DatapointsItem>();
                DatapointsItem datapointsItem = new DatapointsItem();
                Tags tags= new Tags();
                tags.CalisId = holding.CalisId;
                tags.ISBN = holding.ISBN;
                tags.Title = holding.Title;
                tags.PublishDate = holding.PublishDate;
                tags.Publisher = holding.Publisher;
                tags.Price = holding.Price;
                tags.ClassNo = holding.ClassNo;
                tags.Author = holding.Author;
                tags.RecordNo = holding.RecordNo;
                //引用第三方，从MARC中获取CALISID号，但多数没有同步
                tags.CalisId = holding.CalisId;
  
                tags.Save =Save;
                tags.University = University;
                tags.Academy = reader.Academy;
                tags.Major = reader.Major;
                tags.Subject = reader.Subject;
                tags.Grade = reader.Grade;
                tags.Degree = reader.Degree;
                tags.Sex = reader.Sex;


                datapointsItem.field = "ReaderNo";
                datapointsItem.metric = "circulationT";
                datapointsItem.type = "String";
                datapointsItem.value = reader.ReaderNo;
                if(timestamp>1000)
                 {
                    datapointsItem.timestamp = timestamp;
                }
                datapointsItem.tags= tags;
                tsdb.datapoints.Add(datapointsItem);

                string retCloud = Newtonsoft.Json.JsonConvert.SerializeObject(tsdb);
                await _mqttHttp.HttpPublish(retCloud);
            }
            catch (Exception ex)
            {
                msg.code = 3;
                msg.message = ex.Message;
                msg.data = ex.StackTrace;
                msg.reflection = "ReaderHoldingController.get";
                Logger.Error(msg.ToString(), ex);
                return Ok(msg);

            }
            Logger.Info(msg.ToString());
            return Ok(retLocal);
        }

        /// <summary>
        /// 接收SIP2代理传过来的原始数,查询读者和书目信息，并入库和推送
        /// </summary>
        /// <param name="mqttMsg">SIP2消息</param>
        /// <param name="isCustomized">是否实现了书目和读者的信息接口</param>
        /// <returns></returns>
        [HttpPost]
        public async Task<ActionResult> Post([FromBody] MqttMsg mqttMsg,bool isCustomized)
        {
            Msg msg= new Msg();
            //如果获取读者和书目信息没有本地定制，则直接发送SIP2中的10，12消息头到本地
            if (!isCustomized)
            {
                try
                {
                    //推送到本地
                    await MqttPublish.MqttPulishAsync(Newtonsoft.Json.JsonConvert.SerializeObject(mqttMsg), "local/readerAndholding/" + _configuration["MqttOption:UniversityName"].ToString());
                }
                catch (Exception ex)
                {
                    msg.code = 4;
                    msg.message = ex.Message;
                    msg.source = "发送MQTTMsg到本地错误";
                    msg.data = ex.StackTrace;
                    msg.reflection = "ReaderHoldingController.post";
                    Logger.Error(msg.ToString(), ex);
                    return Ok(msg);
                    
                }
                return Ok(true);
            }
            //获取读者和书目信息没有本地定制,推送到本地和云端
            else
            {
                try
                {
                    var temp = mqttMsg.Msg.Split("|");
                    string readerNo = "";
                    string name;
                    string barcode = "";
                    string title;
                    string remark;
                    for (int i = 0; i < temp.Length; i++)
                    {
                        if (temp[i].Substring(0, 2) == "AA")
                        {
                            readerNo = temp[i].Substring(2, temp[i].Length - 2);
                        }
                        else if (temp[i].Substring(0, 2) == "AB")
                        {
                            barcode = temp[i].Substring(2, temp[i].Length - 2);
                        }
                        else if (temp[i].Substring(0, 2) == "AE")
                        {
                            name = temp[i].Substring(2, temp[i].Length - 2);
                        }
                        else if (temp[i].Substring(0, 2) == "AF")
                        {
                            remark = temp[i].Substring(2, temp[i].Length - 2);
                        }
                        else if (temp[i].Substring(0, 2) == "AJ")
                        {
                            title = temp[i].Substring(2, temp[i].Length - 2);
                        }
                    }

                    //将教师信息屏蔽
                    if (readerNo.Length < 7)
                    {
                        msg.code = 1;
                        msg.message = "屏蔽教职工数据";
                        msg.reflection = "ReaderHoldingController.get";
                        Logger.Warn(msg.ToString());
                        return Ok(msg);
                    }
                    string retLocal = "{'msg':'发送错误'}";
                    Hashtable ht = new Hashtable();
                    try
                    {
                        //获取读者信息
                        Reader reader = new Reader().getReader(readerNo);
                        //获取书目信息
                        Holding holding = new Holding().GetHolding(barcode);

                        ht.Add("reader", reader);
                        ht.Add("holding", holding);
                        //如果没有查到读者或者书目，直接返回
                        if (reader.ReaderNo == null || holding.RecordNo == null)
                        {
                            msg.code = 2;
                            msg.message = "读者或书目记录为空";
                            msg.reflection = "ReaderHoldingController.get";
                            Logger.Warn(msg.ToString());
                            return Ok(msg);
                        }
                        mqttMsg.Data = ht;
                        //先推送到本地
                        retLocal = Newtonsoft.Json.JsonConvert.SerializeObject(mqttMsg);
                        await MqttPublish.MqttPulishAsync(retLocal, "local/readerAndholding/" + University);
                        //再推送到云端
                        Tsdb tsdb = new Tsdb();
                        tsdb.datapoints = new List<DatapointsItem>();
                        DatapointsItem datapointsItem = new DatapointsItem();
                        Tags tags = new Tags();
                        tags.CalisId = holding.CalisId;
                        tags.ISBN = holding.ISBN;
                        tags.Title = holding.Title;
                        tags.PublishDate = holding.PublishDate;
                        tags.Publisher = holding.Publisher;
                        tags.Price = holding.Price;
                        tags.ClassNo = holding.ClassNo;
                        tags.Author = holding.Author;
                        tags.RecordNo = holding.RecordNo;
                        //引用第三方，从MARC中获取CALISID号，但多数没有同步
                        tags.CalisId = holding.CalisId;

                        tags.Save = Save;
                        tags.University = University;
                        tags.Academy = reader.Academy;
                        tags.Major = reader.Major;
                        tags.Subject = reader.Subject;
                        tags.Grade = reader.Grade;
                        tags.Degree = reader.Degree;
                        tags.Sex = reader.Sex;


                        datapointsItem.field = "ReaderNo";
                        datapointsItem.metric = "circulationT";
                        datapointsItem.type = "String";
                        datapointsItem.value = reader.ReaderNo;
                        datapointsItem.tags = tags;
                        TimeSpan ts = DateTime.UtcNow - new DateTime(1970, 1, 1, 0, 0, 0, 0);
                        datapointsItem.timestamp = Convert.ToInt64(ts.TotalMilliseconds);
                        tsdb.datapoints.Add(datapointsItem);
                        retLocal = Newtonsoft.Json.JsonConvert.SerializeObject(tsdb);
                        await _mqttHttp.HttpPublish(retLocal);
                    }
                    catch (Exception ex)
                    {
                       Logger.Error(msg.ToString(),ex);
                        return NoContent();
                    }
                }
                catch (Exception ex)
                {
                    msg.code = 3;
                    msg.message = ex.Message;
                    msg.data = ex.StackTrace;
                    msg.reflection = "ReaderHoldingController.post";
                    Logger.Error(msg.ToString(), ex);
                    return Ok(msg);
                }
                Logger.Info(msg.ToString());    
                return Ok(msg);
            }
        }

        /// <summary>
        /// 按年度回溯借阅日志（只适合INTERLIB）
        /// </summary>
        /// <param name="year">年</param>
        /// <returns></returns>
        [HttpGet]
        [Route("updateYear")]
        public async Task<ActionResult> updateYear(string year)
        {
            Msg msg= new Msg();
            int i = 0;
            int j = 0;
            using (OracleConnection conn = new OracleConnection())
            {
                string oracleConnstr = "Data Source = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.203.1.237)(PORT = 1521)))(CONNECT_DATA = (SERVICE_NAME = interlib))); User Id = inter; Password = win ; ";
                conn.ConnectionString = oracleConnstr;
                conn.Open();
                OracleCommand comm = new OracleCommand();
                comm.Connection = conn;
                string commstr ;
                commstr = "select data2,data3,to_char(regtime,'yyyy-mm-dd hh24:mi:ss') from log_cir where to_char(regtime, 'yyyy') = :year and logtype = '30001'";
                comm.CommandText = commstr;
                //取出long类型中的字符串
                comm.Parameters.Clear();
                comm.Parameters.Add("@year", year);
                OracleDataReader dataReader= comm.ExecuteReader();
                HttpClient hc= new HttpClient();
                string readerNo="";
                string barcode="";
                Int64 timestamp = 0;

                while(dataReader.Read())
                {
                    try
                    {
                        i++;
                        readerNo = dataReader.GetString(0);
                        Console.WriteLine(i.ToString()+":"+readerNo);
                        barcode = dataReader.GetString(1);
                        timestamp = (Convert.ToDateTime(dataReader.GetString(2)).ToUniversalTime().Ticks - 621355968000000000) / 10000;
                        string url = string.Format("http://127.0.0.1:2016/api/readerholding?readerNo={0}&barcode={1}&timestamp={2}", readerNo, barcode, timestamp);
                        var httpResponseMessage = await hc.GetAsync(url);
                        if (httpResponseMessage.IsSuccessStatusCode)
                        {
                            var tempStr = await httpResponseMessage.Content.ReadAsStringAsync();
                            j++;
                        }
                    }
                    catch(Exception ex)
                    {
                        msg.message = ex.StackTrace;
                        msg.source = "在循环处理年度日志中发生错误";
                        msg.reflection = "ReaderHoldingController:updateyear";
                        msg.code = 1;
                       Logger.Error(msg.ToString(), ex);
                    }
                }
            }
            msg.message = string.Format("i:{0};j:{1}",i,j);
            Logger.Info(msg.ToString());
            return Ok(msg);
        }
    }
}

//主程序,实现的MQTT协议和WEBSOCKER的数据转发，可以提供WEB端的大屏幕实时数据展示，其中推送到BAIDU IOT 平台采用了HTTP的方式，为了节约MQTT的连接数

using MQTTnet.AspNetCore;
using MQTTnet.Server;
using readerAndholding.Mqtt;

var builder = WebApplication.CreateBuilder(args);

var host = Host.CreateDefaultBuilder(Array.Empty<string>())
            .ConfigureWebHostDefaults(
                webBuilder =>
                {
                    webBuilder.UseKestrel(
                        o =>
                        {
                            // This will allow MQTT connections based on TCP port 1883.
                            o.ListenAnyIP(2012, v => v.UseMqtt());

                            // This will allow MQTT connections based on HTTP WebSockets with URI "localhost:5000/mqtt"
                            // See code below for URI configuration.
                            o.ListenAnyIP(2014); // Default HTTP pipeline
                            
                        });

                    webBuilder.UseStartup<Startup>();
                });
host.RunConsoleAsync();

// Add services to the container.

builder.Services.AddControllers();
builder.Services.AddHttpClient();
builder.Services.AddSingleton<MqttHttp>();

var app = builder.Build();

// Configure the HTTP request pipeline.
app.UseRouting();

app.UseAuthorization();

app.UseStaticFiles();

app.MapControllers();

app.Run("http://*:2016");

//运行截图

 

 　　1.3 订阅百度IOT数据，通过Z3950协议查询ISBN，记录下CALISID，将书目和读者数据入库并提供MQTT和websocket数据推送

  //主程序

using Microsoft.EntityFrameworkCore;
using Mqtt.Models.Z39;
using MQTTnet.AspNetCore;
using z39web.Models.DB;
using z39web.Models.Mqtt;
using z39web.Models.Z39;

var builder = WebApplication.CreateBuilder(args);
//添加MQTT服务
var host = Host.CreateDefaultBuilder(Array.Empty<string>())
        .ConfigureWebHostDefaults(
            webBuilder => {
                webBuilder.UseKestrel(
                    o => {
                        o.ListenAnyIP(2020, v => v.UseMqtt());
                        o.ListenAnyIP(2022);
                    
                    });
                webBuilder.UseStartup<Startup>();
            });
host.RunConsoleAsync(); 


// Add services to the container.

/*MYSQL,直接用原始SQL语句插入JSON

var connStr =builder.Configuration["ConnectionStrings:MysqlConnection"];
builder.Services.AddDbContext<MyDBContext>(options =>
    options.UseMySQL(connStr)
);
*/

//isbn,calisId查询
builder.Services.AddSingleton<IZ39,Z39>();
builder.Services.AddSingleton<IMqttReceive,MqttReceive>();
builder.Services.AddControllers();
builder.Services.AddHttpClient();

var app = builder.Build();

// Configure the HTTP request pipeline.
app.UseRouting();
app.UseAuthorization();
app.UseStaticFiles();
app.MapControllers();

app.Run("http://*:2018");

//CALIS查询比对和入库,用到的是群友dp2的开源代码，其中封装好的z39的检索和解析库

using System.Text;
using System.Reflection.Metadata;
using System;

using MySql.Data;
using MySql.EntityFrameworkCore;
using MySql;

using DigitalPlatform.Z3950;
using static DigitalPlatform.Z3950.ZClient;
using DigitalPlatform.Text;
using DigitalPlatform.Net;
using System.Web;
using DigitalPlatform.Marc;
using System.Xml;
using z39web.Models.DB;
using Microsoft.EntityFrameworkCore;
using MySql.Data.MySqlClient;
using ConfigurationManager = Microsoft.Extensions.Configuration.ConfigurationManager;

namespace z39web.Models.Z39
{
    public class Z39 : IZ39
    {
        public ZClient _zclient;
        public TargetInfo _targetInfo;

        public Z39()
        {
            //初始化变量
            ConfigurationManager configurationManager = new ConfigurationManager();
            configurationManager.AddJsonFile("appsettings.json", true, reloadOnChange: false);
            string MysqlConnection = configurationManager["ConnectionStrings:MysqlConnection"].ToString();

            _targetInfo = new TargetInfo();
            _targetInfo.HostName = configurationManager["Calis:Host"].ToString();
            _targetInfo.Password = configurationManager["Calis:Password"].ToString();
            _targetInfo.UserName = configurationManager["Calis:UserName"].ToString();
            _targetInfo.Port = Convert.ToInt32(configurationManager["Calis:Port"]);
            _targetInfo.DbNames = new string[] { configurationManager["Calis:DbNames"].ToString() };
            _targetInfo.AuthenticationMethod = Convert.ToInt32(configurationManager["Calis:AuthenticationMethod"]);
            Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
            _targetInfo.DefaultQueryTermEncoding = Encoding.GetEncoding(936);
            _targetInfo.DefaultRecordsEncoding = Encoding.GetEncoding(936);
            _targetInfo.PresentPerBatchCount = 1;

            //全局连接
            _zclient = new ZClient();
            
            InitialResult result = _zclient.TryInitialize(_targetInfo).GetAwaiter().GetResult();
            Console.WriteLine("calis z39初始化结果："+result.ToString());
            if (result.Value == -1)
            {
                Console.WriteLine("全局联机错误:{0}", result.ErrorInfo);
            }
        }

        //接口实现，指定查询类型isbn和calisId,isbn不加-，calisId不加CAL_
        public Msg Search(string typ, string str)
        {
            Msg msg = new Msg();

            msg = LocalSearch(typ, str);
            if (msg.Code == 0)
            {
                return msg;
            }
            else
            {
                msg = CalisSearch(typ, str);
                msg.Source = "calis";
            }
            return msg;
        }

        //本地检索,若没有对应的calisId,
        private Msg LocalSearch(string typ, string str)
        {
            Msg msg = new Msg();
            string commstr = "";
            Biblios biblios = new Biblios();
            if (string.IsNullOrWhiteSpace(typ) && string.IsNullOrWhiteSpace(str))
            {
                msg.Code = 1;
                msg.Message = "查询类型和变量不符合要求";
                return msg;
            }
            if (typ == "calisId")
            {
                commstr = "select * from biblios where calisId=@calisId";
            }
            else if (typ == "isbn")
            {
                commstr = "select * from biblios where isbn=@isbn";
            }
            else
            {
                msg.Code = 1;
                msg.Message = "查询类型和变量不符合要求";
                return msg;
            }
            //从本地数据库中查找MARC记录
            try
            {
                using (MySqlConnection conn = new MySqlConnection())
                {
                    string connstr = "server=10.40.6.13;user id=root;Password=this;sslmode=None;database=calisbiblios";
                    conn.ConnectionString = connstr;
                    conn.Open();
                    MySqlCommand comm = new MySqlCommand();
                    comm.Connection = conn;
                    comm.CommandText = commstr;
                    comm.Parameters.Clear();
                    if (typ == "calisId")
                    {
                        comm.Parameters.Add("@calisId", MySqlDbType.VarString).Value = str;
                    }
                    else
                    {
                        comm.Parameters.Add("@isbn", MySqlDbType.VarString).Value = str.Replace("-", "");
                    }
                    MySqlDataReader dr = comm.ExecuteReader();
                    if (dr.Read())
                    {
                        biblios.CalisId = dr["calisId"].ToString();
                        biblios.Isbn = dr["isbn"].ToString();
                        biblios.Title = dr["title"].ToString();
                        biblios.Author = dr["author"].ToString();
                        biblios.Publisher = dr["publisher"].ToString();
                        biblios.PublishDate = dr["publishDate"].ToString();
                        biblios.ClassNo = dr["classNo"].ToString();
                        biblios.Price = dr["price"].ToString();
                        biblios.Marc = Convert.ToBase64String(Encoding.UTF8.GetBytes(dr["marc"].ToString()));
                        msg.Data = biblios;
                    }
                    else
                    {
                        msg.Code = 2;
                        msg.Message = "本地没有找到记录";
                    }
                    dr.Close();
                    conn.Close();
                }
            }
            catch (Exception ex)
            {
                msg.Code = 3;
                msg.Message = ex.Message;
            }
            return msg;
        }

        //calis检索，则查询CALIS并插入本地库
        private Msg CalisSearch(string typ, string str)
        {
            Msg msg = new Msg();
            /*
            //初始化连接
            InitialResult result = _zclient.TryInitialize(_targetInfo).GetAwaiter().GetResult();
            //连接错误返回
            if (result.Value == -1)
            {
                Console.WriteLine("初始化连接发送错误:{0}", result.ErrorInfo);
                msg.Code = 1;
                msg.Message = result.ErrorInfo;
                return msg;
            }*/
            //参数检查
            string strQueryString = "";
            if (typ == "isbn")
            {
                strQueryString = "\"" + str + "\"/1=7";
            }
            else if (typ == "calisId")
            {
                strQueryString = "\"" + str + "\"/1=12";
            }
            else
            {
                msg.Code = 1;
                msg.Message = "参数不合要求";
                return msg;
            }
            //查询
            SearchResult search_result = _zclient.Search(
                    strQueryString,
                    _targetInfo.DefaultQueryTermEncoding,
                    _targetInfo.DbNames,
                    _targetInfo.PreferredRecordSyntax,
                    "default").Result;
            Console.WriteLine("calis z39查询结果："+search_result.ToString());
            if (search_result.ResultCount < 1)
            {
                msg.Message = "没有检索到数据";
                return msg;
            }
            //检索错误，返回
            if (search_result.Value == -1 || search_result.Value == 0)
            {
                Console.WriteLine("检索发送错误:{0}", search_result.ErrorInfo);
                msg.Code = 2;
                msg.Message = search_result.ErrorInfo;
                return msg;
            }
            else
            {
                Console.WriteLine("检索共命中记录:{0} ", search_result.ResultCount);
            }
            //取回数据
            long _resultCount = search_result.ResultCount;
            int _fetched = 0;
            PresentResult present_result = _zclient.Present(
                        "default",
                        _fetched,
                        Math.Min((int)_resultCount - _fetched, 1),
                        1,
                        "F",
                        _targetInfo.PreferredRecordSyntax).Result;

            //取回数据错误，返回
            if (present_result.Value == -1)
            {
                Console.WriteLine("present result:{0}", present_result.ToString());
                msg.Code = 3;
                msg.Message = present_result.ErrorInfo;
                return msg;
            }
            else
            {
                //如果查到MARC数据，并且在本地数据库中没有，则插入到本地
                if (present_result.Records.Count == 1)
                {
                    Biblios biblios = new Biblios();
                    string? calisIdValue = "";
                    string? isbnValue = "";
                    string? priceValue = "";
                    string? titleValue = "";
                    string? authorValue = "";
                    string? publisherValue = "";
                    string? publishDateValue = "";
                    string? classNoValue = "";
                    string strMarc="", strXml, strError;
                    try
                    {
                        XmlDocument xmlDocument;
                        MarcLoader.ConvertIso2709ToMarcString(present_result.Records[0].m_baRecord, Encoding.GetEncoding(936), false, out strMarc, out strError);
                        MarcUtil.Marc2Xml(strMarc, "unimarc", out xmlDocument, out strError);
                        var nsmgr = new XmlNamespaceManager(xmlDocument.NameTable);
                        nsmgr.AddNamespace("unimarc", "http://dp2003.com/UNIMARC");
                        var json = Newtonsoft.Json.JsonConvert.SerializeXmlNode(xmlDocument);
                        //取出索引字段
                        var calisIdxml = xmlDocument.SelectSingleNode("//unimarc:controlfield[@tag='001']", nsmgr);
                        calisIdValue=calisIdxml==null?"":calisIdxml.InnerText;
                        var isbnxml = xmlDocument.SelectSingleNode("//unimarc:datafield[@tag='010']/unimarc:subfield[@code='a']", nsmgr);
                        isbnValue=isbnxml==null?"":isbnxml.InnerText;
                        var pricexml = xmlDocument.SelectSingleNode("//unimarc:datafield[@tag='010']/unimarc:subfield[@code='d']", nsmgr);
                        priceValue= pricexml == null ? "" : pricexml.InnerText;
                        var titlexml = xmlDocument.SelectSingleNode("//unimarc:datafield[@tag='200']/unimarc:subfield[@code='a']", nsmgr);
                        titleValue = titlexml == null ? "" : titlexml.InnerText;
                        var authorxml = xmlDocument.SelectSingleNode("//unimarc:datafield[@tag='200']/unimarc:subfield[@code='f']", nsmgr);
                        authorValue = authorxml == null ? "" : authorxml.InnerText;
                        var publisherxml = xmlDocument.SelectSingleNode("//unimarc:datafield[@tag='210']/unimarc:subfield[@code='c']", nsmgr);
                        publisherValue = publisherxml == null ? "" : publisherxml.InnerText;
                        var publishDatexml = xmlDocument.SelectSingleNode("//unimarc:datafield[@tag='210']/unimarc:subfield[@code='d']", nsmgr);
                        publishDateValue = publishDatexml == null ? "" : publishDatexml.InnerText;
                        //因为下载限制，使用采用简编数据填充并标识
                        var classNoxml = xmlDocument.SelectSingleNode("//unimarc:datafield[@tag='690']/unimarc:subfield[@code='a']", nsmgr);
                        classNoValue = classNoxml == null ? "simple" : classNoxml.InnerText;
                        //返回biblios
                        biblios.Author = authorValue;
                        biblios.Publisher = publisherValue;
                        biblios.PublishDate = publishDateValue;
                        biblios.Price = priceValue;
                        biblios.CalisId = calisIdValue.Substring(4);
                        biblios.Title = titleValue;
                        biblios.Isbn = isbnValue;
                        biblios.Marc = Convert.ToBase64String(Encoding.UTF8.GetBytes(strMarc));
                        msg.Data = biblios;
                        //简编没有分类号
                        biblios.ClassNo = classNoValue;
                    }
                    catch (Exception ex)
                    {
                        msg.Code = 100;
                        msg.Message = ex.Message;
                        msg.Source = "解析MARC数据错误";
                        msg.ToString();
                    }
                    //数据插入
                    using (MySqlConnection conn = new MySqlConnection())
                    {
                        string connstr = "server=10.40.6.13;user id=root;Password=this;sslmode=None;database=calisbiblios";
                        conn.ConnectionString = connstr;
                        conn.Open();
                        MySqlCommand comm = new MySqlCommand();
                        comm.Connection = conn;
                        string commstr = "insert into biblios  (calisId,isbn,title,author,publisher,publishDate,price,classNo,marc) values (@calisId,@isbn,@title,@author,@publisher,@publishDate,@price,@classNo,@marc)";
                        comm.CommandText = commstr;
                        comm.Parameters.Clear();
                        comm.Parameters.Add("@calisId", MySqlDbType.VarString).Value = calisIdValue.Substring(4);
                        comm.Parameters.Add("@isbn", MySqlDbType.VarString).Value = isbnValue.Replace("-", "");
                        comm.Parameters.Add("@title", MySqlDbType.VarString).Value = titleValue;
                        comm.Parameters.Add("@author", MySqlDbType.VarString).Value = authorValue;
                        comm.Parameters.Add("@publisher", MySqlDbType.VarString).Value = publisherValue;
                        comm.Parameters.Add("@publishDate", MySqlDbType.VarString).Value = publishDateValue;
                        comm.Parameters.Add("@price", MySqlDbType.VarString).Value = priceValue;
                        comm.Parameters.Add("@classNo", MySqlDbType.VarString).Value = classNoValue;
                        comm.Parameters.Add("@marc", MySqlDbType.VarString).Value = strMarc;
                        try
                        {
                            int i = comm.ExecuteNonQuery();
                        }
                        catch (Exception ex)
                        {
                            Console.WriteLine(ex.Message);
                            msg.Code = 4;
                            msg.Message = ex.Message;
                        }
                        finally
                        {
                            conn.Close();
                        }
                    }
                }
                else
                {
                    msg.Code = 5;
                    msg.Message = "calis书目数据中未能查到记录";
                }
            }
            return msg;
        }
    }
}

//运行截图

 

 

　　2、后期的基于CCBD数据和FLINK CDC方式采集数据

　　 在早期想通过CALIS的书目数据来统一规范书目数据，但在实施过程中，由于超过了下载额度而暂停。另外在读取读者数据和书目数据时需要直接读取数据库，也有图书馆意愿和安全风险考虑，前期只做了一些试验后就停止了，虽然还在一直运行着。

　　接触到CCBD的数据后，并在业内大佬指点下，试验了一下FLINK CDC的方式采集图书管理系统中的数据，想在各馆各自独立管理数据互不联通和全面采用下一代图书管理平台的中间时段做一些数据整合的尝试，先把纸质图书管理好、利用好是图书馆的基本工作。

　　2.1 FLINK CDC同步图书管理系统中ORACLE数据，本地端通过BINLOG的方式捕获到数据变化后上传到云平台

            1、ORACLE数据库设置，打开归档模式，并启用监听

　　　　图创的归档指导性文件（word）

　　　　

目的
为了进一步加强数据库的安全性 ,建议需要开启数据库的归档模式。
这样做的好处有：
1.数据库文件出现因为磁盘坏块出现坏块，能快速应用归档日志进行恢复。
2.数据库服务器掉电产生数据库文献错误需要进行介质恢复的之后，可以直接应用归档恢复。
3.归档日志开启建议有条件的所有用户开启，oracle默认没有开启。
注意事项：
1.Imp批导入大量数据，大数据量批操作的时候建议关闭归档模式
2.建议磁盘空间够用的情况下，将归档日志目录设置到最大值。比如磁盘剩余200G那么归档日志存放目录大小就设置为200G
**（重要说明：开启归档日志是需要图书馆技术人员有维护能力，因为归档日志如果没有做好任务计划导致满了，那么数据库将自动宕机。
以下命令涉及数据库的重启，所以如果需要操作建议在非业务时间做好数据库全备份的前提下进行）**
开启方式:
步骤一: 闭馆后在数据库服务器上备份数据库，并把数据库备份文件拷贝到其他机器保存。
步骤二：进入命令窗口 输入sqlplus “/as sysdba” 进入超级用户sql环境
步骤三：运行 archive log list 命令得出下列界面，如果是 非存档模式就需要做调整
步骤四：shutdown immediate 关闭数据库，然后 startup mount 打开数据库到加载模式
步骤五：alter database archivelog;改变归档模式为开启归档
步骤六：使用alter system set db_recovery_file_dest_size=200G scope=both;
更改归档目录大小；
alter database open;启动打开数据库
步骤七：设置自动任务计划每天任务计划删除7天前的过久的归档日志，释放磁盘空间。
脚本文件内容如下，附件中也附带脚本：
delete_arch.sh 文件内容：
#!bin/sh
source /home/oracle/.bash_profile
rman target / cmdfile=/opt/app/oracle/scrip_delete_arch/del_arch.sh
del_arch.sh 文件内容：
delete archivelog all completed before 'sysdate-7';
windows环境下对于的脚本：
在根盘符创建一个delete_arch的目录，比如D盘。
Delete_arch目录里面包含两个文件：
第一个文件：delte_arch.bat文件，文件内容是：
rman target / cmdfile=D:\ delete_arch\del_arch.sql
第二个文件：del_arch.sql
文件内容是：
delete archivelog all completed before 'sysdate-7';
将delte_arch.bat加入任务计划，每天晚上执行。

 

    　　 https://debezium.io/documentation/reference/2.2/connectors/oracle.html

　　　2、FLINK CDC数据捕获上传到云平台,运行在各图书馆，数据为测试环境，通过DOCKER部署ORALCE 19C，然后将真实环境的备份数据还原到测试环境中。

package com.example.sendmessage;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.alibaba.fastjson.parser.Feature;
import com.alibaba.fastjson.parser.ParserConfig;
import com.alibaba.fastjson.serializer.SerializerFeature;
import com.google.gson.Gson;
import com.ververica.cdc.connectors.oracle.OracleSource;
import com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.source.SourceRecord;
import com.ververica.cdc.debezium.JsonDebeziumDeserializationSchema;

import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.io.InputStream;
import java.util.Properties;
import java.util.Scanner;

public class TEST {
    public static void main(String[] args) throws Exception {
        Properties pro=new Properties();
        pro.put("database.pdb.name","inte");
        pro.put("log.mining.strategy","online_catalog");
        //pro.put("log.mining.continuous.mine","true");
        SourceFunction<String> sourceFunction = OracleSource.<String>builder()
                .hostname("10.40.6.131")
                .url("jdbc:oracle:thin:@10.40.6.13:1521/inter")
                .port(1521)
                .database("inter") // monitor XE database
                .schemaList("inter") // monitor inventory schema
                //.startupOptions(StartupOptions.latest())
                .tableList("inter.loan_work") // monitor products table
                .debeziumProperties(pro)
                .username("c##dbzuser")
                .password("this")
                .deserializer(new JsonDebeziumDeserializationSchema()) // converts SourceRecord to JSON String
                .build();
        // Configuration configuration=new Configuration();
        //configuration.setString("execution.savepoint.path","file:///D:/checkoracle/cd720959c394888c1bc277e310b35b2d/chk-23");

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //env.enableCheckpointing(30000);
        //env.getCheckpointConfig().setCheckpointStorage("file:///d:/checkoracle");
        // 获取 CDC 数据流
         String URL = "http://reader.yzu.edu.cn/syncdb/api/loan_work";
//        DataStream<String> cdcStream = env.addSource(sourceFunction);
        DataStream<String> mappedStream = processCdcStream(env, sourceFunction);

        mappedStream.addSink(new HttpSink(URL));
//        mappedStream.print();
//        mappedStream.print();



        env.execute();
        String str = System.console().readLine();
    }
   //数据map 处理部分
   public static DataStream<String> processCdcStream(StreamExecutionEnvironment env, SourceFunction<String> source) {
       DataStream<String> cdcStream = env.addSource(source);
       DataStream<String> mappedStream = cdcStream.map(new MapFunction<String, String>() {
           @Override
           public String map(String value) throws Exception {
               // 在这里对输入数据进行处理
                //JOSN数组
               JSONArray jsonArray = new JSONArray();
                //全部的json数据
               JSONObject jsonObject =  JSON.parseObject(value);
               JSONObject allJson = new JSONObject();
               allJson.put("id",2);
               allJson.put("source",jsonObject.toString());
               allJson.put("university","Yangtze University");
               allJson.put("token","4-8-2021");
               jsonArray.add(allJson);


                value=JSON.toJSONString(jsonArray);
               // 返回处理后的输出数据
               return value ;
           }
       });
       return mappedStream;
   }

    public static class HttpSink implements SinkFunction<String> {
        private String URL;

        public HttpSink(String URL) {
            this.URL = URL;
        }

        @Override
        public void invoke(String value, Context context) throws Exception {
            CloseableHttpClient httpclient = HttpClients.createDefault();


            HttpPost httpPost = new HttpPost(URL);

            httpPost.setHeader("Content-Type", "application/json");
            JSONArray jsonArray = JSONArray.parseArray(value);
            String json = jsonArray.toString();
            HttpEntity requestEntity = new StringEntity(json, ContentType.APPLICATION_JSON);
            String requestStr = EntityUtils.toString(requestEntity);
            System.out.println(requestStr);
            httpPost.setEntity(requestEntity);

//

            CloseableHttpResponse response = httpclient.execute(httpPost);
            try {
                HttpEntity responseEntity = response.getEntity();
                if (responseEntity != null) {
                    System.out.println("Response body: " + EntityUtils.toString(responseEntity));
                }
            } finally {
                response.close();
            }
        }

    }
}

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.1.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>sendMessage</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>sendMessage</name>
    <description>sendMessage</description>
    <properties>
        <java.version>8</java.version>
        <flink-version>1.13.0</flink-version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <version>3.0.4</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <version>3.0.4</version>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <version>3.0.4</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink-version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.12</artifactId>
            <version>${flink-version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>${flink-version}</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.32</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.ververica/flink-sql-connector-oracle-cdc -->
        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-sql-connector-oracle-cdc</artifactId>
            <version>2.3.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_2.12</artifactId>
            <version>${flink-version}</version>
        </dependency>
        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-connector-mysql-cdc</artifactId>
            <version>2.2.1</version>
        </dependency>
        <!--老版本的用户打开这个注释，把上面2.0注释掉
        <dependency>
            <groupId>com.alibaba.ververica</groupId>
            <artifactId>flink-connector-mysql-cdc</artifactId>
            <version>1.4.0</version>
        </dependency>-->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.20</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.75</version>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.6</version>
        </dependency>
        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-connector-debezium</artifactId>
            <version>2.3.0</version>
        </dependency>


        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.4.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.6</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpmime</artifactId>
            <version>4.5.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-http_2.12</artifactId>
            <version>${flink-version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.3.1.RELEASE</version>
            </plugin>
        </plugins>
    </build>

</project>

四、云平台（模拟）

     正在编写测试代码中，没有完成，推动相关各方部署（ZOOKEEPER,KAFKA,ORACLE 19C,MONGODB等及管理均用DOCKER部署，相关部署参考DOCKER IMAGE HUB）

　　1，接受各馆管理系统数据(以interlib和汇文为示例)

　　　　1.1 interlib 主要提交数据为BIBLIOS,HOLDING,READER,LOG_CIR,LOAN_WORK表

package org.example;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.ververica.cdc.connectors.base.options.StartupOptions;
import com.ververica.cdc.connectors.oracle.OracleSource;
import com.ververica.cdc.debezium.JsonDebeziumDeserializationSchema;

import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.api.common.functions.MapFunction;

import org.apache.http.HttpEntity;
import org.apache.http.HttpStatus;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.io.InputStream;
import java.util.Properties;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;


public class postDB {
    public static void main(String[] args) throws Exception {
        //log4j2
        Logger logger = LogManager.getLogger(postDB.class.getName());
        //读取配置文件
        InputStream inputStream = ClassLoader.getSystemResourceAsStream("application.properties");
        Properties properties = new Properties();
        properties.load(inputStream);
        System.out.println("====================读取配置文件==========================");
        String pdbname = properties.getProperty("pdbname");
        String  hostname=properties.getProperty("hostname");
        String  url=properties.getProperty("url");
        int port= Integer.parseInt(properties.getProperty("port"));
        String  database=properties.getProperty("database");
        String  schemaList= properties.getProperty("schemaList");
        String  tableList=properties.getProperty("tableList");
        String  username=properties.getProperty("username");
        String  password=properties.getProperty("password");
        int StartupOptionsLatest=Integer.parseInt(properties.getProperty("StartupOptionsLatest"));

        String CheckpointStorage=properties.getProperty("CheckpointStorage");
        String postUrl=properties.getProperty("postUrl");

        String university=properties.getProperty("university");
        String token=properties.getProperty("token");

        //初始化设置
        Properties pro=new Properties();
        //pro.put("database.pdb.name", pdbname);
        pro.put("log.mining.strategy","online_catalog");
        pro.put("offset.flush.interval.ms", "60000");
        //pro.put("log.mining.continuous.mine","true");
        SourceFunction<String> sourceFunction = OracleSource.<String>builder()
                .hostname(hostname)
                .url(url)
                .port(port)
                .database(database) // monitor database
                .schemaList(schemaList) // monitor  schema
                 //.startupOptions(StartupOptions.latest())   //是否只监听最新数据
                .tableList(tableList) // monitor  table
                .debeziumProperties(pro)
                .username(username)
                .password(password)
                .deserializer(new JsonDebeziumDeserializationSchema()) // converts SourceRecord to JSON String
                .build();
        //设置起始点
        // Configuration configuration=new Configuration();
        //configuration.setString("execution.savepoint.path","file:///D:/checkoracle/cd720959c394888c1bc277e310b35b2d/chk-23");

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        //保存读取点
        env.enableCheckpointing(30000);
        env.getCheckpointConfig().setCheckpointStorage(CheckpointStorage);
        // 获取 CDC 数据流
        DataStream<String> mappedStream = processCdcStream(env, sourceFunction,university,token);

        mappedStream.addSink(new HttpSink(postUrl));

        env.execute();
        String str = System.console().readLine();
    }
    //数据map 处理部分
    public static DataStream<String> processCdcStream(StreamExecutionEnvironment env, SourceFunction<String> source,String university,String token) {
        DataStream<String> cdcStream = env.addSource(source);
        DataStream<String> mappedStream = cdcStream.map(new MapFunction<String, String>() {
            @Override
            public String map(String value) throws Exception {
                // 在这里对输入数据进行处理
                //JSONArray jsonArray = new JSONArray();
                //全部的json数据
                //JSONObject jsonObject =  JSON.parseObject(value);
                JSONObject allJson = new JSONObject();
                allJson.put("id",2);
                //allJson.put("source",jsonObject.toString());
                allJson.put("source",value);
                allJson.put("university",university);
                allJson.put("token",token);
                //jsonArray.add(allJson);
                value=JSON.toJSONString(allJson);
                // 返回处理后的输出数据
                return value ;
            }
        });
        return mappedStream;
    }
    public static class HttpSink implements SinkFunction<String> {
        private String URL;

        public HttpSink(String URL) {
            this.URL = URL;
        }

        @Override
        public void invoke(String value, Context context) throws Exception {
            CloseableHttpClient httpclient = HttpClients.createDefault();
            HttpPost httpPost = new HttpPost(URL);
            httpPost.setHeader("Content-Type", "application/json");
            //JSONArray jsonArray = JSONArray.parseArray(value);
            //String json = jsonArray.toString();

                StringEntity s = new StringEntity(value, "UTF-8");
                s.setContentType("application/json");//发送json数据需要设置contentType
                httpPost.setEntity(s);

                CloseableHttpResponse res = httpclient.execute(httpPost);
            try
            {
                if (res.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
                    String result = EntityUtils.toString(res.getEntity());// 返回json格式：
                    System.out.println(JSONObject.parseObject(result));
                }
            }catch (Exception ex)
            {
                System.out.println(ex.toString());
            }
            /*
            HttpEntity requestEntity = new StringEntity(value, ContentType.APPLICATION_JSON);
            String requestStr = EntityUtils.toString(requestEntity);
            System.out.println(requestStr);
            httpPost.setEntity(requestEntity);

            CloseableHttpResponse response = httpclient.execute(httpPost);
            try {
                HttpEntity responseEntity = response.getEntity();
                if (responseEntity != null) {
                    System.out.println("Response body: " + EntityUtils.toString(responseEntity));
                }

            } */
            finally {
                res.close();
            }
        }

    }


}

　　　　1.2 汇文正在寻找合作伙伴，请有兴趣的联系

　　2、推送到KAFKA后提供给以下功能消费订阅

　　　　客户端通过FLINK CDC提交到模拟云平台，服务器端解析后直接推送到KAFKA；

using Confluent.Kafka;
using Microsoft.AspNetCore.Mvc;
using syncdb.Model;
using System.Diagnostics;
using System.Net;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Unicode;

// For more information on enabling Web API for empty projects, visit https://go.microsoft.com/fwlink/?LinkID=397860

namespace syncdb.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class InterlibController : ControllerBase
    {
        // GET: api/<Loan_WorkController>
        [HttpGet]
        public ActionResult<Interlib>  Get()
        {
            Interlib value=new Interlib();
            value.Id= 1;
            value.Source = "source中文";
            value.University = "yangtzeu";
            value.Token= "token";
            bool result= SendKafkaMsg("cn.hbdlib.yangtzeu.interlib", JsonSerializer.Serialize(value, new JsonSerializerOptions()
            {
                Encoder = JavaScriptEncoder.Create(UnicodeRanges.All)
            })).Result;
            if (result)
            {
                return Ok(value);
            }
            else
            {
                return BadRequest(value);
            }
        }

        // GET api/<Loan_WorkController>/5
        [HttpGet("{id}")]
        public Interlib Get(int id)
        {
            Interlib value = new Interlib();
            value.Id = 2;
            value.Source = "source中文";
            value.University = "yangtzeu";
            value.Token = "token";
            return value;
        }

        // POST api/<Loan_WorkController>
        [HttpPost]
        public async Task<Msg> Post(Interlib value)
        {
            Msg msg=new Msg();
            try
            {
                Console.WriteLine("post:"+ value.Id+"|"+value.University);
                string topic = "cn.hbdlib." + value.University+".interlib";
                var result=await SendKafkaMsg(topic, JsonSerializer.Serialize(value, new JsonSerializerOptions()
                {
                    Encoder = JavaScriptEncoder.Create(UnicodeRanges.All)
                }));
                if (!result)
                {
                    msg.Code = 1;
                    msg.Message = "Interlib:Post:在发送到Kafka时没有成功";
                    msg.Data = value;
                }
            }
            catch(Exception ex) {
                msg.Code = 2;
                msg.Message = "Interlib:Post:在发送到Kafka时: "+ ex.Message;
                msg.Data = value;
            }
            return(msg);
        }
        private async Task<bool> SendKafkaMsg(string topic, string message)
        {
            ProducerConfig config = new ProducerConfig
            {
                BootstrapServers = "10.40.6.139:9092",
                ClientId = Dns.GetHostName()
            };

            try
            {
                using (var producer = new ProducerBuilder<Null, string>(config).Build())
                {
                    var result = await producer.ProduceAsync(topic, 
                                new Message<Null, string>
                                {
                                    Value = message
                                }
                        );

                    //Console.WriteLine($"SendKafkaMsg:{ topic}:{message}");
                return await Task.FromResult(true);
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine($"SendKafkaMsg:Error:{ex.Message}");
            }

            return await Task.FromResult(false);
        }

        // PUT api/<Loan_WorkController>/5
        [HttpPut("{id}")]
        public void Put(int id, [FromBody] string value)
        {
        }

        // DELETE api/<Loan_WorkController>/5
        [HttpDelete("{id}")]
        public void Delete(int id)
        {
        }
    }
}

　　3、入库

　　　　3.1 mongodb,直接存放提交的JSON数据，不做修改，用于保存；

// See https://aka.ms/new-console-template for more information
using Confluent.Kafka;
using System.Text.Json.Nodes;
using System.Threading;
using syncdb.Model;
using System.Text.Json;
using System.Text.Encodings.Web;
using System.Text.Unicode;

var config = new ConsumerConfig
{
    BootstrapServers = "10.40.6.139:9092",
    GroupId = "test-group",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
    string topics = "cn.hbdlib.yangtzeu.interlib";
    consumer.Subscribe(topics);
    CancellationToken cancellationToken = new CancellationToken();
    while (true)
    {
        try
        {
            var consumeResult = consumer.Consume(cancellationToken);
            Interlib newtonInterlib = Newtonsoft.Json.JsonConvert.DeserializeObject<Interlib>(consumeResult.Message.Value);
            Console.WriteLine(newtonInterlib.Source.ToString());
            new  mongo().insert(newtonInterlib);
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
    }

    consumer.Close();
}


Console.WriteLine("Hello, World!");

 //消费KAFKA并插入到数据库

using Microsoft.VisualBasic;
using MongoDB.Bson;
using MongoDB.Driver;
using Newtonsoft.Json.Linq;
using System.Text.Json.Nodes;

namespace syncdb.Model
{
    public class Interlib
    {
        public int Id { get; set; }
        public string? Source { set; get; }
        public string? University { set; get; }
        public string? Token { set; get; }
    }

    public interface interlib_mongodb
    {
        public bool insert(Interlib interlib);
    }
    public class mongo : interlib_mongodb
    {
        string connstr = "mongodb://root:th@10.40.6.139:27017";
        MongoClient client;
        public mongo() { 
            client=new MongoClient(connstr);
        }
        public  bool insert(Interlib interlib)
        {
            bool flag=false;

            try
            {
                //取出学校名
                IMongoDatabase db = client.GetDatabase(interlib.University);
                //转换为bson文档
                var document = BsonDocument.Parse(interlib.Source);
                //取出数据库名和表名
                string dbStr=document.GetValue("source").AsBsonDocument.GetValue("db").AsString;
                string tableStr=document.GetValue("source").AsBsonDocument.GetValue("table").AsString;
                //存放到不同的COLLECTION中
                IMongoCollection<BsonDocument> collection = db.GetCollection< BsonDocument>(dbStr+"."+tableStr);
                collection.InsertOne(document);
                flag=true;            
            }
            catch(Exception ex)
            {
                Console.WriteLine(ex.Message);  
            }

            return flag;
        }
    }

    public interface interlib_oracle
    { 
        public bool update(JsonObject json);
        public bool delete(JsonObject json);
        public bool insert(JsonObject json);

    }

}

//原始数据示例

{
  "_id": "ObjectId('64589d0022308d4b25e8c8ba')",
  "before": null,
  "after": {
    "RDID": "300603",
    "BARCODE": "CD11115345",
    "RULESTATE": 1,
    "LOANCOUNT": 0,
    "LOANDATE": 1667407539000000,
    "RETURNDATE": 1675183539434000,
    "DUETIME": 1667407539000000,
    "ATTACHMENT": 0,
    "SENDFLAG": "0",
    "HASHVALUE": null
  },
  "source": {
    "version": "1.6.4.Final",
    "connector": "oracle",
    "name": "oracle_logminer",
    "ts_ms": 1683528934877,
    "snapshot": "true",
    "db": "INTERLIB",
    "sequence": null,
    "schema": "INTERLIB",
    "table": "LOAN_WORK",
    "txId": null,
    "scn": "13206765",
    "commit_scn": null,
    "lcr_position": null
  },
  "op": "r",
  "ts_ms": 1683528934877,
  "transaction": null
}

 

 MONGODB存放的数据

 

　　　　3.2  oracle 19c，解析JSON数据后，存放数据库和各管理系统的源数据库保持一致，为将来系统提供基础数据；

　　4、关联

　　　　4.1 和CCBD数据比对，记录下CCKBID和书目记录号；

　　　　4.2 记录下条码号，为将来馆藏统计提供基础数据

　　5、推送

　　　　5.1 通过MQTT和WEBSOCKET协议实时推送数据，用于大屏幕展示，如借还数据显示；（这个功能能吸引眼球）

　　　　5.2 和其它业务系统对接，微信公众号的消息模板，借还书的信息推送，纸质图书和数字图书的推荐，闸机刷卡签到（这个功能实用）

　　6、分析