随笔 - 394 文章 - 0 评论 - 946 阅读 - 143万

Quartz.net打造信息抽取器

由于最近的一个项目需要定时抽取特定XML信息，然后保存到数据库，最后通过WebApi把手机端要使用的方法给暴露出来，所以去研究了一下Quartz.net。由于项目很小，我没用到Autofac，Repository模式，UOW这些东西，这个小项目中所涉及的知识点有：

1.Quartz.net配置

2.序列化XML信息到类对象中

3.AutoMapper做DomainModel和DTO之间的映射

4.WebApi允许多个Get方法的存在

下面我们一步一步的来进行说明。

让我们先来看看我们待处理的数据:http://price.agridoor.com.cn/nxt_price/uploads/2014/05/09/20140509_37.xml

从上图可以看到，这个xml数据有个根节点NXT_PRICE，并且根节点下面有很多个price子节点。那么如果我们想把这些xml数据反序列化到我们定义的类对象中，该怎么做呢？

其实很简单。

由于一个根节点NXT_PRICE下面有N个price子节点集合，所以我们创建如下的两个实体类来描述这种关系：

[XmlRoot("NXT_PRICE")]
public class NxtPriceModel
{
    public NxtPriceModel()
    {
        priceItems = new List<PriceItemModel>();
    }
 
    [XmlElement("price")]
    public List<PriceItemModel> priceItems { get; set; }
}

public class PriceItemModel
    {
        [XmlElement("seq")]
        public int Seq { get; set; }
 
        [XmlElement("name")]
        public string Name { get; set; }
 
        [XmlElement("type")]
        public string Type { get; set; }
 
        [XmlElement("price")]
        public decimal Price { get; set; }
 
        [XmlElement("unit")]
        public string Unit { get; set; }
 
        [XmlElement("time")]
        public string Time { get; set; }
 
        [XmlElement("first")]
        public string First { get; set; }
 
        [XmlElement("second")]
        public string Second { get; set; }
 
        [XmlElement("area")]
        public string Area { get; set; }
    }

这样两个实体类很简单，相信大家也看出了其中的包含关系。

然后如何反序列化到我们给定的实体类中呢？这里使用XmlSerializer则是再合适不过的事情了。通过XmlSerializer对象，我们可以直接将对应的xml节点解释成实体类属性，并自动将数据保存到类中的集合对象中。

WriteLog("=============================开始反序列化文件========================");
                  XmlSerializer ser = new XmlSerializer(typeof(NxtPriceModel));
                  ms.Position = 0;
                  var result = (NxtPriceModel)ser.Deserialize(ms);

通过上面的代码，我们就能成功的将路径中的xml信息下载并保存到类集合中，非常方便。这样，我们就完成了xml序列化成类对象的功能。

数据都已经保存到类对象中了，下一步就让我们来Consume它。

我们创建一个Asp.net MVC4项目，并利用Install-package quartz命令将其添加到项目中。然后在项目中添加一个MyJob类，继承自IJob对象：

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

public class MyJob : IJob
   {
       private object obj = new object();
 
       public void Execute(IJobExecutionContext context)
       {
           if (ShouldRun())
               Run();
       }
 
       private bool ShouldRun()
       {
           CommandPengEntities cpEntities = new CommandPengEntities();
           try
           {
               string strStart = DateTime.Now.ToString("yyyy-MM-dd") + " 00:00:00";
               string strEnd = DateTime.Now.ToString("yyyy-MM-dd") + " 23:59:59";
 
               DateTime? dtStart = DateTime.Parse(strStart);
               DateTime? dtEnd = DateTime.Parse(strEnd);
 
               var result = (from p in cpEntities.NxtPrice where p.Time >= dtStart && p.Time <= dtEnd select p).FirstOrDefault();
               if (result == null)
               {
                   WriteLog("当天数据不存在，准许插入新数据");
                   return true;  //当天数据未被插入
               }
               WriteLog("当天数据存在，禁止插入新数据");
               return false;   //当天数据已经插入
           }
           catch (Exception ex)
           {
               WriteLog(ex.InnerException.Message);
               return false;
           }
           finally
           {
               DisposeContext(cpEntities);
           }
       }
 
       private void Run()
       {
           string uri = "http://price.agridoor.com.cn/nxt_price/uploads/2014/05/09/20140509_37.xml";
           WriteLog("=============================开始进行文件获取========================");
           var request = HttpWebRequest.Create(uri);
           IAsyncResult iasync = request.BeginGetResponse((iar) =>
           {
               var requestCallBack = (HttpWebRequest)iar.AsyncState;
               var response = requestCallBack.EndGetResponse(iar);
               WriteLog("=============================获取文件内容结束========================");
 
               var stream = response.GetResponseStream();
 
 
               byte[] buffer = null;
 
               //将stream保存到MemoryStream中
               WriteLog("=============================保存内容到内存流========================");
               using (MemoryStream ms = new MemoryStream())
               {
                   int count = 0;
                   do
                   {
                       byte[] buf = new byte[1024];
                       count = stream.Read(buf, 0, 1024);
                       ms.Write(buf, 0, count);
                   }
                   while (stream.CanRead && count > 0);
                   buffer = ms.ToArray();
 
                   //string txt = Encoding.UTF8.GetString(buffer);
 
                   //开始反序列化
                   WriteLog("=============================开始反序列化文件========================");
                   XmlSerializer ser = new XmlSerializer(typeof(NxtPriceModel));
                   ms.Position = 0;
                   var result = (NxtPriceModel)ser.Deserialize(ms);
 
                   //检测是否为空
                   WriteLog("=============================检测文件是否为空========================");
                   if (result == null) return;
                   if (result.priceItems == null) return;
                   if (result.priceItems.Count == 0) return;
 
                   var allItems = result.priceItems;
                   var totalRecords = result.priceItems.Count;
                   int batch = 200;  //每200条批量提交一次
                   int totalExecuteCount = (totalRecords % batch == 0) ? (totalRecords / batch) : (totalRecords / batch + 1);
 
                   //写入数据库
                   lock (obj)
                   {
                       PerformLogicInsert(totalRecords, totalExecuteCount, batch, allItems);
                   }
               }
           }, request);
       }
 
       private void AddEntity(PriceItemModel pModel,CommandPengEntities cpEntities)
       {
           var itemShouldInsert = AutoMapper.Mapper.Map<NxtPrice>(pModel);
           cpEntities.NxtPrice.AddObject(itemShouldInsert);
       }
 
       private bool CommitEntity(CommandPengEntities cpEntities)
       {
           int rowAffected = cpEntities.SaveChanges();
           if (rowAffected > 0)
               return true;
           return false;
       }
 
       private void DisposeContext(CommandPengEntities cpEntities)
       {
           if (cpEntities != null)
               cpEntities.Dispose();
       }
 
       private void PerformLogicInsert(int totalRecords
                                      , int totalExecuteCount
                                     , int batch
                                     , List<PriceItemModel> allItems)
       {
           WriteLog("=============================启用事务控制========================");
 
           CommandPengEntities cpEntities = null;
           try
           {
               using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required, new TimeSpan(0, 3, 0)))
               {
                   cpEntities = new CommandPengEntities();
 
                   for (int i = 0; i < totalExecuteCount; i++)
                   {
                       int remainRecords = totalRecords - i * batch;
                       int isLastBatch = remainRecords / batch;
                       int loopCounter = 0;
                       if (isLastBatch > 0) //不是最后一批数据
                           loopCounter = batch;
                       else              //最后一批数据
                           loopCounter = remainRecords;
 
                       for (int j = 0; j < loopCounter; j++)
                       {
                           var item = allItems[i * batch + j];
                           AddEntity(item, cpEntities);
                       }
 
                       try
                       {
                           bool flag = CommitEntity(cpEntities);
                           if (flag) WriteLog(string.Format("第{0}批数据插入完毕，共计{1}条", i, loopCounter));
                           else WriteLog(string.Format("第{0}批数据插入失败", i));
                       }
                       catch (Exception ex)
                       {
                           WriteLog(ex.Message);
                           WriteLog(ex.InnerException.Message);
                       }
                   }
                   scope.Complete();
                   WriteLog("=============================事务提交成功========================");
               }
           }
           finally
           {
               DisposeContext(cpEntities);
           }
       }
 
       private void WriteLog(string content)
       {
           string logFile = AppDomain.CurrentDomain.BaseDirectory + "\\log" + DateTime.Now.ToString("yyyyMMdd") + ".txt";
           using (FileStream stream = new FileStream(logFile, FileMode.Append))
           {
               byte[] buffer = Encoding.UTF8.GetBytes(DateTime.Now.ToString("yyyy-MM-dd hh:mm:ss") + "  " + content + Environment.NewLine);
               stream.Write(buffer, 0, buffer.Length);
               stream.Flush();
           }
       } 
   }

上面的代码主要是抽取数据,然后将数据写入到数据库中的操作行为.其中需要说到一点的就是Automapper的使用.

由于数据访问层,我直接使用的NxtPrice.edmx,所以会自动生成一个NxtPrice模型出来,这样当我们提交数据的时候,需要将对象进行转换:

private void AddEntity(PriceItemModel pModel,CommandPengEntities cpEntities)
      {
          var itemShouldInsert = AutoMapper.Mapper.Map<NxtPrice>(pModel);
          cpEntities.NxtPrice.AddObject(itemShouldInsert);
      }

由于数据库插入对象时NxtPrice,而我们的数据集合对象是NxtPriceModel,所以这里需要将NxtPriceModel中的字段逐一赋值给NxtPrice对象,由于automapper能够提供类似的行为,所以我们采用automapper自动来进行.automapper会对比两个model的异同,只要是字段相同的话,都会自动进行映射.这样就省去了很多的操作步骤.

AutoMapper进行映射前,我们需要在Global.asax中配置一下,方可使用:

private void ModelMapper()
       {
           //Model and DTO conversion
           AutoMapper.Mapper.CreateMap<PriceItemModel, NxtPrice>();
           AutoMapper.Mapper.CreateMap<NxtPrice,PriceItemModel>();
            
       }

这样就达到我们的自动映射的要求了.

最后需要说明的是,我们的Job写完之后,需要定时运行,这个该如何做呢?

由于Quartz.net中,工作分为创建计划任务->创建工作内容->创建触发条件->启动四个步骤,所以这里我们就按照这四个步骤来进行:

private void StartSchedular()
        {
            IScheduler scheduler = null;
            // start up scheduler
            // construct a factory 
            ISchedulerFactory factory = new StdSchedulerFactory();
            // get a scheduler 
            scheduler = factory.GetScheduler();
            // start the scheduler 
            scheduler.Start();
 
            // create job  
            IJobDetail job = JobBuilder.Create<MyJob>()
                        .WithIdentity("MyJob", "MyJobs")
                        .Build();
 
            // create trigger  
            ITrigger trigger = TriggerBuilder.Create()
                .WithIdentity("MyJobTrigger", "MyJobs")
                // start at 7:30 every day
                .StartAt(DateBuilder.DateOf(7,30,0))
                .WithSimpleSchedule(x => x.WithInterval(TimeSpan.FromMilliseconds(1)).WithRepeatCount(0))
                .Build();
 
            // Schedule the job using the job and trigger   
            scheduler.ScheduleJob(job, trigger);  
        }

注释说的很明白了, 其中scheduler对象就是创建的计划任务.

job对象则是我们刚刚新建的MyJob类及其要执行的内容.

trigger对象则是设置触发条件,这里我们设置为每天7:30开始,运行一次即可.

最后一句则是将工作内容和触发器连接起来,以便进行控制.

这样运行之后,我们运行程序,就能够看到系统正常运行了.

数据库的数据都填充以后,我们开始编写我们的webapi代码了.这里由于不是讲解的重点,我就不多说了,但是需要注意一点的是,webapi中可以通过设置路由来允许多个Get方法并存:

config.Routes.MapHttpRoute(
             name: "ApiByName",
             routeTemplate: "api/{controller}/{action}/{name}",
             defaults: null,
             constraints: new { name = @"^[a-z]+$" }
         );

这样当我们访问多个get方法的时候,就可以通过如下的方式访问了:

http://192.168.0.119/api/PriceItems/GetProductType/?cityName=新乡市

http://192.168.0.119/api/PriceItems/GetProvince/

最后需要说明的是,由于WebApi默认返回JSON格式，所以如果你想接受XML格式的内容，服务端是不需要做任何设置的。你只需要在客户端加上Accept:application/xml即可请求到xml的数据，非常方便。

百度网盘

腾讯网盘