从WEB SERVICE 上返回大数据量的DATASET[转]
众所周知,如果不用其它方法,直接从WEB SERVICE上传回一个10W条记录的DATASET,可想而知的后果是什么,CPU要占用 100%,且要等上几分钟,这是任何一个项目都无法忍受的.在我上网找资料的过程中,试验了几种不同的方法,如通过压缩SOAP改善XML Web service性能,这篇文章所介绍的方法用了SOAP扩展,是通过在WEB SERVICE端用已经过时了的NZIPLIB库来压缩SOAP响应,据称文本压缩率可达80%.文章里面的代码是VB.NET的,费了好大劲翻译成C#的,照上面建项目,但是很可惜,我没有编译成功,总是出错.
这里我找我建好的项目提供大家下载,大家有时间看看是什么问题. SOAP压缩代码下载
而后,找到了用序列化的方式来减少网络传输量,Microsoft .NET Framework 1.x 中内建两种将物件序列化的 Formatter类别,SoapFormatter 和 BinaryFormatter,两种方式均能减轻网络传输量提高性能,但SoapFormatter方式传输的方式其实还是XML形式,加了很多XML标识,因此压缩率不是很理想,BinaryFormatter用纯二进制的方式序列化DATASET,能使压缩率大大提高,这是台湾作者李匡正 (台灣微軟應用架構技術經理提供的例子里对SQL范例库Northwind的测试结果:
在这里,有两种方式:可把序列化后的数据用文件形式保存在客户端硬盘;也可用Byte[]方式传回客户端,以下是代码。
web service 端(文件形式)
[WebMethod(Description="循环获取远程DATASET")]
public void SurrogateReadTable(string TableName)
{
//把DataSet通过Surrogate Class 序列化成 Binary Stream
DataSet ds;
ds=SqlHelper.ExecuteDataset(cnn,CommandType.Text,"select * from "+TableName);
//实例化DataSetSurrogate,传取出的DATASET到构造函数里
sds = new DataSetSurrogate(ds);
//实例化二进制流
BinaryFormatter bf=new BinaryFormatter();
StreamWriter swDat;
//写到本地一个文件里
swDat = new StreamWriter(@"c:\output_surrogate_dataset.dat");
bf.Serialize(swDat.BaseStream, sds);
//这里可以知道序列化后的文件的大小
long size = swDat.BaseStream.Length;
swDat.Close();
}
客户端
private void button1_Click(object sender, System.EventArgs e)
{
label1.Text=DateTime.Now.ToString();
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
button1.Enabled=false;
//反序列化Binary Stream能通过Surrogate Class转换成 DataSet
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
//从WEB SERVICE上读取方法
svs.SurrogateRead("t_busdocbase");
BinaryFormatter bf=new BinaryFormatter();
StreamReader swDat;
swDat = new StreamReader(@"c:\output_surrogate_dataset.dat");
object o=bf.Deserialize(swDat.BaseStream);
DataSet ds;
sds = (DataSetSurrogate) o;
ds = sds.ConvertToDataSet();
dataGrid1.DataSource=ds.Tables[0];
swDat.Close();
}
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
[WebMethod(Description="获取业务资料远程DATASET")]
public byte[] SurrogateRead1()
{
DataSet ds;
ds=SqlHelper.ExecuteDataset(cnn,CommandType.Text,"select * from t_busdocbase");
sds = new DataSetSurrogate(ds);
MemoryStream s= new MemoryStream();
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(s,sds);
byte[] e = s.ToArray();
return e;
}
private void button3_Click(object sender, System.EventArgs e)
{
label1.Text=DateTime.Now.ToString();
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
button3.Enabled=false;
//*反序列化Binary Stream能通过Surrogate Class转换成 DataSet*/
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
//从WEB SERVICE上读取方法
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
byte [] bb=svs.SurrogateRead1();
MemoryStream br=new MemoryStream(bb);
BinaryFormatter bf=new BinaryFormatter();
object o=bf.Deserialize(br);
sds = (DataSetSurrogate) o;
ds = sds.ConvertToDataSet();
dataGrid1.DataSource=ds.Tables[0];
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
br.Close();
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
}
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
这里我找我建好的项目提供大家下载,大家有时间看看是什么问题. SOAP压缩代码下载
而后,找到了用序列化的方式来减少网络传输量,Microsoft .NET Framework 1.x 中内建两种将物件序列化的 Formatter类别,SoapFormatter 和 BinaryFormatter,两种方式均能减轻网络传输量提高性能,但SoapFormatter方式传输的方式其实还是XML形式,加了很多XML标识,因此压缩率不是很理想,BinaryFormatter用纯二进制的方式序列化DATASET,能使压缩率大大提高,这是台湾作者李匡正 (台灣微軟應用架構技術經理提供的例子里对SQL范例库Northwind的测试结果:
|
SoapFormatter |
BinaryFormatter |
Dataset 序列化後 Bytes 數 |
1,953,078 |
1,448,399 |
很显示然BinaryFormatter 明显优于SoapFormatter ,而我也确实用了BinaryFormatter 这种方式实现了提高效率.
再者,用微软提供的DataSetSurrogate 类可以此基础上进一步压缩数据大小,DataSetSurrogate 在.net 2.0里自带。这是比较结果.
SoapFormatter | BinaryFormatter | |
Dataset 序列化後Bytes數 | 1,953,078 | 1,448,399 |
DataSetSurrogate 序列化後Bytes數 | 2,371,942 | 575,684 |
在这里,有两种方式:可把序列化后的数据用文件形式保存在客户端硬盘;也可用Byte[]方式传回客户端,以下是代码。
web service 端(文件形式)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
客户端
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
web service 端(Byte[]方式)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
客户端
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
![](https://www.cnblogs.com/Images/OutliningIndicators/None.gif)
我个人觉得用byte[]方式会安全些,毕竟不用在客户端产生文件,不用担心数据的安全。
当然作为从网络上读取数据来说,10W条是一个不小量,所有的方式包括压缩,序列化等都是权宜之计,而不是长久之计,在使用当中,我用以上的方法虽然能使网络传输量降低,且可在很短时间内就把数据显示在DATAGRID上,但CPU的开销却达到了100%,这是我一直头疼的。我后来又用了分页的方式,把10W条数据在服务器端就分批取出,每次500条,这样读取时间延长了,但CPU开销却未减轻很多,再后来,又用多线程的方式处理,不甚理想。因此最好的方法就是尽可能的不查询10W条数据,通过条件判断等方式减少所需处理的数据量。