按顺序模式下载

最近常在115网盘下载电影, 晚上下载的话有时候速度比较慢, 115上的文件一般会有3个下载地址, 每个地址可以允许3~5个线程下载.

 

btw: 115网盘在3月31号已经彻底水产了 bbs.ylmf.net/forum.php?mod=viewthread&tid=1862346

但迅雷之类的软件只支持人工设置1个地址多线程, 于是祭出大杀器NetTransport, 给每个地址都加上3~5个线程

 

但是比较郁闷的是NetTransport 不支持类似迅雷的顺序模式下载

 

这样的话在视频文件没下完的情况下就不好预览(边下边看)了, 遂决定动手实现一个能按顺序下载支持多地址多线程的下载器, 思考后得出如下设计:

 

首先下载器可以添加下载任务, 而下载任务又可以添加子任务(下载线程), 而下载器本身也是一项任务, 它们都可以开始和停止, 并有已激活或已完成的状态, 每个任务可以会被观察以便我们了解任务过程中的信息, 综合这些抽象出任务接口

publicinterface ITask
2 {
3 event Action<string> Info;
4 event Action<string> Error;
5 bool IsAlive { get; }
6 bool IsComplete { get; }
7 void Start();
8 void Stop();
9 }

而下载器和每项下载任务都可以添加或移除子任务, 这里我们使用组合模式得出一个实现任务接口的任务列表抽象类

publicabstractclass TaskListBase : ITask
 2{
 3public List<ITask> TaskList { get; privateset; }
 4
 5public TaskListBase()
 6    {
 7this.TaskList =new List<ITask>();
 8    }
 9
10publicvirtualvoid Add(ITask task)
11    {
12        task.Info +=this.NotifyInfo;
13        task.Error +=this.NotifyError;
14this.TaskList.Add(task);
15    }
16
17publicvirtualvoid Remove(ITask task)
18    {
19        task.Info -=this.NotifyInfo;
20        task.Error -=this.NotifyError;
21this.TaskList.Remove(task);
22    }
23
24protectedvoid NotifyInfo(string info)
25    {
26if (this.Info !=null)
27        {
28this.Info(info);
29        }
30    }
31protectedvoid NotifyError(string error)
32    {
33if (this.Error !=null)
34        {
35this.Error(error);
36        }
37    }
38
39#region ITask 成员
40
41publicevent Action<string> Info;
42
43publicevent Action<string> Error;
44
45publicbool IsAlive
46    {
47get
48        {
49foreach (var task inthis.TaskList)
50            {
51if (task.IsAlive)
52                {
53returntrue;
54                }
55            }
56returnfalse;
57        }
58    }
59
60publicbool IsComplete
61    {
62get
63        {
64foreach (var task inthis.TaskList)
65            {
66if (!task.IsComplete)
67                {
68returnfalse;
69                }
70            }
71returntrue;
72        }
73    }
74
75publicvirtualvoid Start()
76    {
77foreach (var task inthis.TaskList)
78        {
79if (!task.IsAlive)
80            {
81                task.Start();
82            }
83        }
84    }
85
86publicvirtualvoid Stop()
87    {
88foreach (var task inthis.TaskList)
89        {
90if (task.IsAlive)
91            {
92                task.Stop();
93            }
94        }
95    }
96
97#endregion
98}

 这样就可以把下载器和下载任务的一些公用部分放到任务列表类以减少重复, 在添加和删除子任务的时候, 任务列表订阅和退订子任务的事件,  然后就可以通过观察任务列表得知所有子任务的事件了, 由于任务列表也实现了和子任务相同的接口,我们调用任务的相关操作就可以通过遍历其子任务来调用同类操作, 管理也变得简单了.

[Serializable]
2publicclass Link
3{
4publicstring Uri { get; set; }
5publicstring Referer { get; set; }
6publicstring Cookie { get; set; }
7publicint Count { get; set; }
8}

通过一个简单的链接类来描述多地址多线程的的信息, 每个链接有自己的线程数量,引用页,和Cookie

[Serializable]
 2  publicclass Range
 3 {
 4 publiclong From { get; set; }
 5 publiclong To { get; set; }
 6 publiclong ReadSize { get; set; }
 7 publiclong WriteSize { get; set; }
 8 publicbool IsAlive { get; set; }
 9 publicbool IsComplete
10     {
11 get
12         {
13 returnthis.From +this.WriteSize >this.To;
14         }
15     }
16 public Range(long from, long to)
17     {
18 this.From = from;
19 this.To = to;
20     }
21 }

范围类描述每个线程要下载的范围, 我们把整个下载文件划分出多个块, 所有的下载线程都从分块列表的头部查找未完成的文件块进行下载, 这就是顺序模式下载了

[Serializable]
 2publicclass TaskInfo
 3{
 4publicstring FileName { get; set; }
 5publicstring SaveDir { get; set; }
 6publiclong FileSize { get; set; }
 7publiclong RangeSize { get; set; }
 8publicint WaitTime { get; set; }
 9public List<Range> RangeList { get; set; }
10public List<Link> LinkList { get; set; }
11public TaskInfo()
12    {
13this.RangeList =new List<Range>();
14this.LinkList =new List<Link>();
15this.RangeSize =1024*1024*3;
16this.WaitTime =1000;
17    }
18public Range NextRange()
19    {
20        Range range =null;
21lock (this.RangeList)
22        {
23            range =this.RangeList.FirstOrDefault(it =>!it.IsComplete &&!it.IsAlive);
24if (range !=null)
25            {
26                range.IsAlive =true;
27            }
28        }
29return range;
30    }
31}

任务信息类描述了下载任务的文件名, 存放地址, 文件大小, 分块大小, 重试等待时间等信息, 并包含了一个链接列表和一个文件分块列表, 每个下载线程调用任务信息类的NextRange方法就可以返回距离头部最近的未完成块了,  通过标记[Serializable]特性,  我们就可以在每次文件写入完成后,  序列化保存任务信息类,  以备随时可以恢复未完成的下载,  实现断点续传.

publicclass DownloadThread : ITask
  2{
  3    WebRequest request;
  4    Range range;
  5    Thread thread;
  6
  7public TaskInfo TaskInfo { get; privateset; }
  8publicint ID { get; set; }
  9public Link Link { get; set; }
 10
 11public DownloadThread(TaskInfo taskInfo)
 12    {
 13this.TaskInfo = taskInfo;
 14    }
 15
 16#region 私有方法
 17
 18void StartThread()
 19    {
 20this.thread =new Thread(Request);
 21this.thread.IsBackground =true;
 22this.thread.Start();
 23    }
 24
 25void Request()
 26    {
 27while ((this.range =this.TaskInfo.NextRange()) !=null)
 28        {
 29this.NotifyInfo(string.Format("请求{0}-{1}.", this.range.From +this.range.WriteSize, this.range.To));
 30            HttpWebRequest request =null;
 31            HttpWebResponse response =null;
 32try
 33            {
 34                request = WebRequest.Create(this.Link.Uri) as HttpWebRequest;
 35                request.Timeout =10000;
 36                request.KeepAlive =true;
 37                request.AddRange((int)(this.range.From +this.range.WriteSize), (int)this.range.To);
 38                response = request.GetResponse() as HttpWebResponse;
 39this.Read(response);
 40            }
 41catch (WebException ex)
 42            {
 43this.NotifyInfo(ex.Message);
 44                Thread.Sleep(this.TaskInfo.WaitTime);
 45            }
 46catch (ArgumentOutOfRangeException ex)
 47            {
 48this.NotifyInfo(ex.Message);
 49                Thread.Sleep(this.TaskInfo.WaitTime);
 50            }
 51catch (Exception ex)
 52            {
 53this.NotifyError(ex.Message);
 54            }
 55finally
 56            {
 57this.range.IsAlive =false;
 58if (response !=null)
 59                {
 60                    response.Close();
 61                    response =null;
 62                }
 63if (request !=null)
 64                {
 65                    request.Abort();
 66                    request =null;
 67                }
 68            }
 69        }
 70this.IsComplete =true;
 71this.NotifyInfo("已停止.");
 72    }
 73
 74void Read(HttpWebResponse response)
 75    {
 76        Stream responseStream =null;
 77        MemoryStream stream =null;
 78try
 79        {
 80            responseStream = response.GetResponseStream();
 81            stream =new MemoryStream();
 82int count =0;
 83byte[] buffer =newbyte[32*1024];
 84while ((count = responseStream.Read(buffer, 0, buffer.Length)) >0)
 85            {
 86                stream.Write(buffer, 0, count);
 87this.range.ReadSize += count;
 88if (stream.Length >1024*1024)
 89                {
 90this.Write(stream);
 91                    stream =new MemoryStream();
 92                }
 93            }
 94this.Write(stream);
 95            stream =null;
 96        }
 97catch (Exception ex)
 98        {
 99this.NotifyError(ex.Message);
100        }
101finally
102        {
103if (responseStream !=null)
104            {
105                responseStream.Dispose();
106                responseStream =null;
107            }
108if (stream !=null)
109            {
110                stream.Dispose();
111                stream =null;
112            }
113        }
114    }
115
116void Write(MemoryStream stream)
117    {
118        FileStream fileStream =null;
119try
120        {
121            fileStream =new FileStream(this.TaskInfo.SaveDir +this.TaskInfo.FileName, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);
122            fileStream.Seek(this.range.From +this.range.WriteSize, SeekOrigin.Begin);
123            stream.WriteTo(fileStream);
124this.NotifyInfo(string.Format("写入数据:{0}-{1}", this.range.From +this.range.WriteSize, this.range.From +this.range.WriteSize + stream.Length));
125this.range.WriteSize += stream.Length;
126        }
127catch (Exception ex)
128        {
129this.NotifyError(ex.Message);
130        }
131finally
132        {
133if (stream !=null)
134            {
135                stream.Dispose();
136                stream =null;
137            }
138if (fileStream !=null)
139            {
140                fileStream.Dispose();
141                fileStream =null;
142            }
143        }
144    }
145
146void NotifyInfo(string info)
147    {
148if (this.Info !=null)
149        {
150this.Info("线程["+this.ID +"]: "+ info);
151        }
152    }
153
154void NotifyError(string error)
155    {
156this.Stop();
157this.NotifyInfo("线程["+this.ID +"]已停止.");
158if (this.Error !=null)
159        {
160this.Error("线程["+this.ID +"]: "+ error);
161        }
162this.range.ReadSize =this.range.WriteSize;
163        Thread.Sleep(this.TaskInfo.WaitTime);
164this.Start();
165this.NotifyInfo("线程["+this.ID +"]已重启.");
166    }
167
168#endregion
169
170#region ITask 成员
171
172publicevent Action<string> Info;
173
174publicevent Action<string> Error;
175
176publicbool IsAlive { get; privateset; }
177
178publicbool IsComplete { get; privateset; }
179
180publicvoid Start()
181    {
182if (!this.IsAlive &&!this.IsComplete)
183        {
184this.IsAlive =true;
185this.StartThread();
186this.NotifyInfo("开始下载");
187        }
188    }
189
190publicvoid Stop()
191    {
192if (this.IsAlive)
193        {
194if (this.request !=null)
195            {
196this.request.Abort();
197this.request =null;
198            }
199if (this.thread !=null)
200            {
201this.thread.Abort();
202this.thread =null;
203            }
204this.IsAlive =false;
205        }
206    }
207
208#endregion
209}

publicclass DownloadTask : TaskListBase
  2{
  3public TaskInfo TaskInfo { get; privateset; }
  4
  5public DownloadTask(TaskInfo taskInfo)
  6        : base()
  7    {
  8this.TaskInfo = taskInfo;
  9    }
 10
 11#region 私有方法
 12
 13void InitTaskInfo()
 14    {
 15if (this.TaskInfo.FileSize ==0)
 16        {
 17            HttpWebRequest request =null;
 18            HttpWebResponse response =null;
 19try
 20            {
 21                request = WebRequest.Create(this.TaskInfo.LinkList[0].Uri) as HttpWebRequest;
 22                request.Timeout =10000;
 23                request.KeepAlive =true;
 24                response = request.GetResponse() as HttpWebResponse;
 25this.TaskInfo.FileSize = response.ContentLength;
 26if (this.TaskInfo.FileSize >0&&this.TaskInfo.LinkList.Sum(it => it.Count) >0)
 27                {
 28long offset =0;
 29while (offset <this.TaskInfo.FileSize)
 30                    {
 31if (offset +this.TaskInfo.RangeSize <this.TaskInfo.FileSize)
 32                        {
 33this.TaskInfo.RangeList.Add(new Range(offset, offset +this.TaskInfo.RangeSize));
 34                        }
 35else
 36                        {
 37this.TaskInfo.RangeList.Add(new Range(offset, this.TaskInfo.FileSize -1));
 38                        }
 39                        offset +=this.TaskInfo.RangeSize +1;
 40                    }
 41                }
 42                var disposition = response.Headers["Content-Disposition"];
 43if (disposition !=null)
 44                {
 45                    var match =new Regex(@"attachment;(?:\s+)?filename=""?([^""]+)").Match(disposition);
 46if (match.Groups.Count ==2&& match.Groups[1].Length >0)
 47                    {
 48this.TaskInfo.FileName = Path.GetFileName(HttpUtility.UrlDecode(match.Groups[1].Value));
 49                    }
 50                }
 51if (this.TaskInfo.FileName ==null||this.TaskInfo.FileName.Length ==0)
 52                {
 53this.TaskInfo.FileName = Path.GetFileName(HttpUtility.UrlDecode(request.RequestUri.LocalPath));
 54                }
 55this.NotifyInfo(string.Format(@"取得文件大小:{0}", this.TaskInfo.FileSize));
 56            }
 57catch (Exception ex)
 58            {
 59this.NotifyError(ex.Message);
 60            }
 61finally
 62            {
 63if (response !=null)
 64                {
 65                    response.Close();
 66                    response =null;
 67                }
 68if (request !=null)
 69                {
 70                    request.Abort();
 71                    request =null;
 72                }
 73            }
 74        }
 75    }
 76void InitThread()
 77    {
 78        var id =1;
 79this.TaskInfo.LinkList.ForEach(item =>
 80        {
 81for (var i =0; i < item.Count; ++i)
 82            {
 83                var thread =new DownloadThread(this.TaskInfo);
 84                thread.ID = id++;
 85                thread.Link = item;
 86this.Add(thread);
 87            }
 88        });
 89    }
 90
 91#endregion
 92
 93publicoverridevoid Start()
 94    {
 95if (this.TaskInfo.FileSize ==0)
 96        {
 97this.InitTaskInfo();
 98        }
 99if (this.TaskInfo.FileSize >0)
100        {
101if (this.TaskList.Count ==0)
102            {
103this.InitThread();
104            }
105if (!File.Exists(this.TaskInfo.SaveDir +this.TaskInfo.FileName))
106            {
107using (var fs = File.Create(this.TaskInfo.SaveDir +this.TaskInfo.FileName))
108                {
109long offset = fs.Seek(this.TaskInfo.FileSize -1, SeekOrigin.Begin);
110                    fs.WriteByte(newbyte());
111                }
112            }
113            ServicePointManager.DefaultConnectionLimit =20;
114base.Start();
115        }
116    }
117}

这两个类就需要注意几个地方ServicePointManager.DefaultConnectionLimit 可以设置最大并发数, 默认值为2, 所以需要重新设置

using (var fs = File.Create(this.TaskInfo.SaveDir + this.TaskInfo.FileName))

{

    fs.Seek(this.TaskInfo.FileSize - 1, SeekOrigin.Begin);

    fs.WriteByte(new byte());

}

跳到文件流的最后写入1字节, 可以预先占用磁盘空间, 但在ntfs下建立的是稀疏文件, 并没有实际写入全部数据, 这样会再实际写入的时候引起更多的磁盘碎片, 因此迅雷使用的是全部填入空白数据的方法

FileShare.ReadWrite参数允许在文件被占用的情况下随后打开文件读取或写入数据, 然后我们就可以在下载的过程中播放视频文件, 而不影响下载程序的写入和读取, 还有一个优点就是可以省略线程间争抢写入而需要lock的代码

最后把下载器的代码补上, 简单测试一下

publicclass Downloader : TaskListBase
 2 {
 3 publicvoid AddTask(TaskInfo taskInfo)
 4     {
 5 this.Add(new DownloadTask(taskInfo));
 6     }
 7 
 8 publicvoid ReLoadTask(string path)
 9     {
10 //this.Add(new DownloadTask(Deserialize<TaskInfo>(System.IO.File.ReadAllText(path))));
11      }
12 }

staticvoid Main(string[] args)
 2 {
 3     var downloader =new Downloader();
 4     Action<string> notify = msg => Console.Title = msg;
 5     downloader.Info += notify;
 6     downloader.Error += notify;
 7 
 8     var taskInfo =new TaskInfo();
 9     taskInfo.SaveDir =@"T:\";
10     taskInfo.LinkList.Add(new Link() { Count =4, Uri ="http://dl_dir.qq.com/qqfile/qq/QQ2010/QQIntl1.0.exe" });
11 
12     downloader.AddTask(taskInfo);
13     downloader.Start();
14 
15     var output =new StringBuilder();
16 while (true)
17     {
18         output.AppendFormat("{0}{1}\n", taskInfo.SaveDir, taskInfo.FileName);
19         var isComplete =true;
20 foreach (var item in taskInfo.RangeList)
21         {
22 if (!item.IsComplete)
23             {
24                 isComplete =false;
25             }
26             output.AppendFormat("已读取:{3}\t已写入:{4}\t完成率:{0}%\tbytes {1}-{2}\n", ((decimal)item.ReadSize / (item.To - item.From) *100).ToString("f2"), item.From, item.To, item.ReadSize, item.WriteSize);
27         }
28 if (isComplete)
29         {
30             Console.Title ="下载完成.";
31         }
32         Console.Clear();
33         Console.WriteLine(output);
34         output.Remove(0, output.Length);
35         System.Threading.Thread.Sleep(500);
36     }
37 }