python 3.x I/O多路复用

I/O多路复用

重点学习IO多路复用,因为大部分成熟的框架都使用的是IO多路复用

  • select,poll,epoll
    • select,poll,epoll都是IO多路复用的机制.　　
    • IO多路复用就是通过一种机制,一个进程可以监视多个描述符,一旦某个描述符就绪(一般是可读就绪或者可写就绪),能够通知程序进行相应的读写操作.　　
    • 但select,poll,epoll本质上都是同步I/O,因为他们都需要在读写事件就绪后自己负责进行读写(把数据从内核拷贝到用户空间),也就是说这个读写过程是阻塞的,而异步I/O则无需自己负责进行读写,异步I/O的实现会负责把数据从内核拷贝到用户空间　　
  • select
    • select函数监视的文件描述符分3类,分别是writefds,readfds和exceptfds.　　
    • 调用后select函数会阻塞,直到有描述符就绪(有数据可读,可写,或者有except(异常)),或者超时(timeout指定等待时间,如果立即返回设为null即可),函数返回.　　
    • 当select函数返回后,可以通过遍历fdset,来找到就绪的描述符.　　
    • select目前几乎在所有的平台上支持,其良好跨平台支持也是它的一个优点.　　
    • select的一个缺点在于单进程能够监视的文件描述符的数量存在最大限制,在Linux上一般为1024,可以通过修改宏定义甚至重新编译内核的方式提升这一限制,但这样也会造成效率的降低.　　
    • 调用select()函数时,select会遍历所有的fds的,所以说select的效率比较低.　　
  • poll
    • 不同与select使用三个位图来表示三个fdset的方式,poll使用一个pollfd的指针实现.　　
    • pollfd结构包含了要监视的event和发生的event,不再使用select"参数-值"传递的方式.　　
    • 同时,pollfd并没有最大数量限制(但是数量过大后性能也是会下降).和select函数一样,poll返回后,需要轮询pollfd来获取就绪的描述符.　　
    • 从上面来看,select和poll都需要在返回后,通过遍历文件描述符来获取已就绪的socket.　　
    • 事实上,同时连接的大量客户端在某一时刻可能只有很少的处于就绪状态,因此随着监视的描述符数量的增长,其效率也会线性下降.　　
    • poll最大的优点就是查询效率高,而且没有最大数量限制.　　
    • 所以说poll与select对比,pollfd虽然没有最大数量限制,但是poll的查询效率的提升实际上并不大.　　
  • epoll
    • epoll是在Linux环境下支持的,在windows下是不支持的.　　
    • epoll是在Linux2.6内核中提出的,是之前的select和poll的增强版本.相对于select和poll来说,epoll更加灵活,没有描述符限制.　　
    • epoll使用一个文件描述符管理多个描述符,将用户关系的文件描述符的事件存放到内核的一个事件表中,这样在用户空间和内核空间的copy只需一次.　　
    • epoll的查询使用了数据结构里面的一个性能很高的数据结构,就是红黑树.　　
    • 红黑树效率高,本身是非常复杂的.　　
  • 概念非常多,没必要把概念记住,只需要知道:
    • select,poll,epoll之间的关系是一个不停的进化的一个过程.　　
  • 平常使用高性能的服务器,比如unix实际上使用的就是epoll.
  • epoll并不代表一定比select要好,他们的区别:
    • 在并发高的情况下,(高并发还有一种情况,就是连接活跃度不是很高的情况下)epoll比select好.　　
      • 在并发高,特别是网站或者web系统当中,这个实际上就是一个典型的连接活跃度并不高,因为用户在连接之后,很有可能随时都有可能关闭掉连接.　　　　
      • 比如用户在请求一个网页之后,后期一直不访问这个页面.　　　　
    • 并发性不高,同时连接很活跃这种情况下,select比epoll好.　　
      • 连接活跃:　　　　
        • 就是用户在建立连接之后,不会说建立一次连接就不再管了或者说建立了之后,会明确的告诉你,我会断掉.　　　　　　
        • 在游戏开发中就是一个典型的用例.游戏一但建立连接好之后,断得时候是比较少的,会一直处于连接状态.　　　　　　
        • 而且这个连接实际上整个发送数据很灵活的,这种情况下select比epoll好　　　　　　
  • 虽然epoll在内部的优化上比select好,但是select在某些情况下是比epoll好的.

 

通过阻塞I/O转变成非阻塞I/O,并引出非阻塞I/O

  • 之前在模拟爬虫的时候,实现socket http的案例,在这个里面有几个函数是非常典型的阻塞式I/O.
    • connect()函数　　
      • socket连接的时候,connect()有三次握手.三次握手都会走网络协议.　　　　
      • 如果connect()函数不返回的话,当前的线程会一直停在connect()中.大量的等待网络连接时间.　　　　
    • send()函数　　
    • recv()函数　　
      • 如果服务器没有返回,将一直停在recv()函数中,等到网络数据的返回.　　　　
  • 阻塞式I/O的网络等待的时间远大于cpu操作的时间
  • I/O的时间与cpu操作的时间相差的级别是非常非常的大,所以说在等待的过程中,浪费大量的cpu,
  • 因为在阻塞式I/O当中cpu是属于空闲的,就意味着cpu在大量的等待,这样对cpu的利用率是非常低的.
  • 在整个计算机中最宝贵的资源就是cpu资源.这就网络编程中阻塞式I/O给我们带来最不好的地方.
  • 阻塞式I/O使用起来很简单,时间浪费很严重,就因为I/O阻塞了.
  • 为了解决I/O阻塞这个问题,操作系统提供了对应的方法,就是在socket中设置setblocking(False),将阻塞I/O变为非阻塞I/O
  •  1 import socket
 2 # urlparse的作用是对url进行解析.
 3 from urllib.parse import urlparse
 4 
 5 
 6 def get_url(url):
 7     new_url = urlparse(url)
 8 
 9     host = new_url.netloc
10 
11     path = new_url.path
12 
13     if path == "":
14         path = "/"
15 
16     client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
17 
18     # 为了解决I/O阻塞这个问题
19     client.setblocking(False)
20 
21     # setblocking(False)后,调用connect()方法的时候就会立马返回,不会等待连接好了之后才允许继续执行.
22     client.connect((host, 80))
23 
24     client.send("GET {} HTTP/1.1\r\nHost:{}\r\nConnection:close\r\n\r\n".format(path, host).encode("utf8"))
25 
26     data = b""
27     while True:
28         re_data = client.recv(1024)
29         if re_data:
30             data += re_data
31         else:
32             break
33 
34     data = data.decode("utf8")
35     html_data = data.split("\r\n\r\n")[1]
36 
37     print(html_data)
38     client.close()
39 
40 
41 if __name__ == "__main__":
42     get_url("http://www.baidu.com")
43 
44 """
45 将阻塞I/O变为非阻塞I/O执行以上代码会抛出多个异常需要程序员手动一一处理.
46 """

     

通过非阻塞式I/O,实现模拟http请求

 

  • IO多路复用是并发中应用非常多的一门技术,在介绍IO多路复用前做一个对比,没有对比就内有伤害.
  • 在使用IO多路复用之前,需要尝试一下,使用非阻塞式I/O给我们带来的一个好处.
  • 使用非阻塞式I/O,但依然在整个过程中不停的做while循环去获取connect()的状态.返回数据的时间依然是那么多.并没有提高并发.
  • 根据需求在某些场景之下,非阻塞式I/O是可以提高并发的.

 

 1 import socket
 2 # urlparse的作用是对url进行解析.
 3 from urllib.parse import urlparse
 4 
 5 
 6 def get_url(url):
 7     new_url = urlparse(url)
 8 
 9     host = new_url.netloc
10 
11     path = new_url.path
12 
13     if path == "":
14         path = "/"
15 
16     client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
17 
18     # 解决I/O阻塞这个问题,设置非阻塞
19     client.setblocking(False)
20 
21     # setblocking(False)后,调用connect()方法的时候就会立马返回,不会等待连接好了之后才允许继续执行.
22     # 但是执行代码会抛异常BlockingIOError: [WinError 10035] 无法立即完成一个非阻止性套接字操作。
23     # 这个异常是合理的
24     try:
25         client.connect((host, 80))  # 阻塞不会消耗cpu
26     except BlockingIOError as e:
27         pass
28 
29     # 将connect的异常处理后,再次运行代码,此处报错.因为connect的连接还没有准备好.
30     # OSError: [WinError 10057] 由于套接字没有连接并且
31     # (当使用一个 sendto 调用发送数据报套接字时)没有提供地址，发送或接收数据的请求没有被接受。
32     # 因为send()函数依赖于connect(),所以需要不停的去判断是否连接成功.
33     while True:
34         try:
35             client.send("GET {} HTTP/1.1\r\nHost:{}\r\nConnection:close\r\n\r\n".format(path, host).encode("utf8"))
36             # 连接成功跳出循环
37             break
38         except OSError as e:
39             pass
40 
41     data = b""
42     while True:
43         # send()出去,等待服务器的返回.如果数据没有返回的话,就调用recv()函数,也会报错
44         # BlockingIOError: [WinError 10035] 无法立即完成一个非阻止性套接字操作。
45         try:
46             re_data = client.recv(1024)
47         except BlockingIOError as e:
48             # 继续while循环,不往下执行下面的代码
49             continue
50         if re_data:
51             data += re_data
52         else:
53             break
54 
55     data = data.decode("utf8")
56     html_data = data.split("\r\n\r\n")[1]
57 
58     print(html_data)
59     client.close()
60 
61 
62 if __name__ == "__main__":
63     get_url("http://www.baidu.com")
64 
65 """
66 输出结果,没有任何问题.
67     <!DOCTYPE html><!--STATUS OK-->
68     <html>
69         <head>
70         ...省略...
71         </head>
72         
73         <body ...省略...>
74         </body>
75     </html>
76 """

 

如何使用IO多路复用 

通过select,实现模拟http请求.

  • import select 在实际开发中使用的并不多.
    • select包下有一个select()函数
    • def select(rlist, wlist, xlist, timeout=None):　　
      • rlist : read list　　　　
      • wlist : write list　　　　　　
      • xlist : except list
      • timeout : 超时
  • from selectors import DefaultSelector
    • 使用更多的是Selector包下的DefaultSelector,来完成select的IO多路复用编程.
    • 1.DefaultSelector是在select的基础上包装的一个包,这个包使用更加的方便.
      • 源码:
      •  1                 DefaultSelector = SelectSelector
 2                 class SelectSelector(_BaseSelectorImpl):
 3                     def __init__(self):...
 4                     def register(self, fileobj, events, data=None):...
 5                     def unregister(self, fileobj):...
 6 
 7                     if sys.platform == 'win32':
 8 
 9                         # 内部的实现是select.select(rlist, wlist, xlist, timeout=None)
10                         # 但是将select()函数包装的更加好用.
11                         def _select(self, r, w, _, timeout=None):
12                             r, w, x = select.select(r, w, w, timeout)
13                             return r, w + x, []
14                     else:
15                         _select = select.select
16 
17                     def select(self, timeout=None):...
18                         return ready
    • 2.在windows下有select,但是epoll在Linux下使用,有了Selector,到底是选用epoll还是poll,这都不需要我们关心,会根据平台来自动选择.　　
    1. 如果在windows下运行代码就会选择select,　　　　
    2. 如果在Linux下运行代码就会选择epoll　　
    • 所以推荐大家使用DefaultSelector,这个DefaultSelector除了可以选择IO多路复用的方法之外,还提供了注册机制　　
  • 不使用函数的方式来完成IO多路复用编程的代码(select实现http请求),用类(class Fetcher:)来实现???
    • 是因为selector注册时,需要一个回调函数,而回调函数需要使用之前建立好的socket和其他的变量,　　
    • 如果使用函数的模式来编写IO多路复用代码,回调函数就访问不到建立好的socket和其他的变量.

    • 在定义某些方法的时候,主要针对回调函数,在回调函数里面使用了一些变量,
      如果把这些变量放到类里面,因为在调用回调函数的时候,会把实例传递进来,直接可以取实例里面的变量,
      如果使用函数的话是做不到的或者说是非常难的.所以要学会使用这种方式来处理自己平常的编程过程中所遇到的问题.

  • 事件循环:
    • 如果没有接触回调模式之前,会有人想当然的认为,这里写的回调函数(readable()),当socket变成可读的时候,操作系统会自动调用回调函数readable().大家一定不要这样认为,这是一种错误的思维.　　
    • 回调实际上仍然需要我们自己来做的,就是说当socket变成可读的状态,我们需要自己来调用回调函数.　　
  1 """
  2 回调 + 事件循环 + select(poll/epoll)模式来实现模拟http请求
  3 """
  4 
  5 import socket
  6 # urlparse的作用是对url进行解析.
  7 from urllib.parse import urlparse
  8 from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE
  9 
 10 # 实例化全局selector
 11 selector = DefaultSelector()
 12 
 13 
 14 # Fetcher : 提取器
 15 class Fetcher:
 16 
 17     # 定义一个回调函数
 18     # 告诉selector,如果监控的socket是可以写的时候,就是可以发送数据的时候,我要去执行一个什么逻辑.
 19     def connected(self, key):
 20         # 注销掉
 21         # key.fd在事件循环中会详细说明,为什么这样使用.
 22         selector.unregister(key.fd)
 23 
 24         # 此处说明,为什么把client放到self里面,因为回调函数里面需要用到client,如果把client声明一个局部变量的话,
 25         # 回调函数是拿不到client
 26         # 此处不需要将send()发放进行try...except,是因为使用了事件监听.
 27         # 事件监听时,调用connected,就证明connected处于就绪的状态,可以直接执行send()函数,不再需要像之前一样不停的轮询.
 28         self.client.send(
 29             "GET {} HTTP/1.1\r\nHost:{}\r\nConnection:close\r\n\r\n".format(self.path, self.host).encode("utf8"))
 30 
 31         # send()之后,需要接收数据,就需要再一次的监听socket,监听socket是否是可读的状态
 32         # 接收数据(recv())属于读事件
 33         selector.register(self.client.fileno(), EVENT_READ, self.readable)
 34 
 35     def readable(self, key):
 36         """
 37         定义一个回调函数
 38         当监控到socket可读的时候,执行的逻辑
 39 
 40         readable函数是在连接建立好可以读的情况下调用.
 41         只要socket是可读,会继续调用readable回调函数.继续调用之后,会再一次调用readable函数,再把数据加入data里面
 42         所以data不能放到局部变量里面,如果设置data为局部变量,下一次再调用回调函数,data仍然变空了data = b""
 43         """
 44 
 45         re_data = self.client.recv(1024)
 46         if re_data:
 47             self.data += re_data
 48         else:
 49             # 读出数据为空的时候,代表数据已经读取完毕,取消注册
 50             selector.unregister(key.fd)
 51             # 数据已经读取完毕,进行解码,打印
 52             data = self.data.decode("utf8")
 53             html_data = data.split("\r\n\r\n")[1]
 54             print(html_data)
 55             self.client.close()
 56 
 57     def get_url(self, url):
 58         new_url = urlparse(url)
 59         self.host = new_url.netloc
 60         self.path = new_url.path
 61         # 所以需要把data放到实例变量里面
 62         self.data = b""
 63 
 64         if self.path == "":
 65             # 将path修改为"/",这个是http协议的请求方式
 66             self.path = "/"
 67 
 68         # 将client放到self里面,为什么要将client放到self里面???
 69         # 一会写到回调的时候就会明白.
 70         self.client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 71 
 72         # 需要使用非阻塞式I/O,不要使用阻塞式I/O,如果使用阻塞式I/O,就达不到想要的效果.
 73         # 达不到想去将中间连接建立的时间给节省掉的一个效果
 74         self.client.setblocking(False)
 75 
 76         try:
 77             self.client.connect((self.host, 80))  # 阻塞不会消耗cpu
 78         except BlockingIOError as e:
 79             pass
 80 
 81         # 使用selector注册,将socket注册到selector中去.
 82         #
 83         # def register(self, fileobj, events, data=None)
 84         #     key = super().register(fileobj, events, data)
 85         #         if events & EVENT_READ:
 86         #             self._readers.add(key.fd)
 87         #         if events & EVENT_WRITE:
 88         #             self._writers.add(key.fd)
 89         #         return key
 90         #
 91         #   fileobj : 指的是socket的文件描述符对象
 92         #   events  : 事件,socket是有事件发生的.EVENT_READ和EVENT_WRITE
 93         #   data    ：回调函数,回调函数很重.
 94         #
 95         #  fileno(): 文件描述符
 96         #  事件:首先我们需要知道的是连接建立好之后,需要send()发送数据,发送数据(send)属于写事件,向socket写数据的事件
 97         #  回调函数:
 98         #      回调函数很重要,selector模式都是回调函数组成的,就是当selector变为可写的时候,我们应该执行什么逻辑,
 99         #      需要定义一个函数.
100         #      只能写成回调模式,这个模式很重要,因为我们无法在注册语句的后面写逻辑.
101         #      如果写到注册语句的后面,实际上就是一个阻塞式I/O
102         selector.register(self.client.fileno(), EVENT_WRITE, self.connected)
103         # def connected(self, key):如果这样写,就是一个阻塞式I/O
104 
105 
106 def loop():
107     """
108     loop里面需要自己调用select,去不停的判断,我们哪一个socket已经准备好了,是可读还是可写.
109     如果某一个socket可读或者可写,我们需要去调用对应的回调函数
110 
111     注意:回调是我们自己来做的
112 
113     1.select本身是不支持register模式的,select里只能够去传递一些比如说readable,writeable这些socket句柄进去的.
114       然后操作系统会将一些可读的或者可写的文件句柄给返回回来,但是这个可读或者可写,他本身的回调函数是什么?
115       select本身是不提供的,selector是封装select本身的.所以selector可以完成register.
116       这样的话,我们在拿到某一个socket文件句柄的时候,我们就知道我们应该回调哪一个函数,这就是为什么我们要使用selector
117     2.socket状态变化以后的回调是由程序员来完成的.不是由操作系统完成的.操作系统完成是异步IO,aio
118 
119 
120     这个循环时主循环,会一直运行,会一直在selector里面查询key,mask
121 
122     事件循环,不停的请求socket的状态并调用对应的回调函数,比如我们用到的twisted,Tornado,jevent,协成,think I/O他们都是这种模式
123 
124     注意:
125         事件循环这种模式,在使用IO多路复用的时候都会存在.
126 
127     模式格式:
128         回调 + 事件循环 + select(poll/epoll)
129     """
130 
131     while True:
132         # 此处的select()方法不是select包下的select(4个参数)方法.
133         # 由于selector封装了select,selector提供了register()方法,所以说在注册进来的时候,我们就不需要去传递,哪些socket,
134         # 因为我们已经注册了.selector本身可以找到文件描述符,可读写事件,以及回调函数.
135         #
136         #     def select(self, timeout=None):
137         #         timeout = None if timeout is None else max(timeout, 0)
138         #         ready = []
139         #         try:
140         #             r, w, _ = self._select(self._readers, self._writers, [], timeout)
141         #         except InterruptedError:
142         #             return ready
143         #         r = set(r)
144         #         w = set(w)
145         #         for fd in r | w:
146         #             events = 0
147         #             if fd in r:
148         #                 events |= EVENT_READ
149         #             if fd in w:
150         #                 events |= EVENT_WRITE
151         #
152         #             key = self._key_from_fd(fd)
153         #             if key:
154         #
155         #                 # list中存放的是tuple
156         #                 # tuple存放的是 key ,events 和 key.events的位运算
157         #                 # key是SelectorKey数据类型,
158         #                 # key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
159         #                 # SelectorKey = namedtuple('SelectorKey', ['fileobj', 'fd', 'events', 'data'])
160         #                 ready.append((key, events & key.events))
161         #
162         #         # 返回一个list,list里面存放一个tuple,tuple里面存放两个元素.
163         #         return ready
164         #
165         ready = selector.select()
166         for key, mask in ready:
167             # 可以通过key,获取到当时注册的回调函数data
168             call_back = key.data
169 
170             # 此处的key是namedtuple('SelectorKey', ['fileobj', 'fd', 'events', 'data'])
171             # key里面有一个fd,fd就是当时注册的时候,self.client.fileno()的返回值
172             call_back(key)
173 
174 
175 if __name__ == "__main__":
176     fetcher = Fetcher()
177     fetcher.get_url("http://www.baidu.com")
178     # 必须启动loop,不启动loop是没有效果的.
179     # 不启动loop的话,执行完get_url函数后不会再运行下面的代码
180     # 整个事件的回调是由事件循环来驱动的,这是一个非常经典的模式
181     loop()
182 
183 """
184 以上的代码在Linux下运行是不会报错的,因为Linux使用的是epoll,
185 而在windows下运行会报错,因为windows使用的是select
186 """

 

windows系统下运行select出现OSError的异常

  • OSError: [WinError 10022] 提供了一个无效的参数.
  • 这个异常是在调用ready = selector.select()出现的问题,这个问题实际上是select()函数引起的.
  • 在windows系统下运行此代码,默认使用的是select.
  • select()函数在windows系统下里面有一个问题:
    • 在loop()函数不断的循环的时候,调用select()函数,因为只有一个url,当第二次执行select()函数仍需一个url的host,此时url为空导致获取不到host,所以系统报错:提供一个无效的参数.　　
    • 如果给select(r, w, w, timeout)函数传递的参数为空列表的话,会抛异常　　

 

  1 """
  2 使用IO多路复用实现http请求.
  3     Windows运行select出现的问题.
  4 """
  5 
  6 import socket
  7 # urlparse的作用是对url进行解析.
  8 from urllib.parse import urlparse
  9 from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE
 10 
 11 # 实例化全局selector
 12 selector = DefaultSelector()
 13 
 14 
 15 # Fetcher : 提取器
 16 class Fetcher:
 17 
 18     def connected(self, key):
 19         selector.unregister(key.fd)
 20 
 21         self.client.send(
 22             "GET {} HTTP/1.1\r\nHost:{}\r\nConnection:close\r\n\r\n".format(self.path, self.host).encode("utf8"))
 23 
 24         selector.register(self.client.fileno(), EVENT_READ, self.readable)
 25 
 26     def readable(self, key):
 27 
 28         re_data = self.client.recv(1024)
 29         if re_data:
 30             self.data += re_data
 31         else:
 32             selector.unregister(key.fd)
 33             data = self.data.decode("utf8")
 34             html_data = data.split("\r\n\r\n")[1]
 35             print(html_data)
 36             self.client.close()
 37 
 38     def get_url(self, url):
 39         new_url = urlparse(url)
 40         self.host = new_url.netloc
 41         self.path = new_url.path
 42         self.data = b""
 43         if self.path == "":
 44             self.path = "/"
 45 
 46         self.client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 47 
 48         self.client.setblocking(False)
 49 
 50         try:
 51             self.client.connect((self.host, 80))  # 阻塞不会消耗cpu
 52         except BlockingIOError as e:
 53             pass
 54 
 55         selector.register(self.client.fileno(), EVENT_WRITE, self.connected)
 56 
 57 
 58 def loop():
 59     while True:
 60         ready = selector.select()
 61         for key, mask in ready:
 62             call_back = key.data
 63             call_back(key)
 64 
 65 
 66 if __name__ == "__main__":
 67     fetcher = Fetcher()
 68     fetcher.get_url("http://www.baidu.com")
 69     loop()
 70 
 71 """
 72 输出结果:
 73     输出html所有的内容,但是抛出异常
 74     <html>
 75     <body link="#0000cc"><div id="wrapper" style="display:none;"><div id="u"><a href="//www.baidu.com/gaoji/
 76         preferences.html"  onmousedown="return user_c({'fm':'set','tab':'setting','login':'0'})">搜索设置</a>|
 77         <div id="head"><div class="s_nav"><a href="/" class="s_logo" ......省略>
 78     </body>
 79     </html>
 80     
 81     File "F:/C/PythonProject/PythonHighLevelAndIo/test/chapter12/select_http_request_02.py", line 70, in <module>
 82     loop()
 83     
 84     File "F:/C/PythonProject/PythonHighLevelAndIo/test/chapter12/select_http_request_02.py", line 60, in loop
 85     ready = selector.select()
 86     
 87     File "D:\python3\lib\selectors.py", line 323, in select
 88     r, w, _ = self._select(self._readers, self._writers, [], timeout)
 89     
 90     File "D:\python3\lib\selectors.py", line 314, in _select
 91     r, w, x = select.select(r, w, w, timeout)
 92     
 93 OSError: [WinError 10022] 提供了一个无效的参数.
 94 
 95 这个异常是在调用ready = selector.select()出现的问题,这个问题实际上是select()函数引起的.
 96 在windows系统下运行此代码,默认使用的是select.
 97 select()函数在windows系统下里面有一个问题:
 98     在loop()函数不断的循环的时候,调用select()函数,因为只有一个url,当第二次执行select()函数仍需一个url的host,
 99     此时url为空导致获取不到host,所以系统报错:提供一个无效的参数.
100     如果给select(r, w, w, timeout)函数传递的参数为空列表的话,会抛异常
101 
102 如果将此代码放到Linux系统下运行的时候,在调用DefaultSelector的时候,select会默认选择epoll,然而epoll就不会抛出异常.
103 而在windows下就会一直阻塞在那里,所以说抛出异常的问题只有在windows下才能出现,在Linux下是不会出现的.
104 """

 

select + 回调 + 事件循环的好处

  • 就是并发性高
  • 目前使用的是单线程模式,并没有使用多线程模式抓取我们的url,但是案例里面的单线程这种模式并发性很高.
  • 为什么并发性很高呢???
  • 首先看到驱动整个程序运行的主要是事件循环(loop()函数),也可以理解为心脏,这个心脏不断的跳动,跳动的时候就会知道应该去调用什么代码,去运行什么方法,这里面不会再阻塞到我们去建立连接或等待网络请求的过程中,
    实际上是不停的向操作系统请求哪些socket已经准备好了,一但准备好了之后,我立马就来执行回调方法,通过回调函数可以看到,回调函数里面是没有去做那种费I/O的同步操作的,在回调函数里面都是CPU的操作.CPU的操作是远远高于I/O操作,特别是我们的网络I/O,
    他俩之间是没法对比的.所以说相对我们之前的线程切换,我们CPU的操作,现在的代码中没有线程的切换,我们以前要同时抓取多个url,是需要使用多个线程,比如说有10个url要取10线程,10个线程来回的切换,现在我们有了这种回调和事件循环的模式,我们就在一个线程里面,
    当一个url的连接正在建立的时候,因为是一个非阻塞的方法,就会立马返回,返回之后就会立马注册到selector里面,而selector的select()方法专门找那种准备好的socket,有可能是之前的任意一个url,比如说他是可读或者可写,他的状态都是有可能的,他只会去找那种准备好的socket,
    然后执行他的回调方法,所以说他就不会再有去等待网络I/O的情况,除非所有的url都在阻塞.
  • 所以说在 while not stop: 里面使用单线程的模式,他实际上省去了线程切换的开销,
                            ready = selector.select()
                            for key, mask in ready:
                                    call_back = key.data
                                    call_back(key)
  • 还有就是我们的内存,一个线程占用的内存是远远高于回调函数这种模式的,回调函数无非就是一个指向函数句柄而已,所以说在并发的时候,我们系统里面存在几十个线程以后,线程间的切换就会变的很慢.
    但是我们这种回调模式,成千上万个存在都不会有任何问题,都不会有内存消耗的问题以及切换问题.因为回调函数是单线程的模式.
  • 这个也是我们后续的协成里面的一个核心点,我们的tornado,jevent,异步I/O的框架他们的模式都是select+回调+事件循环.
  • select+回调+事件循环模式,是编程的核心.

 

解决windows下执行select出现异常的问题

 1 import socket
 2 # urlparse的作用是对url进行解析.
 3 from urllib.parse import urlparse
 4 from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE
 5 
 6 # 实例化全局selector
 7 selector = DefaultSelector()
 8 # 1.windows
 9 urls = ["http://www.baidu.com"]
10 # 4.windows,传递一个全局的变量
11 stop = False
12 
13 
14 # Fetcher : 提取器
15 class Fetcher:
16 
17     def connected(self, key):
18         selector.unregister(key.fd)
19 
20         self.client.send(
21             "GET {} HTTP/1.1\r\nHost:{}\r\nConnection:close\r\n\r\n".format(self.path, self.host).encode("utf8"))
22 
23         selector.register(self.client.fileno(), EVENT_READ, self.readable)
24 
25     def readable(self, key):
26 
27         re_data = self.client.recv(1024)
28         if re_data:
29             self.data += re_data
30         else:
31             selector.unregister(key.fd)
32             data = self.data.decode("utf8")
33             html_data = data.split("\r\n\r\n")[1]
34             print(html_data)
35             self.client.close()
36 
37             # 2.windows,如果将数据处理完成,说明一个url处理完成,就直接将自己抓取的url给删除掉
38             # 删除掉抓取的url存在一个问题,我们并没有把抓取到的url给他保存到实例变量里面.
39             urls.remove(self.spider_url)
40 
41             # 3.windows,抓取到一个url之后,就判断urls是否为空
42             if not urls:
43                 # 5.windows,此处很容易出现错误,达不到预期的效果.如果写成这样stop = True,会创建一个局部变量,然后给stop赋值True
44                 # 所以局部变量stop影响不了全局的stop
45                 global stop
46                 # 如果urls列表为空,将stop改为True
47                 stop = True
48 
49     def get_url(self, url):
50         # 3.windows 在这个地方,保存抓取到的url
51         # 为什么不用new_url???是因为new_url接收的是urlparse()函数返回的对象,所以new_url和传递进来的url已经不一样了.
52         self.spider_url = url
53         new_url = urlparse(url)
54         self.host = new_url.netloc
55         self.path = new_url.path
56         self.data = b""
57         if self.path == "":
58             self.path = "/"
59 
60         self.client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
61 
62         self.client.setblocking(False)
63 
64         try:
65             self.client.connect((self.host, 80))  # 阻塞不会消耗cpu
66         except BlockingIOError as e:
67             pass
68 
69         selector.register(self.client.fileno(), EVENT_WRITE, self.connected)
70 
71 
72 def loop():
73     # 6.windows,loop最开始不停的循环,当urls为空时,将loop停止
74     # stop为false时,不停的调用select
75     while not stop:
76         ready = selector.select()
77         for key, mask in ready:
78             call_back = key.data
79             call_back(key)
80 
81 
82 if __name__ == "__main__":
83     fetcher = Fetcher()
84     fetcher.get_url("http://www.baidu.com")
85     loop()
86 
87 """
88 输出结果:
89     获取到完整的html的所有数据,并且不会抛出任何的异常.
90     
91     这就是使用stop模式给我们带来的好处
93 """

 

可能有人还会觉得select+回调+事件循环模式并发不会很高,为了说明select+回调+事件循环模式并发性是高于同步的方式.

  • 需求:准备获取20个url,通过同步方式和select+回调+事件循环模式来展示一下
  • 通过两个案例代码运行的时间对比:
    • 同步的方式的运行时间远远超出select+回调+事件循环模式的运行时间.　　
  1 """
  2 通过select+回调+事件循环模式来完成并发. (使用的是单线程来完成的)
  4 """
  5 
  6 import socket
  7 # urlparse的作用是对url进行解析.
  8 from urllib.parse import urlparse
  9 from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE
 10 
 11 # 实例化全局selector
 12 selector = DefaultSelector()
 13 urls = []
 14 stop = False
 15 
 16 
 17 # Fetcher : 提取器
 18 class Fetcher:
 19 
 20     def connected(self, key):
 21         selector.unregister(key.fd)
 22 
 23         self.client.send(
 24             "GET {} HTTP/1.1\r\nHost:{}\r\nConnection:close\r\n\r\n".format(self.path, self.host).encode("utf8"))
 25 
 26         selector.register(self.client.fileno(), EVENT_READ, self.readable)
 27 
 28     def readable(self, key):
 29 
 30         re_data = self.client.recv(1024)
 31         if re_data:
 32             self.data += re_data
 33         else:
 34             selector.unregister(key.fd)
 35             data = self.data.decode("utf8")
 36             html_data = data.split("\r\n\r\n")[1]
 37             print(html_data)
 38             self.client.close()
 39 
 40             urls.remove(self.spider_url)
 41 
 42             if not urls:
 43                 global stop
 44                 stop = True
 45 
 46     def get_url(self, url):
 47         self.spider_url = url
 48         new_url = urlparse(url)
 49         self.host = new_url.netloc
 50         self.path = new_url.path
 51         self.data = b""
 52         if self.path == "":
 53             self.path = "/"
 54 
 55         self.client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 56 
 57         self.client.setblocking(False)
 58 
 59         try:
 60             self.client.connect((self.host, 80))  # 阻塞不会消耗cpu
 61         except BlockingIOError as e:
 62             pass
 63 
 64         selector.register(self.client.fileno(), EVENT_WRITE, self.connected)
 65 
 66 
 67 def loop():
 68     while not stop:
 69         ready = selector.select()
 70         for key, mask in ready:
 71             call_back = key.data
 72             call_back(key)
 73 
 74 
 75 if __name__ == "__main__":
 76 
 77     import time
 78 
 79     start_time = time.time()
 80 
 81     for url in range(1, 21):
 82         # 获取某一个商品的详情的url,随机获取前20个商品详情的url
 83         url = "http://shop.projectsedu.com/goods/{}/".format(url)
 84         urls.append(url)
 85         fetcher = Fetcher()
 86         fetcher.get_url(url)
 87 
 88     loop()
 89 
 90     print(time.time() - start_time)
 91 
 92 """
 93 输出结果:
 94     {"id":1,"category":{"id":129,"sub_cat":[],"name":"根茎类","code":"gjl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889889.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889264.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889726.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889018.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889287.jpg"}],"goods_sn":"","name":"新鲜水果甜蜜香脆单果约800克","click_num":5078,"sold_num":0,"fav_num":0,"goods_num":-16,"market_price":232.0,"shop_price":156.0,"goods_brief":"食用百香果可以增加胃部饱腹感，减少余热量的摄入，还可以吸附胆固醇和胆汁之类有机分子，抑制人体对脂肪的吸收。因此，长期食用有利于改善人体营养吸收结构，降低体内脂肪，塑造健康优美体态。","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889889.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:52"}
 95     {"id":4,"category":{"id":129,"sub_cat":[],"name":"根茎类","code":"gjl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/47_P_1448946213263.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/47_P_1448946213157.jpg"}],"goods_sn":"","name":"日本蒜蓉粉丝扇贝270克6只装","click_num":3394,"sold_num":0,"fav_num":-1,"goods_num":0,"market_price":156.0,"shop_price":108.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/47_P_1448946213263.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
 96     {"id":16,"category":{"id":145,"sub_cat":[],"name":"饮料/水","code":"yls","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/14_P_1448947354031.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/14_P_1448947354433.jpg"}],"goods_sn":"","name":"52度水井坊臻酿八號500ml","click_num":2758,"sold_num":0,"fav_num":0,"goods_num":-3,"market_price":43.0,"shop_price":36.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/14_P_1448947354031.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
 97     {"id":9,"category":{"id":131,"sub_cat":[],"name":"菌菇类","code":"jgl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/6_P_1448945167279.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/6_P_1448945167015.jpg"}],"goods_sn":"","name":"潮香村澳洲进口牛排家庭团购套餐20片","click_num":3013,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":239.0,"shop_price":199.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/6_P_1448945167279.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
 98     {"id":18,"category":{"id":138,"sub_cat":[],"name":"葡萄酒","code":"ptj","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/46_P_1448946598711.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/46_P_1448946598301.jpg"}],"goods_sn":"","name":"双响炮洋酒JimBeamwhiskey美国白占边","click_num":2721,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":38.0,"shop_price":28.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/46_P_1448946598711.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
 99     {"id":19,"category":{"id":145,"sub_cat":[],"name":"饮料/水","code":"yls","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/21_P_1448946793276.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/21_P_1448946793153.jpg"}],"goods_sn":"","name":"西夫拉姆进口洋酒小酒版","click_num":2709,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":55.0,"shop_price":46.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/21_P_1448946793276.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
100     {"id":7,"category":{"id":132,"sub_cat":[],"name":"进口生鲜","code":"jksx","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/8_P_1448945032810.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/8_P_1448945032646.jpg"}],"goods_sn":"","name":"五星眼肉牛排套餐8片装原味原切生鲜牛肉","click_num":3123,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":150.0,"shop_price":125.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/8_P_1448945032810.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
101     {"id":2,"category":{"id":116,"sub_cat":[{"id":117,"sub_cat":[],"name":"参鲍","code":"cb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":118,"sub_cat":[],"name":"鱼","code":"yu","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":119,"sub_cat":[],"name":"虾","code":"xia","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":120,"sub_cat":[],"name":"蟹/贝","code":"xb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116}],"name":"海鲜水产","code":"hxsc","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/2_P_1448945810202.jpg"},{"image":"http://shop.projectsedu.com/media/15_P_1448947257324.jpg"},{"image":"http://shop.projectsedu.com/media/2_20170719161405_249.jpg"},{"image":"http://shop.projectsedu.com/media/9_P_1448944791617.jpg"}],"goods_sn":"sssss","name":"田然牛肉大黄瓜条生鲜牛肉冷冻真空黄牛","click_num":4121,"sold_num":100,"fav_num":0,"goods_num":-5,"market_price":106.0,"shop_price":88.0,"goods_brief":"前腿+后腿+羊排共8斤，原生态大山放牧羊羔，曾经的皇室贡品，央视推荐，2005年北京招待全球财金首脑。五层专用包装箱+真空包装+冰袋+保鲜箱+顺丰冷链发货，路途保质期8天","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/> </p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/> </p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/> </p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/2_P_1448945810202.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:00"}
102     {"id":3,"category":{"id":124,"sub_cat":[{"id":125,"sub_cat":[],"name":"生菜","code":"sc","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":124},{"id":126,"sub_cat":[],"name":"菠菜","code":"bc","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":124},{"id":127,"sub_cat":[],"name":"圆椒","code":"yj","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":124},{"id":128,"sub_cat":[],"name":"西兰花","code":"xlh","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":124}],"name":"叶菜类","code":"ycl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/7_P_1448945104883.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/7_P_1448945104734.jpg"}],"goods_sn":"","name":"酣畅家庭菲力牛排10片澳洲生鲜牛肉团购套餐","click_num":3305,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":286.0,"shop_price":238.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/7_P_1448945104883.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
103     {"id":17,"category":{"id":141,"sub_cat":[{"id":142,"sub_cat":[],"name":"其他品牌","code":"qtpp","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":141},{"id":143,"sub_cat":[],"name":"黄酒","code":"hj","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":141},{"id":144,"sub_cat":[],"name":"养生酒","code":"ysj","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":141}],"name":"其他酒品","code":"qtjp","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/12_P_1448947547989.jpg"}],"goods_sn":"","name":"53度茅台仁酒500ml","click_num":2698,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":190.0,"shop_price":158.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/12_P_1448947547989.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
104     {"id":14,"category":{"id":130,"sub_cat":[],"name":"茄果类","code":"qgl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/images/201705/goods_img/53_P_1495068879687.jpg"}],"goods_sn":"","name":"帐篷出租","click_num":2770,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":120.0,"shop_price":100.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/53_P_1495068879687.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
105     {"id":6,"category":{"id":130,"sub_cat":[],"name":"茄果类","code":"qgl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/4_P_1448945381985.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/4_P_1448945381013.jpg"}],"goods_sn":"","name":"乌拉圭进口牛肉卷特级肥牛卷","click_num":3081,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":90.0,"shop_price":75.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/4_P_1448945381985.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
106     {"id":15,"category":{"id":146,"sub_cat":[{"id":147,"sub_cat":[],"name":"白兰地","code":"bld","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":146},{"id":148,"sub_cat":[],"name":"威士忌","code":"wsj","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":146}],"name":"红酒","code":"hj","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/16_P_1448947194687.jpg"}],"goods_sn":"","name":"52度茅台集团国隆双喜酒500mlx6","click_num":2837,"sold_num":0,"fav_num":0,"goods_num":-5,"market_price":23.0,"shop_price":19.0,"goods_brief":"贵州茅台酒厂（集团）保健酒业有限公司生产，是以“龙”字打头的酒水。中国龙文化上下8000年，源远而流长，龙的形象是一种符号、一种意绪、一种血肉相联的情感。","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/16_P_1448947194687.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
107     {"id":10,"category":{"id":129,"sub_cat":[],"name":"根茎类","code":"gjl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791617.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791129.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791077.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791229.jpg"}],"goods_sn":"","name":"爱食派内蒙古呼伦贝尔冷冻生鲜牛腱子肉1000g","click_num":2741,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":202.0,"shop_price":168.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791617.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
108     {"id":13,"category":{"id":121,"sub_cat":[{"id":122,"sub_cat":[],"name":"松花蛋/咸鸭蛋","code":"xhd_xyd","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":121},{"id":123,"sub_cat":[],"name":"鸡蛋","code":"jd","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":121}],"name":"蛋制品","code":"dzp","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/5_P_1448945270390.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/5_P_1448945270067.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/5_P_1448945270432.jpg"}],"goods_sn":"","name":"澳洲进口安格斯牛切片上脑牛排1000g","click_num":2752,"sold_num":0,"fav_num":0,"goods_num":-1,"market_price":144.0,"shop_price":120.0,"goods_brief":"澳大利亚是国际公认的没有疯牛病和口蹄疫的国家。为了保持澳大利亚产品的高标准，澳大利亚牛肉业和各级政府共同努力简历了严格的标准和体系，以保证生产的整体化和产品的可追溯性","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/5_P_1448945270390.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
109     {"id":11,"category":{"id":111,"sub_cat":[{"id":112,"sub_cat":[],"name":"羊肉","code":"yr","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":111},{"id":113,"sub_cat":[],"name":"禽类","code":"ql","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":111},{"id":114,"sub_cat":[],"name":"猪肉","code":"zr","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":111},{"id":115,"sub_cat":[],"name":"牛肉","code":"nr","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":111}],"name":"精品肉类","code":"jprl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/3_P_1448945490837.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/3_P_1448945490084.jpg"}],"goods_sn":"","name":"澳洲进口牛尾巴300g新鲜肥牛肉","click_num":2818,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":306.0,"shop_price":255.0,"goods_brief":"新鲜羊羔肉整只共15斤，原生态大山放牧羊羔，曾经的皇室贡品，央视推荐，2005年北京招待全球财金首脑。五层专用包装箱+真空包装+冰袋+保鲜箱+顺丰冷链发货，路途保质期8天","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/3_P_1448945490837.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
110     {"id":20,"category":{"id":139,"sub_cat":[],"name":"洋酒","code":"yj","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/15_P_1448947257324.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/15_P_1448947257580.jpg"}],"goods_sn":"","name":"茅台53度飞天茅台500ml","click_num":532,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":22.0,"shop_price":18.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/15_P_1448947257324.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
111     {"id":12,"category":{"id":116,"sub_cat":[{"id":117,"sub_cat":[],"name":"参鲍","code":"cb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":118,"sub_cat":[],"name":"鱼","code":"yu","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":119,"sub_cat":[],"name":"虾","code":"xia","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":120,"sub_cat":[],"name":"蟹/贝","code":"xb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116}],"name":"海鲜水产","code":"hxsc","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/48_P_1448943988970.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/48_P_1448943988898.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/48_P_1448943988439.jpg"}],"goods_sn":"","name":"新疆巴尔鲁克生鲜牛排眼肉牛扒1200g","click_num":2717,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":126.0,"shop_price":88.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/48_P_1448943988970.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
112     {"id":5,"category":{"id":116,"sub_cat":[{"id":117,"sub_cat":[],"name":"参鲍","code":"cb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":118,"sub_cat":[],"name":"鱼","code":"yu","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":119,"sub_cat":[],"name":"虾","code":"xia","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":120,"sub_cat":[],"name":"蟹/贝","code":"xb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116}],"name":"海鲜水产","code":"hxsc","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/10_P_1448944572085.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/10_P_1448944572532.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/10_P_1448944572872.jpg"}],"goods_sn":"","name":"内蒙新鲜牛肉1斤清真生鲜牛肉火锅食材","click_num":3225,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":106.0,"shop_price":88.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/10_P_1448944572085.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
113     {"id":8,"category":{"id":116,"sub_cat":[{"id":117,"sub_cat":[],"name":"参鲍","code":"cb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":118,"sub_cat":[],"name":"鱼","code":"yu","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":119,"sub_cat":[],"name":"虾","code":"xia","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":120,"sub_cat":[],"name":"蟹/贝","code":"xb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116}],"name":"海鲜水产","code":"hxsc","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/11_P_1448944388277.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/11_P_1448944388034.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/11_P_1448944388201.jpg"}],"goods_sn":"","name":"澳洲进口120天谷饲牛仔骨4份原味生鲜","click_num":3010,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":31.0,"shop_price":26.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/11_P_1448944388277.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
114     
115 运行时间:(在网络接口里面设置了sleep 1秒,才能看到效果,实际上真正的接口不会返回那么慢)
116     2.290130853652954s
117     
118 """

 

 

 1 """
 2 同步的方式
 3 """
 4 
 5 import socket
 6 from urllib.parse import urlparse
 7 
 8 
 9 def get_url(url):
10     new_url = urlparse(url)
11 
12     host = new_url.netloc
13 
14     path = new_url.path
15 
16     if path == "":
17         path = "/"
18 
19     client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
20 
21     # client.setblocking(False)
22 
23     client.connect((host, 80))
24 
25     client.send("GET {} HTTP/1.1\r\nHost:{}\r\nConnection:close\r\n\r\n".format(path, host).encode("utf8"))
26 
27     data = b""
28     while True:
29         re_data = client.recv(1024)
30         if re_data:
31             data += re_data
32         else:
33             break
34 
35     data = data.decode("utf8")
36     html_data = data.split("\r\n\r\n")[1]
37 
38     print(html_data)
39     client.close()
40 
41 
42 if __name__ == "__main__":
43     import time
44 
45     start_time = time.time()
46 
47     for url in range(1, 21):
48         # 获取某一个商品的详情的url,随机获取前20个商品详情的url
49         url = "http://shop.projectsedu.com/goods/{}/".format(url)
50         get_url(url)
51 
52     print(time.time() - start_time)
53 
54 """
55 输出结果:
56     {"id":1,"category":{"id":129,"sub_cat":[],"name":"根茎类","code":"gjl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889889.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889264.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889726.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889018.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889287.jpg"}],"goods_sn":"","name":"新鲜水果甜蜜香脆单果约800克","click_num":5079,"sold_num":0,"fav_num":0,"goods_num":-16,"market_price":232.0,"shop_price":156.0,"goods_brief":"食用百香果可以增加胃部饱腹感，减少余热量的摄入，还可以吸附胆固醇和胆汁之类有机分子，抑制人体对脂肪的吸收。因此，长期食用有利于改善人体营养吸收结构，降低体内脂肪，塑造健康优美体态。","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/1_P_1449024889889.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:52"}
57     {"id":2,"category":{"id":116,"sub_cat":[{"id":117,"sub_cat":[],"name":"参鲍","code":"cb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":118,"sub_cat":[],"name":"鱼","code":"yu","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":119,"sub_cat":[],"name":"虾","code":"xia","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":120,"sub_cat":[],"name":"蟹/贝","code":"xb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116}],"name":"海鲜水产","code":"hxsc","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/2_P_1448945810202.jpg"},{"image":"http://shop.projectsedu.com/media/15_P_1448947257324.jpg"},{"image":"http://shop.projectsedu.com/media/2_20170719161405_249.jpg"},{"image":"http://shop.projectsedu.com/media/9_P_1448944791617.jpg"}],"goods_sn":"sssss","name":"田然牛肉大黄瓜条生鲜牛肉冷冻真空黄牛","click_num":4122,"sold_num":100,"fav_num":0,"goods_num":-5,"market_price":106.0,"shop_price":88.0,"goods_brief":"前腿+后腿+羊排共8斤，原生态大山放牧羊羔，曾经的皇室贡品，央视推荐，2005年北京招待全球财金首脑。五层专用包装箱+真空包装+冰袋+保鲜箱+顺丰冷链发货，路途保质期8天","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/> </p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/> </p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/> </p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/2_P_1448945810202.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:00"}
58     {"id":3,"category":{"id":124,"sub_cat":[{"id":125,"sub_cat":[],"name":"生菜","code":"sc","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":124},{"id":126,"sub_cat":[],"name":"菠菜","code":"bc","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":124},{"id":127,"sub_cat":[],"name":"圆椒","code":"yj","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":124},{"id":128,"sub_cat":[],"name":"西兰花","code":"xlh","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":124}],"name":"叶菜类","code":"ycl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/7_P_1448945104883.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/7_P_1448945104734.jpg"}],"goods_sn":"","name":"酣畅家庭菲力牛排10片澳洲生鲜牛肉团购套餐","click_num":3306,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":286.0,"shop_price":238.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/7_P_1448945104883.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
59     {"id":4,"category":{"id":129,"sub_cat":[],"name":"根茎类","code":"gjl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/47_P_1448946213263.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/47_P_1448946213157.jpg"}],"goods_sn":"","name":"日本蒜蓉粉丝扇贝270克6只装","click_num":3395,"sold_num":0,"fav_num":-1,"goods_num":0,"market_price":156.0,"shop_price":108.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/47_P_1448946213263.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
60     {"id":5,"category":{"id":116,"sub_cat":[{"id":117,"sub_cat":[],"name":"参鲍","code":"cb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":118,"sub_cat":[],"name":"鱼","code":"yu","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":119,"sub_cat":[],"name":"虾","code":"xia","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":120,"sub_cat":[],"name":"蟹/贝","code":"xb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116}],"name":"海鲜水产","code":"hxsc","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/10_P_1448944572085.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/10_P_1448944572532.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/10_P_1448944572872.jpg"}],"goods_sn":"","name":"内蒙新鲜牛肉1斤清真生鲜牛肉火锅食材","click_num":3226,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":106.0,"shop_price":88.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/10_P_1448944572085.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
61     {"id":6,"category":{"id":130,"sub_cat":[],"name":"茄果类","code":"qgl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/4_P_1448945381985.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/4_P_1448945381013.jpg"}],"goods_sn":"","name":"乌拉圭进口牛肉卷特级肥牛卷","click_num":3082,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":90.0,"shop_price":75.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/4_P_1448945381985.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
62     {"id":7,"category":{"id":132,"sub_cat":[],"name":"进口生鲜","code":"jksx","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/8_P_1448945032810.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/8_P_1448945032646.jpg"}],"goods_sn":"","name":"五星眼肉牛排套餐8片装原味原切生鲜牛肉","click_num":3124,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":150.0,"shop_price":125.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/8_P_1448945032810.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
63     {"id":8,"category":{"id":116,"sub_cat":[{"id":117,"sub_cat":[],"name":"参鲍","code":"cb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":118,"sub_cat":[],"name":"鱼","code":"yu","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":119,"sub_cat":[],"name":"虾","code":"xia","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":120,"sub_cat":[],"name":"蟹/贝","code":"xb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116}],"name":"海鲜水产","code":"hxsc","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/11_P_1448944388277.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/11_P_1448944388034.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/11_P_1448944388201.jpg"}],"goods_sn":"","name":"澳洲进口120天谷饲牛仔骨4份原味生鲜","click_num":3011,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":31.0,"shop_price":26.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/11_P_1448944388277.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
64     {"id":9,"category":{"id":131,"sub_cat":[],"name":"菌菇类","code":"jgl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/6_P_1448945167279.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/6_P_1448945167015.jpg"}],"goods_sn":"","name":"潮香村澳洲进口牛排家庭团购套餐20片","click_num":3014,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":239.0,"shop_price":199.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/6_P_1448945167279.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
65     {"id":10,"category":{"id":129,"sub_cat":[],"name":"根茎类","code":"gjl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791617.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791129.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791077.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791229.jpg"}],"goods_sn":"","name":"爱食派内蒙古呼伦贝尔冷冻生鲜牛腱子肉1000g","click_num":2742,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":202.0,"shop_price":168.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/9_P_1448944791617.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
66     {"id":11,"category":{"id":111,"sub_cat":[{"id":112,"sub_cat":[],"name":"羊肉","code":"yr","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":111},{"id":113,"sub_cat":[],"name":"禽类","code":"ql","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":111},{"id":114,"sub_cat":[],"name":"猪肉","code":"zr","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":111},{"id":115,"sub_cat":[],"name":"牛肉","code":"nr","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":111}],"name":"精品肉类","code":"jprl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/3_P_1448945490837.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/3_P_1448945490084.jpg"}],"goods_sn":"","name":"澳洲进口牛尾巴300g新鲜肥牛肉","click_num":2819,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":306.0,"shop_price":255.0,"goods_brief":"新鲜羊羔肉整只共15斤，原生态大山放牧羊羔，曾经的皇室贡品，央视推荐，2005年北京招待全球财金首脑。五层专用包装箱+真空包装+冰袋+保鲜箱+顺丰冷链发货，路途保质期8天","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/3_P_1448945490837.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
67     {"id":12,"category":{"id":116,"sub_cat":[{"id":117,"sub_cat":[],"name":"参鲍","code":"cb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":118,"sub_cat":[],"name":"鱼","code":"yu","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":119,"sub_cat":[],"name":"虾","code":"xia","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116},{"id":120,"sub_cat":[],"name":"蟹/贝","code":"xb","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":116}],"name":"海鲜水产","code":"hxsc","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:33","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/48_P_1448943988970.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/48_P_1448943988898.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/48_P_1448943988439.jpg"}],"goods_sn":"","name":"新疆巴尔鲁克生鲜牛排眼肉牛扒1200g","click_num":2718,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":126.0,"shop_price":88.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/48_P_1448943988970.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:53"}
68     {"id":13,"category":{"id":121,"sub_cat":[{"id":122,"sub_cat":[],"name":"松花蛋/咸鸭蛋","code":"xhd_xyd","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":121},{"id":123,"sub_cat":[],"name":"鸡蛋","code":"jd","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":121}],"name":"蛋制品","code":"dzp","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/5_P_1448945270390.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/5_P_1448945270067.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/5_P_1448945270432.jpg"}],"goods_sn":"","name":"澳洲进口安格斯牛切片上脑牛排1000g","click_num":2753,"sold_num":0,"fav_num":0,"goods_num":-1,"market_price":144.0,"shop_price":120.0,"goods_brief":"澳大利亚是国际公认的没有疯牛病和口蹄疫的国家。为了保持澳大利亚产品的高标准，澳大利亚牛肉业和各级政府共同努力简历了严格的标准和体系，以保证生产的整体化和产品的可追溯性","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/5_P_1448945270390.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
69     {"id":14,"category":{"id":130,"sub_cat":[],"name":"茄果类","code":"qgl","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":110},"images":[{"image":"http://shop.projectsedu.com/media/images/201705/goods_img/53_P_1495068879687.jpg"}],"goods_sn":"","name":"帐篷出租","click_num":2771,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":120.0,"shop_price":100.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/53_P_1495068879687.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
70     {"id":15,"category":{"id":146,"sub_cat":[{"id":147,"sub_cat":[],"name":"白兰地","code":"bld","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":146},{"id":148,"sub_cat":[],"name":"威士忌","code":"wsj","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":146}],"name":"红酒","code":"hj","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/16_P_1448947194687.jpg"}],"goods_sn":"","name":"52度茅台集团国隆双喜酒500mlx6","click_num":2838,"sold_num":0,"fav_num":0,"goods_num":-5,"market_price":23.0,"shop_price":19.0,"goods_brief":"贵州茅台酒厂（集团）保健酒业有限公司生产，是以“龙”字打头的酒水。中国龙文化上下8000年，源远而流长，龙的形象是一种符号、一种意绪、一种血肉相联的情感。","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/16_P_1448947194687.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
71     {"id":16,"category":{"id":145,"sub_cat":[],"name":"饮料/水","code":"yls","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/14_P_1448947354031.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/14_P_1448947354433.jpg"}],"goods_sn":"","name":"52度水井坊臻酿八號500ml","click_num":2759,"sold_num":0,"fav_num":0,"goods_num":-3,"market_price":43.0,"shop_price":36.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/14_P_1448947354031.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
72     {"id":17,"category":{"id":141,"sub_cat":[{"id":142,"sub_cat":[],"name":"其他品牌","code":"qtpp","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":141},{"id":143,"sub_cat":[],"name":"黄酒","code":"hj","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":141},{"id":144,"sub_cat":[],"name":"养生酒","code":"ysj","desc":"","category_type":3,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":141}],"name":"其他酒品","code":"qtjp","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/12_P_1448947547989.jpg"}],"goods_sn":"","name":"53度茅台仁酒500ml","click_num":2699,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":190.0,"shop_price":158.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/12_P_1448947547989.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
73     {"id":18,"category":{"id":138,"sub_cat":[],"name":"葡萄酒","code":"ptj","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/46_P_1448946598711.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/46_P_1448946598301.jpg"}],"goods_sn":"","name":"双响炮洋酒JimBeamwhiskey美国白占边","click_num":2722,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":38.0,"shop_price":28.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/46_P_1448946598711.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
74     {"id":19,"category":{"id":145,"sub_cat":[],"name":"饮料/水","code":"yls","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/21_P_1448946793276.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/21_P_1448946793153.jpg"}],"goods_sn":"","name":"西夫拉姆进口洋酒小酒版","click_num":2710,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":55.0,"shop_price":46.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/21_P_1448946793276.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
75     {"id":20,"category":{"id":139,"sub_cat":[],"name":"洋酒","code":"yj","desc":"","category_type":2,"is_tab":false,"add_time":"2017-07-29T18:56:34","parent_category":133},"images":[{"image":"http://shop.projectsedu.com/media/goods/images/15_P_1448947257324.jpg"},{"image":"http://shop.projectsedu.com/media/goods/images/15_P_1448947257580.jpg"}],"goods_sn":"","name":"茅台53度飞天茅台500ml","click_num":533,"sold_num":0,"fav_num":0,"goods_num":0,"market_price":22.0,"shop_price":18.0,"goods_brief":"","goods_desc":"<p><img src=\"/media/goods/images/2_20170719161405_249.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161414_628.jpg\" title=\"\" alt=\"2.jpg\"/></p><p><img src=\"/media/goods/images/2_20170719161435_381.jpg\" title=\"\" alt=\"2.jpg\"/></p>","ship_free":true,"goods_front_image":"http://shop.projectsedu.com/media/goods/images/15_P_1448947257324.jpg","is_new":false,"is_hot":false,"add_time":"2017-07-31T23:53:54"}
76     
77 运行时间:(在网络接口里面设置了sleep 1秒,才能看到效果,实际上真正的接口不会返回那么慢)
78     22.06326198577881s
79 """

 

回调之痛

使用同步I/O实现http请求代码看起来很清晰,简单.所以说原本一个从上而下的三个阶段(connect,send,recv),
就被select+回调+事件循环这种模式放到三个函数中,将我们的代码割的四分五裂,这个实际上维护代码是非常难的.
所以说回调的这种方式实际上用起来是非常痛苦的.

  • 如果回调函数执行不正常该如何???
    • 我们现在能够看到的是,首先get_url(),传统的编程模式就是如果说整个逻辑出问题了,通过get_url()就会抛出异常.但是现在有了回调函数之后,回调函数是由事件循环调用的.
      如果说connected里面写了一些自己的其他逻辑且抛出了异常,这样的话get_url()函数是不能获取到异常的.因为get_url()函数执行完register之后,就跟get_url()函数没有关系了.
      所以说get_url()函数就不会在得到异常了.但是实际上这个异常是获取url整个流程当中抛出的异常,所以说这个异常的处理应该怎么怎么处理他就是一个很头疼的问题,异常处理在我们工作中实际上是一个重要的点.　　
  • 如果回调里面还要嵌套回调怎么办???要嵌套很多层怎么办???
    • 无非就是通过socket方式去请求一个http页面,这么一件简单的事却有了两层回调,第一层是connected,第二层是readable.connected完成之后回调函数readable,
      所以说回调有两层,就另外多出两个方法.一步一步嵌套的完成回调,所以说代码就变的难以维护,而且变的难以理解.　　
  • 如果嵌套了多层,其中某个环节出错了会造成什么后果???
    • 回调了多层本身代码就很难管理了,如果某一个环节再出了异常,我们整个过程实际上变的非常的麻烦.　　
  • 如果有个数据需要被每个回调都处理怎么办???
    • 在我们的代码中有两个回调connected和readable,在两个回调里面都使用了self.client,这个变量是因为我们用了类的方式.把client放到实例变量里面来,所以说在两个回调函数中可以使用,假设不使用这种类的方式(Fetcher),使用函数的方式,
      整个这个变量维护,特别是函数回调里面和get_url中初始化好的的client,他们之间要共用变量的话,实际上就很头疼了.因为我们不可能一直全部都用全局变量来维护,变量维护实际上是一个很头疼的事.这样的话,代码写起来很牵强.
      所以希望既有回调模式的高性能,又有同步编程模式的体验.就是自上而下的写代码.　　
  • 同步方式实现http请求的代码中的变量实际就非常的简单,send()函数可以使用get_url中的在send()函数之前定义的任何变量.所以说这种传统模式的好处有很多
  • 所以说我们如何才能解决这两个问题共存呢???
    • 第一既有同步I/O这么好的体验,　　
    • 第二又有select+回调+事件循环这种模式的高性能
    • 这就是协成来完成的一件事.协成可以兼容第一和第二的两种情况.　　
  • 怎样使用当前函数中的局部变量???
    • 实际就是一个变量共享的问题　　
  • 回调中有哪些痛点:
  1. 可读性差
  2. 共享状态管理困难
  3. 异常处理困难
  • 使用回调模式虽然他能解决我们的并发问题,但是回调模式使我们的代码本身变的很复杂了.这就是协程要来解决的问题

 

下面的代码是没有任何注释的代码为了看起来清晰

 1 import socket
 2 from urllib.parse import urlparse
 3 from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE
 4 
 5 selector = DefaultSelector()
 6 urls = []
 7 stop = False
 8 
 9 
10 class Fetcher:
11 
12     def connected(self, key):
13         selector.unregister(key.fd)
14 
15         self.client.send(
16             "GET {} HTTP/1.1\r\nHost:{}\r\nConnection:close\r\n\r\n".format(self.path, self.host).encode("utf8"))
17 
18         selector.register(self.client.fileno(), EVENT_READ, self.readable)
19 
20     def readable(self, key):
21 
22         re_data = self.client.recv(1024)
23         if re_data:
24             self.data += re_data
25         else:
26             selector.unregister(key.fd)
27             data = self.data.decode("utf8")
28             html_data = data.split("\r\n\r\n")[1]
29             print(html_data)
30             self.client.close()
31             urls.remove(self.spider_url)
32 
33             if not urls:
34                 global stop
35                 stop = True
36 
37     def get_url(self, url):
38         self.spider_url = url
39         new_url = urlparse(url)
40         self.host = new_url.netloc
41         self.path = new_url.path
42         self.data = b""
43 
44         if self.path == "":
45             self.path = "/"
46 
47         self.client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
48         self.client.setblocking(False)
49 
50         try:
51             self.client.connect((self.host, 80))
52         except BlockingIOError as e:
53             pass
54 
55         selector.register(self.client.fileno(), EVENT_WRITE, self.connected)
56 
57 
58 def loop():
59     while not stop:
60         ready = selector.select()
61         for key, mask in ready:
62             call_back = key.data
63             call_back(key)
64 
65 
66 if __name__ == "__main__":
67 
68     import time
69     start_time = time.time()
70 
71     for url in range(1, 21):
72         url = "http://shop.projectsedu.com/goods/{}/".format(url)
73         urls.append(url)
74         fetcher = Fetcher()
75         fetcher.get_url(url)
76 
77     loop()
78 
79     print(time.time() - start_time)

 

 

 

 

 

 

 

 

 

********

posted on 2019-05-09 10:06  jaydenjune  阅读(84)  评论(0)    收藏  举报

导航