摘要: What is 图片懒加载? 图片懒加载是一种网页优化技术,图片在被请求时也与普通静态资源一样,占用资源,然而当一个图片类型在刚被初次请求的时候不可以一次性加载所有的图片,因为这样太过于浪费网络资源,还增加了页面加载时间,通过前后端配合处理,使图片仅在浏览器当前视窗内出现的时候才加载该图片,达到减少 阅读全文
posted @ 2019-05-09 21:22 FindSoul 阅读(230) 评论(0) 推荐(0) 编辑
摘要: UA池 背景 我们在使用下载中间件处理请求,一般会对请求设置随机的User-Agent,设置随机的代理.目的就是防止爬取网站的反爬虫策略,但是同一类型User-Agent的浏览器还是容易被监测到,开启UA池放置更多类型的User-Agent就能够极大避免反扒机制 作用 - 尽可能多的将scrapy工 阅读全文
posted @ 2019-05-09 20:13 FindSoul 阅读(1883) 评论(0) 推荐(0) 编辑
摘要: What is 分布式,爬虫? 使用多台机器组成一个分布式的机群,在机群中运行同一组程序。进行联合数据的爬取 ! Scrapy怎么实现分布式? 原生Scrapy是无法实现分布式的 ! Reason: - 原生scrapy中的调度器无法共享, - 原生scrapy中的管道不可以被共享 Solution 阅读全文
posted @ 2019-05-09 15:05 FindSoul 阅读(300) 评论(0) 推荐(0) 编辑
摘要: Selenium基础 What is selenium? selenium 英 /sɪ'liːnɪəm/最初是一个自动化测试工具,而爬虫中使用它主要是为了解决requests无法直接执行JavaScript代码的问题 selenium本质是通过驱动浏览器,完全模拟浏览器的操作,比如跳转、输入、点击、 阅读全文
posted @ 2019-05-08 11:28 FindSoul 阅读(492) 评论(0) 推荐(0) 编辑
摘要: scrapy框架基础 - 框架 : 具有很多功能,且具有很强通用性的项目模板 - 环境安装 : 什么是scrapy? Scrapy 是一个为了爬取网站数据,提取结构性数据编写的应用框架,.非常强大,所谓的框架就是一个已经被集成了各种功能(高性能异步下载,队列,分布式,解析,持久化等)的具有很强通用性 阅读全文
posted @ 2019-05-06 23:28 FindSoul 阅读(270) 评论(0) 推荐(0) 编辑
摘要: 如何提升requests模块爬取数据的效率 多进程或者多线程(不建议) 线程池或者进程池(适当使用) 单线程+异步协程(推荐) 示例爬取梨视频 import requests import re from lxml import etree from multiprocessing.dummy im 阅读全文
posted @ 2019-05-06 23:09 FindSoul 阅读(264) 评论(0) 推荐(0) 编辑
摘要: 概述 爬虫是合法的吗? 是的,它是一个计算机的学科!一个工具 什么是爬虫? 通过编写程序,模拟浏览器上网,然后让其去互联网上爬取/获取数据的过程.爬虫爬取的也就是服务端的响应数据 爬虫使用场景的分类 - 通用爬虫 : 爬取一整张页面数据."抓取系统" - 聚焦爬虫 : 爬取页面中指定的内容,建立在通 阅读全文
posted @ 2019-04-30 12:36 FindSoul 阅读(881) 评论(0) 推荐(0) 编辑
摘要: 聚焦爬虫介绍 1.编码流程 指定url 发起请求 获取响应数据 数据解析 持久化存储 聚焦爬虫详情 - 如何实现数据解析? 正则 bs4 xpath - 数据解析的原理 实现标签定位 将标签存储的文本内容或者相关属性值进行提取 实现标签定位 - bs4数据解析 - xpath数据解析 阅读全文
posted @ 2019-04-30 12:35 FindSoul 阅读(798) 评论(0) 推荐(0) 编辑
摘要: elasticsearch背景 https://www.cnblogs.com/Neeo/p/10304892.html 我们建立一个网站或应用程序,并要添加搜索功能,但是想要完成搜索工作的创建是非常困难的。我们希望搜索解决方案要运行速度快,我们希望能有一个零配置和一个完全免费的搜索模式,我们希望能 阅读全文
posted @ 2019-04-25 21:21 FindSoul 阅读(186) 评论(0) 推荐(0) 编辑
摘要: 网段划分 A B C D E F 5类ip段 A 类的默认子网掩码 255.0.0.0 , 一个子网最多容纳1677万多台电脑 B 类的默认子网掩码 255.255.0.0 , 一个子网最多容纳6万台电脑 C 类的默认子网掩码 255.255.255.0 , 一个子网最多容纳254台电脑 待续... 阅读全文
posted @ 2019-04-17 09:02 FindSoul 阅读(149) 评论(0) 推荐(0) 编辑
var RENDERER = { POINT_INTERVAL : 5, FISH_COUNT : 3, MAX_INTERVAL_COUNT : 50, INIT_HEIGHT_RATE : 0.5, THRESHOLD : 50, init : function(){ this.setParameters(); this.reconstructMethods(); this.setup(); this.bindEvent(); this.render(); }, setParameters : function(){ this.$window = $(window); this.$container = $('#jsi-flying-fish-container'); this.$canvas = $('
'); this.context = this.$canvas.appendTo(this.$container).get(0).getContext('2d-disabled'); this.points = []; this.fishes = []; this.watchIds = []; }, createSurfacePoints : function(){ var count = Math.round(this.width / this.POINT_INTERVAL); this.pointInterval = this.width / (count - 1); this.points.push(new SURFACE_POINT(this, 0)); for(var i = 1; i < count; i++){ var point = new SURFACE_POINT(this, i * this.pointInterval), previous = this.points[i - 1]; point.setPreviousPoint(previous); previous.setNextPoint(point); this.points.push(point); } }, reconstructMethods : function(){ this.watchWindowSize = this.watchWindowSize.bind(this); this.jdugeToStopResize = this.jdugeToStopResize.bind(this); this.startEpicenter = this.startEpicenter.bind(this); this.moveEpicenter = this.moveEpicenter.bind(this); this.reverseVertical = this.reverseVertical.bind(this); this.render = this.render.bind(this); }, setup : function(){ this.points.length = 0; this.fishes.length = 0; this.watchIds.length = 0; this.intervalCount = this.MAX_INTERVAL_COUNT; this.width = this.$container.width(); this.height = this.$container.height(); this.fishCount = this.FISH_COUNT * this.width / 500 * this.height / 500; this.$canvas.attr({width : this.width, height : this.height}); this.reverse = false; this.fishes.push(new FISH(this)); this.createSurfacePoints(); }, watchWindowSize : function(){ this.clearTimer(); this.tmpWidth = this.$window.width(); this.tmpHeight = this.$window.height(); this.watchIds.push(setTimeout(this.jdugeToStopResize, this.WATCH_INTERVAL)); }, clearTimer : function(){ while(this.watchIds.length > 0){ clearTimeout(this.watchIds.pop()); } }, jdugeToStopResize : function(){ var width = this.$window.width(), height = this.$window.height(), stopped = (width == this.tmpWidth && height == this.tmpHeight); this.tmpWidth = width; this.tmpHeight = height; if(stopped){ this.setup(); } }, bindEvent : function(){ this.$window.on('resize', this.watchWindowSize); this.$container.on('mouseenter', this.startEpicenter); this.$container.on('mousemove', this.moveEpicenter); this.$container.on('click', this.reverseVertical); }, getAxis : function(event){ var offset = this.$container.offset(); return { x : event.clientX - offset.left + this.$window.scrollLeft(), y : event.clientY - offset.top + this.$window.scrollTop() }; }, startEpicenter : function(event){ this.axis = this.getAxis(event); }, moveEpicenter : function(event){ var axis = this.getAxis(event); if(!this.axis){ this.axis = axis; } this.generateEpicenter(axis.x, axis.y, axis.y - this.axis.y); this.axis = axis; }, generateEpicenter : function(x, y, velocity){ if(y < this.height / 2 - this.THRESHOLD || y > this.height / 2 + this.THRESHOLD){ return; } var index = Math.round(x / this.pointInterval); if(index < 0 || index >= this.points.length){ return; } this.points[index].interfere(y, velocity); }, reverseVertical : function(){ this.reverse = !this.reverse; for(var i = 0, count = this.fishes.length; i < count; i++){ this.fishes[i].reverseVertical(); } }, controlStatus : function(){ for(var i = 0, count = this.points.length; i < count; i++){ this.points[i].updateSelf(); } for(var i = 0, count = this.points.length; i < count; i++){ this.points[i].updateNeighbors(); } if(this.fishes.length < this.fishCount){ if(--this.intervalCount == 0){ this.intervalCount = this.MAX_INTERVAL_COUNT; this.fishes.push(new FISH(this)); } } }, render : function(){ requestAnimationFrame(this.render); this.controlStatus(); this.context.clearRect(0, 0, this.width, this.height); this.context.fillStyle = 'hsl(0, 0%, 95%)'; for(var i = 0, count = this.fishes.length; i < count; i++){ this.fishes[i].render(this.context); } this.context.save(); this.context.globalCompositeOperation = 'xor'; this.context.beginPath(); this.context.moveTo(0, this.reverse ? 0 : this.height); for(var i = 0, count = this.points.length; i < count; i++){ this.points[i].render(this.context); } this.context.lineTo(this.width, this.reverse ? 0 : this.height); this.context.closePath(); this.context.fill(); this.context.restore(); } }; var SURFACE_POINT = function(renderer, x){ this.renderer = renderer; this.x = x; this.init(); }; SURFACE_POINT.prototype = { SPRING_CONSTANT : 0.03, SPRING_FRICTION : 0.9, WAVE_SPREAD : 0.3, ACCELARATION_RATE : 0.01, init : function(){ this.initHeight = this.renderer.height * this.renderer.INIT_HEIGHT_RATE; this.height = this.initHeight; this.fy = 0; this.force = {previous : 0, next : 0}; }, setPreviousPoint : function(previous){ this.previous = previous; }, setNextPoint : function(next){ this.next = next; }, interfere : function(y, velocity){ this.fy = this.renderer.height * this.ACCELARATION_RATE * ((this.renderer.height - this.height - y) >= 0 ? -1 : 1) * Math.abs(velocity); }, updateSelf : function(){ this.fy += this.SPRING_CONSTANT * (this.initHeight - this.height); this.fy *= this.SPRING_FRICTION; this.height += this.fy; }, updateNeighbors : function(){ if(this.previous){ this.force.previous = this.WAVE_SPREAD * (this.height - this.previous.height); } if(this.next){ this.force.next = this.WAVE_SPREAD * (this.height - this.next.height); } }, render : function(context){ if(this.previous){ this.previous.height += this.force.previous; this.previous.fy += this.force.previous; } if(this.next){ this.next.height += this.force.next; this.next.fy += this.force.next; } context.lineTo(this.x, this.renderer.height - this.height); } }; var FISH = function(renderer){ this.renderer = renderer; this.init(); }; FISH.prototype = { GRAVITY : 0.4, init : function(){ this.direction = Math.random() < 0.5; this.x = this.direction ? (this.renderer.width + this.renderer.THRESHOLD) : -this.renderer.THRESHOLD; this.previousY = this.y; this.vx = this.getRandomValue(4, 10) * (this.direction ? -1 : 1); if(this.renderer.reverse){ this.y = this.getRandomValue(this.renderer.height * 1 / 10, this.renderer.height * 4 / 10); this.vy = this.getRandomValue(2, 5); this.ay = this.getRandomValue(0.05, 0.2); }else{ this.y = this.getRandomValue(this.renderer.height * 6 / 10, this.renderer.height * 9 / 10); this.vy = this.getRandomValue(-5, -2); this.ay = this.getRandomValue(-0.2, -0.05); } this.isOut = false; this.theta = 0; this.phi = 0; }, getRandomValue : function(min, max){ return min + (max - min) * Math.random(); }, reverseVertical : function(){ this.isOut = !this.isOut; this.ay *= -1; }, controlStatus : function(context){ this.previousY = this.y; this.x += this.vx; this.y += this.vy; this.vy += this.ay; if(this.renderer.reverse){ if(this.y > this.renderer.height * this.renderer.INIT_HEIGHT_RATE){ this.vy -= this.GRAVITY; this.isOut = true; }else{ if(this.isOut){ this.ay = this.getRandomValue(0.05, 0.2); } this.isOut = false; } }else{ if(this.y < this.renderer.height * this.renderer.INIT_HEIGHT_RATE){ this.vy += this.GRAVITY; this.isOut = true; }else{ if(this.isOut){ this.ay = this.getRandomValue(-0.2, -0.05); } this.isOut = false; } } if(!this.isOut){ this.theta += Math.PI / 20; this.theta %= Math.PI * 2; this.phi += Math.PI / 30; this.phi %= Math.PI * 2; } this.renderer.generateEpicenter(this.x + (this.direction ? -1 : 1) * this.renderer.THRESHOLD, this.y, this.y - this.previousY); if(this.vx > 0 && this.x > this.renderer.width + this.renderer.THRESHOLD || this.vx < 0 && this.x < -this.renderer.THRESHOLD){ this.init(); } }, render : function(context){ context.save(); context.translate(this.x, this.y); context.rotate(Math.PI + Math.atan2(this.vy, this.vx)); context.scale(1, this.direction ? 1 : -1); context.beginPath(); context.moveTo(-30, 0); context.bezierCurveTo(-20, 15, 15, 10, 40, 0); context.bezierCurveTo(15, -10, -20, -15, -30, 0); context.fill(); context.save(); context.translate(40, 0); context.scale(0.9 + 0.2 * Math.sin(this.theta), 1); context.beginPath(); context.moveTo(0, 0); context.quadraticCurveTo(5, 10, 20, 8); context.quadraticCurveTo(12, 5, 10, 0); context.quadraticCurveTo(12, -5, 20, -8); context.quadraticCurveTo(5, -10, 0, 0); context.fill(); context.restore(); context.save(); context.translate(-3, 0); context.rotate((Math.PI / 3 + Math.PI / 10 * Math.sin(this.phi)) * (this.renderer.reverse ? -1 : 1)); context.beginPath(); if(this.renderer.reverse){ context.moveTo(5, 0); context.bezierCurveTo(10, 10, 10, 30, 0, 40); context.bezierCurveTo(-12, 25, -8, 10, 0, 0); }else{ context.moveTo(-5, 0); context.bezierCurveTo(-10, -10, -10, -30, 0, -40); context.bezierCurveTo(12, -25, 8, -10, 0, 0); } context.closePath(); context.fill(); context.restore(); context.restore(); this.controlStatus(context); } }; $(function(){ RENDERER.init(); });