网络爬虫-java-1
一、相关知识介绍
1、HttpClient:HttpClient 是Apache Jakarta Common 下的子项目,可以用来提供高效的、最新的、功能丰富的支持 HTTP 协议的客户端编程工具包,并且它支持 HTTP 协议最新的版本和建议。
2、如果每次请求都要创建HttpClient,会有频繁创建和销毁的问题,可以使用连接池来解决这个问题。
3、Jsoup:jsoup 是一款Java 的HTML解析器,可直接解析某个URL地址、HTML文本内容。它提供了一套非常省力的API,可通过DOM,CSS以及类似于jQuery的操作方法来取出和操作数据。
4、jsoup的主要功能:
1)从一个URL,文件或字符串中解析HTML;
2)使用DOM或CSS选择器来查找、取出数据;
1>DOM:
获取元素:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1. 根据id查询元素getElementById Element element = document.getElementById("city_bj"); //2. 根据标签获取元素getElementsByTag element = document.getElementsByTag("title").first(); //3. 根据class获取元素getElementsByClass element = document.getElementsByClass("s_name").last(); //4. 根据属性获取元素getElementsByAttribute element = document.getElementsByAttribute("abc").first(); element = document.getElementsByAttributeValue("class", "city_con").first();
获取元素的数据:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
/获取元素 Element element = document.getElementById("test"); //1. 从元素中获取id String str = element.id(); //2. 从元素中获取className str = element.className(); //3. 从元素中获取属性的值attr str = element.attr("id"); //4. 从元素中获取所有属性attributes str = element.attributes().toString(); //5. 从元素中获取文本内容text str = element.text();
2>CSS选择器:jsoup elements对象支持类似于CSS (或jquery)的选择器语法,来实现非常强大和灵活的查找功能。这个select 方法在Document, Element,或Elements对象中都可以使用。且是上下文相关的,因此可实现指定元素的过滤,或者链式选择访问。Select方法将返回一个Elements集合,并提供一组方法来抽取和处理结果。
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
/tagname: 通过标签查找元素,比如:span Elements span = document.select("span"); for (Element element : span) { System.out.println(element.text()); } //#id: 通过ID查找元素,比如:#city_bjj String str = document.select("#city_bj").text(); //.class: 通过class名称查找元素,比如:.class_a str = document.select(".class_a").text(); //[attribute]: 利用属性查找元素,比如:[abc] str = document.select("[abc]").text(); //[attr=value]: 利用属性值来查找元素,比如:[class=s_name] str = document.select("[class=s_name]").text(); //el#id: 元素+ID,比如: h3#city_bj String str = document.select("h3#city_bj").text(); //el.class: 元素+class,比如: li.class_a str = document.select("li.class_a").text(); //el[attr]: 元素+属性名,比如: span[abc] str = document.select("span[abc]").text(); //任意组合,比如:span[abc].s_name str = document.select("span[abc].s_name").text(); //ancestor child: 查找某个元素下子元素,比如:.city_con li 查找"city_con"下的所有li str = document.select(".city_con li").text(); //parent > child: 查找某个父元素下的直接子元素, //比如:.city_con > ul > li 查找city_con第一级(直接子元素)的ul,再找所有ul下的第一级li str = document.select(".city_con > ul > li").text(); //parent > * 查找某个父元素下所有直接子元素.city_con > * str = document.select(".city_con > *").text();
3)可操作HTML元素、属性、文本;(爬虫一般不用)
5、说明:虽然使用Jsoup可以替代HttpClient直接发起请求解析数据,但是往往不会这样用,因为实际的开发过程中,需要使用到多线程,连接池,代理等等方式,而jsoup对这些的支持并不是很好,所以我们一般把jsoup仅仅作为Html解析工具使用。
二、测试代码
1、maven项目的依赖
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
<dependencies> <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient --> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.6</version> </dependency> <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <version>1.7.25</version> <!-- <scope>test</scope>--> </dependency> <!-- https://mvnrepository.com/artifact/org.jsoup/jsoup --> <dependency> <groupId>org.jsoup</groupId> <artifactId>jsoup</artifactId> <version>1.11.3</version> </dependency> <!-- junit --> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> <!-- https://mvnrepository.com/artifact/commons-io/commons-io --> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> <version>2.4</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 --> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>3.9</version> </dependency>
2、使用HttpClient连接池管理HttpClient,使用get、post的方式做两次简单的请求
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
package com.me.study; import org.apache.http.client.config.RequestConfig; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.client.methods.HttpPost; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; import org.apache.http.util.EntityUtils; import java.io.IOException; public class Study1 { private static PoolingHttpClientConnectionManager pool = new PoolingHttpClientConnectionManager(); public static void main(String[] args) { // 设置最大连接数 pool.setMaxTotal(200); // 设置每个主机的并发数 pool.setDefaultMaxPerRoute(20); Study1 study1 = new Study1(); study1.httpGet(); study1.httpPost(); } public String httpPostForJsoup() { //使用连接池管理HttpClient CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(pool).build(); //创建HttpGet请求 HttpPost httpPost = new HttpPost("https://www.bookben.net/"); //设置请求参数 RequestConfig requestConfig = RequestConfig.custom() .setConnectTimeout(3000)//设置创建连接的最长时间1s .setConnectionRequestTimeout(1500)//设置获取连接的最长时间0.5s .setSocketTimeout(30 * 1000)//设置数据传输的最长时间10s .build(); httpPost.setConfig(requestConfig); CloseableHttpResponse response = null; String content = ""; try { //使用HttpClient发起请求 response = httpClient.execute(httpPost); //判断响应状态码是否为200 if (response.getStatusLine().getStatusCode() == 200) { //如果为200表示请求成功,获取返回数据 content = EntityUtils.toString(response.getEntity(), "UTF-8"); //打印数据 //System.out.println(content); } } catch (Exception e) { e.printStackTrace(); } finally { //释放连接 if (response == null) { try { response.close(); } catch (IOException e) { e.printStackTrace(); } } } return content; } private void httpPost() { //使用连接池管理HttpClient CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(pool).build(); //创建HttpGet请求 HttpPost httpPost = new HttpPost("https://www.bookben.net/"); //设置请求参数 RequestConfig requestConfig = RequestConfig.custom() .setConnectTimeout(3000)//设置创建连接的最长时间1s .setConnectionRequestTimeout(1500)//设置获取连接的最长时间0.5s .setSocketTimeout(30 * 1000)//设置数据传输的最长时间10s .build(); httpPost.setConfig(requestConfig); CloseableHttpResponse response = null; try { //使用HttpClient发起请求 response = httpClient.execute(httpPost); //判断响应状态码是否为200 if (response.getStatusLine().getStatusCode() == 200) { //如果为200表示请求成功,获取返回数据 String content = EntityUtils.toString(response.getEntity(), "UTF-8"); //打印数据长度 System.out.println(content); } } catch (Exception e) { e.printStackTrace(); } finally { //释放连接 if (response == null) { try { response.close(); } catch (IOException e) { e.printStackTrace(); } } } } public void httpGet() { //使用连接池管理HttpClient CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(pool).build(); //创建HttpGet请求 = 输入网址 HttpGet httpGet = new org.apache.http.client.methods.HttpGet("https://www.bookben.net/"); //设置请求参数 RequestConfig requestConfig = RequestConfig.custom() .setConnectTimeout(3000)//设置创建连接的最长时间1s .setConnectionRequestTimeout(1500)//设置获取连接的最长时间0.5s .setSocketTimeout(30 * 1000)//设置数据传输的最长时间10s .build(); httpGet.setConfig(requestConfig); CloseableHttpResponse response = null; try { //使用HttpClient发起请求 获取响应 response response = httpClient.execute(httpGet); //判断响应状态码是否为200 if (response.getStatusLine().getStatusCode() == 200) { //如果为200表示请求成功,获取返回数据 String content = EntityUtils.toString(response.getEntity(), "UTF-8"); //打印数据长度 System.out.println(content); } } catch (Exception e) { e.printStackTrace(); } finally { //释放连接 if (response == null) { try { response.close(); } catch (IOException e) { e.printStackTrace(); } } } } }
3、jsoup简单解析
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
package com.me.study; import org.jsoup.Jsoup; import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; import java.net.URL; public class Study2 { public static void main(String[] args) throws Exception { Study2 study2 = new Study2(); study2.Jsoup_url(); study2.Jsoup_String(); } public void Jsoup_url() throws Exception { // 解析url地址 Document document = Jsoup.parse(new URL("https://wp.m.163.com/163/page/news/virus_report/index.html"), 10*1000); //获取title的内容 Element title = document.getElementsByTag("title").first(); System.out.println(title.text()); } private void Jsoup_String() throws Exception{ Study1 study1 = new Study1(); String string = study1.httpPostForJsoup(); Document document = Jsoup.parse(string); //获取title的内容 Element title = document.getElementsByTag("title").first(); System.out.println(title.text()); } }
4、jsoup-dom解析
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
private void Jsoup_dom() throws Exception { Study1 study1 = new Study1(); String string = study1.httpPostForJsoup(); Document document = Jsoup.parse(string); //根据id获取元素 Element mnav = document.getElementById("mnav"); System.out.println("根据id获取元素的内容:"+mnav.text()); //根据标签获取元素 Elements h1 = document.getElementsByTag("h1"); System.out.println("根据标签获取元素的内容:"+h1.first().text()); //根据class....类似 //根据属性获取元素 Elements target = document.getElementsByAttribute("target"); String text = target.first().text(); //从元素中获取该元素属性的值 String href = target.first().attr("href"); System.out.println("根据属性获取元素的内容:"+text+":"+href); }
5、jsoup-select解析
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
private void Jsoup_select() throws Exception{ Study1 study1 = new Study1(); String string = study1.httpPostForJsoup(); Document document = Jsoup.parse(string); //通过标签查找元素 Elements tr = document.select("tr"); for (Element element : tr) { Elements td = element.select("td"); for (Element element1 : td) { System.out.print(element1.text()+" "); } System.out.println(); } //通过id找到元素 因为参数是字符串它不知道id唯一所有是 Elements Elements bd = document.select("#bd"); System.out.println(bd.text()); //查找某个元素下子元素 Elements lis = document.select(".nav_mod li"); for (Element element : lis) { System.out.println(element.text()); } //查找某个元素下直系子元素 Elements lis2 = document.select(".nav_mod > ul > li"); for (Element element : lis2) { System.out.println(element.text()+"123"); } }
三、结果