java Jsoup爬取静态网站 

 1    public static void configUrl() throws IOException {
 2         setAgent("10.1.111.14", "1080");
 3         System.setProperty("https.protocols", "TLSv1.2,TLSv1.1,SSLv3");
 4         Document document = Jsoup.connect(url)
 5                 .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
 6                 .header("Accept-Encoding", "gzip, deflate, sdch")
 7                 .header("Accept-Language", "zh-CN,zh;q=0.8")
 8                 .header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36")
 9                 .get();
10         urlElementList = new ArrayList<Element>();
11         Elements aSrcs = document.select("#News");
12         for (Element aSrc : aSrcs) {
13             Elements aes = aSrc.getElementsByTag("a");
14             for (Element ac : aes) {
15                 urlElementList.add(ac);
16             }
17 
18         }
19     }

header()内是为了模拟请求头，setAgent(ip,port)是设置代理服务器（第一篇有介绍），用来爬取国外网站，国内的不用加。

加入jsoup的maven依赖

<!-- https://mvnrepository.com/artifact/org.jsoup/jsoup -->
    <dependency>
      <groupId>org.jsoup</groupId>
      <artifactId>jsoup</artifactId>
      <version>1.8.3</version>
    </dependency>

 

posted @ 2018-04-13 10:24  半柠檬  阅读(314)  评论(0)  编辑  收藏  举报