webClient.getPage爬取csdn页面,卡死

最近在写一下简单的爬虫,然后使用webClient.getPage的时候,一请求就直接挂掉,很长时间没有反应,让人很头疼,然后去掉js,返回的又是一些看不懂的:
获取列表页的文章列表:
https://blog.csdn.net/qq_38366063/article/list/1

正确的应该是:
在这里插入图片描述

这是获取列表页返回错误的:
在这里插入图片描述

<html><script>
var arg1='0BC3DE07BEF4716BE7116BB1EE989B8E4EDBB0D5';
var _0x4818=['\x63\x73\x4b\x48\x77\x71\x4d\x49','\x5a\x73\x4b\x4a\x77\x72\x38\x56\x65\x41\x73\x79','\x55\x63\x4b\x69\x4e\x38\x4f\x2f\x77\x70\x6c\x77\x4d\x41\x3d\x3d','\x4a\x52\x38\x43\x54\x67\x3d\x3d','\x59\x73\x4f\x6e\x62\x53\x45\x51\x77\x37\x6f\x7a\x77\x71\x5a\x4b\x65\x73\x4b\x55\x77\x37\x6b\x77\x58\x38\x4f\x52\x49\x51\x3d\x3d','\x77\x37\x6f\x56\x53\x38\x4f\x53\x77\x6f\x50\x43\x6c\x33\x6...
function reload(x) {setCookie("acw_sc__v2", x);document.location.reload();}
</script></html>

然后就很烦,然后想了想,要是直接get请求返回流将流转字符串转dom对象是不是就可以了,然后试了一下,真可以,美滋滋,解决了爬虫最关键问题:

 public static InputStream doGet(String urlstr, Map<String, String> headers) throws IOException {
        URL url = new URL(urlstr);
        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
        conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 " +
                "(KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
        conn.setRequestProperty("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp," +
                "image/apng,*/*;q=0" +
                ".8");
        if (headers != null) {
            Set<String> keys = headers.keySet();
            for (String key : keys) {
                conn.setRequestProperty(key, headers.get(key));
            }
        }
        Random random = new Random();
        String ip =
                (random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100);
        conn.setRequestProperty("x-forwarded-for", ip);
        InputStream inputStream = conn.getInputStream();
        return inputStream;
    }

返回的InputStream直接转String,然后直接Jsoup解析即可:

   public static String inputStreamToString(InputStream is, String charset) throws IOException {
        byte[] bytes = new byte[1024];
        int byteLength = 0;
        StringBuffer sb = new StringBuffer();
        while ((byteLength = is.read(bytes)) != -1) {
            sb.append(new String(bytes, 0, byteLength, charset));
        }
        return sb.toString();
    }

在这里插入图片描述
还是有问题,有的页面可以直接流的方式直接写,有的不行,不知道什么原因=-=.
博客的详情页面就不行…

csdn有验证token,打开控制台,找到对应token,加上去就好了,token一般更新时间为半个小时…否则会被屏蔽…
,csdn爬取,加上token即可,自己通过webClient.getPage,就可以获取.,或者直接通过httpRequest,直接get请求也是可以的…记得传token即可.

posted @ 2019-09-30 10:06  你就像甜甜的益达  阅读(286)  评论(0)  编辑  收藏  举报