js & anti craw & crawler spam

js & anti craw & crawler spam

demo & X-Sign

    , function(t, e, n) {
        "use strict";
        var r = n(126)
          , o = n.n(r)
          , i = "WSUDD"
          , a = "X"
          , s = "/fe_api/";
        e.a = {
            name: "crawler-spam",
            install: function(t, e) {
                e.isBrowser && e.http.interceptors.dispatch.use(function(t) {
                    return t.url.indexOf(s) > -1 && (t.headers["X-Sign"] = function(t, e) {
                        var n = arguments.length > 2 && void 0 !== arguments[2] ? arguments[2] : a
                          , r = t.url
                          , u = void 0 === r ? "" : r
                          , c = t.params
                          , f = t.paramsSerializer;
                        return u = u.slice(u.indexOf(s), u.length),
                        n === a ? "" + n + o()(e(u, c, f) + i) : ""
                    }(t, e.http.buildURL)),
                    t
                })
            }
        }
    }

1. step 1: find crawler spam js file name & get json data

js & XHR

2. open sources, set break points

js files

3. debug, find the logic

step by step, look for the roots(溯源)

4. mock / fake, craw datas

blogs

crawler-spam

https://www.xiaohongshu.com/page/hot
https://www.xiaohongshu.com/explore

https://www.edificeautomotive.com/blog/2016/02/26/ghost-and-crawler-spam/

referral exclusion list

https://support.google.com/analytics/answer/2795830?hl=en

Referral exclusions & 推荐排除

https://www.liquidlight.co.uk/blog/crawler-spam-referrals-how-to-filter-them-out-from-google-analytics/

---

hack methods

github

https://github.com/topics/xiaohongshu

https://github.com/lonngxiang/xiaohongshu-spider

https://github.com/vinchu/xiaohongshu-2

https://github.com/No-bb-just-do-it/xiaohongshu

npm

pm formula-static/@xhs/launcher

---

---

©xgqfrms 2012-2020

www.cnblogs.com 发布文章使用：只允许注册用户才可以访问！

---