playwright结合adblocker进行广告拦截
现在基本网站都会有埋点（统计分析）或者不少有广告的，对于爬虫场景可能不太方便，社区已经提供了相关的插件
我们可以直接使用，以下是一个简单的使用说明
环境准备
基于browserless
- docker-compose
version: "3"
services:
browser:
image: ghcr.io/browserless/chromium:latest
environment:
- CONCURRENT=40
- QUEUED=20
- CORS=true
- CORS_MAX_AGE=300
- DATA_DIR=/tmp/my-profile
- TOKEN=6R0W53R135510
volumes:
- ./my-profile:/tmp/my-profile
ports:
- "3000:3000"
使用
- packageg.json
{
"name": "local_storage",
"version": "1.0.0",
"main": "index.js",
"license": "MIT",
"dependencies": {
"@cliqz/adblocker-playwright": "^1.27.3",
"cross-fetch": "^4.0.0",
"playwright": "^1.44.0",
},
"scripts": {
"start": "node dalong.js"
}
}
- dalong.js
const { chromium } = require("playwright");
const {PlaywrightBlocker} = require("@cliqz/adblocker-playwright");
const fetch = require("cross-fetch");
(async () => {
let browser = await chromium.connectOverCDP(
"ws://localhost:3000?token=6R0W53R135510"
);
let bContext = await browser.newContext({
userAgent:
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
});
let page = await bContext.newPage();
const blocker = await PlaywrightBlocker.fromPrebuiltAdsAndTracking(fetch);
blocker.enableBlockingInPage(page);
await page.goto("https://www.cnblogs.com/rongfengliang", {
waitUntil: "networkidle",
});
let resutlv2 = await page.locator("#content").all();
for (const row of await resutlv2) {
console.log(await row.innerHTML());
}
await browser.close();
})();
- 请求效果
拦截的
说明
以上是一个简单使用，对于广告进行拦截在一些场景还是比较有用的，比如减少网络开销，提升处理速度，实际上adblocker 提供的能力还是不少的，我们还可以自己进行拦截的自定义，实现灵活的处理
参考资料
https://www.npmjs.com/package/@cliqz/adblocker-playwright
https://playwright.dev/docs/locators#lists
