常见网页爬虫

1.Baiduspider

百度爬虫

"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Baiduspider-render

"Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)

2.Googlebot

Googlebot 是谷歌的搜索引擎蜘蛛

"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

3.bingbot

微软

"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

4.Bytespider

字节跳动旗下头条搜索的爬虫

"Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

5.Sogou web spider

"Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

6.iaskspider

新浪爱问蜘蛛

"GET //xmlrpc.php HTTP/1.1" 302 15 "http://www.rebatesme.com/" "Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)"

7.Yahoo! Slurp China

"http://rebatesme.com" "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html\xA1\xB1)"

8.msnbot

"msnbot/1.0 ( http://search.msn.com/msnbot.htm)"