爬虫 教训
1.对于一些爬取后数据为空,但在网页上f12 上有的 ,可以查看网页源代码 ,发现使用js写的
则只能用模拟浏览器selenium
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('......略')
include_title = []
driver.implicitly_wait(20)
author = driver.find_element_by_xpath('//*[@id="main"]/div/div/div[1]/div[1]/div[1]/div[1]/div[1]/h4/a[1]').text
date = driver.find_element_by_xpath('//*[@id="main"]/div/div/div[1]/div[1]/div[1]/div[1]/div[1]/h4/a[2]').text
driver.find_element_by_xpath('//*[@id="main"]/div/div/div[1]/div[2]/div[1]/button[2]').click()
print(author, date)
2.使用scrapy 如果不模拟浏览器,则只能爬取静态网页
3.对于有内容的网页但response url 打开后为404 则表明不希望你访问,暂时没有办法爬取
4.对于scrapy 在setting 里可以设置头
DEFAULT_REQUEST_HEADERS={
"Accept": "*/*",
"Accept-Encodingv": "gzip, deflate, br",
"Accept-Language": "zh-CN,zh;q=0.9",
"Connectionvkeep-alive":"keep-alive",
"Host": "bbs.tju.edu.cn",
"Referer":"https://bbs.tju.edu.cn/",
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36"
}
5. 普通的导入mongodb
import pymongo
client = pymongo.MongoClient('localhost',27017)
mydb = client['mydb']
taobao = mydb['taobao']
#抓取略
commodity = {
'goods': goods,#抓取之后
'price': price,
'sell': sells,
'shop': shop,
'address': address
}
taobao.insert_one(commodity)
6.scrapy 导入mongodb
#在pipelines
import pymongo
class JianshuPipeline(object):
def __init__(self):
client = pymongo.MongoClient('localhost', 27017)
test = client['test']
jianshu = test['jianshu']
self.post = jianshu
def process_item(self, item, spider):
info = dict(item)
self.post.insert(info)
return item