python_爬虫_爬取京东商品信息
一
爬取京东商品信息
代码：
import requests
# url = "https://item.jd.com/2967929.html"
url = "https://item.jd.com/100011585270.html"
try:
r = requests.get(url)
r.raise_for_status()
r.encoding = r.apparent_encoding
print(r.text[:1000])
except:
print("爬取失败")
运行结果1：
<script>window.location.href='https://passport.jd.com/uc/login?ReturnUrl=http%3A%2F%2Fitem.jd.com%2F100011585270.html'</script>
运行结果2：
能爬取到信息，但是信息不够全面。结果2只出现过一次，没有及时保存。
曾经怀疑结果1的出现是因为没有登陆，可是登陆后仍然会出现结果1。故排除该可能。
由于偶然出现结果2，所以怀疑可能是网络原因，或者爬虫被禁止。
想尝试更改header，模拟浏览器进行访问，但是由于现在要做scratch的分型雪花，所以暂时搁置。
以上止步于python网络爬虫与信息获取（嵩天老师_MOOC）第一周第三单元第一个视频
二
把url换成了淘宝的一个链接：https://item.taobao.com/item.htm?spm=a1z0d.6639537.1997196601.24.77b47484qxHVRi&id=620107543829
爬取结果：
<!doctype html>
<html><!-- cph -->
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge"/>
<meta charset="gbk"/>
<meta name="format-detection" content="telephone=no, address=no">
<link rel="dns-prefetch" href="//g.alicdn.com">
<link rel="dns-prefetch" href="//gtms01.alicdn.com">
<link rel="dns-prefetch" href="//gtms02.alicdn.com">
<link rel="dns-prefetch" href="//gtms03.alicdn.com">
<link rel="dns-prefetch" href="//gtms04.alicdn.com">
<link rel="dns-prefetch" href="//gd1.alicdn.com">
<link rel="dns-prefetch" href="//gd2.alicdn.com">
<link rel="dns-prefetch" href="//gd3.alicdn.com">
<link rel="dns-prefetch" href="//gd4.alicdn.com">
<html><!-- cph -->
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge"/>
<meta charset="gbk"/>
<meta name="format-detection" content="telephone=no, address=no">
<link rel="dns-prefetch" href="//g.alicdn.com">
<link rel="dns-prefetch" href="//gtms01.alicdn.com">
<link rel="dns-prefetch" href="//gtms02.alicdn.com">
<link rel="dns-prefetch" href="//gtms03.alicdn.com">
<link rel="dns-prefetch" href="//gtms04.alicdn.com">
<link rel="dns-prefetch" href="//gd1.alicdn.com">
<link rel="dns-prefetch" href="//gd2.alicdn.com">
<link rel="dns-prefetch" href="//gd3.alicdn.com">
<link rel="dns-prefetch" href="//gd4.alicdn.com">
<link rel="canonical" href="https://item.taobao.com/item.htm?id=620107543829" />
<link rel="amphtml" hreflang="zh-Hans" href="https://www.taobao.com/list/item-amp/620107543829.htm"/>
<link rel="alternate" hreflang="zh-Hant" href="https://world.taobao.com/item/620107543829.htm" />
<link rel="amphtml" hreflang="zh-Hans" href="https://www.taobao.com/list/item-amp/620107543829.htm"/>
<link rel="alternate" hreflang="zh-Hant" href="https://world.taobao.com/item/620107543829.htm" />
<meta name="renderer" content="webkit"/>
<meta name="refer
<meta name="refer
三
shell模式下
>>> import requests
>>> r = requests.get("https://item.taobao.com/item.htm?spm=a1z0d.6639537.1997196601.24.77b47484qxHVRi&id=620107543829")
>>> r.encoding
'gb18030'
>>> r.apparent_encoding
'GB2312'
>>> r.encoding = r.apparent_encoding
>>> r.text[:1000]
'\r\n\r\n\r\n<!doctype html>\n<html><!-- cph -->\n <head>\n <meta http-equiv="X-UA-Compatible" content="IE=edge"/>\n<meta charset="gbk"/>\n<meta name="format-detection" content="telephone=no, address=no">\n<link rel="dns-prefetch" href="//g.alicdn.com">\n<link rel="dns-prefetch" href="//gtms01.alicdn.com">\n<link rel="dns-prefetch" href="//gtms02.alicdn.com">\n<link rel="dns-prefetch" href="//gtms03.alicdn.com">\n<link rel="dns-prefetch" href="//gtms04.alicdn.com">\n<link rel="dns-prefetch" href="//gd1.alicdn.com">\n<link rel="dns-prefetch" href="//gd2.alicdn.com">\n<link rel="dns-prefetch" href="//gd3.alicdn.com">\n<link rel="dns-prefetch" href="//gd4.alicdn.com">\n\n<link rel="canonical" href="https://item.taobao.com/item.htm?id=620107543829" />\n<link rel="amphtml" hreflang="zh-Hans" href="https://www.taobao.com/list/item-amp/620107543829.htm"/>\n<link rel="alternate" hreflang="zh-Hant" href="https://world.taobao.com/item/620107543829.htm" />\n\n<meta name="renderer" content="webkit"/>\n<meta name="refer'
>>> r.encoding
'GB2312'
>>> import requests
>>> r = requests.get("https://item.taobao.com/item.htm?spm=a1z0d.6639537.1997196601.24.77b47484qxHVRi&id=620107543829")
>>> r.encoding
'gb18030'
>>> r.apparent_encoding
'GB2312'
>>> r.encoding = r.apparent_encoding
>>> r.text[:1000]
'\r\n\r\n\r\n<!doctype html>\n<html><!-- cph -->\n <head>\n <meta http-equiv="X-UA-Compatible" content="IE=edge"/>\n<meta charset="gbk"/>\n<meta name="format-detection" content="telephone=no, address=no">\n<link rel="dns-prefetch" href="//g.alicdn.com">\n<link rel="dns-prefetch" href="//gtms01.alicdn.com">\n<link rel="dns-prefetch" href="//gtms02.alicdn.com">\n<link rel="dns-prefetch" href="//gtms03.alicdn.com">\n<link rel="dns-prefetch" href="//gtms04.alicdn.com">\n<link rel="dns-prefetch" href="//gd1.alicdn.com">\n<link rel="dns-prefetch" href="//gd2.alicdn.com">\n<link rel="dns-prefetch" href="//gd3.alicdn.com">\n<link rel="dns-prefetch" href="//gd4.alicdn.com">\n\n<link rel="canonical" href="https://item.taobao.com/item.htm?id=620107543829" />\n<link rel="amphtml" hreflang="zh-Hans" href="https://www.taobao.com/list/item-amp/620107543829.htm"/>\n<link rel="alternate" hreflang="zh-Hant" href="https://world.taobao.com/item/620107543829.htm" />\n\n<meta name="renderer" content="webkit"/>\n<meta name="refer'
>>> r.encoding
'GB2312'
我的想法是正确的，京东应该是有来源审查，将headers换成mozilla就可以了。正确代码放到评论里
橘子Jane
【推荐】国内首个AI IDE，深度理解中文开发场景，立即下载体验Trae
【推荐】编程新体验，更懂你的AI，立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包，你的智能百科全书，全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell：AI 加持，快人一步
· 开发者必知的日志记录最佳实践
· SQL Server 2025 AI相关能力初探
· Linux系列：如何用 C#调用 C方法造成内存泄露
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 记一次.NET内存居高不下排查解决与启示
· 阿里最新开源QwQ-32B，效果媲美deepseek-r1满血版，部署成本又又又降低了！
· 开源Multi-agent AI智能体框架aevatar.ai，欢迎大家贡献代码
· Manus重磅发布：全球首款通用AI代理技术深度解析与实战指南
· 被坑几百块钱后，我竟然真的恢复了删除的微信聊天记录！
· AI技术革命，工作效率10个最佳AI工具