The bytes/str dichotomy in Python 3

 

The bytes/str dichotomy in Python 3 - Eli Bendersky's website https://eli.thegreenplace.net/2012/01/30/the-bytesstr-dichotomy-in-python-3

 Arguably the most significant new feature of Python 3 is a much cleaner separation between text and binary data. Text is always Unicode and is represented by the str type, and binary data is represented by thebytes type. What makes the separation particularly clean is that str and bytes can't be mixed in Python 3 in any implicit way. You can't concatenate them, look for one inside another, and generally pass one to a function that expects the other. This is a good thing.

 

import requests
import re
import time
from redis import Redis

REDIS_HOST, REDIS_PORT, PASSWORD = '192.168.2.51', '6379', 'mypwd'
rds = Redis(host=REDIS_HOST, port=REDIS_PORT, password=PASSWORD)

f, url_l, filter_replace_l = 'DISTINCT_url.txt', [], ['\n', '\t', ' ']
with open(f, 'r', encoding='utf-8') as fr:
    for i in fr:
        try:
            for ii in filter_replace_l:
                i = i.replace(ii, '')
            rds.sadd('chk_url_all', i)
        except Exception as e:
            print(e)

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
while True:
    try:
        url = rds.spop('chk_url_all').decode('utf-8')
        s = 'http://'
        if s not in url:
            url = '{}{}'.format(s, url)
        print(url)
        r = requests.get(url, headers=headers, timeout=50)
        print(r.status_code)
    except Exception as e:
        print(e)



从redis取值结果为bytes type