Python爬取页面时遇到的字符编码问题

在利用Python爬取微博页面时，遇到错误UnicodeEncodeError: ‘UCS-2’ codec can’t encode characters in position 73-73: Non-BMP character not supported in Tk，在网上找了一些资料，但都比较复杂，现我已成功解决该问题，且较简单，故通过我的实例分享经验。

原代码

from urllib.parse import urlencode
from pyquery import PyQuery as pqimport requests
base_url = 'https://m.weibo.cn/api/container/getindex?'headers = {'Host': 'm.weibo.cn','Referer': 'https://m.weibo.cn/u/3908167020','User-Agent': 'Mozilla/5.0 (Macintosh;Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/52.0.2743.116 Safari/537.36','X-Requested-With': 'XMLHttpRequest',}def get_page(page):params = {'type': 'uid','value': '1989519725','containerid': '1076031989519725','page': page}url = base_url + urlencode(params)try:response = requests.get(url, headers = headers)if response.status_code == 200:return response.json()except requests.ConnectionError as e:print('Error',e.args)def parse_page(json):if json:items = json.get('data').get('cards')for item in items:item = item.get('mblog')weibo = {}weibo['id'] = item.get('id')weibo['text'] = pq(item.get('text')).text()weibo['comments'] = item.get('comments_count')weibo['reposts'] = item.get('reposts_count')yield weiboif __name__ == '__main__':for page in range(1,11):json = get_page(page)results = parse_page(json)for result in results:print(result)

运行后显示错误：

出现错误的原因是：
我们爬取的HTML页面中包含了Unicode下无法识别的字符（这是我自己的理解，如果理解有误，欢迎指正！）
解决办法是：

1. import sys（新代码第6行）
2. 定义non_bmp_map = dict.fromkeys(range(0x10000, sys.maxunicode + 1), 0xfffd)（新代码第8行）
3. 对要输出的结果(本实例中为result)translate，即在输出result之前，增加语句result = str(result).translate(non_bmp_map)（新代码第55行）

新代码

from urllib.parse import urlencode
from pyquery import PyQuery as pqimport requestsimport sysnon_bmp_map = dict.fromkeys(range(0x10000,sys.maxunicode + 1),0xfffd)base_url = 'https://m.weibo.cn/api/container/getindex?'headers = {'Host': 'm.weibo.cn','Referer': 'https://m.weibo.cn/u/3908167020','User-Agent': 'Mozilla/5.0 (Macintosh;Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/52.0.2743.116 Safari/537.36','X-Requested-With': 'XMLHttpRequest',}def get_page(page):params = {'type': 'uid','value': '1989519725','containerid': '1076031989519725','page': page}url = base_url + urlencode(params)try:response = requests.get(url, headers = headers)if response.status_code == 200:return response.json()except requests.ConnectionError as e:print('Error',e.args)def parse_page(json):if json:items = json.get('data').get('cards')for item in items:item = item.get('mblog')weibo = {}weibo['id'] = item.get('id')weibo['text'] = pq(item.get('text')).text()weibo['comments'] = item.get('comments_count')weibo['reposts'] = item.get('reposts_count')yield weiboif __name__ == '__main__':for page in range(1,11):json = get_page(page)results = parse_page(json)for result in results:result = str(result).translate(non_bmp_map)print(result)

这样问题就完美解决啦!

参考资料来源

1: https://stackoverflow.com/questions/32442608/ucs-2-codec-cant-encode-characters-in-position-1050-1050.
2: https://www.2cto.com/kf/201805/748337.html.
3: https://blog.csdn.net/qq_16272049/article/details/79492020.