使用requests+代理获取某个网站内容时,一直乱码,源代码是这样的:
headers:
"headers": {"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding": "gzip, deflate, br","accept-language": "zh-CN,zh;q=0.9","cache-control": "max-age=0","sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"97\", \"Chromium\";v=\"97\"","sec-ch-ua-mobile": "?0","sec-ch-ua-platform": "\"Windows\"","sec-fetch-dest": "document","sec-fetch-mode": "navigate","sec-fetch-site": "same-origin","sec-fetch-user": "?1","upgrade-insecure-requests": "1","user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"}
调用的时候是这样的:
proxies = {"HTTP": "http://{}:{}@{}:{}".format(pconfig['proxyUser'], pconfig['proxyPass'], pconfig['proxyHost'], pconfig['proxyPort']),"HTTPS": "https://{}:{}@{}:{}".format(pconfig['proxyUser'], pconfig['proxyPass'], pconfig['proxyHost'], pconfig['proxyPort'])
}
result = requests.get(url=url, headers=headers, timeout=timeout, proxies=proxies, verify=False)
if result is not None and result.status_code == 200:errMsg = ""urlhtml = result.textrealurl = result.url # 重定向地址break
else:
但是获取到的结果是这样的:
result.content= "b'c\x00\xa0a\x00@6s\xd6\x1b\x1f\x7f\xd5\x07\x14\x17\x17C\x07\x88\xaa\xa3&\xea\x18r\xb6\xb2|\xff\x97\tr\xc0G\xd4\xbdT=\xd0\x13\xd9\xdd\xf9WRz\xa1V\xe6\xcf\xfeMoX\x9b]\x82y\xb1}\x9e\x8a!h\x85\x0e8D\x01\x8fH\x1e\xcbu\x91?\x97\x84\x08\xe2%n?D9z\xf4\xf6\xe1\xa7\xef\xef\x1e\xc3\xb3O\xaf_\xd5P....."
result.text="c �a @6s��C���&�r��|�� r�GԽT=����WRz�V���MoX�]�y�}��!h�8D�H�u�?���%n?D9z�����óO�_�P����+�������٨����� �h����0�dV!�>�aH�����q8x�������36ZD��cC"M��aSC�H�7@k"��=��b�~N(�s/�Fj�!G3�$а�w���O���n��GQщ۩......"
看了一下result.apparent_encoding,然而:result.apparent_encoding=None (吐血!)
当时搜了各种解决方案(以下给大家参考,也许可用):
你的apparent_encoding 还准确么? - 灰信网(软件开发博客聚合)
Python requests响应内容乱码解决方案 | 漏洞人生
解决Requests中文乱码_chaowanghn的博客-CSDN博客_requests 乱码
https://segmentfault.com/a/1190000012984536
但是上述办法对我毫无效果,直到第二天看到这篇文章:
requests 中文编码错误_chaoge_dgqb的博客-CSDN博客_requests编码错误
终于发现我的headers设置了一个多余的:
"accept-encoding": "gzip, deflate, br",
把它删掉之后,就立马好了。
所以有时候少即是多....(说点假道理,其实多用不同方式搜索问题,才是正确解法)