Python爬虫使用实例-wallpaper

embedded/2024/11/9 16:43:03/

1/ 排雷避坑

🥝 中文乱码问题

python">print(requests.get(url=url,headers=headers).text)

在这里插入图片描述

出现中文乱码

原因分析:

<meta charset="gbk" />

解决方法:
法一:

python">response = requests.get(url=url,headers=headers)
response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码
print(response.text)

法二:

python">print(requests.get(url=url,headers=headers).content.decode('gbk'))

2/ 数据来源

css解析
在这里插入图片描述

python">for li in lis:href = li.css('a::attr(href)').get()title = li.css('b::text').get()print(href, title)

在这里插入图片描述
在这里插入图片描述
删掉标题为空的那一张图


在这里插入图片描述
获取图片url


有的网站,保存的数据是裂开的图片,可能是因为这个参数:
在这里插入图片描述

3/ 正则处理

处理图片url和标题的时候用了re模块

电脑壁纸
通过匹配非数字字符并在遇到数字时截断字符串

python">title1 = selector1.css('.photo .photo-pic img::attr(title)').get()
modified_title = re.split(r'\d', title1, 1)[0].strip()

re.split(r'\d', title, 1)将 title 字符串按第一个数字进行分割。返回的列表的第一个元素就是数字前面的部分。strip() 去掉字符串首尾的空白字符。


url图片路径替换,因为从点开图片到达的那个页面无法得到的图片路径还是html页面,不是https://····.jpg,所以替换成另一个可以获取到的页面。

python">https://sj.zol.com.cn/bizhi/detail_{num1}_{num2}.html

正则替换修改为

python">https://app.zol.com.cn/bizhi/detail_{num1}.html

例如 https://sj.zol.com.cn/bizhi/detail_12901_139948.html 转换为 https://app.zol.com.cn/bizhi/detail_12901.html .

python"># https://sj.zol.com.cn/bizhi/detail_12901_139948.html
url = "https://sj.zol.com.cn/bizhi/detail_12901_139948.html"
pattern = r'https://sj\.zol\.com\.cn/bizhi/detail_(\d+)_\d+\.html'
replacement = r'https://app.zol.com.cn/bizhi/detail_\1.html'
new_url = re.sub(pattern, replacement, url)
print(url,new_url)

4/ 电脑壁纸

🥝 单线程单页

适用于当页面和第一页

python"># python单线程爬取高清4k壁纸图片
import os
import re
import requests
import parsel
url = 'https://pic.netbian.com/4kmeinv/' # 请求地址
# 模拟伪装
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}
# response = requests.get(url=url,headers=headers)
# response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码
# print(response.text)
html_data = requests.get(url=url,headers=headers).content.decode('gbk')
# print(html_data)
selector = parsel.Selector(html_data)
lis = selector.css('.slist li')
for li in lis:#href = li.css('a::attr(href)').get()title = li.css('b::text').get()if title:href = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response = requests.get(url=href, headers=headers)#print(href, title)# 这里只是获取页面# img_content = requests.get(url=href, headers=headers).content# 不可行, 都是同一张图 https://pic.netbian.com/uploads/allimg/230813/221347-16919360273e05.jpghref = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response1 = requests.get(url=href, headers=headers).content.decode('gbk')selector1 = parsel.Selector(response1)# 若要标题乱码,此处可不解码# response1 = requests.get(url=href, headers=headers)# selector1 = parsel.Selector(response1.text)# img_url = selector1.css('.slist li img::attr(src)').get()# 这一步错了, 要去href页面找img_url, 这是在原来的url页面找了img_url = 'https://pic.netbian.com' + selector1.css('.photo .photo-pic img::attr(src)').get()img_content = requests.get(url=img_url,headers=headers).content# 顺便更新一下title, 因为原来的是半截的, 不全title1 = selector1.css('.photo .photo-pic img::attr(title)').get()modified_title = re.split(r'\d', title1, 1)[0].strip()with open('img\\'+modified_title+'.jpg',mode='wb') as f:f.write(img_content)#print(href, title)print('正在保存:', modified_title, img_url)

🥝 单线程多page

适用于从第二页开始的多页

python"># python单线程爬取高清4k壁纸图片
import os
import re
import time
import requests
import parsel
# url的规律
# https://pic.netbian.com/new/index.html
# https://pic.netbian.com/new/index_1.html
# https://pic.netbian.com/new/index_2.html
# ...
start_time = time.time()
for page in range(2,10):print(f'--------- 正在爬取第{page}的内容 ----------')url = f'https://pic.netbian.com/4kmeinv/index_{page}.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}# response = requests.get(url=url,headers=headers)# response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码# print(response.text)html_data = requests.get(url=url, headers=headers).content.decode('gbk')# print(html_data)selector = parsel.Selector(html_data)lis = selector.css('.slist li')for li in lis:# href = li.css('a::attr(href)').get()title = li.css('b::text').get()if title:href = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response = requests.get(url=href, headers=headers)# print(href, title)# 这里只是获取页面# img_content = requests.get(url=href, headers=headers).content# 不可行, 都是同一张图 https://pic.netbian.com/uploads/allimg/230813/221347-16919360273e05.jpghref = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response1 = requests.get(url=href, headers=headers).content.decode('gbk')selector1 = parsel.Selector(response1)# 若要标题乱码,此处可不解码# response1 = requests.get(url=href, headers=headers)# selector1 = parsel.Selector(response1.text)# img_url = selector1.css('.slist li img::attr(src)').get()# 这一步错了, 要去href页面找img_url, 这是在原来的url页面找了img_url = 'https://pic.netbian.com' + selector1.css('.photo .photo-pic img::attr(src)').get()img_content = requests.get(url=img_url, headers=headers).content# 顺便更新一下title, 因为原来的是半截的, 不全title1 = selector1.css('.photo .photo-pic img::attr(title)').get()modified_title = re.split(r'\d', title1, 1)[0].strip()with open('img\\' + modified_title + '.jpg', mode='wb') as f:f.write(img_content)# print(href, title)print('正在保存:', modified_title, img_url)stop_time = time.time()
print(f'耗时:{int(stop_time)-int(start_time)}秒')

运行效果:

在这里插入图片描述
在这里插入图片描述

🥝 多线程多页

python"># python多线程爬取高清4k壁纸图片
import os
import re
import time
import requests
import parsel
import concurrent.futuresdef get_img(url):# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}# response = requests.get(url=url,headers=headers)# response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码# print(response.text)html_data = requests.get(url=url, headers=headers).content.decode('gbk')# print(html_data)selector = parsel.Selector(html_data)lis = selector.css('.slist li')for li in lis:# href = li.css('a::attr(href)').get()title = li.css('b::text').get()if title:href = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response = requests.get(url=href, headers=headers)# print(href, title)# 这里只是获取页面# img_content = requests.get(url=href, headers=headers).content# 不可行, 都是同一张图 https://pic.netbian.com/uploads/allimg/230813/221347-16919360273e05.jpghref = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response1 = requests.get(url=href, headers=headers).content.decode('gbk')selector1 = parsel.Selector(response1)# 若要标题乱码,此处可不解码# response1 = requests.get(url=href, headers=headers)# selector1 = parsel.Selector(response1.text)# img_url = selector1.css('.slist li img::attr(src)').get()# 这一步错了, 要去href页面找img_url, 这是在原来的url页面找了img_url = 'https://pic.netbian.com' + selector1.css('.photo .photo-pic img::attr(src)').get()img_content = requests.get(url=img_url, headers=headers).content# 顺便更新一下title, 因为原来的是半截的, 不全title1 = selector1.css('.photo .photo-pic img::attr(title)').get()modified_title = re.split(r'\d', title1, 1)[0].strip()img_folder = 'img1\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + modified_title + '.jpg', mode='wb') as f:f.write(img_content)# print(href, title)print('正在保存:', modified_title, img_url)
def main(url):get_img(url)start_time = time.time()
executor = concurrent.futures.ThreadPoolExecutor(max_workers=5)
for page in range(2, 12):print(f'--------- 正在爬取第{page}的内容 ----------')url = f'https://pic.netbian.com/4kmeinv/index_{page}.html'  # 请求地址executor.submit(main, url)
executor.shutdown()
stop_time = time.time()
print(f'耗时:{int(stop_time) - int(start_time)}秒')

5/ 手机壁纸

类似地,另一个网站,图片集合多页,点开之后里面有多张图片
先试图获取外部的,再获取里面的,然后2个一起

🥝 单线程单页0

python">import os
import re
import requests
import parsel
url = 'https://sj.zol.com.cn/bizhi/5/' # 请求地址
# 模拟伪装
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}
# response = requests.get(url=url,headers=headers)
# response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码
# print(response.text)
response = requests.get(url=url,headers=headers)
#print(response.text)
selector = parsel.Selector(response.text)
lis = selector.css('.pic-list2 li')
#img_name=1
for li in lis:#href = li.css('a::attr(href)').get()title = li.css('.pic img::attr(title)').get()#href = li.css('.pic img::attr(src)').get()#print(title, href)if title:#href = 'https://sj.zol.com.cn' +li.css('a::attr(href)').get()# https://sj.zol.com.cn/bizhi/detail_12901_139948.html# https://app.zol.com.cn/bizhi/detail_12901_139948.html#p1#href = 'https://app.zol.com.cn' + li.css('a::attr(href)').get() + '#p1'href=li.css('img::attr(src)').get()#print(href, title)#href = 'https://app.zol.com.cn' + li.css('a::attr(href)').get() + '#p1'#response1 = requests.get(url=href, headers=headers).content.decode('utf-8')#selector1 = parsel.Selector(response1)#img_url=selector1.css('.gallery li img::attr(src)').get()#print(img_url)# 这里只是获取页面img_content = requests.get(url=href, headers=headers).content# 不可行, 都是同一张图 https://pic.netbian.com/uploads/allimg/230813/221347-16919360273e05.jpg# https://sj.zol.com.cn/bizhi/detail_12901_139948.html# https://app.zol.com.cn/bizhi/detail_12901_139948.html#p1#href= selector1.css('.photo-list-box li::attr(href)').get()#href = 'https://app.zol.com.cn' +  + '#p1'#response2 = requests.get(url=href, headers=headers)#selector2 = parsel.Selector(response2.text)#print(href)# 若要标题乱码,此处可不解码# response1 = requests.get(url=href, headers=headers)# selector1 = parsel.Selector(response1.text)# img_url = selector1.css('.slist li img::attr(src)').get()# 这一步错了, 要去href页面找img_url, 这是在原来的url页面找了#img_url = selector1.css('.gallery img::attr(src)').get()#img_content = requests.get(url=img_url, headers=headers).content#print(img_url)# 顺便更新一下title, 因为原来的是半截的, 不全# title1 = selector1.css('.photo .photo-pic img::attr(title)').get()img_folder = 'img3\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + title + '.jpg', mode='wb') as f:f.write(img_content)# print(href, title)print('正在保存:', title, href)#img_name += 1

🥝 单线程单页1

python"># 下载子页面全部
import os
import requests
import parselurl = 'https://app.zol.com.cn/bizhi/detail_12901.html' # 请求地址
# 模拟伪装
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}
response = requests.get(url=url,headers=headers)
selector = parsel.Selector(response.text)
lis = selector.css('.album-list li')
i = 0
for li in lis:# Get all img elements within the current liimg_tags = li.css('img::attr(src)').getall()  # This gets all the img src attributesfor href in img_tags:  # Iterate over all img src attributesimg_content = requests.get(url=href, headers=headers).contentimg_folder = 'img4\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + str(i) + '.jpg', mode='wb') as f:f.write(img_content)# print(href, i)print('正在保存:', i, href)i += 1  # Increment i for each image saved

🥝 单线程单页

python">import os
import re
import requests
import parselurl = 'https://sj.zol.com.cn/bizhi/5/' # 请求地址
# 模拟伪装
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}
response = requests.get(url=url,headers=headers)
#print(response.text)
selector = parsel.Selector(response.text)
#lis = selector.css('.pic-list2 li')
# 筛除包含的底部 3个 猜你喜欢
lis=selector.css('.pic-list2 .photo-list-padding')
for li in lis:#href = li.css('a::attr(href)').get()title = li.css('.pic img::attr(title)').get()href = li.css('a::attr(href)').get()#print(title, href)# https://sj.zol.com.cn/bizhi/detail_12901_139948.html#url = "https://sj.zol.com.cn/bizhi/detail_12901_139948.html"pattern = r'/bizhi/detail_(\d+)_\d+\.html'replacement = r'https://app.zol.com.cn/bizhi/detail_\1.html'new_url = re.sub(pattern, replacement, href)#print(href, new_url)#url = 'https://app.zol.com.cn/bizhi/detail_12901.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=new_url, headers=headers)selector = parsel.Selector(response.text)lis1 = selector.css('.album-list li')i = 0for li1 in lis1:# Get all img elements within the current liimg_tags = li1.css('img::attr(src)').getall()  # This gets all the img src attributesfor href in img_tags:  # Iterate over all img src attributesimg_content = requests.get(url=href, headers=headers).contentimg_folder = 'img5\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + title+'_'+str(i) + '.jpg', mode='wb') as f:f.write(img_content)# print(href, i)print('正在保存:',title+'_'+str(i), href)i += 1  # Increment i for each image saved

在这里插入图片描述

🥝 单线程多页

python">import os
import re
import requests
import parselfor page in range(1,3):print(f'--------- 正在爬取第{page}的内容 ----------')if page==1:url = 'https://sj.zol.com.cn/bizhi/5/'  # 请求地址else:url = f'https://sj.zol.com.cn/bizhi/5/{page}.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=url, headers=headers)# print(response.text)selector = parsel.Selector(response.text)# lis = selector.css('.pic-list2 li')# 筛除包含的底部 3个 猜你喜欢lis = selector.css('.pic-list2 .photo-list-padding')for li in lis:# href = li.css('a::attr(href)').get()title = li.css('.pic img::attr(title)').get()href = li.css('a::attr(href)').get()# print(title, href)# https://sj.zol.com.cn/bizhi/detail_12901_139948.html# url = "https://sj.zol.com.cn/bizhi/detail_12901_139948.html"pattern = r'/bizhi/detail_(\d+)_\d+\.html'replacement = r'https://app.zol.com.cn/bizhi/detail_\1.html'new_url = re.sub(pattern, replacement, href)# print(href, new_url)# url = 'https://app.zol.com.cn/bizhi/detail_12901.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=new_url, headers=headers)selector = parsel.Selector(response.text)lis1 = selector.css('.album-list li')i = 0for li1 in lis1:# Get all img elements within the current liimg_tags = li1.css('img::attr(src)').getall()  # This gets all the img src attributesfor href in img_tags:  # Iterate over all img src attributesimg_content = requests.get(url=href, headers=headers).contentimg_folder = 'img6\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + title + '_' + str(i) + '.jpg', mode='wb') as f:f.write(img_content)# print(href, i)print('正在保存:', title + '_' + str(i), href)i += 1  # Increment i for each image saved

🥝 多线程多页

python">import os
import re
import time
import requests
import parsel
import concurrent.futuresdef get_imgs(url):# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=url, headers=headers)# print(response.text)selector = parsel.Selector(response.text)# lis = selector.css('.pic-list2 li')# 筛除包含的底部 3个 猜你喜欢lis = selector.css('.pic-list2 .photo-list-padding')for li in lis:# href = li.css('a::attr(href)').get()title = li.css('.pic img::attr(title)').get()href = li.css('a::attr(href)').get()# print(title, href)# https://sj.zol.com.cn/bizhi/detail_12901_139948.html# url = "https://sj.zol.com.cn/bizhi/detail_12901_139948.html"pattern = r'/bizhi/detail_(\d+)_\d+\.html'replacement = r'https://app.zol.com.cn/bizhi/detail_\1.html'new_url = re.sub(pattern, replacement, href)# print(href, new_url)# url = 'https://app.zol.com.cn/bizhi/detail_12901.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=new_url, headers=headers)selector = parsel.Selector(response.text)lis1 = selector.css('.album-list li')i = 0for li1 in lis1:# Get all img elements within the current liimg_tags = li1.css('img::attr(src)').getall()  # This gets all the img src attributesfor href in img_tags:  # Iterate over all img src attributesimg_content = requests.get(url=href, headers=headers).contentimg_folder = 'img7\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + title + '_' + str(i) + '.jpg', mode='wb') as f:f.write(img_content)# print(href, i)print('正在保存:', title + '_' + str(i), href)i += 1  # Increment i for each image saveddef main(url):get_imgs(url)start_time = time.time()
executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)
for page in range(1, 9):#print(f'--------- 正在爬取第{page}的内容 ----------')if page == 1:url = 'https://sj.zol.com.cn/bizhi/5/'  # 请求地址else:url = f'https://sj.zol.com.cn/bizhi/5/{page}.html'  # 请求地址executor.submit(main, url)
executor.shutdown()
stop_time = time.time()
print(f'耗时:{int(stop_time) - int(start_time)}秒')

http://www.ppmy.cn/embedded/110551.html

相关文章

B-树底层原理

一、B-树介绍 定义&#xff1a; B-树&#xff08;B-Tree&#xff09;是一种自平衡的树形数据结构&#xff0c;广泛应用于数据库和操作系统中。它的设计目标是减少搜索、顺序访问、插入和删除操作中比较次数和移动次数&#xff0c;特别适合于磁盘中数据的存储和检索。 性质&a…

更换ip地址是什么意思?ip地址怎么更换

在互联网的世界里&#xff0c;IP地址就像是每台设备的“身份证”&#xff0c;它确保了网络信息的准确传输。但有时候&#xff0c;出于某种考虑&#xff0c;我们可能需要更换这个“身份证”。那么&#xff0c;更换IP地址究竟意味着什么&#xff1f;又该如何操作呢&#xff1f;本…

二次规划及其MATLAB实现

引言 二次规划&#xff08;Quadratic Programming, QP&#xff09;是一类重要的优化问题&#xff0c;其目标函数为二次函数&#xff0c;约束条件为线性不等式或等式。二次规划问题在工程、经济、金融等领域有广泛应用&#xff0c;如投资组合优化、人脸表情动画的权重求解、机械…

Pycharm 输入三个引号没有自动生成函数(方法)注释

配置项路径&#xff1a;pycharm–>Settins–>Tools–>Python Integrated Tools–>Docstrings–>Docstrings format选择对应的工程&#xff0c;如果有多个工程的话将 Docstrings format 的值从 Plain 换成 reStructuredText

解决Matlab报错:MEX 文件 ‘D:\MATLAB\toolbox\maple\maplemex.mexw64‘ 无效: 缺少依赖共享库

安装Maple之后&#xff0c;再使用MATLAB就报了以上错误。 按照以下解决方法可以正常运行&#xff1a; 1. 在添加路径下删除D:\matlab\toolbox\Maple 2. 再添加路径 D:\matlab\toolbox\symbolic 3. 保存

JVM中运行时数据区

1.示例代码 无注释版本&#xff08;14行&#xff09; public class JVMMemoryModelDemo { private static int staticVar 10; private int instanceVar 20; public static void main(String[] args) { new JVMMemoryModelDemo().methodCall(); } public void method…

MySQL系统库——mysql库

mysql库 mysql库是什么&#xff1f;储存用户信息和权限的表&#xff1a;user表db表tables_priv表column_priv表运行规律&#xff1a; mysql库是什么&#xff1f; MySQL库是MySQL服务器内置的一个特殊的数据库。它包含了用于管理和控制MySQL服务器的系统表和系统变量。MySQL库存…

wifiip地址可以随便改吗?wifi的ip地址怎么改变

对于普通用户来说&#xff0c;WiFi IP地址的管理和修改往往显得神秘而复杂。本文旨在深入探讨WiFi IP地址是否可以随意更改&#xff0c;以及如何正确地改变WiFi的IP地址。虎观代理小二将详细解释WiFi IP地址的基本概念、作用以及更改时需要注意的事项&#xff0c;帮助用户更好地…