Python爬虫使用实例-wallpaper

news/2024/12/22 15:14:30/

1/ 排雷避坑

🥝 中文乱码问题

python">print(requests.get(url=url,headers=headers).text)

在这里插入图片描述

出现中文乱码

原因分析:

<meta charset="gbk" />

解决方法:
法一:

python">response = requests.get(url=url,headers=headers)
response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码
print(response.text)

法二:

python">print(requests.get(url=url,headers=headers).content.decode('gbk'))

2/ 数据来源

css解析
在这里插入图片描述

python">for li in lis:href = li.css('a::attr(href)').get()title = li.css('b::text').get()print(href, title)

在这里插入图片描述
在这里插入图片描述
删掉标题为空的那一张图


在这里插入图片描述
获取图片url


有的网站,保存的数据是裂开的图片,可能是因为这个参数:
在这里插入图片描述

3/ 正则处理

处理图片url和标题的时候用了re模块

电脑壁纸
通过匹配非数字字符并在遇到数字时截断字符串

python">title1 = selector1.css('.photo .photo-pic img::attr(title)').get()
modified_title = re.split(r'\d', title1, 1)[0].strip()

re.split(r'\d', title, 1)将 title 字符串按第一个数字进行分割。返回的列表的第一个元素就是数字前面的部分。strip() 去掉字符串首尾的空白字符。


url图片路径替换,因为从点开图片到达的那个页面无法得到的图片路径还是html页面,不是https://····.jpg,所以替换成另一个可以获取到的页面。

python">https://sj.zol.com.cn/bizhi/detail_{num1}_{num2}.html

正则替换修改为

python">https://app.zol.com.cn/bizhi/detail_{num1}.html

例如 https://sj.zol.com.cn/bizhi/detail_12901_139948.html 转换为 https://app.zol.com.cn/bizhi/detail_12901.html .

python"># https://sj.zol.com.cn/bizhi/detail_12901_139948.html
url = "https://sj.zol.com.cn/bizhi/detail_12901_139948.html"
pattern = r'https://sj\.zol\.com\.cn/bizhi/detail_(\d+)_\d+\.html'
replacement = r'https://app.zol.com.cn/bizhi/detail_\1.html'
new_url = re.sub(pattern, replacement, url)
print(url,new_url)

4/ 电脑壁纸

🥝 单线程单页

适用于当页面和第一页

python"># python单线程爬取高清4k壁纸图片
import os
import re
import requests
import parsel
url = 'https://pic.netbian.com/4kmeinv/' # 请求地址
# 模拟伪装
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}
# response = requests.get(url=url,headers=headers)
# response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码
# print(response.text)
html_data = requests.get(url=url,headers=headers).content.decode('gbk')
# print(html_data)
selector = parsel.Selector(html_data)
lis = selector.css('.slist li')
for li in lis:#href = li.css('a::attr(href)').get()title = li.css('b::text').get()if title:href = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response = requests.get(url=href, headers=headers)#print(href, title)# 这里只是获取页面# img_content = requests.get(url=href, headers=headers).content# 不可行, 都是同一张图 https://pic.netbian.com/uploads/allimg/230813/221347-16919360273e05.jpghref = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response1 = requests.get(url=href, headers=headers).content.decode('gbk')selector1 = parsel.Selector(response1)# 若要标题乱码,此处可不解码# response1 = requests.get(url=href, headers=headers)# selector1 = parsel.Selector(response1.text)# img_url = selector1.css('.slist li img::attr(src)').get()# 这一步错了, 要去href页面找img_url, 这是在原来的url页面找了img_url = 'https://pic.netbian.com' + selector1.css('.photo .photo-pic img::attr(src)').get()img_content = requests.get(url=img_url,headers=headers).content# 顺便更新一下title, 因为原来的是半截的, 不全title1 = selector1.css('.photo .photo-pic img::attr(title)').get()modified_title = re.split(r'\d', title1, 1)[0].strip()with open('img\\'+modified_title+'.jpg',mode='wb') as f:f.write(img_content)#print(href, title)print('正在保存:', modified_title, img_url)

🥝 单线程多page

适用于从第二页开始的多页

python"># python单线程爬取高清4k壁纸图片
import os
import re
import time
import requests
import parsel
# url的规律
# https://pic.netbian.com/new/index.html
# https://pic.netbian.com/new/index_1.html
# https://pic.netbian.com/new/index_2.html
# ...
start_time = time.time()
for page in range(2,10):print(f'--------- 正在爬取第{page}的内容 ----------')url = f'https://pic.netbian.com/4kmeinv/index_{page}.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}# response = requests.get(url=url,headers=headers)# response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码# print(response.text)html_data = requests.get(url=url, headers=headers).content.decode('gbk')# print(html_data)selector = parsel.Selector(html_data)lis = selector.css('.slist li')for li in lis:# href = li.css('a::attr(href)').get()title = li.css('b::text').get()if title:href = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response = requests.get(url=href, headers=headers)# print(href, title)# 这里只是获取页面# img_content = requests.get(url=href, headers=headers).content# 不可行, 都是同一张图 https://pic.netbian.com/uploads/allimg/230813/221347-16919360273e05.jpghref = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response1 = requests.get(url=href, headers=headers).content.decode('gbk')selector1 = parsel.Selector(response1)# 若要标题乱码,此处可不解码# response1 = requests.get(url=href, headers=headers)# selector1 = parsel.Selector(response1.text)# img_url = selector1.css('.slist li img::attr(src)').get()# 这一步错了, 要去href页面找img_url, 这是在原来的url页面找了img_url = 'https://pic.netbian.com' + selector1.css('.photo .photo-pic img::attr(src)').get()img_content = requests.get(url=img_url, headers=headers).content# 顺便更新一下title, 因为原来的是半截的, 不全title1 = selector1.css('.photo .photo-pic img::attr(title)').get()modified_title = re.split(r'\d', title1, 1)[0].strip()with open('img\\' + modified_title + '.jpg', mode='wb') as f:f.write(img_content)# print(href, title)print('正在保存:', modified_title, img_url)stop_time = time.time()
print(f'耗时:{int(stop_time)-int(start_time)}秒')

运行效果:

在这里插入图片描述
在这里插入图片描述

🥝 多线程多页

python"># python多线程爬取高清4k壁纸图片
import os
import re
import time
import requests
import parsel
import concurrent.futuresdef get_img(url):# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}# response = requests.get(url=url,headers=headers)# response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码# print(response.text)html_data = requests.get(url=url, headers=headers).content.decode('gbk')# print(html_data)selector = parsel.Selector(html_data)lis = selector.css('.slist li')for li in lis:# href = li.css('a::attr(href)').get()title = li.css('b::text').get()if title:href = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response = requests.get(url=href, headers=headers)# print(href, title)# 这里只是获取页面# img_content = requests.get(url=href, headers=headers).content# 不可行, 都是同一张图 https://pic.netbian.com/uploads/allimg/230813/221347-16919360273e05.jpghref = 'https://pic.netbian.com' + li.css('a::attr(href)').get()response1 = requests.get(url=href, headers=headers).content.decode('gbk')selector1 = parsel.Selector(response1)# 若要标题乱码,此处可不解码# response1 = requests.get(url=href, headers=headers)# selector1 = parsel.Selector(response1.text)# img_url = selector1.css('.slist li img::attr(src)').get()# 这一步错了, 要去href页面找img_url, 这是在原来的url页面找了img_url = 'https://pic.netbian.com' + selector1.css('.photo .photo-pic img::attr(src)').get()img_content = requests.get(url=img_url, headers=headers).content# 顺便更新一下title, 因为原来的是半截的, 不全title1 = selector1.css('.photo .photo-pic img::attr(title)').get()modified_title = re.split(r'\d', title1, 1)[0].strip()img_folder = 'img1\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + modified_title + '.jpg', mode='wb') as f:f.write(img_content)# print(href, title)print('正在保存:', modified_title, img_url)
def main(url):get_img(url)start_time = time.time()
executor = concurrent.futures.ThreadPoolExecutor(max_workers=5)
for page in range(2, 12):print(f'--------- 正在爬取第{page}的内容 ----------')url = f'https://pic.netbian.com/4kmeinv/index_{page}.html'  # 请求地址executor.submit(main, url)
executor.shutdown()
stop_time = time.time()
print(f'耗时:{int(stop_time) - int(start_time)}秒')

5/ 手机壁纸

类似地,另一个网站,图片集合多页,点开之后里面有多张图片
先试图获取外部的,再获取里面的,然后2个一起

🥝 单线程单页0

python">import os
import re
import requests
import parsel
url = 'https://sj.zol.com.cn/bizhi/5/' # 请求地址
# 模拟伪装
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}
# response = requests.get(url=url,headers=headers)
# response.encoding = response.apparent_encoding # 自动转码, 防止中文乱码
# print(response.text)
response = requests.get(url=url,headers=headers)
#print(response.text)
selector = parsel.Selector(response.text)
lis = selector.css('.pic-list2 li')
#img_name=1
for li in lis:#href = li.css('a::attr(href)').get()title = li.css('.pic img::attr(title)').get()#href = li.css('.pic img::attr(src)').get()#print(title, href)if title:#href = 'https://sj.zol.com.cn' +li.css('a::attr(href)').get()# https://sj.zol.com.cn/bizhi/detail_12901_139948.html# https://app.zol.com.cn/bizhi/detail_12901_139948.html#p1#href = 'https://app.zol.com.cn' + li.css('a::attr(href)').get() + '#p1'href=li.css('img::attr(src)').get()#print(href, title)#href = 'https://app.zol.com.cn' + li.css('a::attr(href)').get() + '#p1'#response1 = requests.get(url=href, headers=headers).content.decode('utf-8')#selector1 = parsel.Selector(response1)#img_url=selector1.css('.gallery li img::attr(src)').get()#print(img_url)# 这里只是获取页面img_content = requests.get(url=href, headers=headers).content# 不可行, 都是同一张图 https://pic.netbian.com/uploads/allimg/230813/221347-16919360273e05.jpg# https://sj.zol.com.cn/bizhi/detail_12901_139948.html# https://app.zol.com.cn/bizhi/detail_12901_139948.html#p1#href= selector1.css('.photo-list-box li::attr(href)').get()#href = 'https://app.zol.com.cn' +  + '#p1'#response2 = requests.get(url=href, headers=headers)#selector2 = parsel.Selector(response2.text)#print(href)# 若要标题乱码,此处可不解码# response1 = requests.get(url=href, headers=headers)# selector1 = parsel.Selector(response1.text)# img_url = selector1.css('.slist li img::attr(src)').get()# 这一步错了, 要去href页面找img_url, 这是在原来的url页面找了#img_url = selector1.css('.gallery img::attr(src)').get()#img_content = requests.get(url=img_url, headers=headers).content#print(img_url)# 顺便更新一下title, 因为原来的是半截的, 不全# title1 = selector1.css('.photo .photo-pic img::attr(title)').get()img_folder = 'img3\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + title + '.jpg', mode='wb') as f:f.write(img_content)# print(href, title)print('正在保存:', title, href)#img_name += 1

🥝 单线程单页1

python"># 下载子页面全部
import os
import requests
import parselurl = 'https://app.zol.com.cn/bizhi/detail_12901.html' # 请求地址
# 模拟伪装
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}
response = requests.get(url=url,headers=headers)
selector = parsel.Selector(response.text)
lis = selector.css('.album-list li')
i = 0
for li in lis:# Get all img elements within the current liimg_tags = li.css('img::attr(src)').getall()  # This gets all the img src attributesfor href in img_tags:  # Iterate over all img src attributesimg_content = requests.get(url=href, headers=headers).contentimg_folder = 'img4\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + str(i) + '.jpg', mode='wb') as f:f.write(img_content)# print(href, i)print('正在保存:', i, href)i += 1  # Increment i for each image saved

🥝 单线程单页

python">import os
import re
import requests
import parselurl = 'https://sj.zol.com.cn/bizhi/5/' # 请求地址
# 模拟伪装
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}
response = requests.get(url=url,headers=headers)
#print(response.text)
selector = parsel.Selector(response.text)
#lis = selector.css('.pic-list2 li')
# 筛除包含的底部 3个 猜你喜欢
lis=selector.css('.pic-list2 .photo-list-padding')
for li in lis:#href = li.css('a::attr(href)').get()title = li.css('.pic img::attr(title)').get()href = li.css('a::attr(href)').get()#print(title, href)# https://sj.zol.com.cn/bizhi/detail_12901_139948.html#url = "https://sj.zol.com.cn/bizhi/detail_12901_139948.html"pattern = r'/bizhi/detail_(\d+)_\d+\.html'replacement = r'https://app.zol.com.cn/bizhi/detail_\1.html'new_url = re.sub(pattern, replacement, href)#print(href, new_url)#url = 'https://app.zol.com.cn/bizhi/detail_12901.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=new_url, headers=headers)selector = parsel.Selector(response.text)lis1 = selector.css('.album-list li')i = 0for li1 in lis1:# Get all img elements within the current liimg_tags = li1.css('img::attr(src)').getall()  # This gets all the img src attributesfor href in img_tags:  # Iterate over all img src attributesimg_content = requests.get(url=href, headers=headers).contentimg_folder = 'img5\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + title+'_'+str(i) + '.jpg', mode='wb') as f:f.write(img_content)# print(href, i)print('正在保存:',title+'_'+str(i), href)i += 1  # Increment i for each image saved

在这里插入图片描述

🥝 单线程多页

python">import os
import re
import requests
import parselfor page in range(1,3):print(f'--------- 正在爬取第{page}的内容 ----------')if page==1:url = 'https://sj.zol.com.cn/bizhi/5/'  # 请求地址else:url = f'https://sj.zol.com.cn/bizhi/5/{page}.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=url, headers=headers)# print(response.text)selector = parsel.Selector(response.text)# lis = selector.css('.pic-list2 li')# 筛除包含的底部 3个 猜你喜欢lis = selector.css('.pic-list2 .photo-list-padding')for li in lis:# href = li.css('a::attr(href)').get()title = li.css('.pic img::attr(title)').get()href = li.css('a::attr(href)').get()# print(title, href)# https://sj.zol.com.cn/bizhi/detail_12901_139948.html# url = "https://sj.zol.com.cn/bizhi/detail_12901_139948.html"pattern = r'/bizhi/detail_(\d+)_\d+\.html'replacement = r'https://app.zol.com.cn/bizhi/detail_\1.html'new_url = re.sub(pattern, replacement, href)# print(href, new_url)# url = 'https://app.zol.com.cn/bizhi/detail_12901.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=new_url, headers=headers)selector = parsel.Selector(response.text)lis1 = selector.css('.album-list li')i = 0for li1 in lis1:# Get all img elements within the current liimg_tags = li1.css('img::attr(src)').getall()  # This gets all the img src attributesfor href in img_tags:  # Iterate over all img src attributesimg_content = requests.get(url=href, headers=headers).contentimg_folder = 'img6\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + title + '_' + str(i) + '.jpg', mode='wb') as f:f.write(img_content)# print(href, i)print('正在保存:', title + '_' + str(i), href)i += 1  # Increment i for each image saved

🥝 多线程多页

python">import os
import re
import time
import requests
import parsel
import concurrent.futuresdef get_imgs(url):# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=url, headers=headers)# print(response.text)selector = parsel.Selector(response.text)# lis = selector.css('.pic-list2 li')# 筛除包含的底部 3个 猜你喜欢lis = selector.css('.pic-list2 .photo-list-padding')for li in lis:# href = li.css('a::attr(href)').get()title = li.css('.pic img::attr(title)').get()href = li.css('a::attr(href)').get()# print(title, href)# https://sj.zol.com.cn/bizhi/detail_12901_139948.html# url = "https://sj.zol.com.cn/bizhi/detail_12901_139948.html"pattern = r'/bizhi/detail_(\d+)_\d+\.html'replacement = r'https://app.zol.com.cn/bizhi/detail_\1.html'new_url = re.sub(pattern, replacement, href)# print(href, new_url)# url = 'https://app.zol.com.cn/bizhi/detail_12901.html'  # 请求地址# 模拟伪装headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.139 Safari/537.36'}response = requests.get(url=new_url, headers=headers)selector = parsel.Selector(response.text)lis1 = selector.css('.album-list li')i = 0for li1 in lis1:# Get all img elements within the current liimg_tags = li1.css('img::attr(src)').getall()  # This gets all the img src attributesfor href in img_tags:  # Iterate over all img src attributesimg_content = requests.get(url=href, headers=headers).contentimg_folder = 'img7\\'if not os.path.exists(img_folder):os.makedirs(img_folder)with open(img_folder + title + '_' + str(i) + '.jpg', mode='wb') as f:f.write(img_content)# print(href, i)print('正在保存:', title + '_' + str(i), href)i += 1  # Increment i for each image saveddef main(url):get_imgs(url)start_time = time.time()
executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)
for page in range(1, 9):#print(f'--------- 正在爬取第{page}的内容 ----------')if page == 1:url = 'https://sj.zol.com.cn/bizhi/5/'  # 请求地址else:url = f'https://sj.zol.com.cn/bizhi/5/{page}.html'  # 请求地址executor.submit(main, url)
executor.shutdown()
stop_time = time.time()
print(f'耗时:{int(stop_time) - int(start_time)}秒')

http://www.ppmy.cn/news/1523255.html

相关文章

思科交换机端口安全配置1

#网络安全技术实现# #任务一交换机端口安全配置1# #1配置计算机的IP 地址、子网掩码和网关 #2配置交换机B的主机名称&#xff0c;创建vlan 10和vlan 20&#xff0c;将f0/1、2划入vlan 10&#xff0c;f0/3、4划入vlan 20&#xff0c;将f0/24配置为Trunk Switch(config)#hostna…

学习-编写函数交换两个变量的值

编写一个函数&#xff0c;可以交换两个变量的值。可以使用以下代码实现&#xff1a; def swap_values(a, b):temp aa bb tempreturn a, b# 测试交换两个变量的值 x 10 y 20 print("交换前&#xff1a;x ", x, "y ", y) x, y swap_values(x, y) prin…

SSM框架学习(二:SpringFramework实战指南)

目录 一、SpringFramework介绍 1.总体技术体系 &#xff08;1&#xff09;单一架构 &#xff08;2&#xff09; 分布式架构 2.Spring 和 SpringFramework概念 &#xff08;1&#xff09;广义的 Spring&#xff1a;Spring 技术栈&#xff08;全家桶&#xff09; &#xff…

Curator+Zookeeper实现分布式锁的示例

1. 文章导读 随着互联网的兴起&#xff0c;应用的单服务器部署已无法满足日益增长的用户需求&#xff0c;此时云服务分布式技术应用而生&#xff0c;解决了应用弹性伸缩的问题。 问题来了&#xff0c;分布式不同于单机&#xff0c;在单机应用下可以使用Java的并发编程API实现对…

k8s集群部署:安装 kubeadm

1、确保已经将 SELinux 设置为permissive模式&#xff1a; 这些说明适用于 Kubernetes 1.31。 # Set SELinux in permissive mode (effectively disabling it) sudo setenforce 0 sudo sed -i s/^SELINUXenforcing$/SELINUXpermissive/ /etc/selinux/config2、下载并安装相关…

2024数学建模国赛B题代码

B题已经完成模型代码&#xff01;详情查看文末名片 问题1&#xff1a;可以考虑使用统计学中的“样本量估算”方法&#xff0c;使用二项分布或正态近似来决定最少的样本量&#xff0c;并通过假设检验&#xff08;如单侧检验&#xff09;在95%和90%置信度下进行判断。 import n…

关于 win10安装.NET framework 3.5失败 的解决方法

文为原创文章&#xff0c;转载请注明原文出处 本文章博客地址&#xff1a;https://hpzwl.blog.csdn.net/article/details/142030841 长沙红胖子Qt&#xff08;长沙创微智科&#xff09;博文大全&#xff1a;开发技术集合&#xff08;包含Qt实用技术、树莓派、三维、OpenCV、Op…

C++——重写、重载、隐藏

类成员函数的重写(Overriding)、重载(Overloading)和隐藏(Hiding)是面向对象编程中常见的概念,它们分别具有不同的含义和应用场景。 一、重写(Overriding) 简单来说,就是派生类重写基类的虚函数,在不改变虚函数的返回类型,函数名和参数列表的情况 下,对函数内容…