【python】数据爬虫,抓取并分析豆瓣电影信息

ops/2024/11/15 8:34:42/

url-lib-request模块获取HTML

spider.py
网页下载,链接提取,数据存储

http请求:
网址url,
http-header: (1)用户代理;(2)浏览器版本

网页下载用 urllib库。

python"># 将豆瓣电影top250的第一页下载import urllib.request # 导入url-lib-request模块# 设置网址
url = "https://movie.douban.com/top250"# http头信息
header = {"User-Agent":"Mozilla/5.0(Windows NT 10.0;WOW64)""AppleWebKit/537.36(KHTML, like Gecko)""Chrome/76.0.3809.87 Safari/537.36 SLBrowser/6.0.1.8131"
}# 构造一个Request请求对象
req = urllib.request.Request(url = url, headers = header)# 向豆瓣服务器发送请求,获取到回复response
response = urllib.request.urlopen(req)# 读取回复并使用UTF-8编码进行解码
html = response.read().decode("utf-8")# 打印HTML
print(html)

打印满足条件的网址

python">import urllib.request # 导入url-lib-request模块def download_html(url):# http头信息header = {"User-Agent":"Mozilla/5.0(Windows NT 10.0; WOW64)""AppleWebKit/537.36(KHTML, like Gecko)""Chrome/76.0.3809.87 Safari/537.36 SLBrowser/6.0.1.8131"}# 构造一个Request请求对象req = urllib.request.Request(url = url, headers = header)# 向豆瓣服务器发送请求,获取到回复responseresponse = urllib.request.urlopen(req)# 读取回复并使用UTF-8编码进行解码html = response.read().decode("utf-8")return html# 调用函数download_html,下载豆瓣top250第一页,保存至html
html = download_html("https://movie.douban.com/top250")import re # 导入re正则库# 电影介绍页网址的正则表达式
pattern = 'https://movie.douban.com/subject/[0-9]+/'# 从html中提取全部满足电影介绍页的网址pattern的url
urls = re.findall(pattern, html)urls = set(urls) # 通过set集合去重# 打印出提取出的网址数量和具体的网址
print("urls count = %d" % (len(urls)))
for url in urls:print(url)

douban.txt 文件内容:

https://movie.douban.com/top250
https://movie.douban.com/top250?start=25
https://movie.douban.com/top250?start=50
https://movie.douban.com/top250?start=75
https://movie.douban.com/top250?start=100
https://movie.douban.com/top250?start=125
https://movie.douban.com/top250?start=150
https://movie.douban.com/top250?start=175
https://movie.douban.com/top250?start=200
https://movie.douban.com/top250?start=225

写入到文件中

python">import urllib.request # 导入url-lib-request模块
import re # 导入re正则库def download_html(url):# http头信息header = {"User-Agent":"Mozilla/5.0(Windows NT 10.0; WOW64)""AppleWebKit/537.36(KHTML, like Gecko)""Chrome/76.0.3809.87 Safari/537.36 SLBrowser/6.0.1.8131"}# 构造一个Request请求对象req = urllib.request.Request(url = url, headers = header)# 向豆瓣服务器发送请求,获取到回复responseresponse = urllib.request.urlopen(req)# 读取回复并使用UTF-8编码进行解码html = response.read().decode("utf-8")return html# 调用函数download_html,下载豆瓣top250第一页,保存至html
html = download_html("https://movie.douban.com/top250")# 电影介绍页网址提取的功能封装至函数extract_url中,
# 函数输入html,返回html中提取的url连接
def extract_url(html):# 电影介绍页网址的正则表达式pattern = 'https://movie.douban.com/subject/[0-9]+/'# 从html中提取全部满足电影介绍页的网址pattern的urlurls = re.findall(pattern, html)return set(urls) # 通过set集合去重# 以读取方式打开豆瓣电影列表文件
file = open('douban.txt', 'r')
# 然后以写模式打开movie.txt,用来存储提取出的电影介绍页网址
output = open('movie.txt', 'w')# 将douban中的网址按行读入
lines = file.readlines()for url in lines:# 遍历每一行url,通过strip函数去掉首尾空字符url = url.strip()print(url) # 打印url作为调试信息# 调用download_html获取url对应的html代码html = download_html(url)# 调用extract_url提取html中的电影介绍页链接网址urls = extract_url(html)for url in urls: # 遍历url集合output.write(url + '\n') # 将每个url都写入movie文件
# 最后关闭douban和movie两个文件
file.close()
output.close()

movie.txt 结果:

https://movie.douban.com/subject/1889243/
https://movie.douban.com/subject/1849031/
https://movie.douban.com/subject/1291561/
https://movie.douban.com/subject/6786002/
https://movie.douban.com/subject/20495023/
https://movie.douban.com/subject/1291841/
https://movie.douban.com/subject/3011091/
https://movie.douban.com/subject/1307914/
https://movie.douban.com/subject/1291549/
https://movie.douban.com/subject/1292213/
https://movie.douban.com/subject/1295124/
https://movie.douban.com/subject/1292063/
https://movie.douban.com/subject/2131459/
https://movie.douban.com/subject/1292052/
https://movie.douban.com/subject/1291546/
https://movie.douban.com/subject/1292064/
https://movie.douban.com/subject/25662329/
https://movie.douban.com/subject/1292001/
https://movie.douban.com/subject/3793023/
https://movie.douban.com/subject/1292720/
https://movie.douban.com/subject/1295644/
https://movie.douban.com/subject/5912992/
https://movie.douban.com/subject/1296141/
https://movie.douban.com/subject/3541415/
https://movie.douban.com/subject/1292722/
https://movie.douban.com/subject/1292365/
https://movie.douban.com/subject/1305487/
https://movie.douban.com/subject/1293182/
https://movie.douban.com/subject/21937452/
https://movie.douban.com/subject/1291560/
https://movie.douban.com/subject/1291552/
https://movie.douban.com/subject/1296736/
https://movie.douban.com/subject/3742360/
https://movie.douban.com/subject/1291572/
https://movie.douban.com/subject/1295038/
https://movie.douban.com/subject/1293172/
https://movie.douban.com/subject/1299398/
https://movie.douban.com/subject/30170448/
https://movie.douban.com/subject/1291858/
https://movie.douban.com/subject/26387939/
https://movie.douban.com/subject/1300267/
https://movie.douban.com/subject/3319755/
https://movie.douban.com/subject/26752088/
https://movie.douban.com/subject/2129039/
https://movie.douban.com/subject/1298624/
https://movie.douban.com/subject/1851857/
https://movie.douban.com/subject/1308807/
https://movie.douban.com/subject/1929463/
https://movie.douban.com/subject/25958717/
https://movie.douban.com/subject/1291583/
https://movie.douban.com/subject/1291548/
https://movie.douban.com/subject/1293839/
https://movie.douban.com/subject/27060077/
https://movie.douban.com/subject/3011235/
https://movie.douban.com/subject/1301753/
https://movie.douban.com/subject/21937445/
https://movie.douban.com/subject/1652587/
https://movie.douban.com/subject/1292849/
https://movie.douban.com/subject/1418019/
https://movie.douban.com/subject/1299131/
https://movie.douban.com/subject/26580232/
https://movie.douban.com/subject/1292402/
https://movie.douban.com/subject/1292220/
https://movie.douban.com/subject/1485260/
https://movie.douban.com/subject/1291818/
https://movie.douban.com/subject/1293350/
https://movie.douban.com/subject/1291543/
https://movie.douban.com/subject/3008247/
https://movie.douban.com/subject/1291571/
https://movie.douban.com/subject/1291828/
https://movie.douban.com/subject/1292000/
https://movie.douban.com/subject/1306029/
https://movie.douban.com/subject/1900841/
https://movie.douban.com/subject/1294408/
https://movie.douban.com/subject/1291843/
https://movie.douban.com/subject/1298070/
https://movie.douban.com/subject/1306249/
https://movie.douban.com/subject/11525673/
https://movie.douban.com/subject/1302425/
https://movie.douban.com/subject/1292343/
https://movie.douban.com/subject/6985810/
https://movie.douban.com/subject/1292224/
https://movie.douban.com/subject/1292434/
https://movie.douban.com/subject/1291544/
https://movie.douban.com/subject/2334904/
https://movie.douban.com/subject/3442220/
https://movie.douban.com/subject/1300299/
https://movie.douban.com/subject/1780330/
https://movie.douban.com/subject/1292656/
https://movie.douban.com/subject/5322596/
https://movie.douban.com/subject/1296996/
https://movie.douban.com/subject/1303021/
https://movie.douban.com/subject/1292223/
https://movie.douban.com/subject/1291832/
https://movie.douban.com/subject/1293544/
https://movie.douban.com/subject/1294371/
https://movie.douban.com/subject/1297192/
https://movie.douban.com/subject/1865703/
https://movie.douban.com/subject/1292262/
https://movie.douban.com/subject/1292679/
https://movie.douban.com/subject/3395373/
https://movie.douban.com/subject/1418834/
https://movie.douban.com/subject/1305164/
https://movie.douban.com/subject/1294639/
https://movie.douban.com/subject/21318488/
https://movie.douban.com/subject/10777687/
https://movie.douban.com/subject/2149806/
https://movie.douban.com/subject/1787291/
https://movie.douban.com/subject/10577869/
https://movie.douban.com/subject/1395091/
https://movie.douban.com/subject/11026735/
https://movie.douban.com/subject/4202302/
https://movie.douban.com/subject/1291990/
https://movie.douban.com/subject/27010768/
https://movie.douban.com/subject/1297447/
https://movie.douban.com/subject/1292337/
https://movie.douban.com/subject/1291999/
https://movie.douban.com/subject/1297359/
https://movie.douban.com/subject/1292370/
https://movie.douban.com/subject/1293359/
https://movie.douban.com/subject/26799731/
https://movie.douban.com/subject/1296339/
https://movie.douban.com/subject/1300992/
https://movie.douban.com/subject/1297630/
https://movie.douban.com/subject/1291875/
https://movie.douban.com/subject/4268598/
https://movie.douban.com/subject/1309055/
https://movie.douban.com/subject/1292215/
https://movie.douban.com/subject/6307447/
https://movie.douban.com/subject/1303037/
https://movie.douban.com/subject/10533913/
https://movie.douban.com/subject/10437779/
https://movie.douban.com/subject/3287562/
https://movie.douban.com/subject/25814707/
https://movie.douban.com/subject/26628357/
https://movie.douban.com/subject/1828115/
https://movie.douban.com/subject/25986180/
https://movie.douban.com/subject/25814705/
https://movie.douban.com/subject/2353023/
https://movie.douban.com/subject/1294240/
https://movie.douban.com/subject/1297518/
https://movie.douban.com/subject/1297052/
https://movie.douban.com/subject/1291579/
https://movie.douban.com/subject/5989818/
https://movie.douban.com/subject/1307315/
https://movie.douban.com/subject/1418200/
https://movie.douban.com/subject/1858711/
https://movie.douban.com/subject/3072124/
https://movie.douban.com/subject/1461403/
https://movie.douban.com/subject/1292274/
https://movie.douban.com/subject/21360417/
https://movie.douban.com/subject/1793929/
https://movie.douban.com/subject/1578507/
https://movie.douban.com/subject/24733428/
https://movie.douban.com/subject/10463953/
https://movie.douban.com/subject/1417598/
https://movie.douban.com/subject/1292208/
https://movie.douban.com/subject/1293181/
https://movie.douban.com/subject/1302467/
https://movie.douban.com/subject/1316510/
https://movie.douban.com/subject/27119724/
https://movie.douban.com/subject/26325320/
https://movie.douban.com/subject/4920389/
https://movie.douban.com/subject/4917726/
https://movie.douban.com/subject/1307856/
https://movie.douban.com/subject/25917973/
https://movie.douban.com/subject/1306861/
https://movie.douban.com/subject/1300374/
https://movie.douban.com/subject/26683290/
https://movie.douban.com/subject/26611804/
https://movie.douban.com/subject/1291557/
https://movie.douban.com/subject/1907966/
https://movie.douban.com/subject/1297574/
https://movie.douban.com/subject/1295399/
https://movie.douban.com/subject/1291545/
https://movie.douban.com/subject/1291822/
https://movie.douban.com/subject/1293460/
https://movie.douban.com/subject/1292329/
https://movie.douban.com/subject/1292226/
https://movie.douban.com/subject/1307106/
https://movie.douban.com/subject/1304447/
https://movie.douban.com/subject/1291585/
https://movie.douban.com/subject/1295409/
https://movie.douban.com/subject/2209573/
https://movie.douban.com/subject/1959195/
https://movie.douban.com/subject/1867345/
https://movie.douban.com/subject/10808442/
https://movie.douban.com/subject/25773932/
https://movie.douban.com/subject/24750126/
https://movie.douban.com/subject/3792799/
https://movie.douban.com/subject/1291578/
https://movie.douban.com/subject/27622447/
https://movie.douban.com/subject/1291992/
https://movie.douban.com/subject/1307811/
https://movie.douban.com/subject/27059130/
https://movie.douban.com/subject/25895901/
https://movie.douban.com/subject/6534248/
https://movie.douban.com/subject/1862151/
https://movie.douban.com/subject/1310177/
https://movie.douban.com/subject/3011051/
https://movie.douban.com/subject/1291879/
https://movie.douban.com/subject/1292401/
https://movie.douban.com/subject/3592854/
https://movie.douban.com/subject/4739952/
https://movie.douban.com/subject/1438652/
https://movie.douban.com/subject/1308857/
https://movie.douban.com/subject/1303394/
https://movie.douban.com/subject/1297880/
https://movie.douban.com/subject/25934014/
https://movie.douban.com/subject/1293908/
https://movie.douban.com/subject/5300054/
https://movie.douban.com/subject/25980443/
https://movie.douban.com/subject/1959877/
https://movie.douban.com/subject/1422283/
https://movie.douban.com/subject/3075287/
https://movie.douban.com/subject/1309163/
https://movie.douban.com/subject/4848115/
https://movie.douban.com/subject/1291844/
https://movie.douban.com/subject/1978709/
https://movie.douban.com/subject/34874432/
https://movie.douban.com/subject/1292287/
https://movie.douban.com/subject/1296909/
https://movie.douban.com/subject/1291870/
https://movie.douban.com/subject/1302827/
https://movie.douban.com/subject/1292328/
https://movie.douban.com/subject/1428175/
https://movie.douban.com/subject/2213597/
https://movie.douban.com/subject/1305690/
https://movie.douban.com/subject/1419936/
https://movie.douban.com/subject/1308767/
https://movie.douban.com/subject/26430107/
https://movie.douban.com/subject/1304141/
https://movie.douban.com/subject/2051007/
https://movie.douban.com/subject/25724855/
https://movie.douban.com/subject/3552028/
https://movie.douban.com/subject/1300117/
https://movie.douban.com/subject/1293323/
https://movie.douban.com/subject/25807345/
https://movie.douban.com/subject/2222996/
https://movie.douban.com/subject/25864085/
https://movie.douban.com/subject/1307394/
https://movie.douban.com/subject/26374197/
https://movie.douban.com/subject/1293318/
https://movie.douban.com/subject/1304102/
https://movie.douban.com/subject/1293964/
https://movie.douban.com/subject/26615208/
https://movie.douban.com/subject/26787574/
https://movie.douban.com/subject/4202982/
https://movie.douban.com/subject/1457217/
https://movie.douban.com/subject/2133323/

http://www.ppmy.cn/ops/114416.html

相关文章

docker基本(仅供自己参考)

一、大型项目部署的问题: 1、大型项目的组件比较多,运行环境很复杂,部署通常会遇到各种问题: (1):依赖关系复杂,容易出现兼容性问题 (2):开发、…

电子元件制造5G智能工厂物联数字孪生平台,推进制造业数字化转型

5G智能工厂与物联数字孪生平台的融合应用,不仅为电容器制造业注入了新的活力,更为整个制造业的数字化转型树立了新的标杆。电子元件制造过程中,数字孪生平台通过实时监测生产线的各个环节,实现了生产流程的可视化监控。管理人员可…

【工具】Java Excel转图片

【工具】Java Excel转图片 package com.yj.luban.modules.office.excel;import org.apache.poi.ss.usermodel.*; import org.apache.poi.xssf.usermodel.XSSFWorkbook;import javax.imageio.ImageIO; import java.awt.Color; import java.awt.Font; import java.awt.*; import …

IO流中的异常捕获

Java简化 接口AutoCloseable

Android命令行启动settings

在Android设备上,你可以使用命令行启动“设置”应用。以下是通过ADB(Android调试桥)启动“设置”应用的方法: adb shell am start -a android.settings.SETTINGS这个命令将启动设备的“设置”应用。如果你想启动设置中的特定页面…

VB中的垃圾回收(Garbage Collection)机制

在Visual Basic(特别是VB.NET)中,垃圾回收(Garbage Collection, GC)机制是.NET Framework和.NET Core(以及.NET 5/6/7等后续版本)提供的一项核心功能,用于自动管理内存。这一机制对于…

链动321模式小程序开发源码

链动31模式概述 链动31模式是一种基于技术的新型商业模式,它通过激励用户分享和推广,实现用户、企业和平台的共赢。该模式通常涉及商品展示、积分系统、分享推广和排行榜等功能,旨在通过用户之间的社交裂变来扩大销售和品牌影响力。如何开发这…

c++primer 第八章函数编程答案

题一 #include <iostream> using namespace std;void print(char *str, int n 0);int main() {char str[20] "leonardo liu";print(str);print(str, 5);print(str, 16);return 0; }void print(char *str, int n) {static int flag 0; // 唯一初始化flag;if …