当前位置：首页 > news >正文

Python爬取TMDB电影数据：从登录到数据存储的全过程

news 来源：原创 2025/6/26 19:14:21

在当今数据驱动的时代，获取电影数据对于推荐系统、市场分析和个人项目都至关重要。本文将详细介绍如何使用Python构建一个完整的TMDB（The Movie Database）爬虫，从登录认证到数据解析和存储的全过程。（本来博主也想在CSDN里面上白嫖结果没有一篇文章，然后......）

1. 项目概述

TMDB是一个广受欢迎的电影数据库网站，包含了丰富的电影信息、演员数据和用户评分。我们的目标是构建一个爬虫，能够：

自动登录TMDB账号
抓取电影列表页数据
深入获取每部电影的详细信息
将数据保存为结构化的JSON格式

2. 技术栈

Requests：处理HTTP请求和会话管理
BeautifulSoup：HTML解析和数据提取
正则表达式：辅助提取特定格式的数据
类型提示：提高代码可读性和可维护性
随机延时：模拟人类行为，避免被封禁

3. 核心代码解析

3.1 初始化与登录

class TMDBScraper:def __init__(self):self.base_url = "https://www.themoviedb.org"self.username = "eqwaak"  # TMDB账号self.password = "xyx000"  # TMDB密码self.session = requests.Session()# 设置请求头self.headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...','Accept-Language': 'zh-CN,zh;q=0.9,zh-TW;q=0.8','Referer': 'https://www.themoviedb.org/','Origin': 'https://www.themoviedb.org'}self.session.headers.update(self.headers)self.request_delay = (1, 3)# 手动登录获取Cookieself._manual_login()

登录过程是爬虫的关键部分，TMDB采用了CSRF防护机制：

def _manual_login(self):# 获取登录页面以获取CSRF令牌login_url = f"{self.base_url}/login"login_page = self.session.get(login_url)soup = BeautifulSoup(login_page.text, 'html.parser')# 查找CSRF令牌csrf_token = soup.find('input', {'name': 'authenticity_token'}).get('value')# 准备登录数据login_data = {'username': self.username,'password': self.password,'authenticity_token': csrf_token,'remember': 'on','commit': 'Login'}# 提交登录请求login_response = self.session.post(login_url, data=login_data, allow_redirects=False)if login_response.status_code == 302 and 'session_id' in login_response.cookies:print("✅ 登录成功！")self._set_language('zh-CN')else:print("❌❌ 登录失败，尝试备用方法")self._fallback_login()

3.2 数据抓取与解析

电影列表页解析：

def parse_movie_list(self, html: str) -> List[Dict]:soup = BeautifulSoup(html, 'html.parser')cards = soup.select('.card.style_1')movies = []for card in cards:title = self._parse_text('h2', card)rating = self._parse_attribute('.user_score_chart', 'data-percent', card)year = self._parse_text('.release_date', card)href = self._parse_attribute('a', 'href', card, default="")# 提取海报图片URLposter_element = card.select_one('img.poster')poster_url = Noneif poster_element:poster_url = poster_element.get('src') or poster_element.get('data-src')if poster_url:poster_url = self._complete_url(poster_url)if title and href:movies.append({'title': title,'rating': rating,'year': year,'poster_url': poster_url,'detail_url': self.base_url + href + "?language=zh-CN"})return movies

电影详情页解析：

def parse_movie_details(self, html_content: str) -> Optional[Dict]:soup = BeautifulSoup(html_content, 'html.parser')# 获取标题和年份title = self._parse_text('.title h2 a', soup)year_info = self._extract_year(soup)details = {'title': title,'rating': self._parse_attribute('.user_score_chart', 'data-percent', soup) or "0",'year': year_info,'runtime': self._parse_text('.facts .runtime', soup) or "未知时长",'genres': [genre.get_text(strip=True) for genre in soup.select('.genres a')],'overview': self._parse_text('.overview p', soup) or "暂无简介",'cast': self._extract_cast(soup),'poster_url': self._extract_image_url(soup, '.poster img'),'backdrop_url': self._extract_image_url(soup, '.backdrop img')}return details

3.3 主运行逻辑

def run(self, max_pages: int = 2, save_path: str = 'tmdb_movies.json') -> None:all_movies = []# 抓取列表页for page in range(1, max_pages + 1):movies = self.fetch_page(f"{self.base_url}/movie", page)all_movies.extend(movies)time.sleep(random.uniform(2, 5))# 抓取详情页detailed_movies = []for movie in all_movies:detail_html = self._get_html(movie['detail_url'])if detail_html:details = self.parse_movie_details(detail_html)if details:detailed_movies.append(details)time.sleep(random.uniform(3, 6))# 保存数据with open(save_path, 'w', encoding='utf-8') as f:json.dump(detailed_movies, f, ensure_ascii=False, indent=2)

4. 关键技术与优化

4.1 反爬虫策略应对

请求头设置：模拟真实浏览器访问
随机延时：time.sleep(random.uniform(2, 5))避免频繁请求
会话保持：使用requests.Session()维持登录状态
CSRF处理：从页面提取并提交CSRF令牌

4.2 数据提取的健壮性

def _parse_text(self, selector: str, soup: BeautifulSoup) -> Optional[str]:"""通用文本解析方法"""element = soup.select_one(selector)return element.get_text(strip=True) if element else Nonedef _parse_attribute(self, selector: str, attr: str, soup: BeautifulSoup, default=None) -> Optional[str]:"""通用属性解析方法（带默认值）"""element = soup.select_one(selector)return element[attr] if element and attr in element.attrs else default

4.3 错误处理与日志

def _get_html(self, url: str) -> Optional[str]:"""内部请求方法（带重试机制和语言参数）"""for attempt in range(3):  # 重试3次try:response = self.session.get(url, timeout=10)response.raise_for_status()# 验证返回内容是否为中文if "中文" in response.text or "电影" in response.text:return response.textelse:with open('response.html', 'w', encoding='utf-8') as f:f.write(response.text)return Noneexcept requests.RequestException as e:if attempt < 2:time.sleep(2 ** attempt)continueprint(f"请求失败（{url}）: {str(e)}")return None