网站建设有技术的公司长沙靠谱seo优化

🙋‍♀️Tiktok APP的基于关键字检索的视频及评论信息爬虫共分为两期，希望对大家有所帮助。
第一期：基于关键字检索的视频信息爬取
第二期见下文。

1.Node.js环境配置

首先配置 JavaScript 运行环境（如 Node.js），用于执行加密签名代码。
Node.js下载网址：https://nodejs.org/en
Node.js的安装方法（环境配置非常关键，决定了后面的程序是否可以使用）：https://blog.csdn.net/liufeifeihuawei/article/details/132425239

2. Py环境配置

import random
from tqdm import tqdm
import requests
from urllib.parse import urlparse, urlencode
import warnings
from urllib3.exceptions import InsecureRequestWarning
import time# 忽略 InsecureRequestWarning 警告
warnings.filterwarnings("ignore", category=InsecureRequestWarning)

3. 基于视频URL的评论信息爬取

在上期中，已经给出了如何获得指定视频的URL，下面给出根据URL获得视频评论的信息，允许在爬取的过程中对评论进行翻页。

1. 主程序

爬单个URL的评论信息的方法：

if __name__ == '__main__':'''单条数据'''req_url = "https://www.tiktok.com/@resep_debm/video/7475545671383174406"tiktok_comment = TiktokComment()x = tiktok_comment.get_comment_list(req_url)print(x)

爬多个URL的评论信息的方法。通过读取videosInfo.json文件中保存的URL信息，将最后的结果保存到videos_comments.json文件中：

if __name__ == '__main__':'''多条数据'''data = read_json('../results/videosInfo.json')print(len(data))tiktok_comment = TiktokComment()new_data = data.copy()for i in tqdm(range(len(data))):if 'comments' not in data[i].keys():  #  and i > 1695comments = tiktok_comment.get_comment_list(data[i]['video_url'])if comments != []:new_data[i]['comments'] = commentselse:continueif i % 10 == 0:write_json('../results/videos_comments.json', new_data)# 循环结束后再保存一次，确保所有数据都被写入write_json('../results/videos_comments.json', new_data)

2. 定义TiktokComments类
允许获得的评论信息7个字段，包括：
🎰评论ID；
💬评论内容；
🙋评论是否被作者点赞；
😍评论是否热门；
👍评论的点赞数
👀评论的回复数目
⌛评论发布的时间；

class TiktokComments:def __init__(self):# self.config = read_config()self.common_utils = CommonUtils()self.cookies = cookie_str_to_dict(read_cookie())# self.proxies = self.config.get("proxies", None)  # 代理配置self.comment_list_headers = {'sec-ch-ua': '"Google Chrome";v="123", "Not:A-Brand";v="8", "Chromium";v="123"','sec-ch-ua-mobile': '?0','User-Agent': self.common_utils.user_agent,'sec-ch-ua-platform': '"Windows"','Accept': '*/*','Sec-Fetch-Site': 'same-origin','Sec-Fetch-Mode': 'cors','Sec-Fetch-Dest': 'empty','Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',}

通过 cursor_num设置翻页，count={comment_num}表示希望获得的评论总数目。

    def get_comment_list(self, video_url, comments_num=100):aweme_id = urlparse(video_url).path.split("/")[-1]ms_token = self.cookies['msToken']req_comments = []max_retries = 3  # 最大重试次数for i in range(comments_num // 20):cursor_num = i * 20comment_num = 20req_url = f"https://www.tiktok.com/api/comment/list/?WebIdLastTime=1715249710&aid=1988&app_language=ja-JP&app_name=tiktok_web&aweme_id={aweme_id}&browser_language=zh-CN&browser_name=Mozilla&browser_online=true&browser_platform=Win32&browser_version=5.0%20%28Windows%20NT%2010.0%3B%20Win64%3B%20x64%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Chrome%2F123.0.0.0%20Safari%2F537.36&channel=tiktok_web&cookie_enabled=true&" \f"count={comment_num}&current_region=JP&cursor={cursor_num}&device_id=7366941338308609569&device_platform=web_pc&enter_from=tiktok_web&focus_state=true&fromWeb=1&from_page=video&history_len=2&is_fullscreen=false&is_non_personalized=false&is_page_visible=true&odinId=7367172442253296673&os=windows&priority_region=&referer=&region=GB&screen_height=1080&screen_width=1920&tz_name=Asia%2FShanghai&webcast_language=zh-Hans&msToken={ms_token}"xbogus = self.common_utils.get_xbogus(req_url, self.common_utils.user_agent)req_url += f'&X-Bogus={xbogus}&_signature=_02B4Z6wo000016M20awAAIDAnp.LMKuZmC-jNtUAAI6L17'for retry in range(max_retries):try:response = requests.request('GET',req_url,headers=self.comment_list_headers,# cookies=self.cookies,verify=False,timeout=random.randint(3, 7),# proxies=self.proxies)if response.status_code != 200:continuereq_json = response.json()comments = req_json.get('comments', [])# print(f"评论数目：{req_json.get('total')}")if not comments:print(f"No comments found for cursor {cursor_num}.")breakfor comment_item in comments:req_comments.append({"cid": comment_item.get('cid'),"comment": comment_item.get('text'),"comments_is_author_like": comment_item.get('is_author_digged'),"comments_is_hot": comment_item.get('is_comment_translatable'),"comments_like": comment_item.get('digg_count'),"comments_reply": comment_item.get('reply_comment_total'),"comments_time": comment_item.get('create_time')})break  # 成功获取数据，退出重试循环except Exception as e:print(f"Error: {e}. Retrying ({retry + 1}/{max_retries})...")if retry == max_retries - 1:print("Max retries reached. Skipping this request.")return req_comments