当前位置：首页 > news >正文

【网络与爬虫 38】Apify全栈指南：从0到1构建企业级自动化爬虫平台

news 2025/8/3 8:19:50

关键词： Apify、网页自动化、数据提取平台、爬虫即服务、Playwright集成、无服务器爬虫、Actor开发、云端部署、数据管道、企业级爬虫

摘要： 本文全面解析Apify这一强大的网页自动化与数据提取平台，从传统爬虫开发的复杂性出发，详细介绍如何利用Apify构建企业级自动化爬虫系统。文章涵盖平台架构、Actor开发、实战应用和最佳实践，帮助读者快速掌握现代化爬虫开发的核心技能。

文章目录

- 引言：爬虫开发的演进之路
- 什么是Apify？
- - 核心理念：自动化即服务(Automation as a Service)
  - Apify vs 传统爬虫对比
- 快速上手：10分钟搭建第一个爬虫
- - 方式一：使用现成的Actor
  - 方式二：创建自定义Actor
  - Actor配置文件
- Actor深度开发指南
- - 1. 高级数据提取技术
  - 2. 智能反反爬虫策略
  - 3. 动态内容处理
- 企业级应用场景
- - 1. 电商竞品监控系统
  - 2. 新闻舆情监控系统
- 性能优化与最佳实践
- - 1. 并发控制与资源管理
  - 2. 成本优化策略
- 总结与展望
- - 技术发展趋势
  - 选择建议

引言：爬虫开发的演进之路

想象一下这样的场景：你是一名数据工程师，公司需要从500个电商网站实时监控竞品价格。传统方式下，你需要：

开发阶段：

为每个网站编写独立的爬虫脚本
处理各种反爬虫机制
搭建分布式爬虫架构
实现数据存储和清洗

运维阶段：

监控500个爬虫的运行状态
处理网站结构变化导致的脚本失效
应对IP封禁和验证码问题
维护服务器和扩容资源

这个过程可能需要数月时间和庞大的技术团队。但如果告诉你，有一个平台可以让你用可视化界面在几分钟内创建爬虫，并且自动处理所有运维问题，你相信吗？

这就是Apify要解决的问题——将复杂的爬虫开发简化为简单的配置和部署。

什么是Apify？

Apify是一个革命性的网页自动化与数据提取平台，它将爬虫开发从传统的"代码驱动"模式转变为"平台驱动"模式。

核心理念：自动化即服务(Automation as a Service)

传统爬虫开发就像手工制作汽车，每个零件都需要自己制造。而Apify就像现代化的汽车工厂，提供标准化的生产线和可复用的组件。

Apify的三大核心组件：

1. Actor Store（应用商店）

1000+预构建的爬虫应用
覆盖主流网站和应用场景
即插即用，无需编程

2. Apify Platform（云平台）

无服务器执行环境
自动扩缩容
内置数据存储和API

3. Apify SDK（开发工具包）

基于Playwright/Puppeteer
丰富的辅助工具
本地开发和云端部署无缝集成

Apify vs 传统爬虫对比

维度	传统爬虫	Apify平台
开发时间	数周到数月	几分钟到几小时
技术门槛	高（需要深度编程）	低（可视化配置）
运维复杂度	极高	零运维
扩展性	手动扩容	自动弹性扩展
成本	高（人力+基础设施）	按使用量计费
维护难度	持续投入	平台自动维护

快速上手：10分钟搭建第一个爬虫

让我们从最简单的例子开始，感受Apify的魅力。

方式一：使用现成的Actor

// 1. 安装Apify CLI
npm install -g apify-cli// 2. 登录Apify平台
apify login// 3. 运行预构建的爬虫
apify call apify/web-scraper --input '{"startUrls": [{"url": "https://example.com"}],"pageFunction": "async function pageFunction(context) { return { title: await context.page.title() }; }"
}'

方式二：创建自定义Actor

// main.js
const Apify = require('apify');Apify.main(async () => {// 获取输入参数const input = await Apify.getInput();const { startUrls, maxCrawledPages = 10 } = input;// 创建请求队列const requestQueue = await Apify.openRequestQueue();for (const startUrl of startUrls) {await requestQueue.addRequest({ url: startUrl.url });}// 配置爬虫const crawler = new Apify.PlaywrightCrawler({requestQueue,maxRequestsPerCrawl: maxCrawledPages,async requestHandler({ page, request }) {console.log(`Processing: ${request.url}`);// 等待页面加载await page.waitForLoadState('networkidle');// 提取数据const title = await page.title();const description = await page.$eval('meta[name="description"]', el => el.content).catch(() => '');// 保存数据await Apify.pushData({url: request.url,title,description,timestamp: new Date().toISOString()});},async failedRequestHandler({ request }) {console.log(`Request ${request.url} failed too many times.`);},});// 启动爬虫await crawler.run();console.log('Crawler finished.');
});

Actor配置文件

{"actorSpecification": 1,"name": "my-first-scraper","title": "我的第一个爬虫","description": "使用Playwright的基础网页爬虫","version": "1.0.0","meta": {"templateId": "playwright-node"},"input": "./input_schema.json","dockerfile": "./Dockerfile"
}

仅仅几十行代码，我们就创建了一个功能完整的爬虫！这在传统开发中可能需要数百行代码和复杂的配置。

Actor深度开发指南

1. 高级数据提取技术

class AdvancedDataExtractor {constructor() {this.selectors = {title: ['h1', '.title', '#title', '[data-title]'],price: ['.price', '.cost', '[data-price]', '.amount'],image: ['img[src]', '.image img', '.photo img'],description: ['.description', '.desc', '.summary']};}async extractWithFallback(page, fieldName) {const selectors = this.selectors[fieldName] || [];for (const selector of selectors) {try {const element = await page.$(selector);if (element) {let value;if (fieldName === 'image') {value = await element.getAttribute('src');} else {value = await element.textContent();}if (value && value.trim()) {return this.cleanValue(value, fieldName);}}} catch (error) {console.log(`Selector ${selector} failed:`, error.message);}}return null;}cleanValue(value, fieldName) {value = value.trim();switch (fieldName) {case 'price':// 提取价格数字const priceMatch = value.match(/[\d,]+\.?\d*/);return priceMatch ? parseFloat(priceMatch[0].replace(/,/g, '')) : null;case 'description':// 限制描述长度return value.length > 500 ? value.substring(0, 500) + '...' : value;default:return value;}}async extractPageData(page) {const data = {};// 并行提取所有字段const extractions = Object.keys(this.selectors).map(async (field) => {data[field] = await this.extractWithFallback(page, field);});await Promise.all(extractions);// 添加元数据data.url = page.url();data.extractedAt = new Date().toISOString();data.userAgent = await page.evaluate(() => navigator.userAgent);return data;}
}// 在主爬虫中使用
Apify.main(async () => {const extractor = new AdvancedDataExtractor();const crawler = new Apify.PlaywrightCrawler({async requestHandler({ page, request }) {const data = await extractor.extractPageData(page);await Apify.pushData(data);}});await crawler.run();
});

2. 智能反反爬虫策略

class AntiDetectionManager {constructor() {this.userAgents = ['Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36','Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36'];this.viewports = [{ width: 1920, height: 1080 },{ width: 1366, height: 768 },{ width: 1440, height: 900 }];}getRandomUserAgent() {return this.userAgents[Math.floor(Math.random() * this.userAgents.length)];}getRandomViewport() {return this.viewports[Math.floor(Math.random() * this.viewports.length)];}async setupBrowserContext(context) {// 设置随机用户代理await context.setExtraHTTPHeaders({'User-Agent': this.getRandomUserAgent(),'Accept-Language': 'en-US,en;q=0.9','Accept-Encoding': 'gzip, deflate, br',});// 设置随机视窗大小const viewport = this.getRandomViewport();await context.setViewportSize(viewport);// 模拟人类行为await context.addInitScript(() => {// 覆盖webdriver属性Object.defineProperty(navigator, 'webdriver', {get: () => undefined,});// 添加缺失的插件Object.defineProperty(navigator, 'plugins', {get: () => [1, 2, 3, 4, 5],});});}async humanLikeDelay(min = 1000, max = 3000) {const delay = Math.random() * (max - min) + min;await new Promise(resolve => setTimeout(resolve, delay));}async simulateHumanBehavior(page) {// 随机鼠标移动await page.mouse.move(Math.random() * 800,Math.random() * 600);// 随机滚动await page.evaluate(() => {window.scrollBy(0, Math.random() * 500);});// 人类化延迟await this.humanLikeDelay();}
}// 集成到爬虫中
Apify.main(async () => {const antiDetection = new AntiDetectionManager();const crawler = new Apify.PlaywrightCrawler({launchContext: {useChrome: true,launchOptions: {headless: true,args: ['--no-sandbox', '--disable-setuid-sandbox']}},async requestHandler({ page, request }) {// 设置反检测await antiDetection.setupBrowserContext(page.context());// 模拟人类行为await antiDetection.simulateHumanBehavior(page);// 执行数据提取const data = await page.evaluate(() => {return {title: document.title,content: document.body.innerText.slice(0, 1000)};});await Apify.pushData(data);}});await crawler.run();
});

3. 动态内容处理

class DynamicContentHandler {async waitForDynamicContent(page, options = {}) {const {selector = null,timeout = 30000,waitForFunction = null,minLoadTime = 2000} = options;// 等待最小加载时间await new Promise(resolve => setTimeout(resolve, minLoadTime));if (selector) {// 等待特定元素出现await page.waitForSelector(selector, { timeout });}if (waitForFunction) {// 等待自定义条件await page.waitForFunction(waitForFunction, { timeout });}// 等待网络空闲await page.waitForLoadState('networkidle');}async handleInfiniteScroll(page, maxScrolls = 10) {let scrollCount = 0;let lastHeight = 0;while (scrollCount < maxScrolls) {// 滚动到底部await page.evaluate(() => {window.scrollTo(0, document.body.scrollHeight);});// 等待新内容加载await new Promise(resolve => setTimeout(resolve, 2000));// 检查页面高度是否变化const newHeight = await page.evaluate(() => document.body.scrollHeight);if (newHeight === lastHeight) {break; // 没有新内容了}lastHeight = newHeight;scrollCount++;}console.log(`完成无限滚动，共滚动 ${scrollCount} 次`);}async handlePagination(page, maxPages = 5) {const results = [];let currentPage = 1;while (currentPage <= maxPages) {console.log(`处理第 ${currentPage} 页`);// 提取当前页数据const pageData = await page.evaluate(() => {return Array.from(document.querySelectorAll('.item')).map(item => ({title: item.querySelector('.title')?.textContent,link: item.querySelector('a')?.href}));});results.push(...pageData);// 查找下一页按钮const nextButton = await page.$('.next-page, .pagination-next, [aria-label="Next"]');if (!nextButton) {console.log('没有找到下一页按钮，停止翻页');break;}// 点击下一页await nextButton.click();// 等待页面加载await this.waitForDynamicContent(page, {selector: '.item',timeout: 10000});currentPage++;}return results;}async handleAjaxContent(page, ajaxConfig) {const { triggerSelector, resultSelector, maxWaitTime = 10000 } = ajaxConfig;// 监听网络请求const responses = [];page.on('response', response => {if (response.url().includes('api') || response.url().includes('ajax')) {responses.push(response);}});// 触发AJAX请求if (triggerSelector) {await page.click(triggerSelector);}// 等待AJAX响应await page.waitForTimeout(2000);// 等待结果元素出现if (resultSelector) {await page.waitForSelector(resultSelector, { timeout: maxWaitTime });}return responses;}
}// 使用示例
Apify.main(async () => {const contentHandler = new DynamicContentHandler();const crawler = new Apify.PlaywrightCrawler({async requestHandler({ page, request }) {const url = request.url;if (url.includes('infinite-scroll')) {// 处理无限滚动页面await contentHandler.handleInfiniteScroll(page, 5);} else if (url.includes('pagination')) {// 处理分页const allData = await contentHandler.handlePagination(page, 3);await Apify.pushData({ url, items: allData });} else {// 处理一般动态内容await contentHandler.waitForDynamicContent(page, {selector: '.content-loaded',timeout: 15000});}// 提取最终数据const finalData = await page.evaluate(() => {return {title: document.title,itemCount: document.querySelectorAll('.item').length};});await Apify.pushData(finalData);}});await crawler.run();
});

企业级应用场景

1. 电商竞品监控系统

class EcommerceMonitor {constructor() {this.competitors = [{ name: 'Amazon', baseUrl: 'https://amazon.com' },{ name: 'eBay', baseUrl: 'https://ebay.com' },{ name: 'Walmart', baseUrl: 'https://walmart.com' }];}async monitorProducts(productKeywords) {const results = [];for (const competitor of this.competitors) {console.log(`监控 ${competitor.name} 平台`);const competitorData = await this.scrapeCompetitor(competitor, productKeywords);results.push({platform: competitor.name,products: competitorData,scrapedAt: new Date().toISOString()});}return results;}async scrapeCompetitor(competitor, keywords) {const products = [];// 这里使用Apify的Actor来爬取特定平台const input = {startUrls: keywords.map(keyword => ({url: `${competitor.baseUrl}/search?q=${encodeURIComponent(keyword)}`})),maxItems: 50,extendOutputFunction: `async ($, record) => {return {...record,competitor: '${competitor.name}',priceHistory: [],alerts: []};}`};// 调用预构建的电商爬虫Actorconst run = await Apify.call('apify/amazon-product-scraper', input);const { items } = await Apify.client.dataset(run.defaultDatasetId).listItems();return items;}async generatePriceAlerts(currentData, historicalData) {const alerts = [];currentData.forEach(current => {const historical = historicalData.find(h => h.asin === current.asin);if (historical && historical.price) {const priceChange = ((current.price - historical.price) / historical.price) * 100;if (Math.abs(priceChange) > 10) { // 价格变化超过10%alerts.push({productId: current.asin,productName: current.title,oldPrice: historical.price,newPrice: current.price,changePercent: priceChange.toFixed(2),alertType: priceChange > 0 ? 'PRICE_INCREASE' : 'PRICE_DECREASE',timestamp: new Date().toISOString()});}}});return alerts;}async saveToDatastore(data) {// 保存到Apify Datasetawait Apify.pushData(data);// 同时保存到外部数据库const webhookUrl = await Apify.getValue('WEBHOOK_URL');if (webhookUrl) {await fetch(webhookUrl, {method: 'POST',headers: { 'Content-Type': 'application/json' },body: JSON.stringify(data)});}}
}// 主执行逻辑
Apify.main(async () => {const input = await Apify.getInput();const { productKeywords = ['laptop', 'smartphone'] } = input;const monitor = new EcommerceMonitor();// 监控竞品const currentData = await monitor.monitorProducts(productKeywords);// 获取历史数据进行比较const historicalData = await Apify.getValue('LAST_SCRAPE_DATA') || [];// 生成价格告警for (const platform of currentData) {const historicalPlatform = historicalData.find(h => h.platform === platform.platform);if (historicalPlatform) {const alerts = await monitor.generatePriceAlerts(platform.products, historicalPlatform.products);platform.alerts = alerts;}}// 保存数据await monitor.saveToDatastore(currentData);// 更新历史数据await Apify.setValue('LAST_SCRAPE_DATA', currentData);console.log(`监控完成，处理了 ${currentData.length} 个平台的数据`);
});

2. 新闻舆情监控系统

class NewsMonitoringSystem {constructor() {this.newsSources = [{ name: 'CNN', url: 'https://cnn.com', selector: '.card-content' },{ name: 'BBC', url: 'https://bbc.com/news', selector: '.media__content' },{ name: 'Reuters', url: 'https://reuters.com', selector: '.story-content' }];this.keywords = [];this.sentimentAnalyzer = new SentimentAnalyzer();}async monitorNews(keywords, timeRange = '24h') {this.keywords = keywords;const results = [];for (const source of this.newsSources) {console.log(`监控 ${source.name} 新闻源`);try {const articles = await this.scrapeNewsSource(source, timeRange);const relevantArticles = this.filterRelevantArticles(articles, keywords);const analyzedArticles = await this.analyzeArticles(relevantArticles);results.push({source: source.name,articles: analyzedArticles,summary: this.generateSourceSummary(analyzedArticles)});} catch (error) {console.error(`Error scraping ${source.name}:`, error);}}return results;}async scrapeNewsSource(source, timeRange) {const requestQueue = await Apify.openRequestQueue();await requestQueue.addRequest({ url: source.url });const articles = [];const crawler = new Apify.PlaywrightCrawler({requestQueue,async requestHandler({ page }) {await page.waitForSelector(source.selector);const pageArticles = await page.evaluate((selector) => {return Array.from(document.querySelectorAll(selector)).map(article => {const titleEl = article.querySelector('h1, h2, h3, .title');const linkEl = article.querySelector('a');const timeEl = article.querySelector('time, .time, .date');const summaryEl = article.querySelector('.summary, .excerpt, p');return {title: titleEl ? titleEl.textContent.trim() : '',link: linkEl ? linkEl.href : '',publishTime: timeEl ? timeEl.textContent.trim() : '',summary: summaryEl ? summaryEl.textContent.trim() : '',source: window.location.hostname};});}, source.selector);articles.push(...pageArticles);}});await crawler.run();// 过滤时间范围return this.filterByTimeRange(articles, timeRange);}filterRelevantArticles(articles, keywords) {return articles.filter(article => {const content = `${article.title} ${article.summary}`.toLowerCase();return keywords.some(keyword => content.includes(keyword.toLowerCase()));});}async analyzeArticles(articles) {const analyzed = [];for (const article of articles) {try {// 获取完整文章内容const fullContent = await this.getFullArticleContent(article.link);// 情感分析const sentiment = await this.sentimentAnalyzer.analyze(fullContent);// 关键词提取const extractedKeywords = this.extractKeywords(fullContent);// 实体识别const entities = this.extractEntities(fullContent);analyzed.push({...article,fullContent: fullContent.substring(0, 1000), // 限制长度sentiment,keywords: extractedKeywords,entities,relevanceScore: this.calculateRelevanceScore(fullContent)});} catch (error) {console.error(`Error analyzing article ${article.link}:`, error);analyzed.push(article); // 保留原始数据}}return analyzed;}async getFullArticleContent(url) {// 使用Apify的文章提取Actorconst run = await Apify.call('apify/web-scraper', {startUrls: [{ url }],pageFunction: `async function pageFunction(context) {const { page } = context;// 等待页面加载await page.waitForLoadState('networkidle');// 提取文章内容const content = await page.evaluate(() => {const selectors = ['article','.article-content','.post-content','.entry-content','.content'];for (const selector of selectors) {const element = document.querySelector(selector);if (element) {return element.innerText;}}return document.body.innerText;});return { content };}`});const { items } = await Apify.client.dataset(run.defaultDatasetId).listItems();return items[0]?.content || '';}calculateRelevanceScore(content) {let score = 0;const contentLower = content.toLowerCase();this.keywords.forEach(keyword => {const keywordLower = keyword.toLowerCase();const occurrences = (contentLower.match(new RegExp(keywordLower, 'g')) || []).length;score += occurrences * 10; // 每次出现+10分});return Math.min(score, 100); // 最高100分}generateSourceSummary(articles) {const totalArticles = articles.length;const positiveCount = articles.filter(a => a.sentiment?.polarity > 0.1).length;const negativeCount = articles.filter(a => a.sentiment?.polarity < -0.1).length;const neutralCount = totalArticles - positiveCount - negativeCount;const avgRelevance = articles.reduce((sum, a) => sum + (a.relevanceScore || 0), 0) / totalArticles;return {totalArticles,sentimentDistribution: {positive: positiveCount,negative: negativeCount,neutral: neutralCount},averageRelevanceScore: avgRelevance.toFixed(2),topKeywords: this.getTopKeywords(articles),timeRange: {earliest: Math.min(...articles.map(a => new Date(a.publishTime).getTime())),latest: Math.max(...articles.map(a => new Date(a.publishTime).getTime()))}};}getTopKeywords(articles) {const keywordCount = {};articles.forEach(article => {if (article.keywords) {article.keywords.forEach(keyword => {keywordCount[keyword] = (keywordCount[keyword] || 0) + 1;});}});return Object.entries(keywordCount).sort(([,a], [,b]) => b - a).slice(0, 10).map(([keyword, count]) => ({ keyword, count }));}
}// 简化的情感分析器
class SentimentAnalyzer {constructor() {this.positiveWords = ['好', '棒', '优秀', '成功', '增长', 'good', 'great', 'excellent'];this.negativeWords = ['坏', '差', '失败', '下降', '问题', 'bad', 'terrible', 'failed'];}async analyze(text) {const words = text.toLowerCase().split(/\s+/);let positiveScore = 0;let negativeScore = 0;words.forEach(word => {if (this.positiveWords.includes(word)) positiveScore++;if (this.negativeWords.includes(word)) negativeScore++;});const totalWords = words.length;const polarity = (positiveScore - negativeScore) / totalWords;return {polarity,subjectivity: (positiveScore + negativeScore) / totalWords,classification: polarity > 0.1 ? 'positive' : polarity < -0.1 ? 'negative' : 'neutral'};}
}// 主执行函数
Apify.main(async () => {const input = await Apify.getInput();const { keywords = ['technology', '科技'], timeRange = '24h',webhookUrl = null } = input;const monitor = new NewsMonitoringSystem();console.log(`开始监控关键词: ${keywords.join(', ')}`);const results = await monitor.monitorNews(keywords, timeRange);// 生成综合报告const report = {monitoringPeriod: timeRange,keywords,sources: results,timestamp: new Date().toISOString(),summary: {totalSources: results.length,totalArticles: results.reduce((sum, r) => sum + r.articles.length, 0),overallSentiment: monitor.calculateOverallSentiment(results)}};// 保存结果await Apify.pushData(report);// 发送Webhook通知if (webhookUrl) {await fetch(webhookUrl, {method: 'POST',headers: { 'Content-Type': 'application/json' },body: JSON.stringify(report)});}console.log(`监控完成，共处理 ${report.summary.totalArticles} 篇文章`);
});

性能优化与最佳实践

1. 并发控制与资源管理

class PerformanceOptimizer {constructor() {this.maxConcurrency = 10;this.requestDelay = 1000;this.memoryThreshold = 0.8; // 80%内存使用率}async createOptimizedCrawler(options = {}) {const {maxRequestsPerCrawl = 1000,maxConcurrency = this.maxConcurrency,requestHandlerTimeoutSecs = 60} = options;return new Apify.PlaywrightCrawler({maxRequestsPerCrawl,maxConcurrency,requestHandlerTimeoutSecs,// 浏览器池配置browserPoolOptions: {maxOpenPagesPerBrowser: 5,retireBrowserAfterPageCount: 100,operationTimeoutSecs: 60},// 预导航钩子preNavigationHooks: [async ({ page }) => {// 禁用不必要的资源await page.route('**/*', (route) => {const resourceType = route.request().resourceType();if (['image', 'font', 'media'].includes(resourceType)) {route.abort();} else {route.continue();}});}],// 后导航钩子postNavigationHooks: [async ({ page }) => {// 等待关键内容加载await page.waitForLoadState('domcontentloaded');// 检查内存使用await this.checkMemoryUsage();}]});}async checkMemoryUsage() {const memInfo = await Apify.getMemoryInfo();const usageRatio = memInfo.usedBytes / memInfo.totalBytes;if (usageRatio > this.memoryThreshold) {console.log(`内存使用率过高: ${(usageRatio * 100).toFixed(2)}%`);// 触发垃圾回收if (global.gc) {global.gc();}// 清理Apify缓存await Apify.utils.sleep(2000);}}async implementRetryLogic(requestQueue, failedRequests = []) {const retryLimit = 3;for (const failedRequest of failedRequests) {if (failedRequest.retryCount < retryLimit) {failedRequest.retryCount = (failedRequest.retryCount || 0) + 1;// 指数退避const delay = Math.pow(2, failedRequest.retryCount) * 1000;await Apify.utils.sleep(delay);await requestQueue.addRequest(failedRequest);}}}async monitorPerformance(crawler) {const stats = {requestsCompleted: 0,requestsFailed: 0,averageResponseTime: 0,totalDataExtracted: 0};crawler.on('requestCompleted', ({ request }) => {stats.requestsCompleted++;stats.averageResponseTime = (stats.averageResponseTime + request.responseTime) / stats.requestsCompleted;});crawler.on('requestFailed', ({ request }) => {stats.requestsFailed++;});// 定期输出统计信息const interval = setInterval(() => {console.log('性能统计:', {...stats,successRate: `${((stats.requestsCompleted / (stats.requestsCompleted + stats.requestsFailed)) * 100).toFixed(2)}%`,avgResponseTime: `${stats.averageResponseTime.toFixed(2)}ms`});}, 30000); // 每30秒输出一次return { stats, interval };}
}// 使用示例
Apify.main(async () => {const optimizer = new PerformanceOptimizer();const crawler = await optimizer.createOptimizedCrawler({maxConcurrency: 5,maxRequestsPerCrawl: 500});const { stats, interval } = await optimizer.monitorPerformance(crawler);// 设置请求处理器crawler.requestHandler = async ({ page, request }) => {try {const data = await page.evaluate(() => ({title: document.title,url: window.location.href,timestamp: new Date().toISOString()}));await Apify.pushData(data);stats.totalDataExtracted++;} catch (error) {console.error(`Error processing ${request.url}:`, error);throw error;}};await crawler.run();clearInterval(interval);console.log('最终统计:', stats);
});

2. 成本优化策略

class CostOptimizer {constructor() {this.costTracker = {computeUnits: 0,datasetOperations: 0,storageUsed: 0,estimatedCost: 0};}async optimizeDataStorage() {// 智能数据去重const dataset = await Apify.openDataset();const existingData = await dataset.getData();const uniqueData = this.deduplicateData(existingData.items);if (uniqueData.length < existingData.items.length) {console.log(`去重: ${existingData.items.length} -> ${uniqueData.length}`);// 清空数据集并重新存储await dataset.drop();const newDataset = await Apify.openDataset();for (const item of uniqueData) {await newDataset.pushData(item);}}}deduplicateData(items) {const seen = new Set();return items.filter(item => {const key = this.generateItemKey(item);if (seen.has(key)) {return false;}seen.add(key);return true;});}generateItemKey(item) {// 根据URL或其他唯一标识生成键return item.url || JSON.stringify(item);}async implementSmartCaching() {const cache = await Apify.openKeyValueStore('smart-cache');return {async get(key) {const cached = await cache.getValue(key);if (cached && cached.expiry > Date.now()) {return cached.data;}return null;},async set(key, data, ttlMinutes = 60) {await cache.setValue(key, {data,expiry: Date.now() + (ttlMinutes * 60 * 1000),createdAt: new Date().toISOString()});}};}async optimizeRequestQueue(urls) {// 按域名分组以避免过度请求同一服务器const domainGroups = {};urls.forEach(url => {const domain = new URL(url).hostname;if (!domainGroups[domain]) {domainGroups[domain] = [];}domainGroups[domain].push(url);});// 平衡负载const optimizedUrls = [];const maxPerDomain = Math.ceil(urls.length / Object.keys(domainGroups).length);Object.entries(domainGroups).forEach(([domain, domainUrls]) => {const limitedUrls = domainUrls.slice(0, maxPerDomain);optimizedUrls.push(...limitedUrls);});return optimizedUrls;}trackResourceUsage(operation, cost) {this.costTracker.computeUnits += cost;console.log(`操作: ${operation}, 成本: ${cost}, 总计: ${this.costTracker.computeUnits}`);}async generateCostReport() {const report = {...this.costTracker,timestamp: new Date().toISOString(),recommendations: this.getCostOptimizationRecommendations()};await Apify.setValue('COST_REPORT', report);return report;}getCostOptimizationRecommendations() {const recommendations = [];if (this.costTracker.computeUnits > 1000) {recommendations.push({type: 'COMPUTE_OPTIMIZATION',message: '考虑增加缓存时间以减少重复计算',priority: 'HIGH'});}if (this.costTracker.datasetOperations > 500) {recommendations.push({type: 'STORAGE_OPTIMIZATION', message: '建议批量写入以减少操作次数',priority: 'MEDIUM'});}return recommendations;}
}// 集成使用
Apify.main(async () => {const costOptimizer = new CostOptimizer();const cache = await costOptimizer.implementSmartCaching();const input = await Apify.getInput();const { startUrls } = input;// 优化URL队列const optimizedUrls = await costOptimizer.optimizeRequestQueue(startUrls);const requestQueue = await Apify.openRequestQueue();for (const url of optimizedUrls) {await requestQueue.addRequest({ url });}const crawler = new Apify.PlaywrightCrawler({requestQueue,async requestHandler({ page, request }) {const cacheKey = request.url;// 检查缓存let data = await cache.get(cacheKey);if (!data) {// 缓存未命中，执行爬取costOptimizer.trackResourceUsage('SCRAPE', 1);data = await page.evaluate(() => ({title: document.title,content: document.body.innerText.slice(0, 500)}));// 存储到缓存await cache.set(cacheKey, data, 120); // 2小时缓存} else {console.log(`缓存命中: ${request.url}`);}await Apify.pushData(data);}});await crawler.run();// 优化存储await costOptimizer.optimizeDataStorage();// 生成成本报告const report = await costOptimizer.generateCostReport();console.log('成本报告:', report);
});