Crawlee + Playwright + Camoufox
Web scraper example with Crawlee, Playwright and Camoufox. Camoufox is a custom stealthy fork of Firefox. Try this template if you're facing anti-scraping challenges.
src/main.js
src/routes.js
1/**
2 * This template is a production ready boilerplate for developing with `PlaywrightCrawler`.
3 * Use this to bootstrap your projects using the most up-to-date code.
4 * If you're looking for examples or want to learn more, see README.
5 */
6
7// For more information, see https://docs.apify.com/sdk/js
8import { Actor } from 'apify';
9// For more information, see https://crawlee.dev
10import { PlaywrightCrawler } from 'crawlee';
11// this is ESM project, and as such, it requires you to specify extensions in your relative imports
12// read more about this here: https://nodejs.org/docs/latest-v18.x/api/esm.html#mandatory-file-extensions
13import { router } from './routes.js';
14import { firefox } from 'playwright';
15import { launchOptions as camoufoxLaunchOptions } from 'camoufox-js';
16
17// Initialize the Apify SDK
18await Actor.init();
19
20const {
21 startUrls = ['https://crawlee.dev'],
22} = await Actor.getInput() ?? {};
23
24const proxyConfiguration = await Actor.createProxyConfiguration();
25
26const crawler = new PlaywrightCrawler({
27 proxyConfiguration,
28 requestHandler: router,
29 launchContext: {
30 launcher: firefox,
31 launchOptions: await camoufoxLaunchOptions({
32 headless: true,
33 // fonts: ['Times New Roman'], // <- custom Camoufox options
34 }),
35 }
36});
37
38await crawler.run(startUrls);
39
40// Exit successfully
41await Actor.exit();
PlaywrightCrawler + Camoufox template
This template is a production-ready boilerplate for developing an Actor with PlaywrightCrawler
. It has Camoufox - a stealthy fork of Firefox - preinstalled. Note that Camoufox might consume more resources than the default Playwright-bundled Chromium or Firefox.
Use this template to bootstrap your projects using the most up-to-date code.
We decided to split Apify SDK into two libraries, Crawlee and Apify SDK v3. Crawlee will retain all the crawling and scraping-related tools and will always strive to be the best web scraping library for its community. At the same time, Apify SDK will continue to exist, but keep only the Apify-specific features related to building actors on the Apify platform. Read the upgrading guide to learn about the changes.
Resources
If you're looking for examples or want to learn more visit:
- Crawlee + Apify Platform guide
- Documentation and examples
- Node.js tutorials in Academy
- Scraping single-page applications with Playwright
- How to scale Puppeteer and Playwright
- Integration with Zapier, Make, GitHub, Google Drive and other apps
- Video guide on getting data using Apify API
- A short guide on how to create Actors using code templates:
Scrape single page with provided URL with Axios and extract data from page's HTML with Cheerio.
A scraper example that uses Cheerio to parse HTML. It's fast, but it can't run the website's JavaScript or pass JS anti-scraping challenges.
Example of a Puppeteer and headless Chrome web scraper. Headless browsers render JavaScript and are harder to block, but they're slower than plain HTTP.
Web scraper example with Crawlee, Playwright and headless Chrome. Playwright is more modern, user-friendly and harder to block than Puppeteer.
Skeleton project that helps you quickly bootstrap `CheerioCrawler` in JavaScript. It's best for developers who already know Apify SDK and Crawlee.
Example of running Cypress tests and saving their results on the Apify platform. JSON results are saved to Dataset, videos to Key-value store.