Use the following form to configure Web Scraper. The Start URLs and Page function are required and all other fields are optional. If you want to use Pseudo URLs, you also need to Use request queue. To learn more about the different options, click on their title or on the nearby question marks. For details about the Page function visit the actor's README in the ACTOR INFO tab. We also have a <a href="https://apify.com/docs/scraping/tutorial/introduction" target="_blank">step by step tutorial</a> to help you with the basics.

If you get stuck and it seems that you can no longer progress with the Web Scraper, try <a href="https://apify.com/docs/scraping/tutorial/puppeteer-scraper" target="_blank">Puppeteer Scraper</a>. It's a more powerful tool.

Console messages from the Browser will be included in the log. This may result in the log being flooded by error messages, warnings and other messages of little value, especially with high concurrency.

Browser log

This object will be available on pageFunction's context as customData.

Custom data

Debug messages will be included in the log. Use <code>context.log.debug('message')</code> to log your own debug messages.

Debug log

Scraper will download CSS stylesheets. Disabling this may speed up the scrape, but certain websites could stop working correctly.

Download CSS

Scraper will download media such as images, fonts, videos and sounds. Disabling this may speed up the scrape, but certain websites could stop working correctly.

Settings that help to disable downloading web page resources.

Download media

Scraper will ignore CSP (content security policy) and CORS (cross origin resource sharing) settings of visited pages and requested domains. This enables you to freely use XHR/Fetch to make HTTP requests from the scraper.

Ignore CORS and CSP

Scraper will ignore SSL certificate errors.

Various options that help you disable browser security features such as HTTPS certificates and CORS.

Ignore SSL errors

The provided cookies will be pre-set to all pages the scraper opens.

Initial cookies

The jQuery library will be injected into each page. If the page already uses jQuery, conflicts may arise.

Settings that help inject 3rd party libraries into scraper pageFunction.

jQuery

The Underscore.js library will be injected into each page. If the page already uses Underscore.js (or other libraries that attach to '_', such as Lodash), conflicts may arise.

Underscore

URL fragments (the parts of URL after a <code>#</code>) are not considered when the scraper determines whether a URL has already been visited. This means that when adding URLs such as <code>https://example.com/#foo</code> and <code>https://example.com/#bar</code>, only the first will be visited. Turn this option on to tell the scraper to visit both.

Keep URL fragments

CSS selector matching elements with 'href' attributes that should be enqueued. To enqueue urls from <code><div class="my-class" href=...></code> tags, you would enter <strong>div.my-class</strong>. Leave empty to ignore all links.

Link selector

Defines how many pages can be processed by the scraper in parallel. The scraper automatically increases and decreases concurrency based on available system resources. Use this option to set a hard limit.

Max concurrency

Defines how many links away from the StartURLs will the scraper descend. 0 means unlimited.

Max crawling depth

Maximum number of pages that the scraper will open. 0 means unlimited.

Max pages per run

Maximum number of times the request for the page will be retried in case of an error. Setting it to 0 means that the request will be attempted once and will not be retried if it fails.

Max request retries

Maximum number of results that will be saved to dataset. The scraper will terminate afterwards. 0 means unlimited.

Max result records

Page function

Maximum time the scraper will wait for the page function to execute in seconds.

Page function timeout

Maximum time the scraper will allow a web page to load in seconds.

Page load timeout

Choose to use no proxy, Apify Proxy, or provide custom proxy URLs.

Proxy configuration

Pseudo-URLs to match links in the page that you want to enqueue. Combine with Link selector to tell the scraper where to find links. Omitting the Pseudo-URLs will cause the scraper to enqueue all links matched by the Link selector.

Pseudo-URLs

Start URLs

The scraper will use a real Chrome browser instead of a Chromium masking as Chrome. Using this option may help with bypassing certain anti-scraping protections, but risks that the scraper will be unstable or not work at all.

Settings that help mask as a real user and prevent scraper detection, but increase the chances of the scraper not working at all with some websites.

Use Chrome

Request queue enables recursive crawling and the use of Pseudo-URLs, Link selector and <code>context.enqueueRequest()</code>.

Use request queue

The scraper will apply various browser emulation techniques to match a real user as closely as possible. This feature works best in conjunction with the Use Chrome option and also carries the risk of making the scraper unstable.

Use Stealth

The scraper will wait until the selected events are triggered in the page before executing the page function. Available events are <code>domcontentloaded</code>, <code>load</code>, <code>networkidle2</code> and <code>networkidle0</code>. <a href="https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options" target="_blank">See Puppeteer docs</a>.

Web Scraper

You might also like these Actors

Test earnings tab

users-me

test-ppr-2

user-me-2

Legacy Phantomjs Crawler

Crawler Stats

Temp PPR

web-server

No Trial Actor

Where next?