website-content-crawler

One or more URLs of pages where the crawler will start. Note that the Actor will additionally only crawl sub-pages of these URLs. For example, for start URL https://www.example.com/blog, it will crawl pages like https://example.com/blog/article-1, but will skip https://example.com/docs/something-else.

Select the crawling engine:

• Headless web browser (default) - Useful for modern websites with anti-scraping protections and JavaScript rendering. It recognizes common blocking patterns like CAPTCHAs and automatically retries blocked requests through new sessions. However, running web browsers is more expensive as it requires more computing resources and is slower. It is recommended to use at least 8 GB of RAM.
• Stealthy web browser - Headless web browser with antiblocking measures enabled. Try this if you encounter bot protection while scraping. For best performance, use with Apify proxy servers.
• Raw HTTP client - High-performance crawling mode that uses raw HTTP requests to fetch the pages. It is faster and cheaper, but it might not work on all websites.

The maximum number of links starting from the start URL that the crawler will recursively descend. The start URLs have depth 0, the pages linked directly from the start URLs have depth 1, and so on.

This setting is useful to prevent accidental crawler runaway. By setting it to 0, the actor will only crawl start URLs.

The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.

The maximum number of results to store. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway. If both maxCrawlPages and maxResults are defined, then the crawler will finish when the first limit is reached. Note that the crawler skips pages with a canonical URL of a page that has already been crawled, so it might crawl more pages than there are results.

The maximum number of web browsers or HTTP clients running in parallel. This setting is useful to avoid overloading the target websites and to avoid getting blocked.

The initial number of web browsers or HTTP clients running in parallel. The system then scales the concurrency up and down based on the actual performance and memory limit. If the value is set to 0, the Actor uses the default settings for the specific crawler type.

Note that if you set this value too high, the Actor will run out of memory and crash. If too low, it will be slow at start before it scales the concurrency up.

Enables loading the websites from IP addresses in specific geographies and to circumvent blocking.

Select the text parser:

• HTML to text (default) - Extracts full page text after removing HTML tags and side text from navigation, header and footer. Best parser to ensure all text is extracted
• Unfluff - More strict article focused parser. Extracts only main article text. Doesn't work well on more complex pages like API reference. Can be too slow on very long pages.
• Extractus - Extracts only main article text. Includes HTML tags.

You can examine output of all parsers by enabling debug mode.

• Mozilla Readability - Extracts the main contents of the webpage. Utilizes similar logic as the Firefox Reader View.

When enabled, the crawler will prune content lines that are very similar to the ones already crawled. This is useful strip repeating content in the scraped data like menus, headers, footers, etc. In inprobable edge cases, it might remove relevant content from some pages.

A CSS selector matching HTML elements that will be removed from the DOM, before converting it to text, Markdown, or saving as HTML. This is useful to skip irrelevant page content.

By default, the Actor removes headers, menus, or footers. You can disable the removal by setting it to some non-existent CSS selector like dummy.

Adds a sleep before the page contents are processed, to allow for dynamic loading to settle. Defaults to 10s, and will resolve after 2s when the page content stops changing. This option is ignored with cheerio crawler type. We always wait for the window load event, this sleep adds additional time after it.

If enabled, the crawler downloads and stores document files linked from the web pages to the key-value store. The metadata about the files is stored in the output dataset, similarly as for normal pages.

Only files whose URL ends with the following file extensions are stored: PDF, DOC, DOCX, XLS, XLSX, and CSV.

If enabled, the crawler stores full HTML of all pages found, under the html field in the output dataset. This is useful for debugging, but reduces performance and increases storage costs.

If enabled, the crawler stores a screenshot for each article page to the default key-value store. The link to the screenshot is stored under the screenshotUrl field in the output dataset. It is useful for debugging, but reduces performance and increases storage costs.

Note that this feature only works with headless browser crawler type!

If enabled, the crawler stores Markdown of all pages found, under the markdown field in the output dataset.

Applies only to "Readable text" text extractor. The number of characters an article must have in order to return a result.

If enabled, the actor will store the output of all types of extractors, including the ones that are not used by default and it will store HTML to Key-value Store with a link. All this data is stored under the debug field.