Apify Crawler Puppeteer

• How it works
• Input
• Page function
  • context
    • Data structures:
    • Functions:
    • Class instances:
      • Global Store
• Output
  • Dataset

How it works

Crawler Puppeteer is the most powerful crawler tool in our arsenal (aside from developing your own actors). It uses the Puppeteer library to programmatically control a headless Chrome browser and it can make it do almost anything. If using the Crawler does not cut it, Crawler Puppeteer is what you need.

The downside is that Puppeteer is a Node.js library, so knowledge of Node.js and its paradigms is expected when working with the Crawler Puppeteer.

If you need either a more performant, or a simpler tool, see the crawler-cheerio for unmatched performance, or crawler for a plain old JavaScript tool.

Input

Input is provided via the pre-configured UI. See the tooltips for more info on the available options.

Page function

Page function is a single JavaScript function that enables the user to control the Crawler's operation, manipulate the crawled pages and extract data as needed. It is invoked with a context object containing the following properties:

1const context = {
2    // USEFUL DATA
3    input, // Unaltered original input as parsed from the UI
4    env, // Contains information about the run such as actorId or runId
5    customData, // Value of the 'Custom data' Crawler option.
6    request, // Apify.Request object.
7    response, // Response object holding the status code and headers.
8    
9    // EXPOSED FUNCTIONS
10    saveSnapshot, // Saves a screenshot and full HTML of the current page to the key value store.
11    skipLinks, // Prevents enqueueing more links via Pseudo URLs on the current page.
12    skipOutput, // Prevents saving the return value of the pageFunction to the default dataset.
13    enqueuePage, // Adds a page to the request queue.
14    jQuery, // A reference to the jQuery $ function (if injectJQuery was used).
15    
16    // EXPOSED OBJECTS
17    globalStore, // Represents an in memory store that can be used to share data across pageFunction invocations.
18    requestList, // Reference to the run's default Apify.RequestList.
19    requestQueue, // Reference to the run's default Apify.RequestQueue.
20    dataset, // Reference to the run's default Apify.Dataset.
21    keyValueStore, // Reference to the run's default Apify.KeyValueStore.
22    log, // Reference to Apify.utils.log 
23    underscoreJs, // A reference to the Underscore _ object (if injectUnderscore was used).
24}

context

The following tables describe the context object in more detail.

Data structures:

Functions:

Class instances:

Global Store

globalStore represents an instance of a very simple in memory store that is not scoped to the individual pageFunction invocation. This enables you to easily share global data such as API responses, tokens and other. Since the stored data need to cross the from the Browser to the Node.js process, they cannot be any data, but always need to be JSON stringifiable. Therefore, you cannot store DOM objects, live class instances, functions etc. Only a JSON representation of the passed object will be stored, with all the relevant limitations.

Output

Ouput is a dataset containing extracted data for each scraped page.

Dataset

For each of the scraped URLs, the dataset contains an object with results and some metadata. If you were scraping the HTML <title> of IANA it would look like this:

1{
2  "title": "Internet Assigned Numbers Authority",
3  "#error": false,
4  "#debug": {
5    "url": "https://www.iana.org/",
6    "method": "GET",
7    "retryCount": 0,
8    "errorMessages": null,
9    "requestId": "e2Hd517QWfF4tVh"
10  }
11}

The metadata are prefixed with a #. Soon you will be able to exclude the metadata from the results by providing an API flag.