Apify Puppeteer Scraper

• How it works
• Input
• Page function
• context
  • Data structures
  • Functions
  • Class instances and namespaces
    • Page
    • Request
    • Response
    • PuppeteerPool
    • AutoscaledPool
    • Global Store
    • Log
    • Apify
• Output
  • Dataset

How it works

Puppeteer Scraper is the most powerful scraper tool in our arsenal (aside from developing your own actors). It uses the Puppeteer library to programmatically control a headless Chrome browser and it can make it do almost anything. If using the Web Scraper does not cut it, Puppeteer Scraper is what you need.

Puppeteer is a Node.js library, so knowledge of Node.js and its paradigms is expected when working with the Puppeteer Scraper.

If you need either a faster, or a simpler tool, see the Cheerio Scraper for speed, or Web Scraper for simplicity.

Input

Input is provided via the pre-configured UI. See the tooltips for more info on the available options.

Page function

Page function is a single JavaScript function that enables the user to control the Scraper's operation, manipulate the visited pages and extract data as needed. It is invoked with a context object containing the following properties:

const context = {
    // USEFUL DATA
    input, // Unaltered original input as parsed from the UI
    env, // Contains information about the run such as actorId or runId
    customData, // Value of the 'Custom data' scraper option.
    
    // EXPOSED OBJECTS
    page, // Reference to the Puppeteer.Page.
    request, // Apify.Request object.
    response, // Response object holding the status code and headers.
    puppeteerPool, // Reference to the Apify.PuppeteerPool instance running the browsers.
    autoscaledPool, // Reference to the Apify.AutoscaledPool instance managing concurrency.
    globalStore, // Represents an in memory store that can be used to share data across pageFunction invocations.
    log, // Reference to Apify.utils.log
    Apify, // Reference to the full power of Apify SDK.
    
    // EXPOSED FUNCTIONS
    setValue, // Reference to the Apify.setValue() function.
    getValue, // Reference to the Apify.getValue() function.
    saveSnapshot, // Saves a screenshot and full HTML of the current page to the key value store.
    skipLinks, // Prevents enqueueing more links via Pseudo URLs on the current page.
    enqueueRequest, // Adds a page to the request queue.
    
}

context

The following tables describe the context object in more detail.

Data structures

Functions

The context object provides several helper functions that make scraping and saving data easier and more streamlined. All of the functions are async so make sure to use await with their invocations.

Class instances and namespaces

The following are either class instances or namespaces, which is just a way of saying objects with functions on them.

Page

Reference to the Puppeteer Page object, which enables you to use the full power of Puppeteer in your Page functions.

Request

Apify uses a request object to represent metadata about the currently crawled page, such as its URL or the number of retries. See the Request class for a preview of the structure and full documentation.

Response

The response object is produced by Puppeteer. Currently, we only pass the HTTP status code and the response headers to the context.

PuppeteerPool

A reference to the running instance of the PuppeteerPool class. See Apify SDK docs for more information.

AutoscaledPool

A reference to the running instance of the AutoscaledPool class. See Apify SDK docs for more information.

Global Store

globalStore represents an instance of a very simple in memory store that is not scoped to the individual pageFunction invocation. This enables you to easily share global data such as API responses, tokens and other. Since the stored data need to cross from the Browser to the Node.js process, it cannot be any kind of data, but only JSON stringifiable objects. You cannot store DOM objects, functions, circular objects and so on.

globalStore in Puppeteer Scraper is just a Map.

Log

log is a reference to Apify.utils.log. You can use any of the logging methods such as log.info or log.exception. log.debug is special, because you can trigger visibility of those messages in the scraper's Log by the provided Debug log input option.

Apify

A reference to the full power of the Apify SDK. See the docs for more information and all the available functions and classes.

Caution: Since we're making the full SDK available, and Puppeteer Scraper runs using the SDK, some edge case manipulations may lead to inconsistencies. Use Apify with caution and avoid making global changes unless you know what you're doing.

Output

Output is a dataset containing extracted data for each scraped page. To save data into the dataset, return an Object or an Object[] from the pageFunction.

Dataset

For each of the scraped URLs, the dataset contains an object with results and some metadata. If you were scraping the HTML <title> of Apify and returning the following object from the pageFunction

return {
  title: "Web Scraping, Data Extraction and Automation - Apify"
}

it would look like this:

{
  "title": "Web Scraping, Data Extraction and Automation - Apify",
  "#error": false,
  "#debug": {
    "requestId": "fvwscO2UJLdr10B",
    "url": "https://apify.com",
    "loadedUrl": "https://apify.com/",
    "method": "GET",
    "retryCount": 0,
    "errorMessages": null,
    "statusCode": 200
  }
}

You can remove the metadata (and results containing only metadata) from the results by selecting the Clean items option when downloading the dataset.

The result will look like this:

{
  "title": "Web Scraping, Data Extraction and Automation - Apify"
}