Trawl

Scraping

Trawl lets you build automated scrapers that extract data from any website, run on a schedule, and notify you when things change.

What is a scrap?

A scrap is a scraping job made of three parts:

  • Title & description — what the scrap is for
  • Workers (containers) — the browser workers that execute the job (you must assign at least one)
  • Request — a Puppeteer script that runs inside the worker and returns structured data via returnData(result)

Each execution is recorded in the history with its status, duration, and result.

Creating your first scrap

  1. In the Trawl web app, open Scraps and click New scrap.
  2. Fill in the title and description.
  3. Select one or more workers in the Workers field.
  4. Click Create Scrap — you are redirected to the scrap workspace.
  5. Open the Request tab and write your Puppeteer script.

A minimal example that scrapes Hacker News titles:

Node
var page = await browser.newPage();

await page.goto('https://news.ycombinator.com/', { waitUntil: 'domcontentloaded' });

const result = await page.$$eval('span.titleline > a', els =>
  els.map(el => ({ title: el.textContent.trim() }))
);

returnData(result);

Call returnData(array) with an array of objects — this is the result stored after each run.

Available variables

Inside the request script, Trawl injects a TRAWL object with every variable you can use to drive the run:

  • TRAWL.url — the target URL configured on the scrap. This is a mandatory scrap setting and cannot be overridden by a request body at trigger time.
  • TRAWL.<name> — any custom parameter you defined in Settings → Parameters (see the Scraping Advanced guide). For a parameter named query, use TRAWL.query.
Node
var page = await browser.newPage();

await page.goto(TRAWL.url, { waitUntil: 'domcontentloaded' });

const result = await page.$$eval('span.titleline > a', els =>
  els.map(el => ({ title: el.textContent.trim() }))
);

returnData(result);

The legacy @variable syntax (e.g. @url, @query) is no longer supported — always use TRAWL.*.

Running a scrap

In the workspace, click Run (the green play button). The worker executes the request and the result appears in the output panel. The execution is added to the history.

To view past executions, open the History tab of the scrap.

Understanding the status

After each run, the history row records two fields:

  • status — tri-state boolean: true (success), false (failure), null (pending / in-flight)
  • statusDetail — fine-grained enum: success, error, empty, regression
statusDetail Meaning
success Run completed and returned at least one item
error The script threw an error — check the output for the stack trace
empty Script ran without errors but returnData([]) — the selector likely broke
regression Result count dropped significantly compared to previous run

empty is a signal the scrap needs fixing, not a success. Sites change their HTML; use AI Fix (see below) or update the selector manually.

See Results & Payload for the full history row schema.

Scheduling with cron

Open Settings → Automation and enable Schedule. Pick a preset frequency or enter a custom cron expression. The scrap runs automatically at the chosen interval. All executions appear in History.

See Schedule for the full list of presets, custom expressions, and timezone configuration.

Notifications

Two notification types are available in Settings → Automation:

Alert — sends an email when the scrap status changes to error or returns 0 items.

Email results — sends the last run data by email (CSV or JSON) after each cron execution.

See Notifications for recipient configuration and trigger details.

AI Fix

When a run fails or returns 0 items, the Fix button appears in the workspace toolbar. The AI rewrites the script to adapt to the new page structure.

To enable automatic fixes, toggle Auto-fix with AI in Settings → AI Features (max 6 attempts/hour, disabled after 3 consecutive failures). See AI Features for the full guardrail details.

Digest

AI Digest sends an AI-generated summary of each scheduled run by email. Enable it in Settings → AI Features → AI Digest with a custom prompt.

See AI Features for prompt examples and test preview.

Versions

Every time you save a change to the request or parameters, Trawl creates a version snapshot. Open the Versions tab to:

  • Browse the full change history with author and timestamp
  • Show Diff — compare a version to the previous one (unified diff format)
  • Rollback — restore the scrap to any previous version (creates a new version entry tagged rollback)

When saving, you can optionally add a version message to describe the change.

See also: Scraping Advanced · Schedule · Parameters

Next step → Advanced scraping

Quickstart
Scraping Advanced