This guide covers advanced features: dynamic parameters, proxy tiers, data validation, and direct API access.

Contents: Dynamic parameters · CSS selectors · Proxy tiers · Data validation · API integration · Account credentials

Dynamic parameters

Parameters let you inject variables into your request script at runtime without editing the code. Manage them in Settings → Parameters.

Each parameter becomes a TRAWL.<name> global inside the request. Multiple parameter sets can be defined — for cron runs, one set is picked at random.

Two special value helpers are available:

• DATE(±N, unit, format) — resolves to a date relative to now, e.g. DATE(-1, days, YYYY-MM-DD) → yesterday
• RANDOM(a, b, c) — picks one value at random from the list

Example — searching different keywords daily:

[
  { "query": "javascript frameworks" },
  { "query": "rust webassembly" },
  { "query": "AI tools 2026" }
]

In the request, reference the variable directly:

var page = await browser.newPage();

await page.goto(`https://news.ycombinator.com/search?q=${TRAWL.query}`, { waitUntil: 'domcontentloaded' });

const result = await page.$$eval('.titleline > a', els =>
  els.map(el => ({ title: el.textContent.trim(), href: el.href }))
);

returnData(result);

Parameters are also passed when calling the API endpoint (see below). The legacy @variable syntax (e.g. @query) is no longer supported — always use TRAWL.*.

CSS selectors

Use page.$$eval for lists and page.$eval for single elements:

// Multiple elements
const items = await page.$$eval('ul.results li', els =>
  els.map(el => ({
    title: el.querySelector('h2')?.textContent.trim(),
    price: el.querySelector('.price')?.textContent.trim(),
    url: el.querySelector('a')?.href,
  }))
);

// Single element
const heading = await page.$eval('h1.main-title', el => el.textContent.trim());

returnData(items);

Wait for dynamic content before querying:

await page.waitForSelector('.results-list', { timeout: 8000 });

Proxy tiers

When a site blocks direct requests, set a proxy tier in the request parameters or configure it on the container. The tier column in History shows which proxy was used for each execution:

Tier	Label	Use case
tier0	Tier 0	No proxy (default)
tier1	Tier 1	Residential IPs for basic bot detection
tier2	Tier 2	Mobile carrier IPs for stricter sites
tier3	Tier 3	Premium rotating proxies for heavily protected sites
tier4	Tier 4	Premium scraping browser via a residential proxy provider for the hardest anti-bot sites

Higher tiers incur additional cost per execution.

Progressive tier downgrade

Once a scrap has been pinned at a tier > 0 for longer than a threshold (server config trawl.tierDowngradeAfterDaysSuccess after a successful run, default 7 days; trawl.tierDowngradeAfterDaysFail after a failing run, default 4 days), the next run will probe the tier one step below before falling back to the persisted tier. This catches sites that have softened their anti-bot stance and saves ongoing proxy cost.

• The probe tries N-1 first; if that fails, the worker escalates back up the ladder normally and the original tier is re-persisted on success.
• The history row records downgradeAttempted: true so the UI can surface a "Downgrade test" badge and so cost dashboards can separate probe runs from regular ones.
• The day-counter resets on every run (success or failure), not only on success, so a failing downgrade probe does not loop forever.
• The probe runs automatically and has no per-scrap opt-out: once a scrap has stayed at the same tier past the threshold, the next run will always attempt N-1 first. To reduce probe frequency on a noisy site, raise scrapInterval or narrow scrapStart / scrapEnd so the scrap runs less often.
• To disable the probe globally, set both trawl.tierDowngradeAfterDaysSuccess and trawl.tierDowngradeAfterDaysFail to 0. Zeroing only one disables only that path (e.g. zero …Fail alone to keep probing successful-but-overpaying scraps while leaving failing ones pinned).

Data validation

Settings → Data Validation → Check data return lets you define an expected JSON shape. After each run, the result structure is compared against the reference — if a key is missing or the shape differs, the execution is flagged as invalid.

Paste a typical result object as the reference:

[
  {
    "title": "Example title",
    "price": "€29.99",
    "url": "https://example.com/item/1"
  }
]

The check runs on every execution. Combine with Alert to be notified when validation fails.

API integration

Every scrap exposes an HTTP endpoint to trigger it externally. Find the cURL example and URL in the API card on the scrap details page.

Trigger a run

curl -X POST https://api.trawl.me/api/scraps/worker/<scrapId> \
  -H "Authorization: Bearer <accessToken>"

Pass parameters at runtime

curl -X POST https://api.trawl.me/api/scraps/worker/<scrapId> \
  -H "Authorization: Bearer <accessToken>" \
  -H "Content-Type: application/json" \
  -d '{ "query": "machine learning", "date": "2026-01-01" }'

These are exposed to the script as TRAWL.query and TRAWL.date. TRAWL.url is always taken from the scrap settings and cannot be overridden by the request body.

Fetch the latest result

curl https://api.trawl.me/api/scraps/<scrapId> \
  -H "Authorization: Bearer <accessToken>"

The history array on the scrap object contains the most recent executions with their status and result reference.

List all scraps

curl https://api.trawl.me/api/scraps/ \
  -H "Authorization: Bearer <accessToken>"

Account credentials

When a scrap has credentials configured (via Settings → Account), the worker decrypts them and exposes them to the request script as VM globals — separate from custom parameters. They are accessible under the TRAWL.account.* sub-namespace (canonical):

• TRAWL.account.username — string, the stored username
• TRAWL.account.password — string, the stored password
• TRAWL.account.session.cookies — array of Puppeteer cookies, present only if a session was previously persisted via saveSession()

Bare account.* (e.g. account.username) still works as a legacy alias.

The saveSession(cookies) async helper persists fresh cookies (encrypted at rest) after a successful login, so subsequent runs can skip the login flow. It is typically called at the end of the login sequence.

Login flow — first run (no saved session)

var page = await browser.newPage();

await page.goto('https://example.com/login', { waitUntil: 'domcontentloaded' });

await page.type('#username', TRAWL.account.username);
await page.type('#password', TRAWL.account.password);
await Promise.all([
  page.click('button[type=submit]'),
  page.waitForNavigation({ waitUntil: 'networkidle2' }),
]);

// Persist the authenticated cookies for reuse on the next run
await saveSession(await page.cookies());

Reusing a saved session

var page = await browser.newPage();

if (TRAWL.account?.session?.cookies) {
  // Skip the login flow — restore the previous session cookies
  await page.setCookie(...TRAWL.account.session.cookies);
  await page.goto('https://example.com/dashboard', { waitUntil: 'domcontentloaded' });
} else {
  // Fall back to a full login on the first run
  await page.goto('https://example.com/login', { waitUntil: 'domcontentloaded' });
  await page.type('#username', TRAWL.account.username);
  await page.type('#password', TRAWL.account.password);
  await Promise.all([
    page.click('button[type=submit]'),
    page.waitForNavigation({ waitUntil: 'networkidle2' }),
  ]);
  await saveSession(await page.cookies());
}

Credentials and saved cookies are always encrypted at rest. Use Settings → Account in the web app to upsert credentials or clear the saved session.

---

See also: Scraping · Parameters · Account Namespace

Next step → Parameters