This guide covers advanced features: dynamic parameters, proxy tiers, and direct API access.

Contents: Dynamic parameters · CSS selectors · Proxy tiers · API integration · Account credentials

Dynamic parameters

Parameters let you inject variables into your request script at runtime without editing the code. Manage them in Settings → Parameters.

Each parameter becomes a TRAWL.<name> global inside the request. Multiple parameter sets can be defined — for cron runs, one set is picked at random.

Two special value helpers are available:

• DATE(±N, unit, format) — resolves to a date relative to now, e.g. DATE(-1, days, YYYY-MM-DD) → yesterday
• RANDOM(a, b, c) — picks one value at random from the list

Example — searching different keywords daily:

[
  { "query": "javascript frameworks" },
  { "query": "rust webassembly" },
  { "query": "AI tools 2026" }
]

In the request, reference the variable directly:

var page = await browser.newPage();

await page.goto(`https://news.ycombinator.com/search?q=${TRAWL.query}`, { waitUntil: 'domcontentloaded' });

const result = await page.$$eval('.titleline > a', els =>
  els.map(el => ({ title: el.textContent.trim(), href: el.href }))
);

returnData(result);

Parameters are also passed when calling the API endpoint (see below). The legacy @variable syntax (e.g. @query) is no longer supported — always use TRAWL.*.

CSS selectors

Use page.$$eval for lists and page.$eval for single elements:

// Multiple elements
const items = await page.$$eval('ul.results li', els =>
  els.map(el => ({
    title: el.querySelector('h2')?.textContent.trim(),
    price: el.querySelector('.price')?.textContent.trim(),
    url: el.querySelector('a')?.href,
  }))
);

// Single element
const heading = await page.$eval('h1.main-title', el => el.textContent.trim());

returnData(items);

Wait for dynamic content before querying:

await page.waitForSelector('.results-list', { timeout: 8000 });

Proxy tiers

When a site blocks direct requests, set a proxy tier in the request parameters or configure it on the container. The tier column in History shows which proxy was used for each execution:

Higher tiers incur additional cost per execution.

Progressive tier downgrade

Once a scrap has been pinned at a tier > 0 for longer than a threshold (server config trawl.tierDowngradeAfterDaysSuccess after a successful run, default 7 days; trawl.tierDowngradeAfterDaysFail after a failing run, default 4 days), the next run will probe the tier one step below before falling back to the persisted tier. This catches sites that have softened their anti-bot stance and saves ongoing proxy cost.

• The probe tries N-1 first; if that fails, the worker escalates back up the ladder normally and the original tier is re-persisted on success.
• The history row records downgradeAttempted: true so the UI can surface a "Downgrade test" badge and so cost dashboards can separate probe runs from regular ones.
• The day-counter resets on every run (success or failure), not only on success, so a failing downgrade probe does not loop forever.
• The probe runs automatically and has no per-scrap opt-out: once a scrap has stayed at the same tier past the threshold, the next run will always attempt N-1 first. To reduce probe frequency on a noisy site, raise scrapInterval or narrow scrapStart / scrapEnd so the scrap runs less often.
• To disable the probe globally, set both trawl.tierDowngradeAfterDaysSuccess and trawl.tierDowngradeAfterDaysFail to 0. Zeroing only one disables only that path (e.g. zero …Fail alone to keep probing successful-but-overpaying scraps while leaving failing ones pinned).

API integration

Every scrap exposes an HTTP endpoint to trigger it externally. Find the cURL example and URL in the API card on the scrap details page.

Trigger a run

curl -X POST https://api.trawl.me/api/scraps/worker/<scrapId> \
  -H "Authorization: Bearer <accessToken>"

Pass parameters at runtime

curl -X POST https://api.trawl.me/api/scraps/worker/<scrapId> \
  -H "Authorization: Bearer <accessToken>" \
  -H "Content-Type: application/json" \
  -d '{ "query": "machine learning", "date": "2026-01-01" }'

These are exposed to the script as TRAWL.query and TRAWL.date. TRAWL.url is always taken from the scrap settings and cannot be overridden by the request body.

Fetch the latest result

curl https://api.trawl.me/api/scraps/<scrapId> \
  -H "Authorization: Bearer <accessToken>"

The history array on the scrap object contains the most recent executions with their status and result reference.

List all scraps

curl https://api.trawl.me/api/scraps/ \
  -H "Authorization: Bearer <accessToken>"

Account credentials

When a scrap has credentials configured (via Settings → Account), the worker decrypts them and exposes them to the request script as VM globals — separate from custom parameters. They are accessible under the TRAWL.account.* sub-namespace (canonical):

• TRAWL.account.username — string, the stored username
• TRAWL.account.password — string, the stored password
• TRAWL.account.session — object, always present once an account is configured; cookies/origins are [] before a session is captured

Bare account.* (e.g. account.username) still works as a legacy alias.

The worker replays a stored session automatically, before your script's first navigation — cookies onto the browser context, local storage onto each page (including any page or popup your script opens itself). Your script does not need to restore cookies onto the page itself — the page already carries the session by the time your script runs, and that direct per-page restore call is deprecated Puppeteer surface besides. Its job is to detect whether the page is already logged in, and only run the credential login flow when it isn't. See Account & sessions for the full session model, and how to get a session onto a scrap (trawl scraps account session capture <id> is the recommended path).

The saveSession(cookies) async helper persists fresh cookies (encrypted at rest, cookies only — it doesn't touch local storage) after a successful login, so subsequent runs can skip the login flow. It is typically called at the end of the login sequence.

Detect logged-in state, log in only if needed

var page = await browser.newPage();

// The worker has already replayed any stored session onto this page.
// Check for a marker that only appears when logged in, rather than
// assuming a saved session means you're logged in.
await page.goto('https://example.com/dashboard', { waitUntil: 'domcontentloaded' });
const loggedIn = await page.$('.account-menu') !== null;

if (!loggedIn) {
  await page.goto('https://example.com/login', { waitUntil: 'domcontentloaded' });
  await page.type('#username', TRAWL.account.username);
  await page.type('#password', TRAWL.account.password);
  await Promise.all([
    page.click('button[type=submit]'),
    page.waitForNavigation({ waitUntil: 'networkidle2' }),
  ]);

  // Persist the authenticated cookies so a script-driven login
  // doesn't have to repeat next time (session capture already
  // covers this — this matters only when the script itself logs in).
  await saveSession(await page.cookies());
}

Credentials and saved sessions are always encrypted at rest. Use Settings → Account in the web app to upsert credentials, or trawl scraps account session capture <id> / session set to attach a session — see Account & sessions.

---

See also: Scraping · Parameters · Account Namespace

Next step → Parameters