Trawl

Quickstart

Goal: your first structured JSON result in minutes. You need two things — an API key and a scrap id — then one curl returns live data. The fastest way to get both is the setup below.

Setup

  1. Sign up at trawl.me (free, no card — you start with 3,000 compute units).
  2. Click New scrap, paste a URL, describe what to extract in plain English. The AI wizard writes the Puppeteer script, tests it on the live page and rewrites it from what it saw until it comes back with data, then runs the first scrap for you — this step blocks until that first run finishes (typically 60-400s, longer on a multi-page site). Copy the scrap id from the URL (trawl.me/scraps/<id> — a 24-char hex string).
  3. Go to Settings → API keys, click Create key, give it the trawl:scraps:run scope. Copy the key — it starts with trawl_ and is shown once.

You now have a <scrap_id> and a trawl_... key. That's everything.

Run your first scrap — copy, paste, run

Trigger a real run over the REST API. POST /api/scraps/worker/:scrapId executes the scrap synchronously and returns the result inline:

cURL
curl -X POST https://api.trawl.me/api/scraps/worker/<scrap_id> \
  -H "Authorization: Bearer trawl_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{}'

Blocking call: this runs synchronously and does not return until the run finishes — typically 30-250s. Set your HTTP client's timeout accordingly (curl itself has no default, but many libraries do).

That's it — no SDK, no config file. The request body is an optional JSON object of TRAWL.* parameters (see Parameters); send {} to run with the scrap's defaults.

What you get back

A success envelope. data.history.data is the extracted result as a JSON-encoded array — parse it client-side:

JSON
{
  "type": "success",
  "message": "scrap loaded",
  "data": {
    "scrap": {
      "id": "507f1f77bcf86cd799439011",
      "title": "Hacker News — AI search",
      "url": "https://news.ycombinator.com/search",
      "proxyTier": "tier1"
    },
    "history": {
      "id": "65a0f2e3b5c6d7e8f9a0b1c2",
      "status": true,
      "statusDetail": "success",
      "length": 3,
      "time": 4210,
      "triggeredBy": "api",
      "data": "[{\"title\":\"Show HN: Trawl\",\"points\":128},{\"title\":\"AI search engines\",\"points\":94},{\"title\":\"Scraping at scale\",\"points\":61}]"
    }
  }
}

Read it in one line:

cURL
curl -s -X POST https://api.trawl.me/api/scraps/worker/<scrap_id> \
  -H "Authorization: Bearer trawl_your_key_here" -d '{}' \
  | jq -r '.data.history.data | fromjson'

statusDetail tells you the outcome at a glance: success (data returned), empty (0 items — page changed or needs JS wait), error (script threw), regression (item count dropped vs the last run). Full field reference: Results & payload.

Prefer the web app? Run it from the dashboard

You don't have to touch HTTP. In the scrap workspace, click the green Run button — the worker browser opens, executes, and the JSON appears in the output panel once the run finishes (typically 30-250s). The History tab keeps every run (timestamp, status, item count, full payload on click).

Schedule it so it runs without you

Open Settings → Automation, enable Schedule, and pick a preset (or enter a cron expression):

Preset Cron
Every hour 0 * * * *
Twice a day 0 8,20 * * *
Daily at 9am 0 9 * * *
Every Monday 0 9 * * 1

A random jitter (up to 5 min) avoids thundering-herd on shared workers. Full reference: Scheduling.

Common gotchas

401 Unauthorized. The key is missing, malformed, or lacks the trawl:scraps:run scope. Recreate it under Settings → API keys with the run scope. See API keys.

Result is empty (statusDetail: "empty"). The page loads content via JavaScript after the initial HTML. Add a waitForSelector for a key element before extracting. See Scraping basics.

Result looks right once, then breaks. Site HTML changed. Enable Auto-fix under Settings — the AI rewrites the script when it detects a failure. See AI features.

The URL needs a login. Store credentials or inject cookies via Account & sessions so the worker authenticates before extracting. See Account & sessions.

What's next

See also: Scraping basics · API keys · Using Trawl with Claude

Next step → Scraping basics

Welcome
Scraping