Trawl

Quickstart

Goal: your first structured JSON result in minutes. You need two things — an API key and a scrap id — then one curl returns live data. The fastest way to get both is the setup below.

Setup

  1. Sign up at trawl.me (free, no card — you start with 500 compute units).
  2. Click New scrap, paste a URL, describe what to extract in plain English. The AI wizard writes the Puppeteer script and runs the first scrap for you — this step blocks until that first run finishes (typically 30-250s). Copy the scrap id from the URL (trawl.me/scraps/<id> — a 24-char hex string).
  3. Go to Settings → API keys, click Create key, give it the trawl:scraps:run scope. Copy the key — it starts with trawl_ and is shown once.

You now have a <scrap_id> and a trawl_... key. That's everything.

Run your first scrap — copy, paste, run

Trigger a real run over the REST API. POST /api/scraps/worker/:scrapId executes the scrap synchronously and returns the result inline:

cURL
curl -X POST https://api.trawl.me/api/scraps/worker/<scrap_id> \
  -H "Authorization: Bearer trawl_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{}'

Blocking call: this runs synchronously and does not return until the run finishes — typically 30-250s. Set your HTTP client's timeout accordingly (curl itself has no default, but many libraries do).

That's it — no SDK, no config file. The request body is an optional JSON object of TRAWL.* parameters (see Parameters); send {} to run with the scrap's defaults.

What you get back

A success envelope. data.history.data is the extracted result as a JSON-encoded array — parse it client-side:

JSON
{
  "type": "success",
  "message": "scrap loaded",
  "data": {
    "scrap": {
      "id": "507f1f77bcf86cd799439011",
      "title": "Hacker News — AI search",
      "url": "https://news.ycombinator.com/search",
      "proxyTier": "tier1"
    },
    "history": {
      "id": "65a0f2e3b5c6d7e8f9a0b1c2",
      "status": true,
      "statusDetail": "success",
      "length": 3,
      "time": 4210,
      "triggeredBy": "api",
      "data": "[{\"title\":\"Show HN: Trawl\",\"points\":128},{\"title\":\"AI search engines\",\"points\":94},{\"title\":\"Scraping at scale\",\"points\":61}]"
    }
  }
}

Read it in one line:

cURL
curl -s -X POST https://api.trawl.me/api/scraps/worker/<scrap_id> \
  -H "Authorization: Bearer trawl_your_key_here" -d '{}' \
  | jq -r '.data.history.data | fromjson'

statusDetail tells you the outcome at a glance: success (data returned), empty (0 items — page changed or needs JS wait), error (script threw), regression (item count dropped vs the last run). Full field reference: Results & payload.

Prefer the web app? Run it from the dashboard

You don't have to touch HTTP. In the scrap workspace, click the green Run button — the worker browser opens, executes, and the JSON appears in the output panel once the run finishes (typically 30-250s). The History tab keeps every run (timestamp, status, item count, full payload on click).

Schedule it so it runs without you

Open Settings → Automation, enable Schedule, and pick a preset (or enter a cron expression):

Preset Cron
Every hour 0 * * * *
Twice a day 0 8,20 * * *
Daily at 9am 0 9 * * *
Every Monday 0 9 * * 1

A random jitter (up to 5 min) avoids thundering-herd on shared workers. Full reference: Scheduling.

Common gotchas

401 Unauthorized. The key is missing, malformed, or lacks the trawl:scraps:run scope. Recreate it under Settings → API keys with the run scope. See API keys.

Result is empty (statusDetail: "empty"). The page loads content via JavaScript after the initial HTML. Add a waitForSelector for a key element before extracting. See Scraping basics.

Result looks right once, then breaks. Site HTML changed. Enable Auto-fix under Settings — the AI rewrites the script when it detects a failure. See AI features.

The URL needs a login. Store credentials or inject cookies via Account & sessions so the worker authenticates before extracting. See Account & sessions.

What's next

See also: Scraping basics · API keys · Using Trawl with Claude

Next step → Scraping basics

Welcome
Scraping