Trawl

Using Trawl with Claude

Trawl was built agent-first: every feature has a CLI command, a Claude Code skill, and an MCP tool. This guide walks you through the agent-first end-to-end story — from discovering a URL to scrape, to having Claude monitor it for you and iterate when things break.

The three integration paths

Trawl meets Claude wherever you work:

Path Best for How
MCP server Chat-style exploration, quick queries, trigger-on-demand Connect Claude Desktop or Cursor to https://api.trawl.me/api/mcp. Claude discovers your scraps as tools and can list, read, and trigger runs without leaving the conversation.
Skills Project-driven dev, script authoring, local testing Install @trawlme/skills into Claude Code. Claude auto-triggers the right workflow — design, account, local test, banner — based on what you ask.
CLI CI/CD, headless usage, scripted pipelines trawl command in any terminal. Scriptable, composable, CI-friendly.

The three paths compose: MCP for discovery, skills for authoring, CLI for deployment and automation.

End-to-end walkthrough

1. Discover — find the URL you want to watch

In Claude Desktop with MCP connected, paste the URL and ask:

"Watch the pricing table on https://example.com/pricing weekly and alert me if anything changes."

Claude calls trawl_list_scraps to check if a matching scrap exists. If not, it moves to design.

2. Design — scaffold the Puppeteer script

In a Claude Code session with @trawlme/skills installed, the trawl-scrap-design skill activates:

"Create a scrap that extracts the plan names and prices from https://example.com/pricing."

Claude writes the Puppeteer script, handles selector strategy, defines the returnData shape, and suggests parameters for reusability.

For sites behind a login, the trawl-scrap-account skill adds cookie injection or credential setup on top. See Account & sessions.

3. Test — validate locally before spending proxy budget

The trawl-scrap-local-test skill spins up a headed Chrome session on your machine with the same globals the worker injects (browser, TRAWL, account, returnData). You iterate on selectors for free, without touching the worker.

"Test this script locally on https://example.com/pricing."

Claude runs scripts/run-local.mjs, shows you the returnData output, and flags issues before you publish.

4. Deploy — push to Trawl

Once the local run looks right, deploy via the CLI:

cURL
trawl scraps create -t "Example pricing" -u https://example.com/pricing
trawl scraps update <id> --cron "0 9 * * 1"  # every Monday at 9am

Or ask Claude via the trawl-cli skill:

"Create the scrap with a weekly Monday 9am schedule."

Claude runs the trawl scraps create + trawl scraps update commands for you.

5. Monitor — get notified when things change

Enable notifications in Settings → Automation:

  • Alert — email on error or 0 items. See Notifications.
  • AI Digest — AI-generated summary of each run, with your custom prompt. See AI features.

From Claude Desktop via MCP, you can also query history on demand:

"Show me the last 10 runs of my pricing scrap and flag any regressions."

Claude calls trawl_get_scrap_history and summarizes what changed.

6. Iterate — let Claude fix what breaks

Sites change. When a run fails or returns 0 items, the Fix button appears in the dashboard. Alternatively, paste the error in Claude Code:

"My pricing scrap has been returning 0 items since yesterday. Here's the error: [...]"

The trawl-scrap-design skill diagnoses the broken selector, rewrites the script, and you re-test with trawl-scrap-local-test before pushing the updated version via CLI or MCP trawl_trigger_scrap_run.

Authentication setup

MCP — Claude Desktop (macOS)

Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

JSON
{
  "mcpServers": {
    "trawl": {
      "transport": {
        "type": "http",
        "url": "https://api.trawl.me/api/mcp",
        "headers": {
          "Authorization": "Bearer trawl_your_key_here"
        }
      }
    }
  }
}

Create the API key under Developers → API Keys in the Trawl app. Restart Claude Desktop — the trawl server appears in the tool picker.

MCP — Cursor

In Cursor settings → Model Context Protocol, add:

JSON
{
  "mcpServers": {
    "trawl": {
      "url": "https://api.trawl.me/api/mcp",
      "headers": {
        "Authorization": "Bearer trawl_your_key_here"
      }
    }
  }
}

Skills — Claude Code

cURL
npx @trawlme/skills install

Or if @trawlme/cli is installed:

cURL
trawl skills install

Restart Claude Code. Skills auto-trigger when you mention Trawl, scraps, or describe a scraping task.

CLI

cURL
npm install -g @trawlme/cli
trawl login           # interactive email + password
# — or —
export TRAWL_TOKEN=<jwt>   # CI/CD, bypasses prompt

See CLI for the full auth reference and environment variables.

Practical example prompts

These prompts exercise the full Trawl + Claude integration stack:

1. Monitor Hacker News for competitor mentions

"Scrape the front page of Hacker News every hour. If any headline mentions 'Apify' or 'Browserless', send me an alert."

Chain: trawl-scrap-design (selector) → trawl-cli (create + set alert) → MCP trawl_get_scrap_history (review)

2. Track a product price drop

"Watch the prices on https://shop.example.com/products weekly. Email me if any product drops by more than 10% from the previous run."

Chain: trawl-scrap-design (price extractor) → trawl-cli (create, schedule) → Notifications (alert threshold)

3. Monitor GitHub Issues for new bug reports

"Scrape my GitHub Issues feed weekly. Use AI Digest to send me a summary of all issues opened in the past 7 days."

Chain: trawl-scrap-design (GitHub issues page) → trawl-cli (create, schedule) → AI features (digest prompt)

4. Audit a competitor's pricing page

"Create a one-off scrap of https://competitor.example.com/pricing. Run it now and show me the result."

In Claude Desktop: describe what to scrape → MCP trawl_trigger_scrap_run → trawl_get_scrap_history to read result

MCP tool reference

Tool What it does
trawl_whoami Verify auth: returns your user, org, and scrap-id allowlist
trawl_health_ping Liveness probe — confirms you're talking to the right server
trawl_list_scraps List scraps this key can access (filterable, paginated)
trawl_get_scrap Fetch full scrap definition by id
trawl_trigger_scrap_run Run a scrap on demand, with optional param overrides (requires trawl:scraps:run scope)
trawl_scrap_latest_result Return the most recent run result for a scrap without re-triggering it (requires trawl:history:read scope)
trawl_create_scrap Generate extraction code via AI from a URL + goal, create the scrap, and trigger its first run (requires trawl:scraps:create scope, metered)
trawl_get_scrap_history Read run history (cursor-paginated, date-filterable)
trawl_get_run Diagnose a single run (error, failed selector, block status, auto-fix outcome)

Timeout note: trawl_trigger_scrap_run and trawl_create_scrap block synchronously until the run finishes — typically 30-250s. Set your MCP client's tool-call timeout accordingly.

For full MCP reference including JSON-RPC shapes, error codes, and rate limits, see MCP Server.

Troubleshooting

Symptom Likely cause Fix
MCP server not connecting in Claude Desktop Config JSON syntax error or wrong file path Validate JSON — common culprit is a trailing comma. Config path on macOS: ~/Library/Application Support/Claude/claude_desktop_config.json.
Tools not appearing after adding MCP config Claude Desktop not restarted Fully quit and reopen Claude Desktop after editing config.
Skills not triggering in Claude Code Skills not installed or Claude Code not restarted Run npx @trawlme/skills install then restart Claude Code. Confirm with trawl skills list.
trawl: command not found CLI not installed or not in PATH Run npm install -g @trawlme/cli, then verify with which trawl.
401 Unauthorized on MCP calls API key missing, expired, or wrong scope Create a fresh key under Developers → API Keys. Scoped keys need trawl:scraps:read to call trawl_list_scraps. See API Keys.
trawl_trigger_scrap_run returns -32001 forbidden API key doesn't have trawl:scraps:run scope or scrap not in allowlist Add trawl:scraps:run scope to the key, or create a key scoped to the specific scrap id.

See also: MCP Server · Skills · CLI · API keys

MCP Server (Model Context Protocol)