Automate your scraps so they run on their own — no manual clicks, no cron tabs on your server, just set it and forget it.

Enabling a schedule

Open your scrap's Settings → Automation and toggle Schedule. Pick a preset frequency or enter a custom cron expression.

Preset frequencies

Presets receive a per-scrap jitter offset so runs are spread across the window instead of piling up at the top of the minute. The expression stored on the scrap is the canonical one; the runtime applies the offset transparently when dispatching.

Custom cron expressions

Need something specific? Enter any valid 5-field cron expression:

┌───────── minute (0-59)
│ ┌─────── hour (0-23)
│ │ ┌───── day of month (1-31)
│ │ │ ┌─── month (1-12)
│ │ │ │ ┌─ day of week (0-6, Sunday=0)
│ │ │ │ │
* * * * *

Examples:

• 30 9 * * 1-5 — weekdays at 9:30 AM
• 0 8,20 * * * — twice a day at 8 AM and 8 PM
• 0 0 1 * * — first day of every month at midnight

Timezone

Schedules run in UTC by default. Configure a timezone in Settings → Automation → Timezone to align with your local time. All standard IANA timezone names are supported (e.g. Europe/Paris, America/New_York).

Quota and subscription

Scheduling requires an active subscription with available quota. Each scheduled execution counts against your plan's run allowance. If your quota is exhausted, scheduled runs are paused until the next billing cycle or until you upgrade.

When a scrap keeps failing

If a scrap fails three runs in a row, Trawl stops sending its scheduled runs for a while. A scrap whose site is down, blocked, or rebuilt will keep failing at every tick, and there is no reason to spend your quota — or fill your inbox — on it every hour.

Every kind of run counts towards those three: scheduled, manual, and API-triggered. What gets paused is only the scheduled one. A run that finishes but comes back far shorter than usual counts as a failure too — see Data quality.

What this does not do:

• Your schedule is not turned off. The frequency you chose stays exactly as you set it.
• Skipped runs cost you nothing. They do not count against your quota.
• Manual runs are never skipped. You can always run the scrap yourself from the workspace, the API, or the CLI — that is how you test a fix. They do count, though: a manual run that fails moves the scrap closer to the pause, and one that succeeds clears it.

How it recovers on its own:

• If the scrap has not run at all for about two weeks, Trawl lets the next scheduled run through anyway, to check whether the site works again. Skipped runs do not reset that clock; a real run does.
• The first run that comes back clean — that retry, or a manual run of your own — clears the state, and your normal schedule resumes at the next tick. A run that finishes but comes back far too short does not clear it.

In the scrap list, the status icon next to the title turns into a red circle once the third run has counted as a failure — it does not wait for the next scheduled tick to come round. Hover it and it tells you what happened and what Trawl does about it. From a terminal, trawl list --unhealthy lists every scrap currently in this state — see the CLI guide.

You also get an email when a scrap reaches three failures — and again on each later run that fails — provided you have set an alert recipient. That does not depend on anything being paused: a scrap with no schedule has nothing to pause, and still emails you. See Notifications.

What happens on each run

1. The cron triggers the scrap at the scheduled time.
2. A worker executes the request script.
3. The result is stored in History with its status.
4. If configured, notifications are sent (email results, alerts).
5. If enabled, AI Digest summarizes the data and emails it.

If the scrap has failed its last three runs, the scheduled run is skipped instead — see When a scrap keeps failing above. Manual and API-triggered runs always execute.

All executions — manual, scheduled, and API-triggered — appear in the same History timeline.

---

See also: Notifications · AI Features · Scraping

Next step → Notifications