Trawl ships two AI-powered features that keep your scraps healthy and your data actionable — without manual intervention.
Auto-fix with AI
Websites change. Selectors break. Instead of babysitting every scrap, let the AI fix it for you.
How it works
When a run fails (error or 0 items), or returns dramatically fewer items than the recent baseline (silent regression detection), Auto-fix sends the broken request script and the error output to the AI, which rewrites the script to adapt to the new page structure. The fixed script is saved and the scrap is re-run automatically.
Enable it in Settings → AI Features → Auto-fix with AI.
Notifications: when Auto-fix and Alert on failure are both enabled, the generic failure email is suppressed in favour of an outcome-aware email (recovered or needs-review). See Alerts with Auto-fix for the full table.
Safety guardrails
Auto-fix is not a runaway loop — it has built-in limits:
| Guardrail | Value |
|---|---|
| Max attempts per hour | 6 |
| Consecutive failure breaker | 3 (default) — after 3 consecutive failed auto-fix attempts, automatic (cron) fix attempts pause for that scrap. Configurable via trawl.autofix.failureThreshold in server config. |
| Re-arm | Automatic — the pause is derived from the recent fix history: the first fix that works clears it, and failed attempts older than 30 days stop counting, so a paused scrap gets a fresh try once they age out. Configurable via trawl.autofix.failureWindowDays. |
| Regression detection | Triggers when a run returns fewer than 20 % of the recent baseline item count, based on a minimum baseline of 5 items across at least 3 recent successful runs. Counts as an auto-fix trigger (or a notification when auto-fix is off). |
The breaker is evaluated fresh from the scrap's own recent fix history each time — it doesn't change your Auto-fix toggle, and never turns Auto-fix back on for a scrap you disabled yourself. If it pauses, you'll see a notice in the scrap workspace. Fix this still runs one attempt while the scrap is paused — if that fix works, the pause clears by itself and automatic fixes resume, with no toggle to flip. One exception: on a scrap whose Auto-fix setting you turned off yourself, Fix this is refused while the pause is on. Turn Auto-fix back on to get that one attempt.
What Auto-fix does not fix
Auto-fix rewrites your request script — selectors, waits, navigation. So it only helps when the problem is in the page's markup.
It deliberately does not run when the site blocked the request or the page could not be reached. Nothing in your script is wrong in that case, and rewriting a working selector would only make things worse. Trawl detects this before spending anything on a fix attempt, and emails you the ordinary failure alert right away instead of an Auto-fix outcome — see Alerts with Auto-fix.
If that keeps happening, the scrap ends up paused for failing: the site is refusing you, and that calls for a change of approach, not a change of selector.
It also skips a run that landed on a login page instead of the page you asked
for. Trawl recognizes a handful of known login routes (e.g. a redirect to
/accounts/login, /signin) and, when a run lands on one of them, reports it
honestly instead of the generic zero-results message:
"This looks like a login wall — your own session cookies are the likely fix. See:
trawl scraps account session capture <id>(a real Chrome window opens, you log in, and cookies + localStorage are captured),trawl scraps account session set <id> -c cookies.jsonif you're scripting or in CI, or app Settings → Account."
No selector rewrite can get past a login screen, so Auto-fix skips it and the ordinary failure alert still goes out. See Account & sessions for how to capture and attach a session. A captured session is the likely fix, not a guaranteed one — some sites additionally flag a session replayed from a datacenter IP, proxy tier aside. This detection only catches a real redirect to a known login route — a site that renders its own login form inline, at the URL you requested, with an ordinary 200 response, still reads as an empty run.
When to use it
- A site updated its HTML and your selectors stopped matching
- A navigation flow changed (new cookie banner, redirects)
- Timeout errors due to changed page load behavior
- A site partially restructured: results dropped to a small fraction of the usual count without an explicit failure
Manual AI Fix
You can also trigger a one-shot AI fix from the workspace toolbar — click the Fix button that appears after a failed run. This does not count against the auto-fix hourly limit.
AI Digest
Raw data is useful, but sometimes you want the highlights. AI Digest sends an AI-generated summary of each scheduled run to your inbox.
Setting it up
- Open Settings → AI Features → AI Digest and toggle it on.
- Write a digest prompt — tell the AI what to focus on (max 2,000 characters).
- Optionally set a recipient email (defaults to the email results recipient).
Prompt examples
Good prompts are specific and action-oriented:
- "Highlight any price drops greater than 10%. List new items not seen in the previous run."
- "Summarize the top 5 headlines. Flag anything related to AI regulation."
- "Compare today's inventory count to yesterday. Alert if any SKU dropped below 50 units."
Test preview
Before waiting for the next cron run, use the Test Digest button to generate a preview based on the latest result. This lets you iterate on the prompt until the output is exactly what you need.
When does the digest run?
The digest runs after each scheduled (cron) execution — same trigger as email results. Manual runs and API-triggered runs do not generate a digest.
See also: Schedule · Notifications · Scraping · Account & sessions
Next step → Data quality