A scrap that runs is not the same as a scrap that works. This guide is about the second: how to tell a run that came back wrong from one that failed outright, why a scrap stops returning good data, and how to read the scrap list at a glance.
When a scrap stops returning good data
The history list tells you a run went wrong. It does not tell you why, and the why decides what you should do. Four causes look alike there and call for four different actions:
| What happened | How it looks | What to do |
|---|---|---|
| The page changed | The run returns 0 items, or far fewer than usual | Fix the selectors — or let Auto-fix do it |
| The site blocked the request | The run fails, and stops at the same point every time | Not a script problem. Slow the schedule down, or scrape a different entry point. Auto-fix will not touch it |
| The page is behind a login | Same as a block, often with a login form in the result | Configure the scrap's account/session, then run it manually to confirm |
| The site refuses us entirely | Every run fails, including manual ones | The target is out of reach for now. Park the scrap or pick another source |
If the same failure repeats, the scrap is
paused for failing after three runs: Trawl skips
its scheduled runs until one works again, and emails you if you have set an alert
recipient. A scrap you only ever run by hand has no schedule to skip — it still
gets the email. Either way, the first run that comes back clean clears the state.
From a terminal, trawl list --unhealthy lists everything currently in
that state, so you can go through them in one pass.
Reading the list at a glance
Once a scrap has run at least once, it carries a status icon next to its title showing how its most recent run went — except when the scrap is stuck, which is about the runs as a group and outranks any single one:
| Icon | Means | What to do |
|---|---|---|
| Orange spinner | The run is still going | Nothing — come back when it lands |
| Green check | The last run worked | Nothing |
| Orange triangle | The run finished, but returned far fewer items than usual | Look at the results — the page probably changed shape |
| Red cross | The last run failed, was blocked, or returned nothing | One bad run. Worth opening if it repeats |
| Red circle | The last three runs all counted as failures | The scrap is stuck — Trawl skips its scheduled runs until one works (details) |
What "far fewer" means. Trawl takes the median item count of your recent
successful runs and flags anything that comes back under a fifth of it. It needs
a few successful runs to compare against first, and a baseline of at least a
handful of items, so a brand-new scrap and a scrap that only ever returns one or
two rows are never flagged this way. Neither are runs you launch through
trawl trigger or the equivalent API call — that path skips the check
entirely, so a CI pipeline built on it will not see a short run flagged.
An orange triangle counts as a failure too. For the three-run streak, a run that comes back far too short counts exactly like one that failed outright — so three orange triangles in a row give you a red circle, and a short run does not clear the state. Only a clean run does.
The difference between the last two is the one worth internalising: a red cross is a bad run, a red circle is a scrap that will not fix itself.
See also: Scraping Advanced · Notifications · AI Features
Next step → Results & payload