How I stopped babysitting scrapers on multiple machines

Some jobs can't be done through an API. One of mine is a desktop automation that works through a Windows app screen by screen, just like a person would. Each item takes 20 to 60 minutes, so a monthly run of about a thousand items is spread across several computers.

That part worked. Knowing what was actually going on didn't.

Every month I split the work list by hand and copied it onto each machine. To check progress, I had to remote into each one and read its terminal. Results sat on separate machines until the very end, and if one froze overnight, nobody knew until someone happened to look.

Before building anything, I checked last month's run against the original list. The list itself turned out to be the problem: duplicates, names spelled differently from the source system, a few values a spreadsheet had quietly turned into dates, and items marked done with no data behind them.

I cleaned it up and sent the unclear names back to the client to decide. A dashboard is only as honest as the data you feed it.

My first instinct was a server and a database. I didn't need either. Each machine reports a few times an hour and works through its own list, so nothing competes and nothing needs locking. A lightweight shared store that a person can open and fix by hand was plenty.

Each machine keeps its real progress locally and sends a copy to the dashboard. If the network drops, the work carries on. If a report fails, it's logged and skipped.

One page that answers "how's the run going?":

• Overall progress, items left and how many machines are working.
• An estimated finish time, based on the slowest machine still running.
• A card per machine in plain English: "Working on item 102 for 12 minutes" or "Stopped after an error".

My favourite rule: no item has ever taken more than 60 minutes, so a machine stuck on one for over 90 has almost certainly frozen. The machine can't tell you that. The silence can.

It's a small Next.js app behind a login. It refreshes every minute, works on a phone, and is read-only on purpose, so even if someone got in, they couldn't reassign the work.

• Clean your input before you measure anything.
• Use the simplest backend that fits. You might not need a server at all.
• Report failures too, not just successes, or your progress count is quietly wrong.
• Make setup mistakes loud, so a misconfigured machine stops instead of "finishing" an empty list.

Running automation on more than one machine? Get in touch and let's give it a dashboard.