Starting point
A returning client (services industry) runs two near-identical studios on a no-code platform (entities + Deno edge functions, 2-way GitHub sync). Reception registers services, runs the cash register, and pays out employees. After an earlier rescue the system was stable — until volume grew.
Problem
The message that started it: a single payout worth 18,080 CHF covering 212 records had gone through for only about half of them — yet the system reported success and moved on.
- Silent half-failure: only ~100 of 212 records were marked. The function still reached the end and wrote an audit log, so the payout looked complete while the ledger was inconsistent.
- Writes from the browser: the entire payout fired hundreds of individual record updates straight from the client. Past ~200 operations it hit the platform's rate limit.
- Swallowed errors: a
try/catchquietly absorbed every failed write — no alert, no rollback, no status. Nothing told anyone it had only partly worked. - Collateral damage: the manual rescue overwrote 11 day-settings records with the wrong payout reference — a latent double-charge risk.
Diagnosis
The obvious hypothesis was "too much concurrency." I disproved it with a series of controlled load tests:
- Concurrency 10, no retry: 100 / 217 marked, 117 rate-limit errors.
- Concurrency 5 + retries: 25 / 215 — worse; retries only multiply the operation count.
- Concurrency 3, clean 210: 200 / 210.
The pattern was clear: the bottleneck was the total number of write operations (~250+) in a single function — not the concurrency or the delay between them. Tuning would never fix it; the architecture had to change.
Solution
1. Bulk operations instead of per-record writes
The platform SDK exposes bulk endpoints — up to 500 records per single HTTP call. I rewrote the payout to batch every multi-record step (a chunking helper slices into 500-record pages). Roughly 272 calls collapsed to ~10, and a 210-record payout dropped from ~126s (failing) to 3.9 seconds.
2. A real state machine on every payout
Each payout now starts as pending and only becomes
committed when every step succeeds — otherwise failed,
with the exact failed steps recorded. No operation can finish "half-done" and
pretend it didn't. Full idempotency means a retry can never double-charge.
3. Monitoring that speaks up — and only then
A scheduled nightly audit scans recent payouts for inconsistencies and pings the owner on Telegram only when something is wrong. When everything is clean, it stays silent. The employee notification fires only after a successful commit.
4. One-click recovery
Any payout left pending or failed surfaces in the UI as a
red banner with a "Finish payout" button — an idempotent repair routine that
completes the missing writes safely. The rare failure becomes a single click, not
a data-surgery session.
Result
Both studios now run the same engine. A 210-record payout takes ~3.9 seconds and can no longer fail quietly: if anything breaks, the owner knows before an employee does and fixes it with one click. An audit of the previous 30 days confirmed every historical payout but the original incident was clean. The whole rollout happened with zero production downtime — the old mechanism stayed in place as a 30-second rollback, and every change was tested on a throwaway account, never on real employee data.
Takeaways
The fix that mattered wasn't speed — it was visibility. A system that handles money must never be allowed to fail silently. Once payouts move real cash, "it reached the end without throwing" is not the same as "it worked."
No-code platforms scale beautifully until a single operation has to touch hundreds of records at once. That's the line where clicking things together ends and engineering — batching, atomicity, idempotency, monitoring — begins.