The system said “done”, although it had done half the work
The client runs two studios and settles with the staff in an internal system. The problem showed up with one larger payout: CHF 18,080 and 212 items. About half of the records were written, the other half were not, and success appeared on the screen.
This is a more dangerous kind of failure than an ordinary error message. The owner does not know they should react, and partially saved data can later lead to double charging or laborious manual checking.
Why it happened
The system saved every item separately. With a larger payout it sent over 270 requests in a short time, exceeded the limits of the platform in use and lost part of the responses. The errors were caught, but shown nowhere.
Tests with slower writes did not help. The problem was not the pace, but the sheer number of separate operations. What was needed was a change in the way settlements are made, not another delay between attempts.
What I changed
Saving in batches
Instead of updating over 200 items one by one, the system sends them in a few larger batches. The number of connections fell from about 272 to about 10. Settling 210 items now takes 3.9 seconds.
A clear state for every payout
A payout can be started, completed or failed. The “completed” status appears only when all the steps have been confirmed. Repeating the operation does not charge the same money a second time.
A check that speaks up only when there is a problem
Every night the system compares the latest payouts with the source records. When everything adds up, it stays silent. When it detects a difference, the owner gets a Telegram message saying which payout needs checking.
Finishing safely
An unfinished payout is clearly visible in the panel. The “Finish payout” button fills in only the missing records. There is no need to correct data by hand or to ask a developer to step in at every similar event.
Deployment without stopping the studios
The new way of settling was checked on a test account, not on the data of real employees. The previous mechanism stayed ready to be restored until the new one had passed all the trials. The studios worked without a break.
The result
The most important change is not about speed, although that improved more than thirtyfold. A system that handles money stopped pretending that everything is fine. Every payout has a trail, a correctness check and a safe way to repair it.
The numbers on this page come from the event described and from the 210-item test. They are not an average or a promise of a result in another system.
This repair is one of the stages of ongoing care for the studio’s system. See how it started and what came later →