The weekly letter from Systems Thinking Lab: systems thinking insights for junior engineers, framed through the seven building blocks.
How do I handle errors in distributed systems?
Wrong instinct. I watched the right-looking answer fail on my own site this week: a purchase that paid, said thank you, and delivered nothing. The buyer was me, so nobody got hurt. The next one would not have been me.
Here is what happened. A buyer paid $99 from a phone. Stripe took the money and sent my server the message it always sends: this session is paid. My server went to create the buyer's account. The account already existed, so the auth server said so, in slightly different words than my code was expecting. My code caught the error, wrote one line to a log, and told Stripe everything was fine.
Stripe believed it. The buyer landed on a page that said access granted. Nothing had been granted.
No crash. No retry. No alert. One tidy log line that nobody was reading. I only saw it because I was doing what a real customer does, buying from inside the Facebook app on a Friday afternoon and looking at the page they would see.
Look at what the try/catch did. It did its job. It caught the error. That is the whole problem: catching is not handling, and the code had no idea where the error should go next, so it went nowhere.
Errors are not edge cases you catch. They are a force you design around. The payment processor will go down. The maps API will rate-limit you. The webhook will arrive seven times, or once, with a different sentence in the body than the one you tested against. The question is never how do I catch this. The question is which block absorbs it.
A Queue absorbs a slow consumer. The producer keeps moving while the work waits in line.
A Worker absorbs a flaky call. It retries on its own schedule while the user moves on. Idempotency is the Worker's contract: run it twice, get the same result. That contract is what makes the retry safe.
And the handshake with an External Service absorbs the rest, as long as you tell the other side the truth. If my webhook had answered "not done" instead of "fine", Stripe would have knocked again, on a schedule, for three days. The error would have landed in a retry. Instead it landed on the customer, which is the one place an error must never land.
So trace it. For every failure your system can have, ask where it lands. On a Queue, to be picked up later. On a Worker, to be retried. Back on the caller, who will try again. In front of a person who can fix it. Or on the customer, silently. Only the last one is a bug, and it is the one a try/catch produces by default.
Now the part that matters more this year than last. Ask an AI coding agent to make a handler "more robust" and it will do exactly what my code did. Wrap every call. Log every failure. Return 200. It will do this with total confidence, and the result will pass every test you have, because every test checks the happy path and the one error you thought of. You get robust boxes and a silent system.
The question to put to the agent, and to yourself, is one line per failure: if this fails, who finds out, and when? If the answer is a log line, you have not handled the error. You have hidden it.
In Course 0 that line is part of the plan you write with the agent before it writes any code. The agent builds from the plan, so the failures have a place to land before the first function exists. It is a short list. It is also the list that would have granted my buyer the course.
You do not catch errors. You decide where they land.
Kay
P.S. The Queue is the block that absorbs the most failures with the least code, and the one AI forgets first. The full page on it is here: https://systemthinkinglab.ai/learn/building-blocks/queues/