The weekly letter from Systems Thinking Lab: systems thinking insights for junior engineers, framed through the seven building blocks.
The most reliable part of a stack is rarely the part anyone shows off. It has no clever layer to point at. That is the whole reason it never breaks.
A payment service I want to walk through this week is a clean example of that. One queue sat in front of it, catching every incoming charge. One database sat behind it, recording each charge exactly once. That was the entire system, and it had looked that way for four years.
Engineers rotated onto the team and asked the same question every time. Why is this so primitive? No event mesh. No service constellation. No clever caching layer sitting in front to make it feel faster.
Twice a year, someone pitched a redesign. Twice a year, the answer was no.
Then Black Friday arrived. The recommendation engine buckled under the traffic and stopped returning results. Search slowed to a crawl and started serving stale listings. The mobile app threw errors across half its screens, and the support queue backed up.
The payment service kept working. Every single order that reached it got processed, exactly once, without a single duplicate charge.
Boring was never the absence of ambition. It was the design.
The parts doing the real work here have names. Use them. A Service is the piece that answers you while you are standing there waiting on it, the way a waiter takes your order and comes back with a plate. A Queue is a line for work that does not need to happen the instant it shows up, so a sudden rush of it lands somewhere safe instead of getting dropped. A Relational Database is the record book: the one place that can say, with certainty, this specific charge happened, once, and only once. One Service, one Queue, one Relational Database. Three pieces, not ten.
You can see the same restraint inside Stripe's own payment API. Every charge you send Stripe can carry an idempotency key, a value that means "this is the same request if you see it twice." Your network drops the response, you retry the request, and Stripe uses that key to recognize the retry and hand back the original charge instead of billing your customer again. Stripe did not hand you a new service to run to catch duplicates. It handed you a key, and made one place the last word on whether a charge already happened. Same restraint. Same refusal to add a fourth. You do not need Stripe's traffic to use the same idea. You need one place your data trusts to be the last word on whether something already happened, and the discipline to let it do that job alone.
The way a system like this actually gets worse is rarely neglect. It is ambition wearing a good disguise. An engineer joins the team, wants something to show for the quarter, and pitches the event mesh, or the extra caching layer, or a second message broker sitting in front of the first one. Every one of those pitches sounds like seniority. On a slide, an event mesh reads like foresight. Under real load, it is a new component that has to come up cleanly, hold a connection, and survive Black Friday: a new thing that can time out, drop a message, or disagree with the database about what actually happened, at the exact hour agreement matters most. Saying no twice a year was never stubbornness. It was refusing to add one more way to fail during the one week failure would have cost the most.
In Course 1 I teach these seven building blocks before anyone draws a single box, because the skill this payment service is showing off is not stacking parts well. It is knowing when three of them are enough, and having the discipline to stop there.
You do not protect a system by adding to it. You protect it by refusing what it does not need.
P.S. Knowing when three blocks are enough is a skill you can learn deliberately. The decision framework that teaches it is free here: systemthinkinglab.ai/learn/building-blocks/decision-framework/