003 - Writing
Notes from the backend.
Systems, payments, retries, federation, and the boring decisions that keep production calm.
Observability Is Not Three Pillars.
At 02:47 you get paged: checkout is failing for one customer in twenty. You have all the tools. You open the metrics dashboard, and yes, the error rate is climbing. You tail the logs, and there are thousands of lines, some errors, but nothing that tells you *who* is failing or *why*. You have distributed tracing, so you go looking for a broken trace, and the sampler kept one request in a hundred and none of them are the broken ones.
Zero-Downtime Schema Migrations.
At 03:00 you ship a migration that adds one column to the `orders` table. It ran in 8 milliseconds in staging. In production it takes the whole site down for four minutes, and the confusing part is that the migration itself *still* finishes in 8 milliseconds. Nothing in your code changed. You added a column. And every checkout on the site timed out.
Distributed Job Scheduling and Durable Workflows.
Picture a single server that wakes up every night and pays out your merchants. It works fine, right up until the power flickers at 02:00 and the server is asleep when the payout was due. Nobody runs the job. You find out at 09:00, when a merchant asks where their money is.
Next
More notes when they earn their place.