Backend Engineering Is Restoring Determinism

I’ve worked in backend engineering and distributed systems for years.

Accross different systems, teams, and codebases, I've seen the same problems recur.

And I've seen them being solved often in ad hoc ways:

Ad hoc is a Latin phrase meaning literally 'for this'.
In English, it typically signifies a solution designed
for a specific purpose, problem, or task rather than a
generalized solution adaptable to collateral instances.

Specific. Improvised. Brittle.

There is value in that. Engineering requires craft, creativity, and pragmatism.

But engineering is also:

The application of scientific and mathematical
principles to practical ends such as the design,
manufacture, and operation of efficient and economical
structures, machines, processes, and systems.

You cannot build a cathedral ignoring physics. It will fall apart.

You cannot build software ignoring mathematics. It will fall apart... on a Sunday... at 3AM.

And it is common practice to ignore mathematics in software development, unfortunately.

And doing so, puts a very low ceiling on what we can build, and how quickly can we build it.

You cannot build a cathedral. Even less a space ship.

This post explains some foundational ideas I wish I had learned when I was starting my career.

Ideas that make reliable backend systems seem less like a bag of tricks, and more like one coherent subject.

Ideas for you to build reliable software from first principles.

Determinism

A system is deterministic when its behavior is uniquely determined by its input.

A pure function is deterministic:

f : X → Y

For a given input x, it always returns the same output f(x).

It reads no hidden state. It mutates nothing outside itself. Its behavior is fully contained in its input and definition.

That is the simplest form of computation:

one input, one outcome.

Determinism composes

If two pure functions are deterministic,

f : X → Y
g : Y → Z

then their composition is deterministic too:

g ∘ f : X → Z

A system built entirely from pure functions is therefore just a larger pure function:

input → output

State does not destroy determinism

Real programs remember things. They have state.

But state can still be modeled deterministically. Make it explicit:

δ : S × I → S × O

Given:

current state s ∈ S
input i ∈ I

we get exactly:

next state s'
output o

So one step is:

(s, i) → (s', o)

If the initial state is known and the input sequence is known,

i₁, i₂, ..., iₙ

then the entire state trace is known too:

s₀ → s₁ → s₂ → ... → sₙ

A stateful program is still deterministic when its transition function is deterministic and its input order is known.

An unreliable network explodes the execution space

Suppose g() sends five messages:

1, 2, 3, 4, 5

across an unreliable network. On the other side, f() computes over whatever messages arrive.

The network may:

drop messages
reorder messages
later, if retries exist, duplicate messages

So the receiver might see:

1, 5, 4

2, 1

5, 3, 4, 1, 2

or nothing at all.

Even before considering duplicates, five sent messages already produce:

Σₖ₌₀⁵ C(5, k) · k! = 326

possible delivered-and-ordered execution traces.

This is the basic shock of distributed systems:

one logical intent, many physical executions.

Semantics can collapse executions into one outcome

The number of executions may be huge. The number of outcomes depends on the semantics.

If f is commutative, arrival order does not matter:

a ⊕ b = b ⊕ a

If g retries until delivery, message loss does not matter.

If processing the same message twice is idempotent, duplicates do not matter:

a ⊕ a = a

If all three hold, then many physical executions collapse into one logical outcome.

The network may choose among hundreds of schedules, but the program exposes only one meaning.

The system does not eliminate chaotic executions. It chooses semantics that make them equivalent.

That is the seed of reliable distributed systems.

From outcome compaction to trace compaction

For a pure function, correctness means:

same input → same output

For a stateful system, correctness is stronger:

same logical input trace → same logical state trace

But the physical implementation may realize that logical trace through many different executions:

retries
reordered messages
crashes
recovery
leader failover
network partitions

Let:

P be the set of physical execution traces
L be the set of logical traces
π : P → L map each physical trace to its logical meaning

Then a reliable implementation tries to ensure:

π(p₁) = π(p₂)

for all acceptable physical executions p₁ and p₂ of the same logical operation.

I will call this trace compaction:

Trace compaction is the engineering work of collapsing many messy physical executions into one clean logical trace.

Backend engineering is largely the design of these compactors.

Idempotency compacts retries across time

Consider a request:

Deduct $50 from Account A.

The client sends it, then times out.

From the client’s point of view:

maybe applied ∨ maybe not applied

The only practical move is to retry.

But if the first request actually succeeded, the server may now receive the same command twice. Naively:

200 → 150 → 100

Wrong.

The fix is idempotency.

The client attaches a stable operation ID:

operation_id = abc123

The server records the result of the first execution:

abc123 → success, new_balance = 150

If the same operation arrives again, the server returns the recorded result instead of executing it again.

So physically:

one send, two sends, ten retries

all become logically:

one deduction

Idempotency turns “maybe happened” into “retry safely.”

It compacts uncertainty across time.

A write-ahead log compacts crashes

Now suppose the server begins processing the deduction and crashes halfway through.

Without durable structure, the system may awaken in an ambiguous state:

maybe the update happened
maybe it did not
maybe only part of it happened

The fix is a write-ahead log:

durably record the intended operation
apply the state change
recover by replaying or completing logged work

Now these different physical executions:

no crash
crash after logging
crash after applying but before replying
reboot and recover

can all project to the same logical trace:

one completed deduction

A write-ahead log turns crash recovery from guesswork into deterministic continuation.

It compacts uncertainty caused by volatility.

Consensus compacts divergence across space

A single durable machine can still disappear forever. So we replicate state across machines.

But replication introduces a new danger: different replicas may observe different messages in different orders.

During a partition:

one replica may accept one update
another replica may accept a conflicting update

If both histories can commit, the system forks into two valid realities.

The fix is consensus.

Protocols such as Raft and Paxos ensure that only one history becomes canonical. Typically:

a leader proposes operations
a quorum must durably accept them
conflicting histories cannot both commit, because majorities intersect

In a 3-node cluster, a commit requires 2 nodes. Two distinct majorities cannot be disjoint. They must overlap somewhere.

Consensus therefore compacts a spatially distributed system into one authoritative log:

op₁, op₂, op₃, ...

Consensus does not remove failures. It prevents failures from creating two committed histories.

It compacts uncertainty across space.

The essence

At the level of business logic, we want programs to behave like pure functions:

same logical input → same logical result

At the physical level, systems are messy:

networks drop, reorder, and duplicate messages
machines crash
replicas disagree
observers lack complete knowledge

The task of backend engineering is not to make that mess disappear.

It is to make the mess irrelevant to the meaning of the computation.

That is why commutativity matters. That is why retries matter. That is why idempotency matters. That is why logs matter. That is why consensus matters.

They are all ways of saying:

Many physical executions. One logical trace.

That is determinism restored.