Why Clocks Break Distributed Systems
An event can happen later and still receive an earlier timestamp.
Consider two related operations handled by different nodes. Node A records a profile update at 10:00:00.120. The update then leads to an account deletion on Node B, whose clock is slightly behind and records it at 10:00:00.115. The deletion happened later, but sorting by timestamp places it first.
Clock skew does not change the actual execution order. The problem appears when a system treats physical timestamps as authoritative. Conflict resolution, last-write-wins policies, or audit pipelines can then assign the wrong precedence to valid events.
Physical timestamp order, real-time order, and causal order can differ in a distributed system. Reliable event ordering requires an ordering mechanism that provides the guarantee the system needs.
This newsletter examines:
Why Network Time Protocol (NTP) reduces drift without eliminating ordering uncertainty.
How Lamport clocks preserve causal precedence.
How vector clocks expose concurrent updates.
How ordering choices shape consistency guarantees.
How bad ordering corrupts data and audit trails.
A natural first response to clock skew is to synchronize the clocks more closely. NTP does exactly that, but synchronization reduces uncertainty rather than removing it.
NTP reduces drift but leaves uncertainty#
NTP periodically estimates a machine’s offset from time servers and adjusts its local clock. This keeps clocks reasonably aligned, but it does not make them identical.
Several sources of uncertainty remain. Hardware oscillators drift at different rates, synchronization happens periodically, and network delays vary. Asymmetric network paths can also distort offset estimates when a request and its response experience different transit times.
The result is residual uncertainty in the clocks. Two nodes can record nearby events with timestamps that do not reflect their real-time or causal order. NTP does not give application logic a universal hard bound that makes timestamp comparison safe for correctness-sensitive ordering.
The key distinction is between keeping clocks operationally close and using them as a correctness mechanism. NTP is valuable for logging, metrics, scheduling, and correlating activity across machines, but those uses tolerate a level of uncertainty that strict event ordering may not.
Watch out: A physical timestamp can be useful for observability without being authoritative evidence that one distributed event occurred before another.
Reducing clock uncertainty helps operationally, but it does not solve the ordering problem. Systems that need to preserve causal relationships require a mechanism that does not depend on synchronized physical time. Logical clocks provide that next step.
Logical clocks preserve causal order#
If physical clocks cannot reliably establish event order, the system can track ordering without using wall-clock time. Lamport clocks do this with a logical counter maintained by each process.
The counter advances according to two rules:
Local or send event: Increment the counter before the event. A sent message carries the resulting counter value.
Message receive: Set the counter to
max(local_counter, received_counter) + 1.
These rules preserve causal precedence. If event A happened before event B, then:
L(A) < L(B)
The reverse does not hold. A lower Lamport timestamp does not prove that A caused B because independent, concurrent events can also receive different counter values.
This distinction is important. Lamport clocks preserve the ordering required by causal dependencies, but they do not reconstruct the happened-before relation from timestamps alone.
Key limitation: Lamport timestamps preserve causal precedence when a causal relationship exists, but they cannot determine whether two differently timestamped events are causally related or concurrent.
Systems that need to detect concurrent updates therefore need to retain more information than a single counter can represent. Vector clocks extend the same idea by tracking logical progress across multiple participants.
Vector clocks reveal concurrency#
Lamport clocks preserve causal precedence, but they cannot determine whether two differently timestamped events are causally related or concurrent. Vector clocks retain more information, so the system can make that distinction.
In the standard fixed-membership model, each process or replica maintains a vector with one counter per participant. The update rules are:
Local event: Increment the process’s own counter.
Message send: Include the current vector with the message.
Message receive: Take the element-wise maximum of the local and received vectors, then increment the receiver’s own counter.
Two vector timestamps can then be compared directly. Event A happened before event B when every component of V(A) is less than or equal to the corresponding component of V(B), with at least one component strictly lower.
If neither vector dominates the other, the events are concurrent.
That extra information makes concurrent versions visible instead of forcing them into an arbitrary scalar order. The application can then decide whether those versions conflict, commute, can be merged, or require another resolution policy.
The trade-off is metadata overhead. In the basic model, vector size grows with the number of tracked participants, increasing storage, transmission, and comparison costs as membership grows.
Practical point: Vector clocks detect concurrent versions. They do not determine whether those versions represent an application-level conflict or how that conflict should be resolved.
Once a system can distinguish causal order from concurrency, the next question is what ordering guarantee the application actually needs. That choice is closely tied to the consistency model the system is designed to provide.
Consistency depends on ordering#
Vector clocks show that different ordering mechanisms preserve different information. That distinction matters because a consistency model can only provide guarantees that the underlying system can actually enforce.
Common requirements map to ordering in different ways:
Linearizability requires each operation on a linearizable object to appear atomic and to respect real-time precedence. If operation A completes before operation B begins, B cannot appear before A. Replicated implementations often rely on an authoritative leader, coordination, or consensus-based sequencing to maintain this guarantee.
Causal consistency preserves cause-and-effect relationships. If operation A causally precedes operation B, every observer that sees both must observe A before B. Concurrent operations do not need to appear in the same order everywhere.
Eventual consistency allows replicas to temporarily diverge and requires them to converge once updates stop. It does not prescribe a particular event order or conflict-resolution policy.
Per-entity ordering guarantees sequencing only within a key, account, document, partition, or other ownership boundary. Many systems use this narrower guarantee when a global order would add unnecessary coordination.
These guarantees depend on different ordering capabilities. Linearizability needs real-time-respecting coordination, causal consistency depends on causal precedence, eventual consistency may use concurrency detection during reconciliation, and per-entity ordering limits sequencing to a narrower scope.
The ordering primitive should match the required semantics. Physical timestamps work for approximate wall-clock time, Lamport clocks preserve causal precedence, vector clocks expose concurrency, and consensus-backed sequencing can establish an agreed order where needed.
Hybrid Logical Clocks (HLCs) combine wall-clock proximity with logical ordering information, but they do not provide consensus or linearizability on their own.
A mismatch between the claimed consistency guarantee and the available ordering information can produce behavior that appears valid while violating application semantics. The consequences become easier to see when ordering errors reach conflict resolution and audit histories.
Bad ordering corrupts data and audit trails#
Ordering failures can surface directly as application-level errors. A system can accept, store, and replicate every event successfully while still applying the wrong precedence.
Consider a last-write-wins (LWW) policy based on physical timestamps. If an older update is written by a node whose clock is ahead, it can receive a later timestamp and overwrite a more recent value. Clock skew becomes harmful when the system treats timestamp order as authoritative.
The same problem affects audit trails. Sorting records only by physical timestamp can produce a chronology that conflicts with the actual dependencies between operations. Transaction IDs, sequence numbers, traces, or other ordering metadata may still allow reconstruction, but timestamps alone cannot guarantee an authoritative history.
Operational risk: Ordering failures often produce valid-looking records rather than explicit errors, so uptime and latency metrics may remain normal even when data or audit semantics are incorrect.
These failures show why ordering must be designed explicitly. The next step is to turn that principle into a practical choice of ordering mechanism based on the guarantees the application actually needs.
Ordering is infrastructure#
The failures above point to a broader design principle. Event ordering should be chosen as deliberately as storage, replication, or consistency strategy. Raw physical timestamps do not provide a reliable distributed ordering guarantee on their own.
The right mechanism depends on what the application needs:
Approximate wall-clock time: Synchronized physical clocks may be sufficient for logs, metrics, scheduling, and observability.
Causal precedence: Lamport clocks can preserve cause-and-effect ordering.
Concurrency detection: Vector clocks can identify concurrent versions.
Agreed sequencing: An authoritative leader or consensus-based mechanism can establish a shared order within the required scope.
Per-entity ordering: Sequencing is often limited to a key, account, document, or partition rather than coordinating the entire system.
Hybrid Logical Clocks offer another option when a system needs timestamps that remain close to physical time while carrying logical ordering information. They do not synchronize clocks or provide consensus or linearizability on their own.
Before choosing a mechanism, the design should answer a few questions: Does correctness depend on real-time precedence? Must causal relationships be preserved? Must concurrent versions be detected? Is ordering required globally or only within an entity?
Design principle: Clock uncertainty is a normal property of distributed systems. Correctness comes from selecting an ordering mechanism that explicitly provides the guarantees the application depends on.
Once those requirements are explicit, physical time can remain useful without being treated as a global ordering guarantee.