01 / THE PROBLEM
Pricing rules shaped the execution problem.
The requirement was to calculate usage-based charges from billable events, configured unit prices, and configurable slicing and discount tiers. Horizontal slicing evaluated discount tiers per device or card; vertical slicing evaluated them across the tenant's usage.
Starting from that high-level requirement, I independently architected and implemented the core distributed bill-run engine. My implementation covered the mapper, reducer, finalizer, and reporting phases, with work partitioned for parallel execution.
Partitioning the work also meant coordinating when stages could advance. I had to account for checkpointing and recovery from interrupted execution, so that partial progress would not be mistaken for a completed report. Completion needed to be a shared decision that remained meaningful when individual workers stopped or restarted.
02 / A SYSTEM MODEL
Separate the data being processed from the decisions about progress.
I used Cassandra-backed distributed coordination to synchronize execution stages durably. A completion signal held only in one process could not serve as a reliable shared source of truth when independent workers could be interrupted or restarted. Durable coordination let progress and stage-completion decisions survive those interruptions, at the cost of additional state management and coordination overhead.
The conceptual model below separates the data plane, where events are processed and contributions combined, from the control plane, where ownership and stage progression are coordinated. It compresses finalization and reporting into the final output step; it is not an exact four-phase production topology.
The completion requirement carries through each stage: the existence of intermediate output cannot be the sole proof of stage completion. In this model, a reducer needs to know which contributions were expected and which were committed before treating its input as complete. Durable progress alone does not settle whether an overlapping or retried attempt may publish a result. That ownership question is the next part of the design discussion.
03 / OWNERSHIP AND COMPLETION
A lease is not a guarantee that work ran only once.
A lease is a time-bounded claim coordinating which worker should attempt a task. It does not stop a paused worker from resuming after its claim expires. Each execution is a physical attempt; several attempts can belong to the same logical contribution in a particular bill run. Repeating computation need not mean applying that contribution again.
The critical question is where a result becomes authoritative for downstream processing. Checking ownership and publishing later leaves a gap in which authority can change. In the conceptual sequence below, A pauses after checking its claim. B takes over and attempts publication before A resumes. A's earlier check cannot authorize its later write.
A's claim is valid at the check, before publication.
A pauses before writing; its claim expires during the pause.
B takes over, completes the work, and attempts to publish its result.
A attempts to publish using its earlier check, after B has taken over.
At the point a result becomes authoritative, stale attempts must be rejected and the same logical contribution must not be applied twice within its bill run. A successful write alone does not establish that these conditions hold.
These protections have different jobs. A guarded transition or version/fencing check can reject obsolete authority when enforced where the effect is accepted. Guarding a coordination record does not automatically protect a write to another resource. Stable identity associates retries with one logical contribution; idempotent acceptance keeps that contribution from being applied again. Identity alone does not enforce this. Durable completion evidence serves a separate purpose: establishing that every expected contribution is validly complete before advancing a stage. These are conceptual design options, not claims that my implementation used each mechanism.
04 / FAILURE WALKTHROUGHS
Four scenarios worth designing before the happy path is done.
These hypothetical scenarios pair a failure with an acceptance rule and a proposed test. Bounded retries and concurrency can limit retry pressure, but cannot guarantee success. Recovery depends on which failures persist, remaining retry budget, dependency availability, and workers being able to make progress.
01Worker loses ownership
Worker A checks its claim, then pauses before publishing long enough for the claim to expire.
Worker B takes over, but A later uses its earlier check to publish a stale result.
Reject obsolete authority where a result becomes authoritative. A separate ownership check before the write leaves a race; a lease alone cannot protect that boundary.
Pause A after its ownership check. Let its claim expire, let B take over and publish, then resume A. Assert that A cannot become authoritative and the logical contribution is not applied twice.
02Write succeeds; acknowledgement fails
A result write succeeds, but its acknowledgement is lost. The worker cannot tell whether the write took effect.
Treating the missing acknowledgement or timeout as proof of failure can lead a retry to apply the contribution again.
Keep the logical contribution identity stable within its bill run and make acceptance idempotent. Recording acceptance and applying the contribution must agree even after interruption; an identity alone does not ensure this.
Persist a contribution, suppress its acknowledgement, then retry it with the same logical identity. Allow repeated computation, but assert one applied contribution and an unchanged downstream result against a known-correct baseline.
03A stage partially completes
Some expected contributions finish while another remains missing; a completed contribution may also be retried.
An output count can appear complete despite a missing contribution if duplicates are counted as distinct work.
Define the expected contributions for the run. Gate stage completion on valid, durable completion evidence for each one, with no missing contribution or duplicate logical application. Partial output alone is insufficient.
Withhold one required contribution and replay a completed one. Assert that downstream finalization remains blocked. Then complete the missing contribution and verify that valid completion evidence allows the stage to advance without double application.
04Retries compete with useful work
A shared dependency slows down or remains unavailable while multiple workers retry.
Retries consume capacity needed for useful work; independent per-worker limits can still overload the shared dependency.
Bound attempts per logical task and limit aggregate in-flight work against the shared dependency. Define retry spacing and what happens when the budget is exhausted; persistent failure must remain visible rather than appearing complete.
Inject transient and persistent failures. Observe retry rate, aggregate concurrency, useful throughput, and backlog. Verify the configured limits and budget-exhaustion behavior; check recovery only when dependencies recover and retry budget remains.
These examples illustrate possible failure modes; they are not claims about specific bugs or incidents at Thales.
05 / TRADEOFFS
Each reliability decision has a cost.
For the core bill-run engine, I chose Cassandra-backed shared coordination over relying exclusively on completion signals held in worker processes. Independent workers could be interrupted or restarted, so stage progression needed durable evidence that remained available beyond any one worker's lifetime. That choice made persistent coordination state part of my implementation responsibility.
Swipe horizontally to read all table columns →
| Decision | Why it helps | What it costs |
|---|---|---|
| Parallelize mapping | Partitioning offers concurrency; the boundaries still have to preserve the selected billing semantics. | More partitions can add scheduling, synchronization, and aggregation work. Extra workers alone do not establish an end-to-end improvement. |
| Make phase boundaries durable | Shared completion evidence remains available across worker interruption or restart. | Persistent coordination adds operations and state to manage, with potential contention and more complex transitions. |
| Recover abandoned work | Reclaiming abandoned work gives an interrupted run a path to continue when recovery conditions allow. | Lease expiry can overlap attempts; recovery must account for stale authority and duplicate logical contributions. |
| Gate report finalization | A completion barrier keeps finalization dependent on valid evidence for all expected contributions. | Missing contributions can delay the report even when most work has finished. Partial progress must remain distinguishable from completion. |
The table describes the competing considerations around that choice. I accepted additional coordination operations, persistent state management, and more complex transition and recovery reasoning. Shared state can also introduce contention. Their performance impact requires measurement under a defined workload.
06 / VALIDATION
The measurement is only useful if you can explain it.
My project work included the core engine implementation and laboratory performance evaluation. The criteria below explain what would substantiate a result; they are not a record that every check was executed. The preceding fault-injection scenarios remain proposed tests. Reported measurements still require review of their supporting evidence.
Check charges against independently established expected results for the chosen pricing and slicing rules. Verify complete contributions without double application; producing an output alone is insufficient.
Under stated failure and retry assumptions, verify that interrupted work can reach a valid completed report. Distinguish bounded retries that make progress from repeated attempts that never complete.
Specify workload and distribution, reporting scope, environment, concurrency, and timing start/stop boundaries. State exclusions so the elapsed time describes a defined amount of completed work.
Measure stage durations, coordination activity, and waiting for contributions alongside end-to-end time. A total duration alone cannot identify which stage or delay dominates the run.
Public figures are intentionally withheld here until confirmed against test evidence and approved for publication. This avoids turning an internal lab result into an unsupported production-scale claim.
07 / WHAT I TOOK AWAY
I had to account for the whole run.
Starting from a pricing requirement, I had to decide how the processing phases fit together and what completion meant when the workers could change.
Making coordination durable put state management and recovery reasoning inside my core-engine responsibility. Concurrency required me to think about the validity of the whole report as well as the execution of each worker.
The lesson I take forward is to make completion conditions and recovery assumptions explainable, then distinguish what the design intends from what verification actually demonstrates.