← Back to selected work
ENGINEERING STORY / 01BACKEND · CONCURRENT PROCESSING · RELIABILITY

Making distributed billing finish correctly.

Usage-based billing had to apply configured prices and discount tiers per device or across a tenant's usage. I architected and implemented the core distributed bill-run engine behind that requirement. The challenge was to preserve those billing semantics across concurrent work and know when a report was complete, including after an interruption.

In this story
Scope and attribution. My ownership covered the core bill-run engine; other engineers contributed supporting ingestion, bucketing, and API functionality. The diagrams, failure examples, and validation scenarios below are educational abstractions, not documentation of Thales' production architecture or claims that every described test was performed. No internal code, operational configuration, or confidential measurements are reproduced.

01 / THE PROBLEM

Pricing rules shaped the execution problem.

The requirement was to calculate usage-based charges from billable events, configured unit prices, and configurable slicing and discount tiers. Horizontal slicing evaluated discount tiers per device or card; vertical slicing evaluated them across the tenant's usage.

Starting from that high-level requirement, I independently architected and implemented the core distributed bill-run engine. My implementation covered the mapper, reducer, finalizer, and reporting phases, with work partitioned for parallel execution.

THE CENTRAL QUESTIONHow could I preserve the selected billing semantics across concurrent work and know when every required contribution to a report was complete?

Partitioning the work also meant coordinating when stages could advance. I had to account for checkpointing and recovery from interrupted execution, so that partial progress would not be mistaken for a completed report. Completion needed to be a shared decision that remained meaningful when individual workers stopped or restarted.

02 / A SYSTEM MODEL

Separate the data being processed from the decisions about progress.

I used Cassandra-backed distributed coordination to synchronize execution stages durably. A completion signal held only in one process could not serve as a reliable shared source of truth when independent workers could be interrupted or restarted. Durable coordination let progress and stage-completion decisions survive those interruptions, at the cost of additional state management and coordination overhead.

The conceptual model below separates the data plane, where events are processed and contributions combined, from the control plane, where ownership and stage progression are coordinated. It compresses finalization and reporting into the final output step; it is not an exact four-phase production topology.

FIG. 01 / SIMPLIFIED EXECUTION MODELCONCEPTUAL, NOT PRODUCTION TOPOLOGY
DATA PLANE · process contributions
INPUTEventsSource records
FAN INReduceCombine contributions
PUBLISHFinalizeComplete report
CONTROL PLANECoordinatorTask claims · expiry · stage barrier · retry decisions
DURABLE STATEPersistent coordination recordsOwnership and completion survive worker interruption
This diagram illustrates responsibilities and correctness boundaries. It omits internal service names, storage schemas, and deployment details.

The completion requirement carries through each stage: the existence of intermediate output cannot be the sole proof of stage completion. In this model, a reducer needs to know which contributions were expected and which were committed before treating its input as complete. Durable progress alone does not settle whether an overlapping or retried attempt may publish a result. That ownership question is the next part of the design discussion.

03 / OWNERSHIP AND COMPLETION

A lease is not a guarantee that work ran only once.

A lease is a time-bounded claim coordinating which worker should attempt a task. It does not stop a paused worker from resuming after its claim expires. Each execution is a physical attempt; several attempts can belong to the same logical contribution in a particular bill run. Repeating computation need not mean applying that contribution again.

The critical question is where a result becomes authoritative for downstream processing. Checking ownership and publishing later leaves a gap in which authority can change. In the conceptual sequence below, A pauses after checking its claim. B takes over and attempts publication before A resumes. A's earlier check cannot authorize its later write.

ILLUSTRATIVE TIMELINENOT AN INCIDENT REPORT
T0Worker A checks its claim

A's claim is valid at the check, before publication.

T1Worker A pauses

A pauses before writing; its claim expires during the pause.

T2Worker B reclaims task

B takes over, completes the work, and attempts to publish its result.

T3Worker A resumes

A attempts to publish using its earlier check, after B has taken over.

DESIGN INVARIANT

At the point a result becomes authoritative, stale attempts must be rejected and the same logical contribution must not be applied twice within its bill run. A successful write alone does not establish that these conditions hold.

These protections have different jobs. A guarded transition or version/fencing check can reject obsolete authority when enforced where the effect is accepted. Guarding a coordination record does not automatically protect a write to another resource. Stable identity associates retries with one logical contribution; idempotent acceptance keeps that contribution from being applied again. Identity alone does not enforce this. Durable completion evidence serves a separate purpose: establishing that every expected contribution is validly complete before advancing a stage. These are conceptual design options, not claims that my implementation used each mechanism.

04 / FAILURE WALKTHROUGHS

Four scenarios worth designing before the happy path is done.

These hypothetical scenarios pair a failure with an acceptance rule and a proposed test. Bounded retries and concurrency can limit retry pressure, but cannot guarantee success. Recovery depends on which failures persist, remaining retry budget, dependency availability, and workers being able to make progress.

01Worker loses ownership
TRIGGER

Worker A checks its claim, then pauses before publishing long enough for the claim to expire.

WHAT CAN GO WRONG

Worker B takes over, but A later uses its earlier check to publish a stale result.

DESIGN RESPONSE

Reject obsolete authority where a result becomes authoritative. A separate ownership check before the write leaves a race; a lease alone cannot protect that boundary.

HOW TO TEST THE IDEA

Pause A after its ownership check. Let its claim expire, let B take over and publish, then resume A. Assert that A cannot become authoritative and the logical contribution is not applied twice.

02Write succeeds; acknowledgement fails
TRIGGER

A result write succeeds, but its acknowledgement is lost. The worker cannot tell whether the write took effect.

WHAT CAN GO WRONG

Treating the missing acknowledgement or timeout as proof of failure can lead a retry to apply the contribution again.

DESIGN RESPONSE

Keep the logical contribution identity stable within its bill run and make acceptance idempotent. Recording acceptance and applying the contribution must agree even after interruption; an identity alone does not ensure this.

HOW TO TEST THE IDEA

Persist a contribution, suppress its acknowledgement, then retry it with the same logical identity. Allow repeated computation, but assert one applied contribution and an unchanged downstream result against a known-correct baseline.

03A stage partially completes
TRIGGER

Some expected contributions finish while another remains missing; a completed contribution may also be retried.

WHAT CAN GO WRONG

An output count can appear complete despite a missing contribution if duplicates are counted as distinct work.

DESIGN RESPONSE

Define the expected contributions for the run. Gate stage completion on valid, durable completion evidence for each one, with no missing contribution or duplicate logical application. Partial output alone is insufficient.

HOW TO TEST THE IDEA

Withhold one required contribution and replay a completed one. Assert that downstream finalization remains blocked. Then complete the missing contribution and verify that valid completion evidence allows the stage to advance without double application.

04Retries compete with useful work
TRIGGER

A shared dependency slows down or remains unavailable while multiple workers retry.

WHAT CAN GO WRONG

Retries consume capacity needed for useful work; independent per-worker limits can still overload the shared dependency.

DESIGN RESPONSE

Bound attempts per logical task and limit aggregate in-flight work against the shared dependency. Define retry spacing and what happens when the budget is exhausted; persistent failure must remain visible rather than appearing complete.

HOW TO TEST THE IDEA

Inject transient and persistent failures. Observe retry rate, aggregate concurrency, useful throughput, and backlog. Verify the configured limits and budget-exhaustion behavior; check recovery only when dependencies recover and retry budget remains.

These examples illustrate possible failure modes; they are not claims about specific bugs or incidents at Thales.

05 / TRADEOFFS

Each reliability decision has a cost.

For the core bill-run engine, I chose Cassandra-backed shared coordination over relying exclusively on completion signals held in worker processes. Independent workers could be interrupted or restarted, so stage progression needed durable evidence that remained available beyond any one worker's lifetime. That choice made persistent coordination state part of my implementation responsibility.

Swipe horizontally to read all table columns →

DecisionWhy it helpsWhat it costs
Parallelize mappingPartitioning offers concurrency; the boundaries still have to preserve the selected billing semantics.More partitions can add scheduling, synchronization, and aggregation work. Extra workers alone do not establish an end-to-end improvement.
Make phase boundaries durableShared completion evidence remains available across worker interruption or restart.Persistent coordination adds operations and state to manage, with potential contention and more complex transitions.
Recover abandoned workReclaiming abandoned work gives an interrupted run a path to continue when recovery conditions allow.Lease expiry can overlap attempts; recovery must account for stale authority and duplicate logical contributions.
Gate report finalizationA completion barrier keeps finalization dependent on valid evidence for all expected contributions.Missing contributions can delay the report even when most work has finished. Partial progress must remain distinguishable from completion.

The table describes the competing considerations around that choice. I accepted additional coordination operations, persistent state management, and more complex transition and recovery reasoning. Shared state can also introduce contention. Their performance impact requires measurement under a defined workload.

06 / VALIDATION

The measurement is only useful if you can explain it.

My project work included the core engine implementation and laboratory performance evaluation. The criteria below explain what would substantiate a result; they are not a record that every check was executed. The preceding fault-injection scenarios remain proposed tests. Reported measurements still require review of their supporting evidence.

01 / CORRECTNESSValidate the billing result

Check charges against independently established expected results for the chosen pricing and slicing rules. Verify complete contributions without double application; producing an output alone is insufficient.

02 / RECOVERYDemonstrate recoverable completion

Under stated failure and retry assumptions, verify that interrupted work can reach a valid completed report. Distinguish bounded retries that make progress from repeated attempts that never complete.

03 / PERFORMANCEDefine the measurement

Specify workload and distribution, reporting scope, environment, concurrency, and timing start/stop boundaries. State exclusions so the elapsed time describes a defined amount of completed work.

04 / BOTTLENECKSLocate the limiting stage

Measure stage durations, coordination activity, and waiting for contributions alongside end-to-end time. A total duration alone cannot identify which stage or delay dominates the run.

BENCHMARK DISCLOSURE

Public figures are intentionally withheld here until confirmed against test evidence and approved for publication. This avoids turning an internal lab result into an unsupported production-scale claim.

07 / WHAT I TOOK AWAY

I had to account for the whole run.

Starting from a pricing requirement, I had to decide how the processing phases fit together and what completion meant when the workers could change.

Making coordination durable put state management and recovery reasoning inside my core-engine responsibility. Concurrency required me to think about the validity of the whole report as well as the execution of each worker.

The lesson I take forward is to make completion conditions and recovery assumptions explainable, then distinguish what the design intends from what verification actually demonstrates.

THANKS FOR READINGHave an interesting systems problem?
Let's talk