← All solutions

Final Boss - Solution

1. Review the Incident Brief

Final Boss reaches 0 HP in the fight UI, staging records the replay as a win, but production sometimes rejects the same ranked replay. The important clue is that production is now split across regional shards, and those shards do not all fail the same way.

This is not a frontend animation bug. The K.O. animation shows what the client believes happened, but ranked results are decided by the server-side result verifier.

2. Read the Runtime Config Docs

Open the Runtime Config documentation and note the key warning: production is not a single row. Each production region has its own runtime row and its own finalization policy.

The fix has to make every ranked production region deterministic while leaving staging and the internal canary alone.

3. Enumerate the Environments

Use the SQL console to inspect the runtime configuration:

SELECT *
FROM combat_runtime_config;

The important rows are:

environment   combat_mode   combat_workers   result_verify_delay_ms   leaderboard_ranked_mode
staging       synchronous   1                250                      0
prod-emea     async         2                40                       1
prod-amer     async         6                0                        1
prod-apac     async         4                0                        1
prod-canary   async         3                0                        0

Only prod-emea, prod-amer, and prod-apac are ranked production regions. prod-canary is not ranked and should not be promoted.

4. Inspect the Current Finalization Policy

Query the verifier policy table:

SELECT *
FROM result_finalization_policy;

The production ranked regions are using:

verifier_source = match_store
finalization_strategy = read_after_enqueue
commit_barrier_workers = 0
client_authoritative = 0

That means the verifier reads match-store state immediately after enqueueing the final hit. In async production, that read is not ordered behind the worker commit that writes canonical boss HP.

5. Prove the Race from Replay Evidence

Compare verifier reads against combat-worker commits:

SELECT environment, service, event_time_ms, message, canonical_boss_hp, victory
FROM combat_replay_events
ORDER BY environment, event_time_ms;

prod-amer and prod-apac show the failure clearly: the result verifier reads canonicalBossHp=12 and writes victory=false before the combat worker commits canonicalBossHp=0.

prod-emea looks clean in the captured replay because its worker commit happened before the verifier read. That does not make it safe. It is still ranked async production with the same stale-read policy and no commit barrier, so it must be fixed too.

6. Apply the Deterministic Fix

Update every ranked production region so the verifier stops using the stale match-store read and waits for exactly that region's worker commits before finalizing.

One accepted fix is:

UPDATE result_finalization_policy
SET verifier_source = 'committed_events',
    finalization_strategy = 'event_stream_finalization',
    commit_barrier_workers = 2,
    updated_by = 'player'
WHERE environment = 'prod-emea';

UPDATE result_finalization_policy
SET verifier_source = 'committed_events',
    finalization_strategy = 'event_stream_finalization',
    commit_barrier_workers = 6,
    updated_by = 'player'
WHERE environment = 'prod-amer';

UPDATE result_finalization_policy
SET verifier_source = 'committed_events',
    finalization_strategy = 'event_stream_finalization',
    commit_barrier_workers = 4,
    updated_by = 'player'
WHERE environment = 'prod-apac';

Do not update staging. Do not update prod-canary. Do not set client_authoritative = 1. Do not make production synchronous. Do not rely on result_verify_delay_ms; a delay is not an ordering guarantee.

7. Verify the Final State

Run:

SELECT environment, verifier_source, finalization_strategy, commit_barrier_workers, client_authoritative
FROM result_finalization_policy
ORDER BY environment;

The ranked production rows should now be:

prod-emea  committed_events  event_stream_finalization  2  0
prod-amer  committed_events  event_stream_finalization  6  0
prod-apac  committed_events  event_stream_finalization  4  0

The exact barrier values matter. They must match each region's combat_workers value from combat_runtime_config.

8. Deploy

Click Deploy. The validation should pass because:

The root cause is async production finalization reading stale canonical state before the final damage commit was ordered. The remediation is per-region committed-event finalization with a commit barrier equal to the ranked region's worker count.