← All solutions

The Goal That Stuck - Solution

KickCast's scoreboard kept counting a goal that VAR had already disallowed. The infrastructure was healthy and staging used the same config, but production processed two related event streams out of causal order.

1. Start With The Mismatch

The incident says FC Stacktrace's 87th-minute goal was ruled out for offside, but KickCast still showed:

FC Stacktrace 2 - 1 Real Uptime

The correct score is:

FC Stacktrace 1 - 1 Real Uptime

There are no crash loops, no 500s, and no obvious config drift between staging and production. That points away from the broadcast graphics surface and toward the score ledger pipeline.

2. Check The Ledger

Query score_ledger.

Production shows the bad score, while staging replay shows the corrected score. That matters because the graphics service only mirrors the ledger. Editing broadcast_runtime_config or the visible graphics layer will not fix the source of truth.

3. Inspect The 87th-Minute Events

Query match_event_log around minute 87 and compare the goal event with the VAR overturn:

evt_goal_0087
evt_var_0091

The VAR event references the goal:

ref_event_id = evt_goal_0087

But the log note says the overturn was discarded because the referenced goal was not in the ledger yet.

4. Compare Consume Times

The important timestamps are:

evt_var_0091  consumed_at = 21:48:30.520
evt_goal_0087 consumed_at = 21:48:35.300

The overturn was processed 4780ms before the goal it was supposed to cancel arrived.

Production was using:

var_overturn_ordering_mode = fire_and_forget

So scoreboard-service looked for evt_goal_0087, did not find it, discarded the overturn, and later counted the delayed goal when it finally arrived.

5. Explain Why Production Raced

Query event_bus_metrics.

During the Final, match-events-topic lagged heavily because it shared a partition with high-volume ball-tracking and telemetry events. var-decisions-topic stayed fast and low-volume.

That means VAR decisions could arrive at scoreboard-service before the match event they corrected.

6. Check The Validation Cases

Do not tune only for the 4780ms headline incident. deploy_validation_cases includes a worse replay gap:

6150ms

The timeout must cover that worst case while staying under the broadcast delay budget:

6150ms <= timeout <= 8000ms

7. Apply The Fix

Update the production scoreboard-service runtime config:

UPDATE scoreboard_runtime_config
SET var_overturn_ordering_mode = 'causal_wait',
    var_overturn_wait_timeout_ms = 6500,
    updated_by = 'player'
WHERE environment = 'production'
  AND service = 'scoreboard-service';

Other accepted causal-wait modes are:

buffer_and_retry
await_source_event
ordered_replay

Keep this guard enabled:

require_ref_event_match = 1

Disabling exact reference matching is not a fix. It can cancel the wrong event the next time two plays happen close together.

8. Deploy And Verify

Run Deploy. The backend replays all validation cases against the new config.

Expected result:

• The 87th-minute goal is overturned correctly.
• The 62nd-minute card correction still works.
• The 6150ms stress case passes.
• The timeout stays within the 8000ms broadcast delay budget.

9. Root Cause

The root cause was a causal ordering bug between two event streams. scoreboard-service consumed a VAR overturn before it consumed the goal event the overturn referenced, then discarded the correction because production used fire_and_forget handling. When the delayed goal finally arrived, there was no pending overturn left, so the disallowed goal stayed in the ledger.

The durable fix is to make VAR overturn handling wait briefly for the referenced source event while preserving exact ref_event_id matching.