← All solutions

The Sold-Out Concert With 20,000 Empty Seats - Solution

Fans could see paid tickets in the app, and their QR signatures were valid, but gate scanners rejected them with Ticket holder does not match ticket owner. The failure came from QR regeneration racing ahead of authoritative ticket ownership transfer.

1. Start With A Rejected Scan

Use the gate scan evidence and focus on one affected scan, such as:

scan_000771

The important detail is that the QR signature is valid. This is not a cryptographic failure and not a missing ticket.

The scanner rejects the ticket because the signed QR owner does not match the current ticket owner.

2. Compare Ticket Owner And QR Owner

Query the ticket and QR payload for the affected ticket:

SELECT ticket_id,
       current_owner_identity_id,
       owner_updated_at
FROM tickets
WHERE ticket_id = 'tkt_A17_044_12';

SELECT qr_id,
       ticket_id,
       signed_owner_identity_id,
       issued_at,
       signature_valid
FROM qr_payloads
WHERE ticket_id = 'tkt_A17_044_12';

The mismatch is the clue:

tickets.current_owner_identity_id      = registered identity
qr_payloads.signed_owner_identity_id   = guest identity
signature_valid                        = 1

The QR is valid for the wrong owner.

3. Follow The Guest-To-Account Merge

Trace the order and identity lineage:

SELECT order_id,
       ticket_id,
       checkout_identity_id,
       checkout_mode,
       payment_status,
       purchased_at
FROM checkout_orders
WHERE ticket_id = 'tkt_A17_044_12';

SELECT identity_id,
       identity_type,
       merged_into_identity_id,
       merged_at
FROM customer_identities
ORDER BY created_at;

Affected users bought as guests, then created or linked accounts. The ticket correctly ended up owned by the registered identity, but the latest QR was minted against the earlier guest identity.

4. Compare Event Timing

Now compare when the QR service saw the identity merge with when ticket ownership became authoritative:

SELECT merge_event_id,
       guest_identity_id,
       registered_identity_id,
       qr_service_consumed_at
FROM identity_merge_events
ORDER BY qr_service_consumed_at;

SELECT owner_event_id,
       ticket_id,
       new_owner_identity_id,
       published_at
FROM ticket_owner_events
ORDER BY published_at;

The broken sequence is:

customer.merged consumed by qr-issuing-service
QR regenerated immediately
ticket.owner_updated published later

So the QR worker did exactly what it was configured to do, but it read the owner projection before the transfer was ready.

5. Tie It To The Fastlane Deployment

Check the deploy history:

SELECT deployed_at,
       version,
       author,
       summary
FROM deploy_change_log
ORDER BY deployed_at;

The relevant deploy moved reunion-concert QR regeneration into a fastlane and removed the fixed delay that had previously hidden the race. That made wallet refresh faster under normal traffic, but during the guest-to-account spike it allowed stale owner reads.

6. Rule Out The Tempting Wrong Fixes

Do not weaken gate validation.

These are not fixes:

• Setting require_gate_owner_match = 0
• Accepting any valid QR signature for a ticket
• Reissuing from the guest identity because checkout began as guest
• Increasing the wait without changing the ownership barrier

The gate scanner is right to require the QR owner to match the current ticket owner.

7. Check The Runtime Config

Inspect production admission runtime config:

SELECT environment,
       service,
       qr_regeneration_trigger,
       owner_read_policy,
       ownership_ready_barrier,
       max_ready_wait_ms,
       require_gate_owner_match
FROM admission_runtime_config
ORDER BY environment, service;

Production qr-issuing-service is the row to fix. It needs to wait for authoritative ticket ownership and sign QRs for the authoritative owner.

8. Choose A Bounded Wait

Use the validation cases and admission refresh budget:

SELECT case_id,
       required_owner_ready_wait_ms,
       must_use_authoritative_owner
FROM deploy_validation_cases
ORDER BY required_owner_ready_wait_ms DESC;

The wait must be at least the slowest required owner-transfer replay and still inside the wallet refresh budget:

4200ms <= max_ready_wait_ms <= 5000ms

9. Apply The Fix

Update only the production QR issuing runtime config:

UPDATE admission_runtime_config
SET owner_read_policy = 'authoritative_ticket_owner',
    ownership_ready_barrier = 'owner_updated_event',
    max_ready_wait_ms = 4200,
    updated_by = 'incident-response'
WHERE environment = 'production'
  AND service = 'qr-issuing-service';

Any wait from 4200 through 5000 is valid. Keep gate ownership matching enabled.

10. Deploy And Verify

Run Deploy. The replay should pass:

• A17 row 44 guest-to-account tickets are reissued after ticket.owner_updated.
• VIP late-merge cases wait for authoritative owner transfer without timing out.
• Logged-in buyer paths still issue immediately.
• Duplicate merge retries do not mint stale QRs.
• Gate owner matching remains enforced.

11. Root Cause

The root cause was a causal ordering bug in QR regeneration. The fastlane deployment made qr-issuing-service regenerate wallet QRs as soon as it consumed customer.merged, but the ticket owner projection could lag behind that event. During the Blackout Reunion spike, production minted valid QRs for pre-merge guest identities before the authoritative registered owner was visible.

The durable fix is to regenerate QR payloads from the authoritative ticket owner after the owner update barrier is satisfied, while keeping the gate validator's owner match requirement intact.