The JSON That Isn't Equal to Itself - Solution

1. Review the Incident Brief

Ledgerly webhooks are failing signature checks in production after the shared verifier rollout. The dashboard shows the same JSON fields and values, staging accepts the replay, and production rejects it.

The trap is that JSON equality in a parsed viewer is not the same thing as byte equality for an HMAC signature. Ledgerly signs the HTTP entity body it sends, so the verifier must use those exact bytes.

2. Inspect the Capture Evidence

Open the SQL console and start with the webhook capture table:

SELECT *
FROM signature_captures
ORDER BY capture_id;

The important pattern is:

visible_json_equal = 1
raw_sha256 differs from reserialized_sha256
production rows end in signature_mismatch

That means the JSON object still looks equivalent after middleware, but the byte stream used for signing has changed.

3. Check the Verifier Logs

Query the validation events around the incident:

SELECT event_time, service, capture_id, level, message
FROM signature_validation_events
ORDER BY event_time;

The verifier log for production reports an HMAC mismatch where the raw body hash and verifier input hash differ. Another ingress event says the production Ledgerly route is missing a raw body handle and is falling back to parser output.

4. Rule Out Key Rotation

Check Ledgerly's active key IDs:

SELECT *
FROM partner_key_versions
WHERE partner = 'ledgerly'
ORDER BY environment;

Staging and production use the same active key ID, and the last rotation predates the incident. Since staging can verify the same logical replay, key rotation is not the root cause.

5. Review the Middleware Rollout

Look at the middleware changes:

SELECT *
FROM middleware_rollouts
ORDER BY enabled_at;

Production webhook-ingress enabled json-normalizer-v2 at the start of the incident window. Staging has raw-body-tee, which preserves an entity-body handle before the JSON parser. The audit replay canonicalizer is a separate archive path and is not the online verifier.

6. Compare Runtime Config

Query the mutable runtime configuration:

SELECT *
FROM service_runtime_config
ORDER BY environment, service;

The staging online path is the known-good reference:

staging webhook-ingress      body_capture_mode=tee_before_parser preserve_raw_body=1
staging signature-verifier   signature_input=ingress.raw_body canonicalization_policy=none preserve_raw_body=1

The production online path is broken:

production webhook-ingress     body_capture_mode=parser_output preserve_raw_body=0
production signature-verifier  signature_input=parsed_json canonicalization_policy=parse_and_stringify preserve_raw_body=0

Production parses and reserializes the payload before verification. That changes harmless-looking details such as key order, whitespace, Unicode escaping, slash escaping, and float formatting, which changes the HMAC input.

7. Understand the Root Cause

The root cause is not that Ledgerly sent invalid JSON, and it is not that the displayed JSON values changed. The production verifier stopped signing the exact request body that Ledgerly signed.

The correct remediation needs both halves of the online path:

webhook-ingress must preserve the raw entity body before parser middleware.
signature-verifier must compute the HMAC from that preserved raw body without canonicalizing or reserializing it.

Signature verification must stay enabled, and mismatches must still be rejected.

8. Apply the Fix

Update only the production online rows:

UPDATE service_runtime_config
SET body_capture_mode = 'tee_before_parser',
    preserve_raw_body = 1,
    updated_by = 'player'
WHERE environment = 'production'
  AND service = 'webhook-ingress';

UPDATE service_runtime_config
SET signature_input = 'ingress.raw_body',
    canonicalization_policy = 'none',
    preserve_raw_body = 1,
    updated_by = 'player'
WHERE environment = 'production'
  AND service = 'signature-verifier';

Do not update audit-replay. Do not disable verify_signature_enabled. Do not set reject_on_mismatch = 0. Those changes would hide the incident instead of fixing the verifier contract.

9. Verify the Final State

Run:

SELECT environment,
       service,
       body_capture_mode,
       signature_input,
       canonicalization_policy,
       preserve_raw_body,
       verify_signature_enabled,
       reject_on_mismatch
FROM service_runtime_config
ORDER BY environment, service;

The production online rows should now show:

production webhook-ingress      tee_before_parser  n/a               n/a   1  1  1
production signature-verifier   n/a                ingress.raw_body  none  1  1  1

10. Deploy

Click Deploy. The validation should pass because production now verifies the exact received byte stream for cases where the parsed JSON still looks equivalent but the bytes differ.

The root cause was a shared middleware rollout that made production verify parser output instead of Ledgerly's signed request body. The remediation is to preserve the pre-parser body at ingress and verify against that raw body without canonicalization.