The Cheap Rate Window - Solution

Voltaic had enough charge-authz-service capacity by 21:01:30, but the fleet was still stuck two hours later. The outage survived because firmware 4.3.0 turned the original failure into a synchronized retry storm.

1. Notice The Autoscaler Is No Longer The Incident

Start with the timeline and traffic samples:

SELECT sampled_at,
       requests_per_min,
       distinct_devices_calling,
       success_rate_pct,
       instance_count
FROM authz_traffic_samples
ORDER BY sampled_at;

The autoscaler was late to the 21:00 spike, but it had caught up by 21:01:30. After that, capacity rose from 4 to 12 to 36 instances while success stayed around 0.9%.

That means scaling is not the lever. The platform is not short on app servers.

2. Prove The Load Is Self-Generated

Compare requests_per_min with distinct_devices_calling.

Before the window, callers produce about one request per minute. From 21:02 onward, the fleet produces exactly six requests per calling device per minute.

That matches the 4.3.0 firmware behavior:

SELECT firmware_version,
       devices_plugged_in,
       devices_still_awaiting,
       retry_attempts_per_cycle,
       cycle_reentry_seconds,
       honors_retry_after,
       jitter_enabled
FROM wallbox_firmware;

Firmware 4.3.0 retries 3 times, then re-enters the cycle every 30 seconds with no jitter. That is:

3 attempts / 0.5 minutes = 6 requests per minute

Nobody is plugging in new cars at 23:00. The failed devices are creating the traffic themselves.

3. Shed With Backoff, Not Just Shed

The fix has to reduce inbound request rate. More instances, a larger pool, or autoscaler tuning will not converge because failures generate more retries.

The admission mode must do two things:

Shed excess traffic.
Emit Retry-After so firmware 4.3.0 leaves its fixed 30-second loop.

drop_excess is not enough because it returns a bare rejection. Keep retry_after_jitter_enabled = 1; otherwise the fleet returns in phase again.

Valid modes include:

shed_with_backoff
rate_limit_with_backoff
token_bucket_with_retry_after
admission_queue_with_backoff

4. Compute The Minimum Safe Rate

The backlog and deadline live in the charge window:

SELECT devices_awaiting_authorization,
       snapshot_at,
       closes_at
FROM charge_window;

At 23:00, 360,000 wallboxes still need authorization. The cheap rate closes at 05:00, which leaves 360 minutes.

360,000 wallboxes / 360 minutes = 1,000 authorizations per minute

The rate must be at least 1000/min or the backlog will not clear before the tariff closes.

5. Compute The Grid Ceiling

The runbook trap is to use the protected service capacity, 9000/min. That would make software dashboards green and trip grid protection.

You need the highest evidenced-safe grid ramp:

SELECT occurred_on,
       peak_30s_demand_rise_mw,
       dno_notice_ref
FROM grid_ramp_events;

10.4 MW had no DNO notice. 11.8 MW did. Use 10.4 MW as the highest evidenced-safe ramp.

Then compute the real average draw per waiting wallbox:

SELECT model,
       rated_kw,
       devices_still_awaiting
FROM wallbox_models;

The docs talk about 7.4 kW wallboxes, but the waiting fleet also includes 36,000 three-phase VB-22 units at 22 kW:

(324,000 * 7.4 + 36,000 * 22.0) / 360,000 = 8.86 kW

Now convert the grid ramp into authorizations per minute:

10,400 kW / 8.86 kW = 1,173 authorizations per minute

So the valid rate is:

1000 <= admission_rate_limit_per_min <= 1173

6. Apply The Fix

Update only the production charge-authz-service runtime row:

UPDATE charge_authz_runtime_config
SET admission_control_mode = 'shed_with_backoff',
    admission_rate_limit_per_min = 1100,
    retry_after_jitter_enabled = 1,
    updated_by = 'incident-response'
WHERE environment = 'production'
  AND service = 'charge-authz-service';

Any rate from 1000 through 1173 passes. 1100 is a comfortable middle value.

Then run Deploy.

7. What Deploy Checks

The validation requires all of these to be true:

retry_after_jitter_enabled stays 1.
admission_control_mode sheds and emits Retry-After.
The rate is at least 1000/min so the backlog clears by 05:00.
The rate is at most 1173/min so the grid ramp stays below the highest evidenced-safe level.

8. Root Cause

The root cause was a metastable retry storm. The 21:00 autoscaler delay created the first authorization failures, but firmware 4.3.0 turned those failures into a synchronized, endless retry loop: three attempts every 30 seconds with no jitter.

The durable fix is not more service capacity. It is admission control that sheds over-limit requests with jittered Retry-After, at a rate high enough to clear the cheap-rate backlog and low enough to avoid a grid ramp incident.