← All solutions

The Silent Stockout - Solution

1. Review the Incident Brief

Orders placed during the overnight window are being created normally, but they are not being fulfilled. No services are showing errors, and all dashboards look healthy by morning.

2. Open the ShelfLife Terminal

Navigate to the ShelfLife CLI terminal. This challenge is solved by investigating orders, logs, cache state, scheduler jobs, and then recreating the missing job.

3. Query Skipped Orders

Start by checking skipped orders during the incident window:

shelflife orders --status skipped --since 02:00 --until 04:00

The skipped orders all point to reason=insufficient_stock. The important detail is that the orders are skipped, not failing with an exception.

4. Check Fulfillment Logs

Inspect the service that decides whether orders can be fulfilled:

shelflife logs fulfillment-engine --since 02:00 --until 04:00

The logs show fulfillment decisions based on cached_qty=0. The fulfillment engine believes inventory is unavailable because it is reading zero-quantity cache entries.

5. Compare Cache Against Warehouse Data

Check whether the cache agrees with the source of truth:

shelflife cache compare

The output shows mismatches: inventory-cache reports zero or stale quantities while warehouse-db still has stock. The warehouse data is healthy; the cache is wrong.

6. Inspect Cache Details

Look at the cache snapshot directly:

shelflife cache inspect

The cache entries show expired or stale data. This explains why fulfillment silently skips valid orders overnight.

7. Review the Cache Documentation

Open the Cache Layer documentation.

Key insight: inventory-cache is updated by write-through stock mutations and has a 1-hour TTL. That works while warehouse writes are happening, but it needs another refresh path when write traffic stops.

8. Check Warehouse Write Traffic

Inspect warehouse-api logs around the overnight period:

shelflife logs warehouse-api --since 00:00 --until 06:00

Write traffic drops off around 01:00 and resumes closer to morning. With a 1-hour TTL, cache entries can expire around 02:00, exactly when orders start getting skipped.

9. List Scheduled Jobs

Check whether a scheduled refresh job exists:

shelflife scheduler list

Only order-cleanup, analytics-rollup, and health-check-sweep are active. There is no inventory cache refresh job.

10. Review Recent Deploy History

Look at recent scheduler deploys:

shelflife deploys --service scheduler-service

The relevant change is a cleanup that removed unused scheduled tasks about 2 weeks ago, matching when the overnight stockout symptoms began.

11. Find the Removed Job Configuration

Open the linked GitHub PR for the cleanup and inspect the diff for infra/scheduler/jobs.yaml.

The removed job is inventory-sync with these settings:

name: inventory-sync
schedule: */30 * * * *
source: warehouse-db
target: inventory-cache
type: full-refresh

The full-refresh type matters because there are no overnight stock mutations to drive an incremental refresh.

12. Recreate the Missing Scheduler Job

Return to the terminal and recreate the removed job:

shelflife scheduler create --name inventory-sync --schedule "*/30 * * * *" --source warehouse-db --target inventory-cache --type full-refresh

13. Verify the Fix

The command should report that the scheduled job was created and that an initial sync refreshed the cache.

A correct response includes:

14. Challenge Completed

The validation system confirms that the missing overnight inventory refresh job has been restored.

The root cause was not Redis, the fulfillment engine, or warehouse stock. The cache-refresh backstop was removed, so overnight cache entries expired when write traffic stopped.