← All solutions

The Tamagotchi That Dies Overnight - Solution

Pets were fed before bedtime, but production later recorded them dead from starvation. The feed event and the death event were both real. The bug was in how the worker replayed delayed hunger jobs after a dependency change and restart window.

1. Start With The Contradiction

The incident brief gives the key fact pattern:

• Patchy was fed at 22:41.
• Patchy died at 03:12.
• The audit log accepts the feed.
• The worker log records the starvation.

Do not assume one log is lying. The challenge is about explaining how both events can be valid while the system state is still wrong.

2. Compare Staging And Production

Staging runs the same application code and dependency versions, but it does not reproduce the failure. The important difference is the production deploy window:

sleepy-queue 2.3.1 -> 2.4.0
worker stopped during deploy
worker restart loop lasted for hours
worker recovered delayed jobs at 03:12

That points away from UI display, food nutrition, and timezone math. The suspicious path is delayed job recovery after a long worker downtime.

3. Inspect The Queue Dependency Change

Open the sleepy-queue changelog. Version 2.4.0 changed delayed-job restart recovery:

• old behavior: coalesce missed delayed jobs by dedupe key
• new behavior: backfill each missed delayed job, oldest first

This does not kill pets by itself. It makes stale delayed ticks replayable if the application treats the tick payload as source-of-truth state.

4. Compare Queue Snapshots

Look at the queue snapshots before and after deploy. You should notice two hunger job key shapes:

hunger:<petId>:<when>
scheduler:hunger:<petId>:<when>

The scheduler now namespaces repeatable hunger jobs under scheduler:hunger:..., but legacy jobs can still exist with the old hunger:... prefix.

5. Inspect Feed Cancellation

The feed path increments the pet's careEpoch, updates live hunger, then tries to cancel pending hunger ticks. The cancellation prefix only matches the new scheduler namespace:

scheduler:hunger:<petId>:

So when Patchy is fed, the new-format job is cancelled, but an older legacy-format hunger tick remains queued. Production says removed=1, which is true, but it is not the whole queue.

6. Inspect The Worker

The worker's mistake is in apps/worker/job-handlers.js: it advances hunger from the delayed job payload snapshot, not from the current pet.

The stale replayed job still carries the state from before the feed:

payload.hungerAtScheduleTime = 83
payload.careEpoch = 17

But the live pet was already cared for:

pet.hunger = 12
pet.careEpoch = 18

When the worker finally comes back at 03:12, sleepy-queue backfills the uncancelled legacy tick. The worker trusts the stale payload, decays from hunger 83 to 100, drains health to 0, and records starvation over the newer fed state.

7. Rule Out The Red Herrings

Timezone and DST are not the cause; the timestamps parse correctly as UTC instants.

moonberries are lower nutrition, but feeding still leaves Patchy at hunger 12, nowhere near starvation.

The UI mood cache is cosmetic. It does not write the authoritative pet health state.

The OOM restart loop matters only because it creates the delayed-job recovery window. The memory bump fixes rollout health, not the pet-state bug.

8. Apply The Fix

The worker must treat delayed hunger ticks as replayable commands, not as authoritative state. Fix apps/worker/job-handlers.js so it skips stale ticks and decays from live pet state.

Reference shape:

if (payload.careEpoch != null && payload.careEpoch < pet.careEpoch) {
  log.info('hunger.tick.skipped', {
    pet_id: petId,
    reason: 'stale',
    job_epoch: payload.careEpoch,
    pet_epoch: pet.careEpoch,
  });
  return;
}

const reference = pet.lastFedAt || job.scheduledFor;
const hours = Math.max(0, petClock.hoursBetween(reference, firedAtIso));
const result = sim.decay(pet.hunger, pet.health, hours);

You may also widen cancellation to cover the legacy key prefix, but that is not sufficient as the primary fix. Correctness cannot depend on best-effort cancellation.

9. Verify

Run the project checks from the terminal:

npm test
npm run incident:replay
npm run verify

The expected result is that the stale legacy hunger tick is logged and skipped, the incident replay exits successfully, and unattended pets still starve normally.

10. Root Cause

The root cause was a contract violation between delayed jobs and domain state. After sleepy-queue changed restart recovery to backfill every missed delayed job, production replayed an uncancelled legacy hunger tick. The worker treated that tick's stale payload as authoritative pet state instead of comparing it against the live pet's newer careEpoch and current hunger.

The durable fix is worker-side idempotency: skip stale ticks and compute from live state.