Skip to main content
Magistry
Start free
System status
Degraded performance 2026-05-19, 08:02–08:11 UTC 9 minutes

Catalog cycle ran twice on a single store

Resolved — postmortem published

An advisory lock was released a beat early during a deploy; the catalog cycle ran twice on one workspace. The second cycle produced no decisions (idempotency caught it), but the dry-run preview UI flickered. Lock TTL increased from 600s to 900s.

Timeline

  1. 08:02 UTCA deploy restarts the decision worker mid-cycle on one workspace.
  2. 08:03 UTCAn advisory lock guarding the catalog cycle is released a beat early as the process cycles, allowing a second cycle to start.
  3. 08:05 UTCIdempotency guard catches the duplicate: the second cycle produces zero decisions because the subjects already have matching rows.
  4. 08:08 UTCOperators on that workspace see the dry-run preview UI flicker as the duplicate cycle's no-op output renders and clears.
  5. 08:11 UTCCycle completes; advisory lock TTL raised from 600 s to 900 s. Incident resolved.

Root cause

A deploy restarted the worker mid-cycle. The advisory lock TTL (600 s) expired slightly before the cycle's own completion under the deploy's timing, so a second cycle acquired the lock and ran. Idempotency prevented any duplicate writes, but the redundant cycle was visible in the preview UI.

Impact

No incorrect or duplicate decisions were written — idempotency caught the second cycle, which produced zero decisions. The only visible effect was a brief flicker in the dry-run preview UI on one workspace.

Resolution

The duplicate cycle was a confirmed no-op. The advisory lock TTL was increased from 600 s to 900 s to cover worst-case cycle duration during a deploy, eliminating the early-release window.

What we changed

  • Advisory lock TTL increased from 600 s to 900 s to exceed worst-case cycle duration.
  • Deploys now drain in-flight cycles before restarting the worker, rather than cycling mid-run.

// system status

Every incident gets a postmortem.

Root cause, mitigation, what changed in the runbook — published within seven days, not buried. The same discipline we apply to the agents, applied to ourselves.

Back to status
Subscribe to updates

99.98% uptime · 90 days · Postmortems within 7 days · Updated every 60s