On September 15, between 07:45 and 09:34 UTC, our Management Service experienced degraded performance and partial outages. Existing peer connections remained fully operational throughout the incident; impact was limited to new user login, slow peer reconnections and general management operations.
What happened
A scheduled production deployment failed to start due to a version incompatibility with our license validation service. We immediately initiated our automated rollback procedure. Unfortunately, this coincided with an outage of our CI/CD provider, which rendered our automated rollback pipeline unavailable and required us to fall back to a manual recovery process.
During the manual rollback, a configuration inconsistency in the deployment setup prevented the restored version from passing license validation. In parallel, our database cluster came under unusually high load caused by an elevated volume of peer metadata updates. The combination of these factors extended the recovery time significantly.
By 08:50 UTC, parts of the management cluster had recovered and were serving the majority of peers. However, sustained database load continued to cause significant instability across the service. After a series of controlled recovery measures to relieve database pressure, the cluster stabilized at 09:27 UTC and was confirmed fully operational at 09:34 UTC.
Why it took longer than it should have
Several independent issues compounded: a service version incompatibility, an unavailable automated rollback path due to a CI/CD provider outage, a configuration inconsistency, and elevated database load. Individually, each would have been recovered quickly; together, they extended the incident.
Next steps
We have completed a thorough internal review of the incident and are acting on its findings across four areas:
Deployment & rollback resilience: Strengthening our deployment pipeline and ensuring rollback procedures remain fully operational independent of third-party service availability.
Configuration management: Improving how configuration is managed and validated across environments to prevent inconsistencies from reaching production.
Infrastructure capacity: Scaling and tuning our database cluster to better absorb load spikes during recovery scenarios.
Release practices: Scheduling future releases during low-traffic periods and refining our release process to minimize potential impact.
Several of these improvements are already in progress, and we will continue to prioritize them in the coming weeks.
