No issues were detected in the last 12 hours. Systems remain optional. We will continue monitoring things for the next 2 days.
Posted Jun 20, 2020 - 08:54 UTC
Monitoring
We have detected intermittent issues in the web console (console.wassenger.com) making it partially broken for some users.
After further investigation, we detected a CPU exhaustion in some of our API servers consumed by the console app because of a database driver version update that is causing some CPU-bound bottleneck.
We have downgraded the database driver and deployed a new version in production. We will continue monitoring the systems for the next 24 hours in order to ensure there is no further incident.
Fortunately, a total outage did not happen and the API service was able to ingest messages and provide API responses thanks to service replicas that were not impacted by the problem.