Wednesday November 20th between 14:00–16:50 Finnish time, our system experienced significant performance degradation. Users encountered slower operations, delays, and higher error rates. Sympa Classic customers have been seen even bigger problems during this period. The system stabilized after 16:50 after a mitigation action.

Root Cause

The issue stemmed from a database problem where the CPU utilization for our primary database peaked for over two hours. The underlying cause was identified as broken database index statistics . This issue disrupted query optimization, causing a system-wide slowdown for all database queries.

Resolution

Once the root cause was identified, we rebuilt the table statistics in the database. The system recovered within minutes of completing this operation.

Next Steps

1. Improve Query Efficiency: Refactor the code to prevent future database jams by optimizing the database query that had issue
2. Enhance Monitoring for database
3. Faster Recovery Protocols: Leverage today's learnings to ensure quicker recovery in case of recurrence.

Apologies and Acknowledgments

We deeply regret the issue. This was an unfortunate situation, and we sincerely apologize for the disruption caused to customers. Rest assured, Sympa Engineers are committed to implementing the necessary safeguards to prevent similar occurrences in the future.