In this incident, users encountered the API unstable and slow issue.

All times are UTC.

20:44 A new deployment on the production site.

21:30 Monitoring Servers report a check failed issue to the DevOps team.

21:35 Developer investigates the issue.

21:46 DevOps found an error from Core API.

21:51 DevOps found Database performance slow and all API latency is very high.

22:39 DevOps decides to rollback to the previous stable version.

22:44 All services are back to normal.

Root cause:

The root cause is the new migration codes caused a lot of locks on tables and then slow down all API calls.