In this incident, users encountered the API unstable and slow issue.
All times are UTC.
20:44 A new deployment on the production site.
21:30 Monitoring Servers report a check failed issue to the DevOps team.
21:35 Developer investigates the issue.
21:46 DevOps found an error from Core API.
21:51 DevOps found Database performance slow and all API latency is very high.
22:39 DevOps decides to rollback to the previous stable version.
22:44 All services are back to normal.
Root cause:
The root cause is the new migration codes caused a lot of locks on tables and then slow down all API calls.