Update - Our Engineering team have deployed the the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this. Currently, we are working on rolling out maintenance update to recycle the already affected services.
Nov 26, 2022 - 06:49 UTC
Update - Our Engineering team have deployed the the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this. Currently, we are working on rolling out maintenance update to recycle the already affected services.
Nov 26, 2022 - 02:10 UTC
Update - Our Engineering team have deployed the the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this. Currently, we are working on rolling out maintenance update to recycle the already affected services.
Nov 25, 2022 - 22:32 UTC
Update - We are still working towards deploying the patch, which will mitigate the bug found in MySQL 8.0.30. In the meantime, we have detection tooling in place to detect and fix affected services. Affected services will be mitigated when detected so backups can still be created.
Nov 25, 2022 - 13:51 UTC
Update - We are still working towards deploying the patch, which will mitigate the bug found in MySQL 8.0.30

We now have detection tooling in place to detect services running MySQL 8.0.30, and an ALTER statement has been executed. Affected services will be mitigated when detected so backups can still be created.

We will provide further updates when fixes begin to roll out.

Nov 25, 2022 - 07:12 UTC
Update - We are still working towards deploying the patch, which will mitigate the bug found in MySQL 8.0.30

We now have detection tooling in place to detect services running MySQL 8.0.30, and an ALTER statement has been executed. Affected services will be mitigated when detected so backups can still be created.

We will provide further updates when fixes begin to roll out.

Nov 24, 2022 - 23:13 UTC
Update - Our engineering team have devised the patch to fix the backup issue with MySQL 8.0.30 and we are currently fixing the affected services. We are also working simultaneously for the permanent fix that will be rolled out to all the services.
Nov 24, 2022 - 17:49 UTC
Update - Our engineering team is still working on a patch to fix the backup issue on MySQL 8.0.30 services. We will first fix the affected services, then will later roll out a permanent fix to all MySQL services
Nov 24, 2022 - 13:03 UTC
Update - Our engineering team is currently working on a patch to fix the backup issue on MySQL 8.0.30 services
Nov 24, 2022 - 09:15 UTC
Identified - We are aware of a problem affecting MySQL services running on version 8.0.30, preventing the service from taking base backups. Specifically, the backup procedure will fail whenever an ALTER TABLE statement has been made in 8.0.30.

If you have MySQL services with versions below 8.0.30, we recommend that you do not apply any maintenance updates, change the plan or migrate to a different cloud in order to not be affected by this backup problem.

If your MySQL is running on 8.0.30 and notice any issues, you can contact us via support@aiven.io so that we are able to guide you further.

Our engineers are currently working on mitigating the problem with a patch. We apologise for any inconvenience.

Nov 24, 2022 - 07:02 UTC
Past Incidents
Nov 26, 2022

Unresolved incident: MySQL version 8.0.30 will fail to create basebackups.

Nov 25, 2022
Nov 24, 2022
Nov 23, 2022

No incidents reported.

Nov 22, 2022

No incidents reported.

Nov 21, 2022

No incidents reported.

Nov 20, 2022
Resolved - This incident has been resolved.
Nov 20, 05:51 UTC
Monitoring - The issue has been resolved and we are monitoring
Nov 20, 05:41 UTC
Update - The issue has been resolved and we are monitoring
Nov 20, 05:41 UTC
Update - We are continuing to investigate this issue.
Nov 20, 04:16 UTC
Investigating - We are investigating an issue affecting billing updates for Aiven Azure Marketplace users.
Nov 20, 04:01 UTC
Nov 19, 2022

No incidents reported.

Nov 18, 2022

No incidents reported.

Nov 17, 2022

No incidents reported.

Nov 16, 2022

No incidents reported.

Nov 15, 2022
Resolved - The fix has been deployed, and all affected services have completed their scheduled maintenance updates.
This incident has been resolved

Nov 15, 11:19 UTC
Update - We are applying the fix to the affected service.
Nov 15, 10:18 UTC
Identified - We have identified the cause of the issue and are currently working on a fix
Nov 15, 09:05 UTC
Investigating - We have recently discovered a bug in our automation that prevents node replacement for M3DB services during maintenance update. Currently running nodes in your M3DB services remain operational. We will temporarily disable maintenance update on all M3DB services while we are fixing this bug.
Nov 15, 07:57 UTC
Nov 14, 2022

No incidents reported.

Nov 13, 2022

No incidents reported.

Nov 12, 2022

No incidents reported.