MySQL version 8.0.30 will fail to create basebackups
Resolved
Lasted for 6d

We have released the fix into the production environment and maintenance is being rescheduled in the background. This incident is now resolved. If you continue to experience any issues, please reach out to Aiven Support.

Thu, Dec 1, 2022, 05:09 AM
1 year ago
Affected components

No components marked as affected

Updates

Resolved

We have released the fix into the production environment and maintenance is being rescheduled in the background. This incident is now resolved. If you continue to experience any issues, please reach out to Aiven Support.

Thu, Dec 1, 2022, 05:09 AM
6h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detectedNo components were marked as affected by this incident.No components were marked as affected by this incident.

Wed, Nov 30, 2022, 10:09 PM
5h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detectedNo components were marked as affected by this incident.

Wed, Nov 30, 2022, 05:02 PM
3h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected

Wed, Nov 30, 2022, 01:57 PM
3h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected

Wed, Nov 30, 2022, 10:49 AM
5h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected.

Wed, Nov 30, 2022, 05:36 AM
3h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected.

Wed, Nov 30, 2022, 01:38 AM
4h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected.

Tue, Nov 29, 2022, 09:30 PM
7h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected.

Tue, Nov 29, 2022, 02:00 PM
3h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected.

Tue, Nov 29, 2022, 10:11 AM
6h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected.

Tue, Nov 29, 2022, 03:36 AM
4h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected.

Mon, Nov 28, 2022, 11:28 PM
5h earlier...

Identified

Multiple issues have been identified for MySQL 8.0.30 releases, which may prevent scheduled backups from running.

Backups may fail or timeouts may be encountered.Automatic maintenance updates to 8.0.30 have been disabled

Aiven is continuing to monitor for impacted services and manually applying mitigation as they are detected.

Mon, Nov 28, 2022, 05:46 PM
1d earlier...

Identified

Our Engineering team have deployed the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this.

Maintenance updates to recycle the already-affected services are ongoing.

Sun, Nov 27, 2022, 11:17 AM
4h earlier...

Identified

Our Engineering team have deployed the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this.

Maintenance updates to recycle the already-affected services are ongoing.

Sun, Nov 27, 2022, 06:39 AM
8h earlier...

Identified

Our Engineering team have deployed the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this.

Maintenance updates to recycle the already-affected services are ongoing.

Sat, Nov 26, 2022, 09:49 PM
6h earlier...

Identified

Our Engineering team have deployed the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this. Currently, we are working on rolling out a maintenance update to recycle the already-affected services.

Sat, Nov 26, 2022, 03:04 PM
4h earlier...

Identified

Our Engineering team have deployed the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this. Currently, we are working on rolling out a maintenance update to recycle the already-affected services.

Sat, Nov 26, 2022, 10:38 AM
3h earlier...

Identified

Our Engineering team have deployed the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this. Currently, we are working on rolling out a maintenance update to recycle the already-affected services.

Sat, Nov 26, 2022, 06:49 AM
4h earlier...

Identified

Our Engineering team have deployed the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this. Currently, we are working on rolling out a maintenance update to recycle the already-affected services.

Sat, Nov 26, 2022, 02:10 AM
3h earlier...

Identified

Our Engineering team have deployed the permanent fix to the backup failure issue in MySQL 8.0.30. No more services will be affected due to this. Currently, we are working on rolling out a maintenance update to recycle the already-affected services.

Fri, Nov 25, 2022, 10:32 PM
8h earlier...

Identified

We are still working towards deploying the patch, which will mitigate the bug found in MySQL 8.0.30. In the meantime, we have detection tooling in place to detect and fix affected services. Affected services will be mitigated when detected so backups can still be created.

Fri, Nov 25, 2022, 01:51 PM
6h earlier...

Identified

We are still working towards deploying the patch, which will mitigate the bug found in MySQL 8.0.30

We now have detection tooling in place to detect services running MySQL 8.0.30, and an ALTER statement has been executed. Affected services will be mitigated when detected so backups can still be created.

We will provide further updates when fixes begin to roll out.

Fri, Nov 25, 2022, 07:12 AM
7h earlier...

Identified

We are still working towards deploying the patch, which will mitigate the bug found in MySQL 8.0.30

We now have detection tooling in place to detect services running MySQL 8.0.30, and an ALTER statement has been executed. Affected services will be mitigated when detected so backups can still be created.

We will provide further updates when fixes begin to roll out.

Thu, Nov 24, 2022, 11:13 PM
5h earlier...

Identified

Our engineering team have devised the patch to fix the backup issue with MySQL 8.0.30 and we are currently fixing the affected services. We are also working simultaneously for the permanent fix that will be rolled out to all the services.

Thu, Nov 24, 2022, 05:49 PM
4h earlier...

Identified

Our engineering team is still working on a patch to fix the backup issue on MySQL 8.0.30 services. We will first fix the affected services, then will later roll out a permanent fix to all MySQL services

Thu, Nov 24, 2022, 01:03 PM
3h earlier...

Identified

Our engineering team is currently working on a patch to fix the backup issue on MySQL 8.0.30 services

Thu, Nov 24, 2022, 09:15 AM
2h earlier...

Identified

We are aware of a problem affecting MySQL services running on version 8.0.30, preventing the service from taking base backups. Specifically, the backup procedure will fail whenever an ALTER TABLE statement has been made in 8.0.30.

If you have MySQL services with versions below 8.0.30, we recommend that you do not apply any maintenance updates, change the plan or migrate to a different cloud in order to not be affected by this backup problem.

If your MySQL is running on 8.0.30 and notice any issues, you can contact us via support@aiven.io so that we are able to guide you further.

Our engineers are currently working on mitigating the problem with a patch. We apologise for any inconvenience.

Thu, Nov 24, 2022, 07:02 AM