Temporarily suspending upgrades to Aiven for PostgreSQL version 16 due to bug in upstream PostgreSQL
Resolved
Partial outage
Lasted for 1 week

While we continue to work with the PG community to propose a proper fix for the upstream, we have implemented a workaround for Aiven for PostgreSQL 16. The upgrade of existing PostgreSQL services to version 16 is now available again.

Fri, Mar 8, 2024, 11:34 AM
2 months ago
Affected components
Aiven
Updates

Resolved

While we continue to work with the PG community to propose a proper fix for the upstream, we have implemented a workaround for Aiven for PostgreSQL 16. The upgrade of existing PostgreSQL services to version 16 is now available again.

Fri, Mar 8, 2024, 11:34 AM
3d earlier...

Identified

Our engineers discovered a bug in PostgreSQL that was promptly reported upstream (https://www.postgresql.org/message-id/1958255.PYKUYFuaPT%40aivenlaptop). A root cause has been identified and our engineers are working on a workaround to enable upgrading existing Aiven for PostgreSQL services to version 16. In addition, we are also working together with the PG community to propose a proper fix for upstream.

We are confident that the bug can't lead to corruption of customer's data or indexes on any Aiven for PostgreSQL 16 service. However, this issue will show up as a corruption of the Free Space Map (FSM), forcing the database to return errors on write operations.

Current Status:

  • The creation of new Aiven for PostgreSQL 16 services is available again.

  • Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

  • We have identified repair actions where this issue occurs.

  • We have found the root cause and we are working on the fix.

Tue, Mar 5, 2024, 09:27 AM
12h earlier...

Investigating

Our engineers discovered a bug in PostgreSQL that was promptly reported upstream (https://www.postgresql.org/message-id/1958255.PYKUYFuaPT%40aivenlaptop). We are still focused on finding the root cause and continuing on creating a fix.

We did not find any evidence that the bug has led to corruption on any Aiven for PostgreSQL 16 service. However, this issue will show up as a corruption of the Free Space Map (FSM), forcing the database to return errors on write operations.

This scenario appears more frequently on small instances and is only affecting a small number of Aiven for PostgreSQL 16 services.

Current Status:

  • Creation of new Aiven for PostgreSQL 16 services is available again.

  • Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

  • We have identified repair actions where this issue occurs.

  • We are continuously focusing on finding and fixing the root cause of the issue.

Mon, Mar 4, 2024, 09:15 PM
2d earlier...

Investigating

Our engineers discovered a bug in PostgreSQL that was promptly reported upstream (https://www.postgresql.org/message-id/1958255.PYKUYFuaPT%40aivenlaptop). We are still focused on finding the root cause and creating a fix.

We did not find any evidence that the bug has led to corruption on any Aiven for PostgreSQL 16 service. However, this issue will show up as a corruption of the Free Space Map (FSM), forcing the database to return errors on write operations.

This scenario appears more frequently on small instances and is only affecting a small number of Aiven for PostgreSQL 16 services.

Current Status:

  • Creation of new Aiven for PostgreSQL 16 services is available again.

  • Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

  • We have identified repair actions where this issue occurs.

  • We are focusing on finding and fixing the root cause of the issue.

Fri, Mar 1, 2024, 10:33 PM
1d earlier...

Investigating


Our engineers discovered that the source of the issue is a bug in PostgresSQL that was promptly reported upstream.

We did not find any evidence that the bug has led to persistent corruption on any Aiven for PostgreSQL 16 service.

Current Status:

  • Creation of new Aiven for PostgreSQL 16 services is available again.

  • Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

Thu, Feb 29, 2024, 12:35 AM
6h earlier...

Investigating

Our engineers discovered that the source of the issue is a bug in PostgresSQL that was promptly reported upstream.

We did not find any evidence that the bug has led to persistent corruption on any Aiven for PostgreSQL 16 service.

Creation of new Aiven for PostgreSQL 16 services is available again.

Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

Wed, Feb 28, 2024, 06:34 PM
2h earlier...

Investigating

Our engineers discovered that the source of the issue is a bug in PostgresSQL that was promptly reported upstream.

We did not find any evidence that the bug has led to persistent corruption on any Aiven for PostgreSQL 16 service. Shortly, creation of new Aiven for PostgreSQL 16 services will become available again.

Upgrading existing Aiven for PostgreSQL services to version 16 will remain unavailable while our engineers thoroughly examine the issue to ensure it does not negatively interact with the upgrade process.

Wed, Feb 28, 2024, 03:39 PM
16h earlier...

Investigating

We continue to disallow the creation of new Aiven for PostgreSQL services using PostgreSQL version 16. This is due to an ongoing investigation of a potential data integrity issue that may manifest under high load and specific circumstances.

Please note that the PostgreSQL versions 12 to 15 are still available for creation.

We apologise for the inconvenience caused by this issue as we continue our investigation and resolution.

We will provide another update in the next few hours.

Tue, Feb 27, 2024, 11:06 PM
2h earlier...

Investigating

We continue to disallow the creation of new Aiven for PostgreSQL services using PostgreSQL version 16 due to an investigation of a potential data integrity issue that may manifest under high load and specific circumstances.

We apologise for the inconvenience caused by this issue as we continue our investigation and resolution.

We will provide another update in the next few hours.

Tue, Feb 27, 2024, 08:11 PM
6h earlier...

Investigating

We have temporarily prevented the creation of new Aiven for PostgreSQL services using PostgreSQL version 16 due to an investigation of a potential data integrity issue that may manifest under high load and specific circumstances.

We apologise for the inconvenience caused by this issue.

Tue, Feb 27, 2024, 01:28 PM