Ongoing incident
Resolved
Lasted for 17h

This incident has been resolved.

Tue, Jun 29, 2021, 05:05 AM
2 years ago
Affected components

No components marked as affected

Updates

Resolved

This incident has been resolved.

Tue, Jun 29, 2021, 05:05 AM
3h earlier...

Identified

Our team is continuing to roll out a permanent fix to all affected systems

Tue, Jun 29, 2021, 01:46 AM
3h earlier...

Identified

Our team is continuing to roll out a permanent fix to all affected systems.

Mon, Jun 28, 2021, 10:46 PM
1h earlier...

Identified

We are continuing to roll out a permanent fix to all affected systems.

Mon, Jun 28, 2021, 09:27 PM
2h earlier...

Identified

All systems have been patched. We are gradually rolling out the permanent fix to all affected systems.

Mon, Jun 28, 2021, 06:46 PM
2h earlier...

Identified

We have applied fixes for all new nodes and services. A temporary fix has been applied to existing nodes. We are in the process of applying a permanent fix for the replication issue.

Mon, Jun 28, 2021, 04:29 PM
1h earlier...

Identified

We have applied fix for the root cause and no new services or nodes should be affected by this. A temporary fix has been applied to almost all previously affected nodes to fix any immediate replication issues. We are still working on applying a permanent fix for all affected nodes.

Mon, Jun 28, 2021, 03:21 PM
1h earlier...

Identified

We have identified the root cause of the problem and we have prepared a fix, which is ready to be deployed. At the same time we are still making the necessary corrections on the affected nodes in order to restore normality.

Mon, Jun 28, 2021, 02:07 PM
2h earlier...

Identified

PostgreSQL replica nodes fail to connect to new master after failover. We are manually fixing affected services to fix any immediate issues while working on fixing the root cause. There may be higher than expected replication lag for standby and separate read-replica nodes while the issue is ongoing.

Mon, Jun 28, 2021, 11:53 AM