Nodes not creating in Google West Cloud
Resolved
Lasted for 2 weeks

The GCP data center outage has been resolved. Google's Status page can be found here: https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY#73mBtVKKfeJGJ1yaY7hV

Thu, May 11, 2023, 08:09 AM
1 year ago
Affected components

No components marked as affected

Updates

Resolved

The GCP data center outage has been resolved. Google's Status page can be found here: https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY#73mBtVKKfeJGJ1yaY7hV

Thu, May 11, 2023, 08:09 AM
1d earlier...

Monitoring

We are continuing to monitor Google's status updates about the data center outage of April 25th in europe-west9 (Paris) region. Those are available at https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY#73mBtVKKfeJGJ1yaY7hV .

Current status: Data center recovery is still in progress. Impact is now limited to services in europe-west9-a zone. This includes core services such as Compute Engine and Persistent Disk. Impacted services are listed in https://status.cloud.google.com/regional/europe . There is no ETA for full recovery of affected instances at this time.

Workaround: In case Aiven services do not properly fail-over to other zones in europe-west9, project administrators can migrate their services to a different GCP region.

Tue, May 9, 2023, 08:20 AM
1 week earlier...

Monitoring

We are continuing to monitor the status of this incident via Google's Status page which can be found here: https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY#73mBtVKKfeJGJ1yaY7hV

The latest update from Google can be found below:

Incident affecting Anthos Service Mesh, Google Compute Engine, Google Kubernetes Engine, Cloud Memorystore, Google Cloud Bigtable, Persistent Disk, Google Cloud SQL, Cloud Filestore, Google Cloud Dataproc

Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted.

Workaround: Customers can failover to other zones in europe-west9 or to other regions

Tue, May 2, 2023, 05:19 AM
4d earlier...

Monitoring

We continue to closely monitor the status of Google services for the europe-west9 (Paris) region.

For detailed updates on the current status, you can check the Google service status page here: https://status.cloud.google.com/regional/europe

For now, Aiven recommends migrating current workloads out of the affected region, into another functioning region until this is resolved by Google.

If you run into any issues during this time, please reach out to Aiven Support for assistance.

Fri, Apr 28, 2023, 12:00 AM
1d earlier...

Monitoring

GCP is still reporting red flags on west9 (Paris). And other regions seem to be fine. But, we recommend avoiding maintenance updates or restarts at the moment. In case of services blocked in re-building state, migrating it to another region might help.

Wed, Apr 26, 2023, 07:35 PM
5h earlier...

Monitoring

GCP is still reporting red flags in european regions west3 (Frankfurt) and west9 (Paris). We recommend avoiding maintenance updates or restarts at the moment. In case of services blocked in re-building state, migrating it to another region might help.

Wed, Apr 26, 2023, 02:08 PM
3h earlier...

Monitoring

GCP westeurope9 is still experiencing issues. In case you don't have any specific network settings in place, migrating impacted services to a different region might help you quickly restore operations.

Wed, Apr 26, 2023, 10:38 AM
45m earlier...

Monitoring

GCP europe infrastructure is now slowly coming back. We are monitoring impacted services.

Wed, Apr 26, 2023, 09:53 AM
22m earlier...

Investigating

We are still investigating, but mostly dependent on Google which declared red flags in all european regions: https://status.cloud.google.com/regional/europe

Wed, Apr 26, 2023, 09:30 AM
32m earlier...

Investigating

We are still investigating. Other GCP regions than europe-west9 seem to be impacted as well. In case you don't have any specific network settings in place, migrating impacted services to a different cloud provider might help you quickly restoring operations.

Wed, Apr 26, 2023, 08:58 AM
57m earlier...

Investigating

We are still investigating. Since there is an ongoing incident declared by Google for europe-west9, we will exclude it so nodes should provision in other availability zones.

Wed, Apr 26, 2023, 08:00 AM
36m earlier...

Investigating

We are experiencing issues with the creation of service nodes in Google West Cloud region. New and existing services might be impacted.

Wed, Apr 26, 2023, 07:24 AM