Cloud provider core infrastructure unavailability

Incident Report for WellSaid Labs

Resolved

Services are now fully operational, and all systems are performing as expected.
Posted Jun 12, 2025 - 20:46 UTC

Update

Services are now operational, though some requests may still be slower than normal. Clip creation is functioning, but clips generated during the outage remain unavailable. Our team is continuing to monitor the system and investigating options to recover the affected clip data.
Posted Jun 12, 2025 - 20:32 UTC

Monitoring

Services are beginning to recover, and clip creation is now fully operational. However, clips that were created during the outage remain unavailable. Our engineering team is actively investigating options to recover those affected clips.
Posted Jun 12, 2025 - 20:17 UTC

Update

Our cloud provider has reported that most locations have fully recovered. However, some features—such as clip interactions—remain unavailable. We’re actively working to restore full functionality and will provide further updates as they become available.
Posted Jun 12, 2025 - 19:45 UTC

Update

Google Cloud has reported a major service outage across IAM affecting the entire platform https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
Posted Jun 12, 2025 - 19:05 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jun 12, 2025 - 18:40 UTC

Identified

The issue has been identified as outside our infrastructure and reported by our cloud providers.
Posted Jun 12, 2025 - 18:33 UTC

Investigating

We are currently experiencing issues with our Cloud provider's core infrastructure, which is affecting a large portion of network traffic.

We are monitoring this outage at the PaloAltoNetworks level https://status.paloaltonetworks.com/incidents/l4zv6n67r8x6
Posted Jun 12, 2025 - 18:33 UTC
This incident affected: Studio (Editor, API, Clip Downloading), Text to Speech (Developer API), and Google Cloud Platform Google Kubernetes Engine, Google Cloud Platform Google Cloud Storage, Google Cloud Platform Identity & Access Management.