GPU Node Provisioning Failure

Incident Report for WellSaid Labs

Resolved

This incident has been resolved.
Posted Sep 30, 2025 - 23:16 UTC

Monitoring

Provisioning of new GPU nodes in our GKE environments began returning to normal overnight. We are now seeing only minimal delays in node creation, and service performance has largely stabilized. We are working on expanding our infrastructure and are actively monitoring node availability.
Posted Sep 17, 2025 - 04:00 UTC

Identified

We are currently experiencing issues provisioning new GPU nodes in our GKE environments. This is causing a reduced capacity in the number of requests we're able to serve simultaneously, leading to longer response times and clips failing to generate.
Posted Sep 16, 2025 - 15:00 UTC
This incident affected: Text to Speech (Developer API).