Resolved -
This incident has been resolved.
Sep 30, 23:16 UTC
Monitoring -
Provisioning of new GPU nodes in our GKE environments began returning to normal overnight. We are now seeing only minimal delays in node creation, and service performance has largely stabilized. We are working on expanding our infrastructure and are actively monitoring node availability.
Sep 17, 04:00 UTC
Identified -
We are currently experiencing issues provisioning new GPU nodes in our GKE environments. This is causing a reduced capacity in the number of requests we're able to serve simultaneously, leading to longer response times and clips failing to generate.
Sep 16, 15:00 UTC