Stop guessing auto scaling targets for TTS on Amazon SageMaker AI
Teams deploying custom text-to-speech models on Amazon SageMaker AI often guess at autoscaling thresholds, and a wrong guess either wastes GPU spend or breaches a latency SLO. This post deploys Kokoro-82M as a custom container and derives a measured autoscaling policy from two live-endpoint measurements, Q_max and T_total, instead of guesswork.
First seen by Oriel on .
AI summary
The post presents a measured method for deriving Amazon SageMaker AI auto scaling thresholds for a Kokoro-82M TTS endpoint from its per-instance concurrency ceiling and real scale-out time.
- A custom container uses SageMaker AI's native ConcurrentRequestsPerModel metric, with measurements correlating client-observed concurrency to CloudWatch's 10-second Maximum statistic.
- A closed-loop load ladder found Q_max of 50 concurrent requests at a 3,000 ms p95 time-to-first-audio-byte SLO; concurrency 60 breached it.
- A forced scale-out took 260.2 seconds from capacity request to recovered traffic; adding a bounded 60-second policy detection lag yields a planned T_total of 320.2 seconds.
- The plan command derives scaling thresholds and cooldowns from measurement artifacts plus operator inputs, while deployment requires a feasible measured plan before synthesizing scaling resources.
- The sample pins an earlier PyTorch version with a known advisory, PYSEC-2026-2286, fixed in torch 2.10.0; the post says to review and upgrade it before non-proof-of-value use.
#amazon sagemaker ai #text-to-speech #auto scaling #aws cdk
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| AWS Machine Learning Blog | bc1e8aba2e409ec4403d397afa02816caf7e4b6e | https://aws.amazon.com/blogs/machine-learning/stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai/ Read discussion |