Oriel — AI & technology worth noticing, every day.

Stop guessing auto scaling targets for TTS on Amazon SageMaker AI

Web page · https://aws.amazon.com/blogs/machine-learning/stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai

Teams deploying custom text-to-speech models on Amazon SageMaker AI often guess at autoscaling thresholds, and a wrong guess either wastes GPU spend or breaches a latency SLO. This post deploys Kokoro-82M as a custom container and derives a measured autoscaling policy from two live-endpoint measurements, Q_max and T_total, instead of guesswork.

First seen by Oriel on .

AI summary

The post presents a measured method for deriving Amazon SageMaker AI auto scaling thresholds for a Kokoro-82M TTS endpoint from its per-instance concurrency ceiling and real scale-out time.

  • A custom container uses SageMaker AI's native ConcurrentRequestsPerModel metric, with measurements correlating client-observed concurrency to CloudWatch's 10-second Maximum statistic.
  • A closed-loop load ladder found Q_max of 50 concurrent requests at a 3,000 ms p95 time-to-first-audio-byte SLO; concurrency 60 breached it.
  • A forced scale-out took 260.2 seconds from capacity request to recovered traffic; adding a bounded 60-second policy detection lag yields a planned T_total of 320.2 seconds.
  • The plan command derives scaling thresholds and cooldowns from measurement artifacts plus operator inputs, while deployment requires a feasible measured plan before synthesizing scaling resources.
  • The sample pins an earlier PyTorch version with a known advisory, PYSEC-2026-2286, fixed in torch 2.10.0; the post says to review and upgrade it before non-proof-of-value use.

#amazon sagemaker ai #text-to-speech #auto scaling #aws cdk

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
AWS Machine Learning Blogbc1e8aba2e409ec4403d397afa02816caf7e4b6ehttps://aws.amazon.com/blogs/machine-learning/stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai/ Read discussion