Oriel — AI & technology worth noticing, every day.

Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

Web page · https://aws.amazon.com/blogs/machine-learning/building-a-self-service-llm-gateway-with-amazon-bedrock-and-litellm

Build a self-service LLM gateway with LiteLLM on AWS Fargate that puts Amazon Bedrock foundation models behind an OpenAI-compatible API. Developers get instant access through a single API key, while platform teams gain per-user cost controls, quota isolation, and observability.

First seen by Oriel on .

AI summary

This AWS post presents a self-service LLM gateway combining LiteLLM on AWS Fargate with Amazon Bedrock Application Inference Profiles for OpenAI-compatible access, per-user cost controls, and observability.

  • LiteLLM exposes a single OpenAI-compatible endpoint for tools such as Claude Code, Continue, and Cursor, while resolving model names to Bedrock inference profile ARNs.
  • Amazon RDS (PostgreSQL) stores API-key metadata and spend state, ElastiCache for Valkey provides caching and distributed rate limiting, and CloudWatch captures RPM, TPM, and quota metrics.
  • Per-key budgets require explicit input and output token costs when using raw inference-profile ARNs; the example enforces a rolling seven-day $250 limit and returns HTTP 429 after exhaustion.
  • Global cross-Region inference can route requests across supported AWS commercial Regions, but IAM must authorize the application profile and underlying regional and global foundation-model resources.
  • LiteLLM’s self-service key issuance requires LiteLLM Enterprise, while Langfuse adds prompt-level telemetry; production deployments should address responsible AI controls, redaction, retention, and access management.

#amazon bedrock #litellm #llm gateways #cloud observability

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
AWS Machine Learning Blogf5427fcfe4e9eb6589cc6544f1dbc031a6ca40bahttps://aws.amazon.com/blogs/machine-learning/building-a-self-service-llm-gateway-with-amazon-bedrock-and-litellm/ Read discussion