Build a Self-Service LLM Gateway on Amazon Bedrock With LiteLLM, Fargate and Inference Profiles

AWS has published a pattern for a self-service LLM gateway: LiteLLM running on AWS Fargate in front of Amazon Bedrock Application Inference Profiles, so every developer uses one OpenAI-compatible URL and a personal key. The AWS Machine Learning blog post, published November 12, 2026 by Sitaraman Vijay Krishna, Eric Christensen and Saurav Bhattacharya, is rated Advanced (300). The page we could read stopped after the cost-controls section, so this guide covers the architecture, configuration, cost controls and pitfalls, not any later material.

The use case is common. Developers want coding assistants with little setup. Platform teams want latency and error data. Finance wants cost by team and project. Security wants one place to apply access control. A gateway gives all three a single choke point.

TL;DR

The architecture

AWS lists these components:

• Request path: Amazon Route 53, Amazon CloudFront and an Application Load Balancer front LiteLLM on Fargate. Fargate uses target-tracking auto scaling on CPU utilization.
• Persistent state: Amazon RDS for PostgreSQL stores virtual keys and spend data.
• Cache and rate limiting: Amazon ElastiCache for Valkey provides non-persistent response caching and coordinates rate limits across Fargate tasks.
• Secrets: AWS Secrets Manager holds the master key, the database URL and Langfuse credentials.
• Access: A scoped IAM task role invokes Bedrock.
• Monitoring: Amazon CloudWatch tracks requests per minute (RPM), tokens per minute (TPM) and quota alarms. Langfuse provides prompt-level telemetry and latency tracking.
• Self-service portal: LiteLLM's portal lets operators set spend policy and developers self-provision keys, and can integrate with corporate SSO.
• Inference profiles: Each model gets its own profile, managed with AWS CDK, for per-model quota isolation and an abstraction over model versioning.

The inference profile is the quiet hero. A profile per model lets you track cost and isolate quota by use, and lets you move to a new model version by changing a mapping, not every developer's config.

Licensing: where cost enters

The post says LiteLLM's AI Gateway is open source and free for up to five users. Self-service key issuance requires LiteLLM Enterprise, which adds SSO, audit logs, delegated admin roles and support. That line matters. A small team can run the open source gateway for itself. A company that wants developers to mint their own keys through SSO needs the paid tier. We did not see Enterprise pricing in the post, so ask the vendor before you plan.

The configuration, step by step

AWS gives five steps.

1. Developer IDE config. Developers set apiBaseUrl to the gateway's /v1 endpoint, plus their own API key and a model name. Nothing else changes in their tools.

2. LiteLLM config.yaml. It maps model names to inference profile ARNs and sets max_retries: 3. Timeouts are 900 seconds for Sonnet, to allow extended thinking, and 120 seconds for Haiku. It sets max_budget: 250 with budget_duration: 7d, a rolling seven-day window per key, and max_parallel_requests: 10. Caching uses the Redis type, pointed at Valkey on port 6379, and secrets load from environment variables.

3. CDK inference profiles. Create CfnApplicationInferenceProfile resources that copy from the system-defined global. cross-Region profile. Grant bedrock:InvokeModel and InvokeModelWithResponseStream to the task role, scoped to the profile ARN and the foundation model ARNs.

4. Invocation. Bedrock routes requests across Regions transparently.

5. Langfuse. Add success_callback and failure_callback set to ["langfuse"], and set the public key, secret key and host as environment variables.

What a virtual key is, from LiteLLM's docs

AWS's design leans on LiteLLM virtual keys, so it helps to know what they are. LiteLLM's virtual keys documentation describes them as API keys issued by the proxy to control model access and track spend. Spend is recorded per key, user, team and end user, and is calculated in USD from each model's cost. That last point explains the first pitfall below: if the proxy has no cost for a model, it records nothing.

A key can carry a spending cap (max_budget), a period for that cap (budget_duration, such as "10d") and limits on tokens per minute, requests per minute and concurrent requests. Operators can set proxy-wide defaults and upper bounds in litellm_settings, and the docs say values above a bound are rejected, not clamped. Keys come from a /key/generate call on the proxy, made with the master key. The body can name the allowed models and a user or team, plus budget and rate fields. The response contains a new sk- key for the client.

In practice that means three levels of control for your platform team. Operators set the outer bounds. Teams or users get a key with a budget and a model list. Developers use the key like any OpenAI-compatible credential. AWS's self-service portal, which needs the Enterprise tier, lets developers do the middle step themselves instead of filing a ticket. Without it, an operator issues keys by hand or by script, which is workable for a small group.

Cross-Region inference choices

The authors recommend global cross-Region inference when the models support it and policy allows. They say it gives access to a broader capacity pool and potential pricing benefits. Geographic cross-Region inference keeps processing within a defined area, such as US-only Regions. If your compliance rules fix where data may be processed, pick the geographic option and accept the smaller pool.

Cost controls

When a key exceeds its budget, LiteLLM returns HTTP 429 with a clear message. You can check spend through GET /user/daily/activity, GET /user/info?user_id=... or the portal dashboard. A 429 is also what a developer sees when rate limits trip, so make sure the message tells people which limit they hit, or your help desk will field the confusion.

AWS published related cost posts the same day. Tokenomics at scale covers tracking and capping Bedrock spend per IAM principal, and Part 2 of the cost series covers cost per task and agent ROI. For the cost side of agent work beyond keys and budgets, see our guide to measuring cost per task, not per token, on Bedrock, and for org-wide invocation data the Bedrock tokenomics data lake.

Three pitfalls AWS lists

1. Costs read as zero. Raw inference profile ARNs are not in LiteLLM's built-in cost map. You must set per-token costs in model_info. Otherwise spend stays at $0 and budgets never enforce. AWS says its example prices are illustrative only, so enter your real rates.
2. Cross-Region IAM scope. Global cross-Region inference needs IAM permissions on the foundation model ARNs in every Region it might route to. Scoping only to the profile ARN causes authorization errors at invoke time.
3. Prompt logging. Langfuse traces can include prompts and responses. Configure redaction, retention and access controls before you turn logging on, because prompts often hold code, secrets and customer data.

A fourth note is a gap, not a pitfall: the example omits Bedrock Guardrails. AWS recommends them for production.

Who should build this

A gateway pays off when you have more than a handful of developers, several models, and a need to answer "who spent what." It is overkill for one team calling one model. Teams already running agents on AgentCore may get some of this from the platform. Teams building data pipelines around Bedrock, like the ones in our Crawl4AI for Bedrock Knowledge Bases guide and the serverless MCP server on Fargate, already use the same building blocks, so the gateway is a natural add.

What a gateway does not solve

A gateway centralizes access, but it does not make usage wise. A key with a seven-day budget still lets one developer spend the whole budget on a single runaway agent loop. Budgets cap damage, and they do not say whether the spend was worthwhile, which is the question AWS's Part 2 tackles. A gateway also becomes a single point of failure and a high-value target, since it holds a master key and can invoke models for everyone. Treat its secrets, its database and its logs with the same care as any production credential store, and review who can reach the admin portal.

Checklist before you deploy

1. Confirm Bedrock model access and IAM permissions in your account.
2. Decide on global or geographic cross-Region inference with your compliance team.
3. Set real per-token costs in model_info and test that a key hits its budget.
4. Scope IAM to both the profile ARN and foundation model ARNs.
5. Decide whether you need LiteLLM Enterprise for self-service keys and SSO.
6. Set Langfuse redaction and retention before you log.
7. Add Bedrock Guardrails and review the 429 message text.

Caveats

This is vendor guidance from AWS, and the page we read was truncated after the cost-controls section, so we cannot say what follows. The budget and timeout values are the authors' example, not recommendations for your workload. We have not run the stack ourselves, and we did not see load or latency numbers in the part we read.

Related reading

• AWS Part 2: Bedrock cost per task, not per token
• Amazon Bedrock tokenomics data lake
• AWS Bedrock AgentCore runtime
• Crawl4AI for Bedrock Knowledge Bases
• Serverless MCP server on ECS Fargate
• Claude Code token efficiency and prompt cache guide

This post reflects the AWS article available on November 12, 2026.