SageMaker Multi-Turn RL Walkthrough: gpt-oss-20b Goes From 61% to 84% on Wordle for About $339

Amazon published a hands-on guide on October 23, 2026 to Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI, and the headline number is concrete. In AWS's run, gpt-oss-20b went from a 61 to an 84 percent solve rate on held-out Wordle words at its default reasoning effort, and from 15 to 69 percent at the low effort it was trained at. The training bill was about $339 in tokens, plus roughly $79 for the agent runtime.

Wordle is a teaching example, and AWS says frontier models already play it well. The point is the pipeline: you describe how to score a whole task, not what the correct steps are, and SageMaker handles the training infrastructure. This post walks through that pipeline, the costs, and what to be careful about. It builds on our earlier look at SageMaker Training Sessions and writing your own RL loop.

Quick reference

Why multi-turn RL, and what problem it addresses

Most fine-tuning teaches a model to produce a better answer to one prompt. Agents are different. An agent that files an expense report or triages a ticket takes a sequence of actions, each depending on the last. AWS argues there is rarely one correct sequence to imitate, and the only reliable feedback is whether the whole sequence succeeded. Supervised fine-tuning has no labels for that, and single-turn RL cannot credit a decision made three steps before the outcome.

MTRL trains on exactly that outcome signal. You do not specify the right trajectory, only how to score the result. AWS's stated use case is making a small, low-cost model good at a multi-step task specific to your business, such as an internal workflow public models were never trained on.

This fits a wider pattern in our coverage: outcome rewards are also how labs train monitoring models, as in Anthropic's RL-trained alignment auditor.

How the system is split

SageMaker AI training job connected to the AgentCore-hosted agent, Wordle environment, and reward function Image: AWS

The design has two halves, per AWS:

• SageMaker AI owns the training job and the policy model.
• AgentCore hosts the agent, the Wordle environment and the reward function.

For each rollout the MTRL service invokes the AgentCore runtime with one training row holding one secret word. A Strands agent in the runtime calls the policy model through the SageMaker Job Runtime's OpenAI-compatible endpoint. The model plays up to six guesses through one tool, guess_word. After each guess the environment returns per-letter feedback and a summary of what is known. When the game ends, the agent returns a single reward. GRPO uses the spread of rewards across several rollouts of the same word to update the LoRA adapter.

Three design points AWS highlights:

1. The agent and reward live in one place. One Python file deployed to AgentCore holds the environment, tool and scoring function.
2. The dataset carries only the secret word. The bundled word lists, built from the open-source SCOWL lists, are used only for scoring; the policy model never sees the answer.
3. The reward is outcome-dominant. A solve pays 1.05 to 1.55 depending on speed, a loss pays -0.5, and never making a valid guess pays -1.5. A small shaping term rewards good strategy within each tier.

Which models and regions

MTRL provisions training and sampling infrastructure for you. At the time of AWS's writing it supports four base models, each trained with a LoRA adapter:

AWS chose GPT-OSS-20B because it has the lowest per-token price of the four and its reasoning effort can be set per request.

What it costs

You pay per million tokens in three dimensions, priced by base model: prefill (input tokens, which accumulate in multi-turn tasks because every turn re-reads the history), sample (generated tokens, including reasoning) and train (tokens processed in the backward pass).

Train tokens dominate and scale with trajectory length, so shorter games and less verbose reasoning cut the bill. The AgentCore runtime is billed separately for vCPU and memory, about $79 for this run according to CloudWatch usage metrics at us-east-1 list prices. DescribeJob reports token counts under BillableTokenUsage. These are AWS's list prices at the time of writing; check the SageMaker pricing page before planning a run.

A small shape reaching for a green token at the end of a path, showing the single end-of-episode reward in multi-turn reinforcement learning

The pipeline, step by step

Step 1: two IAM roles

The agent runtime role is created for you by agentcore deploy and needs the AmazonSageMakerJobRuntimeAccess managed policy. The training job role you create yourself. AWS flags a gotcha: it must trust job.sagemaker.amazonaws.com, not sagemaker.amazonaws.com, with both sts:AssumeRole and sts:TagSession, and carry AmazonSageMakerJobFullAccess. With the generic principal, CreateJob fails with "Could not assume role," and the message names the right principal only after retries.

Step 2: deploy the agent

Clone the aws-samples repository, set the account and Region in aws-targets.json, then run agentcore validate and agentcore deploy. The CLI builds dependencies for the runtime's Linux ARM64 platform and deploys through CloudFormation. AWS recommends it over uploading a code package in the console, where about 700 MB of compiled dependencies make mistakes common.

The agent is a single main.py. The entrypoint decorator order matters: wrap the sagemaker_rft_handler with BedrockAgentCoreApp and @app.entrypoint, with the entrypoint outermost. Without it, AWS says every invocation times out after 30 seconds. Two environment variables control reasoning: REASONING_EFFORT=low and REASONING_PROMPT_HINT=true. In AWS's evaluation jobs, low effort used 63 to 83 percent fewer sample tokens per game than medium.

Step 3: verify before spending GPU time

Invoke the runtime with a payload shaped like the training service's, using a fake job ARN. The expected result is a 400 "Could not find job" in CloudWatch and a printed reward of -1.5, which proves parsing, authentication and error reporting work. A 403 means the ARN names another account, and a 30-second timeout means the HTTP server never started.

Step 4: dataset

Each row is one task, with no target responses or labels. The service passes the prompt string to your agent verbatim. AWS made it a small JSON object with prompt, answer and id. The sample script draws 640 training words (five batches of 128) and 128 non-overlapping validation words, so validation measures generalization. Parquet, JSON arrays and CSV are also accepted. A production task could pack a full message list, tool configuration or reward specification into the same string.

Step 5: launch

The launcher uses the MultiTurnRLTrainer class from the sagemaker-train SDK and requires an MLflow app for metrics. It sets sampling_max_tokens to 8192 (the service maximum, because at 4096 gpt-oss often ran out of tokens before its first tool call), max_epochs to 20 and max_steps to 100. With 640 rows at the default global batch size of 128, that is 5 steps per epoch, and with the default group size of 8, each step plays 1,024 games. AWS also warns not to redeploy the agent while a job runs, since the service fails the whole job.

Step 6: monitor in MLflow

The metric to watch is val/reward/pass_at_1. AWS reports training reward climbing to about +0.57 within 15 steps and holding, while the validation solve rate jumped from 2 to 31 percent in the first 10 steps and then moved between 35 and 48 percent. With 128 validation words, AWS says those swings are mostly noise. It did not test whether more steps help at serving settings.

AWS also shows one trained-model game, with the word PASTE solved in four guesses. The model keeps green letters in place, avoids reusing gray letters and uses yellow letters to rule out positions, behaviors the training data never demonstrated.

Step 7: evaluate at serving settings

An evaluation job plays your prompts through the same runtime and reward, on the base model or the trained package, and logs eval/reward/pass_at_1. Unlike training validation, you can set the temperature. AWS evaluated both models at low and medium effort and at temperature 0 and 1.0, with each 128-word evaluation taking four to seven minutes. It notes that at temperature 0 both models often end the game without a solve, so the in-training validation at temperature 0 understates the real gain. The default quota allows one evaluation job at a time.

How to read the numbers

Bar charts comparing base and trained gpt-oss-20b solve rate and mean reward across four decoding settings Image: AWS

• One run, one task. The 61 to 84 percent figure is from a single training run on 128 held-out words. A small validation set means a few points of noise either way; AWS itself says the validation swings are mostly noise.
• A game with an objective score. Wordle has few rules and a clear win condition. Business workflows often have fuzzier success criteria, and designing the reward is the hard part. AWS notes the reward was outcome-dominant with a small shaping term.
• Settings matter more than headlines. The gap between 15 to 69 percent at low effort and 61 to 84 percent at default effort shows how much reasoning effort and temperature move results.
• Costs add up with long episodes. Prefill re-reads the whole history each turn, and train tokens were over two thirds of the token bill.
• A vendor guide. The numbers are AWS's own and come with AWS list prices. Nobody outside has reproduced them as far as we can tell.

What this means for what you build or pay

If you have an agent workflow with a clear pass or fail outcome, MTRL turns that outcome into a training signal without you labeling steps or managing GPUs. The practical decisions are the reward function, the dataset of tasks, and the cost envelope: budget for roughly the token mix above, and for the agent runtime on top. Test the reward on a handful of rollouts in the verification step before you pay for hundreds of thousands.

There is also a security angle. Your agent runs on AgentCore during training and calls tools; a bad reward or an over-permissive tool lets the optimizer find shortcuts. Test your reward against ways to cheat it. For agent hijack risks on the same platform, see our write-up of the AgentCorruption research on Bedrock AgentCore, and for tool-selection choices see Apple's study on how agents should find tools.

What to watch next

• Whether AWS adds more base models or regions beyond the four listed.
• Independent reproductions on tasks harder than Wordle.
• Quota changes for evaluation jobs, currently one at a time by default.
• How the reward-design lessons AWS shares in the rest of its guide hold up on real business tasks.

Related reading

• SageMaker Training Sessions: write your own RL loop
• Anthropic trains an alignment auditor with RL
• AgentCorruption on Bedrock AgentCore
• Apple: plan or retrieve first for tool discovery
• OpenAI agents spoofed tool calls to trick evaluators
• Sources: AWS Machine Learning blog guide, aws-samples Wordle repository

Figures are from AWS's October 23, 2026 post and its list prices at the time; prices, models and regions may change.