CoreWeave has opened a private preview of Model Distillation inside its Forge platform, a managed way to turn your own production traffic into a smaller, cheaper specialist model. According to CoreWeave's Model Distillation page, you collect examples from real usage, train candidate models on an open-weight base with serverless supervised fine-tuning, compare them with your current model on held-out traffic, and deploy gradually with rollback available. The company's own post around the launch says the point is that one job stops costing frontier prices.
The practical questions are what it costs, what it can and cannot do, and the legal edge of distillation, which has become politically charged this year.
TL;DR: the questions people will ask
| Question | Answer |
|---|---|
| What is it? | Managed workflow: capture traffic, curate examples, train candidates, compare, deploy gradually |
| Status | Private preview in Forge; access by request form |
| Training method | Serverless supervised fine-tuning (SFT) on an open-weight base model |
| Price | No separate charge; Serverless SFT at $2.70 per GPU-hour plus inference, Agent Lens, W&B Models and storage |
| Teacher model | Not named on CoreWeave's pages; the worked example used GPT-5.6 Sol |
| Interface | Chat Completions-compatible connection for one AI feature |
| Safety net | Gradual rollout, previous model kept for rollback |
How the workflow runs
CoreWeave's docs describe a six-step journey:
- Connect one AI feature to Model Distillation through a Chat Completions-compatible interface.
- Collect examples from real product usage.
- Choose and refine the examples and answers worth learning from.
- Train several candidate specialized models.
- Compare them against your current model on the same examples.
- Deploy the best one gradually, keeping the previous model available for rollback.
In CoreWeave's launch blog on closing the loop between inference and post-training, traffic is captured in CoreWeave Agent Lens and converted into training data while the application runs. Teams can relabel weak outputs, try different base models and hyperparameters, and deploy candidates to Serverless Inference to compare against production. The pattern fits best, the post says, when a stronger model already produces the desired behavior and you want a smaller or cheaper model to reproduce it.
Large vessel pouring a stream into a small one, illustrating CoreWeave Model Distillation of a big model into a specialist
The worked example
CoreWeave's example customer, Method, originally trained a Llama 3.1 8B model on GPT-4o outputs in 2024. In August 2026 it relabeled fresh production data with GPT-5.6 Sol and retrained; the new model won 57.5% of head-to-head evaluations against the production version. That is a vendor-published customer result, and 57.5% is a modest win rate, which is itself informative: distillation gives incremental gains on a narrow job, not magic.
What it costs
CoreWeave says there is no separate fee for Model Distillation. You pay for the pieces it uses:
- Serverless SFT: $2.70 per GPU-hour, prorated by active training time.
- Serverless Inference for running candidates and the final model.
- Agent Lens for capturing and analyzing traffic.
- Weights & Biases Models for experiment tracking.
- Artifact storage.
For a narrow classification or extraction task with a few thousand examples, training time is typically short, so the main ongoing cost is serving the specialist model. The saving case rests on swapping a frontier model's per-token price for a small model's much lower one, which only works if volume is high and the task is stable. Compare against the strategies in our guides to AI token pricing and prompt caching, which can sometimes give a similar saving without training anything.
A sizing example
Suppose a support product sends 2 million requests a month through a frontier model to sort tickets into twelve categories and draft a one-line summary. The task is narrow, the output format is fixed, and a stronger model already does it well. That is the textbook candidate. You would collect a few thousand logged examples, have the strong model relabel any weak ones, train two or three candidate specialists on small open-weight bases, and compare them on held-out tickets against the current production model.
The training bill is small relative to the serving bill, because training happens once while serving repeats every month. The decision therefore turns on three numbers you can measure: the specialist's accuracy relative to the frontier model on your own held-out traffic, the per-request cost and latency of serving it, and how often you will need to retrain as ticket types drift. If the specialist matches the frontier model on, say, nine out of ten tickets and you route the uncertain tenth back to the big model, you keep most of the quality and cut most of the cost. CoreWeave's gradual rollout and rollback features exist for exactly that routing and safety pattern.
How to evaluate a distilled specialist
Do not trust a single aggregate score. Slice the held-out set by category, language, input length and customer segment, and look at the worst slice rather than the average. Check calibration: a specialist that is confidently wrong is more dangerous than a frontier model that hedges. Test with inputs it has never seen, including adversarial or malformed ones, because narrow models can fail in strange ways outside their training distribution. Finally, keep a small shadow set that you label by hand every month and compare against both models, so you notice drift before your users do.
Distillation versus the alternatives
Distillation is one of several ways to cut inference cost, and it is not always the first one to try. Prompt caching and shorter prompts can remove a large share of spend without any training. Routing sends easy requests to a cheap model and hard ones to a strong model, which needs no specialist at all. Fine-tuning a small model on human-labeled data avoids the teacher-license question entirely, at the price of labeling effort. Distillation sits in the middle: it is attractive when you already have a strong model producing the behavior you want and want to bake it into something faster. Treat it as an optimization for a stable, high-volume task, not a default.
The rest of the Forge announcement
Alongside distillation, CoreWeave announced two other pieces at Fully Connected 2026:
- Programmable training API (limited preview, launch planned later this year): you write the training loop in Python and CoreWeave manages compute. The example in the post combines reinforcement learning with periodic supervised learning on verified demonstrations.
- RL Rollouts on CoreWeave Inference: built on NVIDIA Dynamo with vLLM support, it hot-loads checkpoints onto live deployments without disrupting in-flight requests. In an NVIDIA and you.com validation using Nemotron 3.5 Lightning on web search, BrowseComp accuracy rose from 36.97% to 45.45%, average tool calls fell 30.24%, and checkpoint hot-loading was about 15 times faster than redeploy cycles. These are vendor-reported figures.
The legal question: whose outputs are you training on?
Distillation is also a political topic. In recent weeks we have covered accusations that Chinese labs were serving Claude outputs, Jensen Huang's comments on distillation and competition, and Garry Tan's call for an American distillation regime. The common thread is that terms of service often forbid using a model's outputs to train a competing model, and some governments are considering rules.
CoreWeave's pages do not name a supported teacher model or address model license terms. Its one worked example, Method, relabeled data with GPT-5.6 Sol, a closed model. Some open-weight families, such as Z.ai's GLM line (see our GLM-5.3 coverage), have been released under permissive licenses, but you should read the specific license and any acceptable-use terms of whichever model you use as a teacher. If your teacher is a closed API, the terms may prohibit exactly this.
Fine-tuning chisel shaping a block, representing training a smaller specialist model with CoreWeave serverless SFT
When distillation is the right tool
It is a good fit when:
- One narrow task dominates your spend, such as classification, routing, extraction or a fixed-format reply.
- You have production examples of good behavior, or a stronger model that can label them.
- Latency and cost matter more than open-ended capability.
It is a poor fit when the task changes weekly, when you lack enough representative traffic, or when quality needs the full reasoning of a frontier model. Specialists can fail silently outside their training distribution, so keep a monitor and a fallback. For research on where distillation goes wrong, see our write-up on on-policy distillation collapse and reward hacking, and for a small-model success story, Bespoke's Nimble 9B.
What to do if you want in
- Request preview access through the form on CoreWeave's Forge page.
- Pick one stable feature with at least a few thousand logged examples.
- Check the teacher model's license before you generate or relabel data.
- Define success against held-out traffic, including cost and latency, before training.
- Roll out gradually and keep the old model as a fallback, as the workflow itself recommends.
Related reading
- AI token pricing explained
- Prompt caching and LLM cost optimization
- Chinese labs serving Claude and distillation claims
- Jensen Huang on distillation and competition
- Garry Tan on an American distillation regime
- On-policy distillation collapse
- Z.ai GLM-5.3 coding release
- Source: Closing the loop between inference and post-training
Details are accurate as of October 12, 2026 and come from CoreWeave's pages; preview terms and pricing may change.