AI sustainability discussions usually compare a giant model with a small model once. Production systems do something messier: they send every task to a default model, attach too much context, retry failures, call tools, and sometimes escalate to a stronger model anyway.
That means the useful unit is not watt-hours per answer. It is energy per successfully completed task.
Recent research on task-aware model selection reports a 27.8% reduction in energy for a 3.9% utility trade-off in its evaluated setting. Smaller models can be dramatically more efficient, and aggressive quantization can cut energy further. Meanwhile GPT-4-class inference prices have fallen from roughly $30 per million tokens toward less than $0.50 for some comparable classes and workloads over two years.
These are not separate stories. Cost and energy fall along much of the same efficiency curve.
The core equation
For a routed AI workflow:
text
task energy = Σ(attempt tokens × energy per token on chosen hardware)
+ tool and retrieval energy
+ idle and facility overhead allocated to the task
The number that matters is:
text
energy per success = total workflow energy / accepted completed tasks
A 4B model may consume one-tenth the energy per attempt but solve only half the tasks. If every failure is rerun on a 100B-class model with the entire conversation, the routing policy can erase the initial savings.
A worked 10,000-task month
Consider a support-analysis workflow with 10,000 cases. These units are illustrative energy indices, not claims about a named vendor.
Policy A: strongest model for everything
10,000 attempts
10 energy units per attempt
95% accepted without repair
Total: 100,000 energy units
Accepted tasks: 9,500
Energy per immediate success: 10.53 units
Policy B: small model, then full fallback
10,000 small attempts at 2 units = 20,000
30% fall back to strong model: 3,000 × 10 = 30,000
Total: 50,000 units
If combined acceptance reaches 94%, energy per success = 5.32 units
That is roughly half the energy with nearly the same accepted-task rate.
4,000 retries with duplicated context at 3 units = 12,000
3,500 full fallbacks at 12 units = 42,000
Total: 74,000 units
Routing still helps, but poor context and retry design surrender almost half the available saving. The harness matters as much as the model menu. See how AI agents work end to end for the loop mechanics.
What the 27.8% result means
The 2025–2026 model-selection research evaluated whether tasks could be allocated among models based on expected utility and energy. Its reported 27.8% reduction and 3.9% utility cost show a real frontier: many requests are over-served by the default strongest model.
It does not prove that every company can switch models and save exactly 27.8%. Results depend on task distribution, scoring, models, hardware, traffic, and the value assigned to quality. The durable finding is that routing creates an optimization opportunity larger than many facility-level tweaks.
The study also estimated enormous system-level savings—tens of terawatt-hours under its assumptions—because a modest percentage compounds across global demand.
Why small models can be 65.8% more efficient
A smaller model generally moves fewer weights through memory and performs fewer operations per generated token. It can fit on cheaper hardware, batch more requests, and return faster. Research reporting efficiency advantages around 65.8% captures those benefits in a particular comparison.
But parameter count is not enough. A smaller model can lose if it:
writes twice as many tokens;
needs repeated prompting;
cannot use a tool reliably;
produces errors requiring human repair;
runs on poorly utilized hardware;
causes most requests to escalate.
This is why our open-weight versus closed-model framework evaluates task fit, deployment, and operations rather than treating “small” or “open” as automatically superior.
Quantization: less precision, less movement
Quantization stores and computes model values at lower precision. Moving four-bit values can require substantially less memory bandwidth than sixteen-bit values, which often reduces both latency and energy.
A 2026 study of AI energy and quality reported an NVFP4 configuration using 41% less energy with a 2–4% quality loss in the tested settings. That is a useful frontier, not a promise. Some tasks tolerate the loss; exact arithmetic, coding edge cases, and rare knowledge may not.
For local deployment, quantization can be the difference between a model fitting in laptop memory and failing to load. Our top ten laptop-ready open-weight models includes realistic memory bands and warns against equating “runs” with “runs well.”
Prompt and context are energy decisions
Teams often optimize the model while sending a 100,000-token context to every turn. In an agent loop, the history may be resent five or ten times. Cached input can reduce price and some repeated computation, but it does not make oversized context operationally free.
Let a task use:
50,000 input tokens per turn;
2,000 output tokens;
six agent turns.
The workflow processes 300,000 input tokens, not 50,000. If the agent appends tool output each turn, later prompts can be larger still. Trimming retrieved documents, summarizing state, and storing structured memory often saves more than switching two similarly sized models.
Analysts tracking API markets estimate that GPT-4-class inference fell from about $30 per million tokens to below $0.50 in some comparable offerings over roughly two years—a decline above 95%.
Price is not a calibrated energy meter. Vendors discount strategically, cross-subsidize products, depreciate chips, and charge different margins. Quality also changes, so “GPT-4 class” is an approximate capability band.
Still, the drivers overlap:
better accelerators produce more tokens per watt;
quantization reduces memory and compute;
distillation moves capability into smaller models;
batching improves utilization;
kernels and speculative decoding reduce wasted work;
competition passes some efficiency gains to customers.
Cheaper and greener are not identical, but they are often siblings.
A practical routing policy
Do not route by vague labels such as “easy.” Build a measurable ladder.
Step 1: define failure cost
Classify tasks by reversibility and review. Drafting an internal summary can tolerate more error than approving a payment or producing legal advice.
Step 2: create representative evals
Include common cases, long-tail inputs, adversarial instructions, tool failures, and data-access boundaries. Public benchmarks cannot substitute for your workflow.
Step 3: start at the cheapest qualified tier
Choose the smallest model that meets the acceptance threshold, not the smallest available model.
Step 4: escalate on observable signals
Escalate for low confidence, failed schema validation, missing citations, failed tests, or an independent verifier—not because the model says “this is hard.”
Step 5: preserve a compact task state
Send the necessary record, not the entire transcript. Cache stable prefixes where provider terms and privacy permit.
Step 6: measure the entire cascade
Attribute tokens, retries, tool calls, latency, human review, and accepted results to the original task.
A model-selection scorecard
Metric
Why it belongs
Task acceptance rate
Captures useful quality
Cost per accepted task
Prevents cheap failures looking efficient
Tokens per accepted task
Exposes context and verbosity waste
Fallback rate
Tests routing quality
Human repair minutes
Captures shifted labor
P50/P95 latency
Shows operational experience
Energy per accepted task
Connects workload and infrastructure
Data/privacy fit
Prevents unsafe optimization
If direct energy telemetry is unavailable, use GPU-seconds, hardware class, and utilization as a proxy. Label estimates honestly rather than publishing a universal “liters per prompt” number.
The organizational trap
Cloud invoices often belong to engineering, sustainability reports to facilities, and model quality to product. Nobody owns the combined denominator. One team celebrates cheaper tokens while retry volume rises; another buys renewable certificates while every request hits the largest model.
Give a single owner the task-level scorecard. Procurement should expose prices, engineering should expose traces, product should define acceptance, and sustainability should translate compute into energy with documented assumptions.
Verdict
The fastest energy source is the computation you do not need. Right-sizing a model, reducing repeated context, and stopping failed loops can free capacity immediately, while a new power plant takes years.
Do not downgrade high-stakes work to manufacture an energy claim. Route low-risk and well-evaluated tasks to efficient models, preserve escalation, and measure success after retries. That approach turns sustainability from a brand promise into ordinary systems engineering.
A safe first optimization experiment
Take one low-risk, high-volume workflow and randomly route half its traffic to the current default. Route the other half through a smaller qualified model with a strong-model fallback triggered only by schema failure, missing evidence, or a deterministic verifier. Keep prompts, tools, hardware region, and success criteria constant.
After at least a representative business cycle, compare accepted task rate, fallback rate, tokens, cost, latency, and repair time. If direct power telemetry exists, allocate GPU or accelerator energy to the experiment; otherwise label GPU-seconds and hardware class as proxies. Do not convert a rough proxy into an impressive but fictional carbon figure.
The experiment succeeds only if the smaller route preserves the agreed quality floor. A 30% energy saving paired with a flood of silent errors is a failed product change. A 15% saving with equal acceptance, faster latency, and fewer expensive fallbacks is a durable improvement that can expand to the next workflow.
Energy and utility figures describe specific studies and assumptions, not universal model rankings. Hardware, traffic, software, facility overhead, task mix, and evaluation method materially change results. Measure your own accepted-task frontier before making procurement or sustainability claims.
