Tencent Hy3 GGUF — 1-Bit and 4-Bit Quants for Single-GPU llama.cpp

On July 14, 2026, Tencent Hunyuan posted the follow-up many local-LLM builders waited for after Hy3's July 6 launch:

"We've just released the 1-bit & 4-bit version of Hy3, a flagship-scale 295B model that can be served on a single GPU. Run Hy3 with llama.cpp, enable MTP, and experience powerful intelligence on dramatically lower hardware."

The thread (~14K views in its first hours) quoted the original July 6 launch — 295B MoE, Apache 2.0, 256K context, free OpenRouter window — and added the missing piece from that post's self-hosting section: you no longer need 8× H20 GPUs for every inference path.

Critical nuance upfront: single GPU here means 128 GB unified-memory class — DGX Spark, Mac Studio 128 GB, Strix Halo — not a 16 GB RTX 3060. One viral reply promised "can't wait to run this on my 3060 and 16GB RAM." That is not what Tencent's memory math supports.

TL;DR — what people are asking

Why July 14 matters — Hy3 goes from datacenter to desk

When Hy3 launched on July 6, explainx.ai's read was blunt: 299B BF16 parameters need 8-GPU tensor parallel via vLLM or SGLang. The 21B active MoE step is cheap per token; weight storage is not.

Eight days later, Tencent closes the gap:

That places Hy3 in the same July 2026 conversation as GLM-5.2 MIT open weights and Colibrì's disk-streaming GLM path — but Hy3's pitch is interactive tok/s on one big memory pool, not 0.05 tok/s on 25 GB RAM.

Hardware reality — what "single GPU" actually means

X thread questions answered

@adidshaft asked the right question: which card, context length, and tok/s for 1-bit?

@4xdotrip asked about Hy3 on a single DGX Spark — yes, that is the reference class. Community quant author vcruz305 benchmarked IQ2_M on Spark GB10: 18.0 → 22.8 tok/s with MTP (+27%, 90% draft acceptance).

@Macchilust's 3060 + 16 GB plan: treat as API-only via OpenRouter free route or smaller dense models — not this MoE.

llama.cpp stack — hy_v3 architecture + MTP

Hy3 uses a new hy_v3 architecture (295B MoE, 80 decoder layers + 1 MTP/NextN layer, 192 experts top-8). Mainline llama.cpp added support via PR #25395 — base graph, MoE router, and draft-mtp speculative head.

Build (until merged to release tag)

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/25395/head:hy3-mtp && git checkout hy3-mtp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j"$(nproc)"

Apple Silicon: swap -DGGML_CUDA=ON for -DGGML_METAL=ON.

Serve with MTP (copy-paste)

./build/bin/llama-server \
  -m /path/to/Hy3-MTP-IQ2_M.gguf \
  --host 0.0.0.0 --port 8080 \
  --ctx-size 32768 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-p-min 0.75 \
  --parallel 1 \
  --jinja

Why --spec-draft-p-min 0.75 is mandatory: Hy3's MTP head is single-depth trained. At default p-min, draft acceptance collapses (~39%) and speculation becomes slower than baseline. At 0.75, acceptance jumps to ~88–97% and decode wins appear.

Other gotchas from community testing:

• --jinja required — chat template aborts without it
• --parallel 1 required for draft-mtp on Hy3
• First load of 100 GB+ quants: ~7–8 minutes before first token — do not kill early
• Stop string: add <｜hy_eos:opensource｜> — occasional EOG leak without it
• Tool calling: native OpenAI-style tool endpoints may 500; inject tools in prompt until parser lands

OpenAI-compatible smoke test

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="hy3",
    messages=[{"role": "user", "content": "Write a FastAPI health check route."}],
    temperature=0.9,
    top_p=1.0,
)
print(r.choices[0].message.content)

Wire the same baseURL into OpenCode local model config or Kilo Code / Cline — Hy3's July 6 launch targeted agent scaffolds specifically.

Quant ladder — 1-bit vs 4-bit trade-offs

Tencent's announcement names 1-bit and 4-bit; Hugging Face's quant tree (46+ variants as of July 14) fills the rungs:

MoE quantization insight: 192 routed experts are ~97% of parameters but only 8 activate per token — quant recipes put the bit budget on experts while keeping attention, shared expert, and output head at Q6–Q8. That is why aggressive IQ rungs remain usable.

Compare to GLM-5.2 Unsloth 2-bit at ~245 GB minimum — Hy3's smaller total parameter count wins the "fits on one desk" race even though GLM-5.2 leads some coding leaderboards.

Hy3 GGUF vs alternatives — July 2026 local stack

For MacBook vs dedicated GPU shoppers: Hy3 GGUF is the first frontier MoE that treats 128 GB Apple Silicon as a first-class target — not an afterthought.

For Fable 5 local hardware projections: closed-weight parity may lag years; Hy3 GGUF is proof Chinese open weights keep compressing the desk-scale window.

Agent and product context

Hy3's July 6 positioning was agent reliability — SWE-bench variance ≤4% across CodeBuddy, Cline, and KiloCode scaffolds. The GGUF release does not change model weights; it changes who can run those agents locally without routing prompts through Shenzhen-hosted APIs.

OpenClaw v2026.7.1 already ships Hy3 via TokenHub. Self-hosters can now mirror that stack on llama-server for air-gapped labs — same China AI playbook thesis: open weights + commodity inference.

Compliance reminder: Apache 2.0 clears license friction; it does not clear data-sovereignty review for regulated workloads — same framing as Asian AI alternatives.

Who should download today

Do it if:

• You own 128 GB unified memory and want a 295B-class agent offline
• You are A/B testing Hy3 vs GLM-5.2 on identical prompts
• You need 256K context locally for repo-scale agents (watch KV cache math — 32K comfortable on 128 GB)

Skip if:

• You have ≤24 GB VRAM — use hosted API or dense 27B models
• You need production tool-call parsers today — stay on vLLM 8-GPU recipes until llama.cpp Hy3 tool support matures
• You require maximum coding benchmark scores — GLM-5.2 may still edge Hy3 on your eval

Summary

On July 14, 2026, Tencent released 1-bit and 4-bit Hy3 GGUF quants aimed at single-GPU serving through llama.cpp + MTP — turning a model that launched on 8 GPUs into a 128 GB desk-scale option. The X thread's excitement is justified on DGX Spark and 128 GB Mac hardware; it is not a free pass for 16 GB consumer cards.

The load-bearing flags: --spec-type draft-mtp, --spec-draft-p-min 0.75, --parallel 1, --jinja. Get those right and community benchmarks show ~+27–40% tok/s over non-speculative decode. Get them wrong and MTP becomes a tax.

Related on explainx.ai

• PrismML Bonsai 27B — HN harness reality + Gemma 4 12B QAT compare (Jul 2026) — dense 27B at 3.9 GB vs Hy3 MoE at 128 GB
• Tencent Hy3 launch — 295B MoE, Apache 2.0, agent benchmarks
• What is llama.cpp — install, GGUF, llama-server
• GLM-5.2 MIT open source — Code Arena #2
• Colibrì — GLM-5.2 on 25 GB RAM via disk streaming
• Unsloth GLM-5.2 — 256 GB Mac path
• MacBook vs dedicated GPU for local LLMs
• OpenClaw 2026.7.1 — Hy3 TokenHub integration
• China AI playbook — free models, cheap compute

Sources: Tencent Hunyuan X post, Jul 14 2026 · tencent/Hy3 on Hugging Face · llama.cpp PR #25395 — hy_v3 + MTP · YanissAmz/Hy3-295B-A21B-GGUF · satgeze/Hy3-1M-GGUF quant ladder · Hy3 research page

Quant sizes, tok/s benchmarks, and llama.cpp PR merge status reflect community reports as of July 14, 2026. Verify GGUF shard dates and arch metadata (hy_v3) before multi-hour downloads. Free OpenRouter promotions from the July 6 launch may expire — check current pricing.