RF-DETR: Roboflow's Real-Time Detection Transformer, Explained

Roboflow's RF-DETR crossed 8,800 GitHub stars and landed a spot at ICLR 2026 — not bad for a detection model whose core pitch is unglamorous: real-time, accurate, and fine-tunable on your own data without a research team. It's a DINOv2-backed transformer that now covers three tasks — object detection, instance segmentation, and keypoint detection — in one consistent Python API, and it's actively maintained, with a perf fix to mask upsampling shipping just 11 hours before this writeup.

If you've been evaluating real-time detectors against the YOLO family or RT-DETR, here's what RF-DETR actually is, how it benchmarks, and how to get it running.

TL;DR

What RF-DETR actually is

RF-DETR stands apart from the CNN-based YOLO lineage by building on a DINOv2 vision transformer backbone — a self-supervised pretrained encoder from Meta AI — instead of training detection features from scratch. Roboflow's architecture builds on prior work from LW-DETR, DINOv2, and Deformable DETR, and the team credits all three explicitly in the repo.

The practical result: one package, rfdetr, exposes a consistent API across three computer vision tasks:

• Object detection — six sizes, Nano through 2XLarge
• Instance segmentation — the same six sizes, with mask output
• Keypoint detection (preview) — a single COCO-person-keypoints-pretrained model

That consistency matters if you're deploying across a pipeline that needs more than boxes — e.g., detection for triage plus segmentation for precise masking — without switching model families or reworking your inference code.

Benchmarks: COCO and RF100-VL, not just COCO

A detail worth flagging for anyone who's been burned by vendor-reported numbers: Roboflow says all COCO accuracy figures are measured in-house, computed with pycocotools across the full 5,000-image val2017 split, using their own benchmarking harness (roboflow/sab) — so every row in their tables is directly comparable to every other row. Only rows marked with a dagger (†) are pulled from original authors' papers rather than re-measured. Latency numbers all come from an NVIDIA T4, TensorRT, FP16, batch size 1.

They also benchmark on RF100-VL, Roboflow's multi-domain real-world dataset collection — a meaningful signal, since COCO-only benchmarks can flatter models that overfit to COCO's specific object distribution and don't generalize to industrial, medical, or aerial imagery.

Detection sizes

△ Requires pip install rfdetr[plus].

Segmentation sizes

Notably, every segmentation size — including XL and 2XL — stays under Apache 2.0, unlike detection where XL/2XL move to PML 1.0.

Keypoint detection (preview)

Roboflow benchmarks keypoints against YOLO26-pose and YOLO11-pose on MS COCO, reporting AP50:95 under the OKS (Object Keypoint Similarity) metric — the standard for pose comparisons.

How the license actually splits

This trips people up, so it's worth stating plainly: licensing is per-component, not per-repo.

• rfdetr (the core package) → Apache 2.0
• Detection: Nano, Small, Medium, Large → Apache 2.0
• Detection: XLarge, 2XLarge → PML 1.0 (requires rfdetr_plus / pip install rfdetr[plus])
• Segmentation: all six sizes, Nano through 2XLarge → Apache 2.0
• Keypoint preview → Apache 2.0

If you need a fully Apache 2.0 pipeline and don't need the top two detection tiers, you never have to touch the Plus extension at all — Medium or Large detection plus any segmentation size covers most production use cases already at 73–75+ AP50.

Running RF-DETR

Installation is a single pip install rfdetr on Python 3.10+. Detection:

import supervision as sv
from rfdetr import RFDETRMedium
from rfdetr.assets.coco_classes import COCO_CLASSES

model = RFDETRMedium()
detections = model.predict("https://media.roboflow.com/dog.jpg", threshold=0.5)

labels = [f"{COCO_CLASSES[class_id]}" for class_id in detections.class_id]
annotated_image = sv.BoxAnnotator().annotate(detections.metadata["source_image"], detections)
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)

Segmentation swaps in a RFDETRSeg* class and sv.MaskAnnotator():

from rfdetr import RFDETRSegMedium
model = RFDETRSegMedium()
detections = model.predict("https://media.roboflow.com/dog.jpg", threshold=0.5)

Keypoints:

from rfdetr import RFDETRKeypointPreview
model = RFDETRKeypointPreview()
key_points = model.predict("image.jpg", threshold=0.5)

One gotcha worth noting: COCO_CLASSES only resolves names for COCO-pretrained checkpoints. Once you fine-tune on a custom dataset, switch to detections.data["class_name"] instead — it reads class names directly from the checkpoint and works for both COCO and custom label sets.

RF-DETR also ships an rfdetr-* alias family (rfdetr-nano, rfdetr-medium, etc.) for use through Roboflow's Inference package, if you'd rather not manage the model classes directly.

Fine-tuning and NAS on the Roboflow platform

RF-DETR's published sizes weren't hand-picked — they came out of a neural architecture search (NAS) process, and Roboflow has since improved that NAS pipeline and exposed it directly on the platform. Instead of choosing from six fixed sizes, a single NAS training run searches for the architecture best suited to your dataset, and Roboflow reports it now beats both the original paper's NAS results and the open-source named checkpoints on RF100-VL's accuracy-latency Pareto frontier.

Training paths:

1. Google Colab — a guided notebook for quick fine-tuning experiments.
2. Roboflow platform — managed training, including the NAS mode, with a documented step-by-step video walkthrough.

Should you use RF-DETR over YOLO or RT-DETR?

RF-DETR's core argument is the accuracy-latency Pareto position: Roboflow's own benchmarks show it beating comparable real-time YOLO and RT-DETR variants at matched latency on both COCO and RF100-VL. That's a meaningful claim specifically because RF100-VL exists — a lot of "beats YOLO" claims in this space are COCO-only, and COCO's object distribution doesn't necessarily transfer to a warehouse camera feed or a drone survey.

That said, benchmarks measured in-house by the model's own creator — even with a published, open-source harness like roboflow/sab — are not a substitute for testing on your own data and hardware target. If you're choosing a detector for production, the move is still: pull the size that fits your latency budget, fine-tune on a representative slice of your actual dataset, and compare against your current baseline directly.

For teams already running detection at the edge, RF-DETR's Nano and Small sizes (2.3ms–3.5ms on a T4) are worth pairing against setups like Frigate NVR, which currently leans on TensorFlow-based object detection, or dedicated inference hardware covered in our Coral Edge AI guide. Both are areas where a faster, more accurate open-weight detector directly changes what's deployable on constrained hardware.

If your use case sits downstream of detection — you don't need boxes, you need to act on what's found, by blurring it, erasing it, or swapping it out — RF-DETR-Seg's per-instance masks are exactly the kind of precise, per-object selection that tools like BGBlur.com build on for blurring, removing, or replacing any object in a video without hand-rotoscoping every frame.

Update — August 30, 2026: Polymarket reposted Simon Weckert's Digital Camouflage shirt — a live demo of why detector benchmark scores are not the same as adversarial robustness.

Evaluate the detector inside the complete camera pipeline

Measure capture, preprocessing, inference, postprocessing, and display together on the intended device. A published model latency excludes work your application may still need. Test representative lighting, object sizes, occlusion, and camera angles instead of relying on a single attractive example.

For a fine-tuned model, inspect class labels and failure cases on held-out images before choosing the confidence threshold. Keep examples used for tuning separate from those used to report performance. Use the RF-DETR documentation for the supported training and export path, then compare the result against the baseline your application actually uses.

A useful rollout criterion combines quality, runtime, and recovery. Decide what happens when a frame is dropped, no object is returned, or the model selects the wrong class. Those application behaviors matter as much as the benchmark row when detection drives a visible product feature.

Related reading

• This Tennis AI Coach Was Built With Roboflow Agent and Claude Code — a fine-tuned RF-DETR model in a real hobbyist build
• Digital Camouflage — the adversarial shirt that beats AI person-detection cameras
• SweepLED: KAIST's $7 AI hidden camera detector — reflection-pattern analysis as an applied detection model
• Roboflow's VLM benchmark: GPT-5.6 Sol is OpenAI's best vision model — but still loses to Gemini 3.5 Flash
• Frigate NVR: open-source AI camera system for Home Assistant
• Coral Edge AI: Google's edge computing platform
• AI vs machine learning vs deep learning guide
• Go open source AI: Fortune 500 guide
• Google Gemma 4 12B: multimodal local AI guide
• RF-DETR on GitHub
• RF-DETR paper (arXiv:2511.09554)
• RF-DETR documentation

Benchmarks, licensing terms, and package details reflect the RF-DETR repository as of August 1, 2026 (v1.9.0). Check the releases page for the latest version before deploying.