RF-DETR: Roboflow's Real-Time Detection Transformer, Explained

Roboflow's RF-DETR crossed 8,800 GitHub stars and landed a spot at ICLR 2026 — not bad for a detection model whose core pitch is unglamorous: real-time, accurate, and fine-tunable on your own data without a research team. It's a DINOv2-backed transformer that now covers three tasks — object detection, instance segmentation, and keypoint detection — in one consistent Python API, and it's actively maintained, with a perf fix to mask upsampling shipping just 11 hours before this writeup.

If you've been evaluating real-time detectors against the YOLO family or RT-DETR, here's what RF-DETR actually is, how it benchmarks, and how to get it running.

TL;DR

What RF-DETR actually is

RF-DETR stands apart from the CNN-based YOLO lineage by building on a DINOv2 vision transformer backbone — a self-supervised pretrained encoder from Meta AI — instead of training detection features from scratch. Roboflow's architecture builds on prior work from LW-DETR, DINOv2, and Deformable DETR, and the team credits all three explicitly in the repo.

The practical result: one package, rfdetr, exposes a consistent API across three computer vision tasks:

• Object detection — six sizes, Nano through 2XLarge
• Instance segmentation — the same six sizes, with mask output
• Keypoint detection (preview) — a single COCO-person-keypoints-pretrained model

That consistency matters if you're deploying across a pipeline that needs more than boxes — e.g., detection for triage plus segmentation for precise masking — without switching model families or reworking your inference code.

Benchmarks: COCO and RF100-VL, not just COCO

A detail worth flagging for anyone who's been burned by vendor-reported numbers: Roboflow says all COCO accuracy figures are measured in-house, computed with pycocotools across the full 5,000-image val2017 split, using their own benchmarking harness (roboflow/sab) — so every row in their tables is directly comparable to every other row. Only rows marked with a dagger (†) are pulled from original authors' papers rather than re-measured. Latency numbers all come from an NVIDIA T4, TensorRT, FP16, batch size 1.

They also benchmark on RF100-VL, Roboflow's multi-domain real-world dataset collection — a meaningful signal, since COCO-only benchmarks can flatter models that overfit to COCO's specific object distribution and don't generalize to industrial, medical, or aerial imagery.

Detection sizes

△ Requires pip install rfdetr[plus].

Segmentation sizes

Notably, every segmentation size — including XL and 2XL — stays under Apache 2.0, unlike detection where XL/2XL move to PML 1.0.

Keypoint detection (preview)

Roboflow benchmarks keypoints against YOLO26-pose and YOLO11-pose on MS COCO, reporting AP50:95 under the OKS (Object Keypoint Similarity) metric — the standard for pose comparisons.

How the license actually splits

This trips people up, so it's worth stating plainly: licensing is per-component, not per-repo.

• rfdetr (the core package) → Apache 2.0
• Detection: Nano, Small, Medium, Large → Apache 2.0
• Detection: XLarge, 2XLarge → PML 1.0 (requires rfdetr_plus / pip install rfdetr[plus])
• Segmentation: all six sizes, Nano through 2XLarge → Apache 2.0
• Keypoint preview → Apache 2.0

If you need a fully Apache 2.0 pipeline and don't need the top two detection tiers, you never have to touch the Plus extension at all — Medium or Large detection plus any segmentation size covers most production use cases already at 73–75+ AP50.

Running RF-DETR

Installation is a single pip install rfdetr on Python 3.10+. Detection:

import supervision as sv
from rfdetr import RFDETRMedium
from rfdetr.assets.coco_classes import COCO_CLASSES

model = RFDETRMedium()
detections = model.predict("https://media.roboflow.com/dog.jpg", threshold=0.5)

labels = [f"{COCO_CLASSES[class_id]}" for class_id in detections.class_id]
annotated_image = sv.BoxAnnotator().annotate(detections.metadata["source_image"], detections)
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)

Segmentation swaps in a RFDETRSeg* class and sv.MaskAnnotator():

from rfdetr import RFDETRSegMedium
model = RFDETRSegMedium()
detections = model.predict("https://media.roboflow.com/dog.jpg", threshold=0.5)

Keypoints:

from rfdetr import RFDETRKeypointPreview
model = RFDETRKeypointPreview()
key_points = model.predict("image.jpg", threshold=0.5)

One gotcha worth noting: COCO_CLASSES only resolves names for COCO-pretrained checkpoints. Once you fine-tune on a custom dataset, switch to detections.data["class_name"] instead — it reads class names directly from the checkpoint and works for both COCO and custom label sets.

RF-DETR also ships an rfdetr-* alias family (rfdetr-nano, rfdetr-medium, etc.) for use through Roboflow's Inference package, if you'd rather not manage the model classes directly.

Fine-tuning and NAS on the Roboflow platform

RF-DETR's published sizes weren't hand-picked — they came out of a neural architecture search (NAS) process, and Roboflow has since improved that NAS pipeline and exposed it directly on the platform. Instead of choosing from six fixed sizes, a single NAS training run searches for the architecture best suited to your dataset, and Roboflow reports it now beats both the original paper's NAS results and the open-source named checkpoints on RF100-VL's accuracy-latency Pareto frontier.

Training paths:

1. Google Colab — a guided notebook for quick fine-tuning experiments.
2. Roboflow platform — managed training, including the NAS mode, with a documented step-by-step video walkthrough.

Should you use RF-DETR over YOLO or RT-DETR?

RF-DETR's core argument is the accuracy-latency Pareto position: Roboflow's own benchmarks show it beating comparable real-time YOLO and RT-DETR variants at matched latency on both COCO and RF100-VL. That's a meaningful claim specifically because RF100-VL exists — a lot of "beats YOLO" claims in this space are COCO-only, and COCO's object distribution doesn't necessarily transfer to a warehouse camera feed or a drone survey.

That said, benchmarks measured in-house by the model's own creator — even with a published, open-source harness like roboflow/sab — are not a substitute for testing on your own data and hardware target. If you're choosing a detector for production, the move is still: pull the size that fits your latency budget, fine-tune on a representative slice of your actual dataset, and compare against your current baseline directly.

For teams already running detection at the edge, RF-DETR's Nano and Small sizes (2.3ms–3.5ms on a T4) are worth pairing against setups like Frigate NVR, which currently leans on TensorFlow-based object detection, or dedicated inference hardware covered in our Coral Edge AI guide. Both are areas where a faster, more accurate open-weight detector directly changes what's deployable on constrained hardware.

If your use case sits downstream of detection — you don't need boxes, you need to act on what's found, by blurring it, erasing it, or swapping it out — RF-DETR-Seg's per-instance masks are exactly the kind of precise, per-object selection that tools like BGBlur.com build on for blurring, removing, or replacing any object in a video without hand-rotoscoping every frame.

Related reading

• Frigate NVR: open-source AI camera system for Home Assistant
• Coral Edge AI: Google's edge computing platform
• AI vs machine learning vs deep learning guide
• Go open source AI: Fortune 500 guide
• Google Gemma 4 12B: multimodal local AI guide
• RF-DETR on GitHub
• RF-DETR paper (arXiv:2511.09554)
• RF-DETR documentation

Benchmarks, licensing terms, and package details reflect the RF-DETR repository as of August 1, 2026 (v1.9.0). Check the releases page for the latest version before deploying.