fai-rtdetr-m-coco

Overview

The models is a RT-DETR model otimized by FocoosAI for the COCO dataset. It is a object detection model able to detect 80 thing (dog, cat, car, etc.) classes.

Benchmark

Note: FPS are computed on NVIDIA T4 using TensorRT and image size 640x640.

Model Details

The model is based on the RT-DETR architecture. It is a object detection model that uses a transformer-based encoder-decoder architecture.

Neural Network Architecture

The RT-DETR FocoosAI implementation optimize the original neural network architecture for improving the model's efficiency and performance. The original model is fully described in this paper.

RT-DETR is a hybrid model that uses three main components: a backbone for extracting features, an encoder for upscaling the features, and a transformer-based decoder for generating the detection output.

In this implementation:

• the backbone is STDC-2 that show an amazing trade-off between performance and efficiency.
• the encoder is a bi-FPN (bilinear feature pyramid network). With respect to the original paper, we removed the attention modules in the encoder, speeding up the inference while only marginally affecting the accuracy.
• the transformer decoder is a lighter version of the original, having only 3 decoder layers, instead of 6, and we select 300 queries.

Losses

We use the same losses as the original paper:

• loss_vfl: a variant of the binary cross entropy loss for the classification of the classes that is weighted by the correctness of the predicted bounding boxes IoU.
• loss_bbox: an L1 loss computing the distance between the predicted bounding boxes and the ground truth bounding boxes.
• loss_giou: a loss minimizing the IoU the predicted bounding boxes and the ground truth bounding boxes. for more details look here: GIoU.

These losses are applied to each output of the transformer decoder, meaning that we apply it on the output and on each auxiliary output of the transformer decoder layers. Please refer to the RT-DETR paper for more details.

Output Format

The pre-processed output of the model is set of bounding boxes with associated class probabilities. In particular, the output is composed by three tensors:

• class_ids: a tensor of 300 elements containing the class id associated with each bounding box (such as 1 for wall, 2 for building, etc.)
• scores: a tensor of 300 elements containing the corresponding probability of the class_id
• boxes: a tensor of shape (300, 4) where the values represent the coordinates of the bounding boxes in the format [x1, y1, x2, y2]

The model does not need NMS (non-maximum suppression) because the output is already a set of bounding boxes with associated class probabilities and has been trained to avoid overlaps.

After the post-processing, the output is a the output is a Focoos Detections object containing the predicted bounding boxes with confidence greather than a specific threshold (0.5 by default).

Classes

The model is pretrained on the COCO dataset with 80 classes.

Class ID	Class Name	AP
1	person	56.4
2	bicycle	32.8
3	car	44.2
4	motorcycle	47.9
5	airplane	72.5
6	bus	70.4
7	train	69.1
8	truck	39.5
9	boat	27.9
10	traffic light	26.6
11	fire hydrant	68.8
12	stop sign	64.4
13	parking meter	45.5
14	bench	26.0
15	bird	37.4
16	cat	73.4
17	dog	68.6
18	horse	59.5
19	sheep	56.2
20	cow	59.2
21	elephant	67.9
22	bear	77.9
23	zebra	71.4
24	giraffe	72.2
25	backpack	16.7
26	umbrella	43.4
27	handbag	16.6
28	tie	36.5
29	suitcase	44.4
30	frisbee	68.0
31	skis	27.5
32	snowboard	35.0
33	sports ball	46.5
34	kite	44.8
35	baseball bat	29.2
36	baseball glove	38.4
37	skateboard	56.2
38	surfboard	43.3
39	tennis racket	49.5
40	bottle	37.8
41	wine glass	35.7
42	cup	43.1
43	fork	39.0
44	knife	22.6
45	spoon	20.4
46	bowl	43.5
47	banana	27.1
48	apple	22.2
49	sandwich	38.8
50	orange	33.4
51	broccoli	24.7
52	carrot	23.8
53	hot dog	38.4
54	pizza	57.4
55	donut	50.6
56	cake	38.4
57	chair	30.9
58	couch	50.0
59	potted plant	28.9
60	bed	51.4
61	dining table	32.8
62	toilet	67.2
63	tv	59.4
64	laptop	62.7
65	mouse	64.7
66	remote	34.4
67	keyboard	55.8
68	cell phone	38.2
69	microwave	61.4
70	oven	41.8
71	toaster	48.3
72	sink	39.4
73	refrigerator	59.5
74	book	15.5
75	clock	49.0
76	vase	39.3
77	scissors	30.3
78	teddy bear	50.6
79	hair drier	4.8
80	toothbrush	30.2

What are you waiting? Try it!

from focoos import Focoos
import os

# Initialize the Focoos client with your API key
focoos = Focoos(api_key=os.getenv("FOCOOS_API_KEY"))

# Get the remote model (fai-rtdetr-m-coco) from Focoos API
model = focoos.get_remote_model("fai-rtdetr-m-coco")

# Run inference on an image
predictions = model.infer("./image.jpg", threshold=0.5)

# Output the predictions
print(predictions)