autoware_tensorrt_rtmdet#

Purpose#

RTMDet is a real-time instance segmentation model which can be used for detecting objects like cars, pedestrians, bicycles, etc. in a scene. This package provides a ROS 2 interface for RTMDet using TensorRT.

Inner-workings / Algorithms#

Inputs / Outputs#

Input#

Name	Type	Description
`in/image`	`sensor_msgs/Image`	The input image

Output#

Name	Type	Description
`out/objects`	`tier4_perception_msgs/DetectedObjectsWithFeature`	The detected objects with 2D bounding boxes and scores
`out/mask`	`autoware_internal_perception_msgs/SegmentationMask`	The instance segmentation mask
`out/color_mask`	`sensor_msgs/Image`	The colorized image of instance segmentation mask for visualization
`out/debug_image`	`sensor_msgs/Image`	The image with 2D bounding boxes for visualization

Parameters#

Name	Type	Description	Default	Range
is_publish_debug_image	boolean	If true, debug image which contain bounding boxes is published at runtime.	False	N/A
is_publish_color_mask	boolean	If true, debug image which contain colorful masks is published at runtime.	False	N/A
model_path	string	Path to ONNX file.	$(var data_path)/tensorrt_rtmdet/$(var model_name).onnx	N/A
color_map_path	string	Path to color map file.	$(var data_path)/tensorrt_rtmdet/color_map.csv	N/A
plugin_paths	array	Path to plugin files.	['./build/tensorrt_rtmdet/libtensorrt_rtmdet_plugin.so']	N/A
input_image_mean	array	Mean values for the model.	[103.53, 116.28, 123.675]	N/A
input_image_std_dev	array	Standard deviation values for model.	[57.375, 57.12, 58.395]	N/A
score_threshold	float	A threshold value of existence probability score, all of objects with score less than this threshold are ignored.	0.3	≥0.0 ≤1.0
nms_threshold	float	A threshold value of NMS.	0.0	≥0.0 ≤1.0
mask_threshold	float	A threshold value of pixel values in the output mask.	0.7	≥0.0 ≤1.0
precision	string	Operation precision to be used on inference. Valid value is one of: [fp32, fp16, int8].	fp16	N/A
dla_core_id	float	If positive ID value is specified, the node assign inference task to the DLA core.	-1	N/A
quantize_first_layer	boolean	If true, set the operating precision for the first (input) layer to be fp16. This option is valid only when precision==int8.	False	N/A
quantize_last_layer	boolean	If true, set the operating precision for the last (output) layer to be fp16. This option is valid only when precision==int8.	False	N/A
profile_per_layer	boolean	If true, profiler function will be enabled. Since the profile function may affect execution speed, it is recommended to set this flag true only for development purpose.	False	N/A
clip_value	float	If positive value is specified, the value of each layer output will be clipped between [0.0, clip_value]. This option is valid only when precision==int8 and used to manually specify the dynamic range instead of using any calibration.	0.0	N/A
apply_erosion	boolean	If true, apply erosion to the output mask.	False	N/A
calibration_image_list_path	string	Path to a file which contains path to images. Those images will be used for int8 quantization.		N/A

Assumptions / Known limits#

Onnx model#

A sample model is provided in autoware_data folder by ansible script on env preparation stage. If you cannot find the model, you follow instructions from the link to download the model.

The shared model was trained by open-mmlab using the COCO dataset. For more details, see link.

Package acceptable model generation#

Users can generate their own model using official RTMDet repository. Please refer to the official repository

Label file#

A sample label file (named label.txt) and instance segmentation color map file (name color_map.csv) are also downloaded automatically during env preparation process.

These files are used to map the class index to class name and color respectively.

Reference repositories#

https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet#rtmdet-an-empirical-study-of-designing-real-time-object-detectors

Citation#

@misc{lyu2022rtmdetempiricalstudydesigning,
title={RTMDet: An Empirical Study of Designing Real-Time Object Detectors},
author={Chengqi Lyu and Wenwei Zhang and Haian Huang and Yue Zhou and Yudong Wang and Yanyi Liu and Shilong Zhang and Kai Chen},
year={2022},
eprint={2212.07784},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2212.07784},
}