pointpainting_fusion#
Purpose#
The pointpainting_fusion is a package for utilizing the class information detected by a 2D object detection in 3D object detection.
Inner-workings / Algorithms#
The lidar points are projected onto the output of an image-only 2d object detection network and the class scores are appended to each point. The painted point cloud can then be fed to the centerpoint network.
Inputs / Outputs#
Input#
| Name | Type | Description |
|---|---|---|
input |
sensor_msgs::msg::PointCloud2 |
pointcloud |
input/camera_info[0-7] |
sensor_msgs::msg::CameraInfo |
camera information to project 3d points onto image planes |
input/rois[0-7] |
tier4_perception_msgs::msg::DetectedObjectsWithFeature |
ROIs from each image |
input/image_raw[0-7] |
sensor_msgs::msg::Image |
images for visualization |
Output#
| Name | Type | Description |
|---|---|---|
output |
sensor_msgs::msg::PointCloud2 |
painted pointcloud |
~/output/objects |
autoware_perception_msgs::msg::DetectedObjects |
detected objects |
~/debug/image_raw[0-7] |
sensor_msgs::msg::Image |
images for visualization |
Parameters#
Core Parameters#
| Name | Type | Default Value | Description |
|---|---|---|---|
encoder_onnx_path |
string | "" |
path to VoxelFeatureEncoder ONNX file |
encoder_engine_path |
string | "" |
path to VoxelFeatureEncoder TensorRT Engine file |
head_onnx_path |
string | "" |
path to DetectionHead ONNX file |
head_engine_path |
string | "" |
path to DetectionHead TensorRT Engine file |
build_only |
bool | false |
shutdown the node after TensorRT engine file is built |
trt_precision |
string | fp16 |
TensorRT inference precision: fp32 or fp16 |
post_process_params.score_threshold |
double | 0.4 |
detected objects with score less than threshold are ignored |
post_process_params.yaw_norm_thresholds |
list[double] | [0.3, 0.3, 0.3, 0.3, 0.0] | An array of distance threshold values of norm of yaw [rad]. |
post_process_params.iou_nms_target_class_names |
list[string] | ["CAR"] | An array of class names to be target in NMS. |
post_process_params.iou_nms_search_distance_2d |
double | 10.0 | A maximum distance value to search the nearest objects. |
post_process_params.iou_nms_threshold |
double | 0.1 | A threshold value of NMS using IoU score. |
post_process_params.has_twist |
boolean | false | Indicates whether the model outputs twist value. |
densification_params.world_frame_id |
string | map |
the world frame id to fuse multi-frame pointcloud |
densification_params.num_past_frames |
int | 1 |
the number of past frames to fuse with the current frame |
Assumptions / Known limits#
- The multi-frame painting is not implemented yet.
(Optional) Error detection and handling#
(Optional) Performance characterization#
References/External links#
[1] Vora, Sourabh, et al. "PointPainting: Sequential fusion for 3d object detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[2] CVPR'20 Workshop on Scalability in Autonomous Driving] Waymo Open Dataset Challenge: https://youtu.be/9g9GsI33ol8?t=535 Ding, Zhuangzhuang, et al. "1st Place Solution for Waymo Open Dataset Challenge--3D Detection and Domain Adaptation." arXiv preprint arXiv:2006.15505 (2020).