autoware_ptv3#

Purpose#

The autoware_ptv3 package is used for 3D lidar segmentation.

Inner-workings / Algorithms#

This package implements a TensorRT powered inference node for Point Transformers V3 (PTv3) [1]. The sparse convolution backend corresponds to spconv. Autoware installs it automatically in its setup script. If needed, the user can also build it and install it following the following instructions.

Inputs / Outputs#

Input#

Name	Type	Description
`~/input/pointcloud`	`sensor_msgs::msg::PointCloud2`	Input pointcloud topic.

Output#

Name	Type	Description
`~/output/pointcloud/segmentation`	`sensor_msgs::msg::PointCloud2`	`PointXYZCPE` cloud with class ID, probability, and entropy fields.
`~/output/pointcloud/visualization`	`sensor_msgs::msg::PointCloud2`	XYZ cloud with RGB field.
`~/output/pointcloud/filtered`	`sensor_msgs::msg::PointCloud2`	Filtered cloud in the requested `filter.output_format`.
`~/output/objects`	`autoware_perception_msgs::msg::DetectedObjects`	Detected 3D objects after score filtering and IoU NMS.
`debug/cyclic_time_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Cyclic time (ms).
`debug/pipeline_latency_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Pipeline latency time (ms).
`debug/processing_time/preprocess_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Preprocess (ms).
`debug/processing_time/inference_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Inference time (ms).
`debug/processing_time/postprocess_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Postprocess time (ms).
`debug/processing_time/total_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Total processing time (ms).

Segmentation point cloud format#

The ~/output/pointcloud/segmentation topic uses the naturally aligned PointXYZCPE point type. Its point_step is 24 bytes.

Field	Data type	Offset	Description
`x`	`FLOAT32`	0	X coordinate in meters.
`y`	`FLOAT32`	4	Y coordinate in meters.
`z`	`FLOAT32`	8	Z coordinate in meters.
`class_id`	`UINT8`	12	Predicted class encoded as `autoware::point_types::PointCloudClassification`; `INVALID` (255) represents an invalid label.
`probability`	`FLOAT32`	16	Probability of the predicted class.
`entropy`	`FLOAT32`	20	Shannon entropy normalized by the logarithm of the number of classes.

Bytes 13 through 15 are alignment padding. The output is intentionally not packed so that the floating-point fields remain naturally aligned on both the CPU and GPU.

Points with an invalid label (class_id = PointCloudClassification::INVALID, that is 255) have no class distribution: their probability is 0.0 and their entropy is NaN, which is the default value of autoware::point_types::PointXYZCPE::entropy and means "not available". Consumers must therefore guard against NaN before using the entropy field.

Parameters#

PTv3Node node#

Name	Type	Description	Default	Range
plugins_path	string	Path to the TensorRT plugins shared library.		N/A
trt_precision	string	How the TensorRT engines are built. "fp16" and "fp32" request a precision for a weakly typed build, letting the builder assign tensor precisions. "strongly-typed" takes every tensor precision from the model itself.		['fp16', 'fp32', 'strongly-typed']
cloud_capacity	integer	Point cloud buffer capacity. Must be at least the maximum expected number of points per frame.		≥1
encoder.onnx_path	string	Path to the encoder ONNX file.		N/A
encoder.engine_path	string	Path to the encoder TensorRT engine file.		N/A
encoder.workspace_size	integer	TensorRT builder memory-pool limit in bytes for this engine.	8589934592	≥1
encoder.voxels_num	array	Voxel ranges used during inference [min, opt, max].	[16000, 128000, 192000]	N/A
segmentation3d.use_head	N/A	Enable the segmentation head.		N/A
segmentation3d.onnx_path	string	Path to the segmentation head ONNX file.		N/A
segmentation3d.engine_path	string	Path to the segmentation head TensorRT engine file.		N/A
segmentation3d.source_reconstruction	string	Controls mapping of voxel predictions back to source points. 'none': voxel representative points; 'partial': cropped source points inside point_cloud_range; 'full': full input cloud with unknown label 255 for out-of-range points.		['none', 'partial', 'full']
segmentation3d.class_mapping	object	Mapping from each segmentation3d.class_names entry to a PointCloudClassification value. Keys must match class_names exactly and every class_names entry must be present.		N/A
segmentation3d.filter.classes	array	Class names to filter out from the filtered output cloud.		N/A
segmentation3d.filter.output_format	string	Filtered output point format. Empty string preserves the input format.		['', 'xyzi', 'xyzirc', 'xyziradrt', 'xyzircaedt']
segmentation3d.filter.apply_to_segmentation	boolean	Whether to also remove filter.classes from the segmentation pointcloud output.		N/A
segmentation3d.workspace_size	integer	TensorRT builder memory-pool limit in bytes for this engine.	17179869184	≥1
detection3d.use_head	N/A	Enable the detection head.		N/A
detection3d.onnx_path	string	Path to the detection head ONNX file.		N/A
detection3d.engine_path	string	Path to the detection head TensorRT engine file.		N/A
detection3d.post_process_params.iou_nms_search_distance_2d	float	Maximum 2D distance between object centers considered for IoU NMS.		>0.0
detection3d.post_process_params.iou_nms_threshold	float	IoU threshold above which the lower-scoring box is suppressed.		≥0.0 ≤1.0
detection3d.post_process_params.yaw_norm_thresholds	array	Per-class minimum sine/cosine norm for yaw. Boxes below threshold are discarded.		N/A
detection3d.detection_score_thresholds.distance_bin_upper_limits	array	Ordered upper bounds of radial distance bins in meters.		N/A
detection3d.detection_score_thresholds.min_confidence_scores	object	Per-class arrays of minimum confidence scores, one value per distance bin.		N/A
detection3d.allow_remapping_by_area_matrix	array	Flat row-major matrix (num_classes x num_classes) of allowed class remappings by area.		N/A
detection3d.min_area_matrix	array	Flat row-major matrix (num_classes x num_classes) of minimum bounding box areas for remapping.		N/A
detection3d.max_area_matrix	array	Flat row-major matrix (num_classes x num_classes) of maximum bounding box areas for remapping.		N/A
detection3d.workspace_size	integer	TensorRT builder memory-pool limit in bytes for this engine.	4294967296	≥1

PTv3Node model#

Name	Type	Description	Default	Range
encoder.point_cloud_range	array	Range in meters of the pointcloud in meters [min_x, min_y, min_z, max_x, max_y, max_z].	[-76.8, -76.8, -3.0, 76.8, 76.8, 5.0]	N/A
encoder.voxel_size	array	Voxels size [x, y, z] in meters.	[0.3, 0.3, 8.0]	N/A
encoder.serialization_orders	array	Space-filling curve serialization orders used by the exported model.	["z", "z-trans"]	N/A
encoder.pooling_strides	array	Serialized pooling stride for each encoder downsampling stage.	[2, 2, 2, 2]	N/A
encoder.enc_channels	array	Encoder feature channels per stage, finest to deepest (pooling_strides size + 1 entries).	[32, 64, 128, 256, 512]	N/A
segmentation3d.class_names	array	Segmentation class names.		N/A
segmentation3d.palette	array	Flat RGB palette for visualization output. Must contain 3 values per class.		N/A
segmentation3d.dec_depths	array	Decoder attention-block counts per stage; stages with blocks add serialization-metadata inputs to the seg-head engine.	[0, 0, 0, 0]	N/A
detection3d.bbox_voxel_size	array	Voxel size [x, y, z] in meters of the BEV grid on which the detection head regresses box centers.		N/A
detection3d.has_twist	boolean	Whether the model predicts per-object velocity (twist). When false, the velocity of each detection is set to zero.		N/A
detection3d.num_proposals	integer	Number of detection candidates the head outputs and that are evaluated during post-processing.		≥1
detection3d.post_center_range	array	Bounds [x_min, y_min, z_min, x_max, y_max, z_max] in meters applied to decoded detection centers during post-processing. Any detection whose center lies outside is dropped.		N/A
detection3d.class_names	array	Detection class names in the model's output label order.		N/A

filter.* and class_mapping.* parameters are configured in config/ptv3.param.yaml, while class metadata and the visualization palette are configured in config/ml_package_ptv3_seg3d_head.param.yaml.

segmentation3d.class_mapping maps each segmentation class name to the autoware::point_types::PointCloudClassification value published in the segmentation point cloud's class_id field. Its keys must match segmentation3d.class_names exactly, and every entry of class_names must be present — the node fails to start otherwise, naming the missing class. This makes the consolidation of model classes (for example traffic_cone, debris and vertical_thin into HAZARD) a configuration choice rather than a compile-time constant.

The `build_only` option#

The autoware_ptv3 node has a build_only option to build the TensorRT engine file from the specified ONNX file, after which the program exits.

ros2 launch autoware_ptv3 ptv3.launch.xml build_only:=true

The `log_level` option#

The default logging severity level for autoware_ptv3 is info. For debugging purposes, the developer may decrease severity level using log_level parameter:

ros2 launch autoware_ptv3 ptv3.launch.xml log_level:=debug

Assumptions / Known limits#

This node detects the input pointcloud format automatically on the first received message and supports:

XYZIRCAEDT (10 fields)
XYZIRADRT (9 fields)
XYZIRC (6 fields)
XYZI (4 fields)

The filtered output cloud format is controlled by filter.output_format. When it is set to an empty string, the filtered output preserves the same format as the input cloud.

Trained Models#

The model was trained on the T4Dataset using approximately 4,000 frames and is available in the Autoware artifacts.

Troubleshooting#

`Fail to create host memory`#

This error may occur when TensorRT cannot satisfy the memory requirements for building an engine. A workspace_size that is too small can prevent TensorRT from using the tactics needed for the configured input profiles. Conversely, a workspace_size that is too large can allow memory usage to exceed the GPU's available VRAM. A large maximum value in encoder.voxels_num also increases the size of TensorRT profiles and intermediate buffers.

Adjust the workspace_size values and the maximum encoder.voxels_num value in config/ptv3.param.yaml to fit the available GPU memory. Finding a suitable balance between these parameters can resolve the issue.

References/External links#

[1] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. "Point Transformer V3: Simpler, Faster, Stronger." 2024 Conference on Computer Vision and Pattern Recognition.