autoware_lidar_frnet#

Purpose#

The autoware_lidar_frnet package is used for 3D semantic segmentation based on LiDAR data (x, y, z, intensity).

Inner-workings / Algorithms#

The implementation is based on the FRNet [1] project. It uses TensorRT library for data processing and network inference.

We trained the models using AWML [2].

Inputs / Outputs#

Input#

Name	Type	Description
`~/input/pointcloud`	`sensor_msgs::msg::PointCloud2`	Input pointcloud.

Output#

Name	Type	Description
`~/output/pointcloud/segmentation`	`sensor_msgs::msg::PointCloud2`	XYZ cloud with class ID and probability fields.
`~/output/pointcloud/visualization`	`sensor_msgs::msg::PointCloud2`	XYZ cloud with RGB field.
`~/output/pointcloud/filtered`	`sensor_msgs::msg::PointCloud2`	Filtered cloud in the requested `filter.output_format`.
`debug/cyclic_time_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Cyclic time (ms).
`debug/pipeline_latency_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Pipeline latency time (ms).
`debug/processing_time/preprocess_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Preprocess (ms).
`debug/processing_time/inference_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Inference time (ms).
`debug/processing_time/postprocess_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Postprocess time (ms).
`debug/processing_time/total_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	Total processing time (ms).
`/diagnostics`	`diagnostic_msgs::msg::DiagnosticArray`	Node diagnostics with respect to processing time constraints

Parameters#

FRNet node#

Name	Type	Description	Range
onnx_path	string	Path to ONNX model file.	N/A
trt_precision	string	TensorRT engine precision.	['fp16', 'fp32']
filter.class_probability_threshold	float	If any of the filter classes has probability >= this value, the point is excluded from the filtered output cloud.	≥0.0 ≤1.0
filter.classes	array	Class names to filter out when their probability exceeds the threshold.	N/A
filter.output_format	string	Filtered output point format. Empty string preserves the input format.	['', 'xyzi', 'xyzirc', 'xyziradrt', 'xyzircaedt']
filter.ego_crop_box.reference_frame	string	TF frame for ego crop box bounds (e.g. base_link).	N/A
filter.ego_crop_box.min_x	float	Min X bound of crop box in reference frame.	N/A
filter.ego_crop_box.min_y	float	Min Y bound of crop box in reference frame.	N/A
filter.ego_crop_box.min_z	float	Min Z bound of crop box in reference frame.	N/A
filter.ego_crop_box.max_x	float	Max X bound of crop box in reference frame.	N/A
filter.ego_crop_box.max_y	float	Max Y bound of crop box in reference frame.	N/A
filter.ego_crop_box.max_z	float	Max Z bound of crop box in reference frame.	N/A

FRNet model#

Name	Type	Description	Default	Range
fov_up_deg	float	LiDAR's upper elevation angle in degrees.	15.0	≥-90.0 ≤90.0
fov_down_deg	float	LiDAR's lower elevation angle in degrees.	-25.0	≥-90.0 ≤90.0
frustum_width	integer	Width of the FRNet frustum in pixels.	1024	≥1
frustum_height	integer	Height of the FRNet frustum in pixels.	128	≥1
interpolation_width	integer	Width of the FRNet LiDAR's points to 2D plane interpolation in pixels.	4096	≥1
interpolation_height	integer	Height of the FRNet LiDAR's points to 2D plane interpolation in pixels.	128	≥1
class_names	array	An array of class names which will be predicted.		N/A
num_points	array	TensorRT optimization profile for number of points [min, opt, max].	[5000, 80000, 160000]	N/A
num_unique_coors	array	TensorRT optimization profile for number of unique coordinates [min, opt, max].	[3000, 30000, 60000]	N/A
palette	array	A sequence of RGB values for each class name. The length of the array must be 3 times the number of class names.		N/A

FRNet diagnostics#

Name	Type	Description	Default	Range
max_allowed_processing_time_ms	float	A threshold value for the allowed processing time. If the processing time exceeds this value, it will be considered a warning state. [ms]	200	≥0.0
max_acceptable_consecutive_delay_ms	float	A threshold value for the error state. If the duration since the last processing timestamp, which ended within max_allowed_processing_time_ms, exceeds this value, it will be considered an error state. [ms]	1000	≥0.0
validation_callback_interval_ms	float	An interval value for a timer callback that checks whether the current state meets the max_acceptable_consecutive_delay_ms condition. [ms]	100	≥1.0

The `build_only` option#

The autoware_lidar_frnet node has build_only option to build the TensorRT engine file from the ONNX file.

ros2 launch autoware_lidar_frnet lidar_frnet.launch.xml build_only:=true

Assumptions / Known limits#

This library operates on raw cloud data (bytes). It supports multiple input pointcloud formats and automatically detects the format on the first received message. The supported formats are (checked in order from largest to smallest, with exact field count match required):

XYZIRCAEDT (10 fields)#

[
  sensor_msgs.msg.PointField(name='x', offset=0, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='y', offset=4, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='z', offset=8, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='intensity', offset=12, datatype=2, count=1),
  sensor_msgs.msg.PointField(name='return_type', offset=13, datatype=2, count=1),
  sensor_msgs.msg.PointField(name='channel', offset=14, datatype=4, count=1),
  sensor_msgs.msg.PointField(name='azimuth', offset=16, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='elevation', offset=20, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='distance', offset=24, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='time_stamp', offset=28, datatype=6, count=1)
]

XYZIRADRT (9 fields)#

[
  sensor_msgs.msg.PointField(name='x', offset=0, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='y', offset=4, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='z', offset=8, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='intensity', offset=12, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='ring', offset=16, datatype=4, count=1),
  sensor_msgs.msg.PointField(name='azimuth', offset=18, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='distance', offset=22, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='return_type', offset=26, datatype=2, count=1),
  sensor_msgs.msg.PointField(name='time_stamp', offset=27, datatype=8, count=1)
]

XYZIRC (6 fields)#

[
  sensor_msgs.msg.PointField(name='x', offset=0, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='y', offset=4, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='z', offset=8, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='intensity', offset=12, datatype=2, count=1),
  sensor_msgs.msg.PointField(name='return_type', offset=13, datatype=2, count=1),
  sensor_msgs.msg.PointField(name='channel', offset=14, datatype=4, count=1)
]

XYZI (4 fields)#

[
  sensor_msgs.msg.PointField(name='x', offset=0, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='y', offset=4, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='z', offset=8, datatype=7, count=1),
  sensor_msgs.msg.PointField(name='intensity', offset=12, datatype=7, count=1)
]

The filtered output cloud format is controlled by filter.output_format. When it is set to an empty string, the filtered output preserves the same format as the input cloud.

For debug purposes, you can validate your pointcloud topic using simple command:

ros2 topic echo <input_topic> --field fields

Trained Models#

The model was trained on the T4Dataset using approximately 16,000 frames (4,000 frames × 4 surrounding sensors) and is available in the Autoware artifacts for two sensor models:

Hesai OT128
Hesai QT128 Due to the design of FRNet, specifically its use of range images, the input point cloud must be referenced to the sensor's origin. Additionally, any differences in your sensor's specifications, such as FoV or horizontal/vertical resolution, may affect performance.

References/External links#

[1] X. Xu, L. Kong, H. Shuai and Q. Liu, "FRNet: Frustum-Range Networks for Scalable LiDAR Segmentation" in IEEE Transactions on Image Processing, vol. 34, pp. 2173-2186, 2025, doi: 10.1109/TIP.2025.3550011.

[2] https://github.com/tier4/AWML.git

[3] https://xiangxu-0103.github.io/FRNet