autoware_lidar_centerpoint#

Purpose#

autoware_lidar_centerpoint is a package for detecting dynamic 3D objects.

Inner-workings / Algorithms#

In this implementation, CenterPoint [1] uses a PointPillars-based [2] network to inference with TensorRT.

We trained the models using https://github.com/open-mmlab/mmdetection3d.

Inputs / Outputs#

Input#

Name	Type	Description
`~/input/pointcloud`	`sensor_msgs::msg::PointCloud2`	input pointcloud

Output#

Name	Type	Description
`~/output/objects`	`autoware_perception_msgs::msg::DetectedObjects`	detected objects
`debug/cyclic_time_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	cyclic time (msg)
`debug/processing_time_ms`	`autoware_internal_debug_msgs::msg::Float64Stamped`	processing time (ms)

Parameters#

ML Model Parameters#

Note that these parameters are associated with ONNX file, predefined during the training phase. Be careful to change ONNX file as well when changing this parameter. Also, whenever you update the ONNX file, do NOT forget to check these values.

Name	Type	Description	Default	Range
version	string	Model bundle version. The node reads major version 4 from minor version 1 upward; a minor bump may add manifest entries the node requires.	v4.1	N/A
encoder_onnx_path	string	VoxelFeatureEncoder ONNX file, relative to model_path (absolute paths allowed).	pts_voxel_encoder.onnx	N/A
encoder_engine_path	string	VoxelFeatureEncoder TensorRT Engine file, relative to model_path (absolute paths allowed).	pts_voxel_encoder.engine	N/A
head_onnx_path	string	DetectionHead ONNX file, relative to model_path (absolute paths allowed).	pts_backbone_neck_head.onnx	N/A
head_engine_path	string	DetectionHead TensorRT Engine file, relative to model_path (absolute paths allowed).	pts_backbone_neck_head.engine	N/A
class_remapper_param_path	string	Class remapper parameter file, relative to model_path (absolute paths allowed).	detection_class_remapper.param.yaml	N/A
trt_precision	string	TensorRT inference precision.	fp16	['fp32', 'fp16']
model_params.class_names	array	An array of class names will be predicted.	["CAR", "TRUCK", "BUS", "BICYCLE", "PEDESTRIAN"]	N/A
model_params.has_twist	boolean	Indicates whether the model outputs twist value.	false	N/A
model_params.point_feature_size	integer	A number of channels of point feature layer.	4	N/A
model_params.has_variance	boolean	Indicates whether the model outputs variance value.	false	N/A
model_params.max_voxel_size	integer	A maximum size of voxel grid.	40000	N/A
model_params.point_cloud_range	array	The range of the point cloud in [min_x, min_y, min_z, max_x, max_y, max_z] order. [m]	[-76.8, -76.8, -4.0, 76.8, 76.8, 6.0]	N/A
model_params.voxel_size	array	The size of a voxel in [x, y, z] order. [m]	[0.32, 0.32, 10.0]	N/A
model_params.downsample_factor	integer	A scale factor of downsampling points	1	≥1
model_params.encoder_in_feature_size	integer	A size of encoder input feature channels.	9	N/A
model_params.yaw_norm_thresholds	array	A minimum norm of the predicted yaw vector per class, in the order of `class_names`. Objects below the threshold are discarded.	[0.3, 0.3, 0.3, 0.3, 0.0]	N/A
model_params.detection_score_thresholds.distance_bin_upper_limits	array	An array of upper bounds of each distance bin.	[50.0, 90.0, 121.0, 200.0]	N/A
model_params.detection_score_thresholds.min_confidence_scores.CAR	array	A minimum confidence score for CAR in each distance bin.	[0.35, 0.35, 0.35, 0.35]	N/A
model_params.detection_score_thresholds.min_confidence_scores.TRUCK	array	A minimum confidence score for TRUCK in each distance bin.	[0.35, 0.35, 0.35, 0.35]	N/A
model_params.detection_score_thresholds.min_confidence_scores.BUS	array	A minimum confidence score for BUS in each distance bin.	[0.35, 0.35, 0.35, 0.35]	N/A
model_params.detection_score_thresholds.min_confidence_scores.BICYCLE	array	A minimum confidence score for BICYCLE in each distance bin.	[0.35, 0.35, 0.35, 0.35]	N/A
model_params.detection_score_thresholds.min_confidence_scores.PEDESTRIAN	array	A minimum confidence score for PEDESTRIAN in each distance bin.	[0.35, 0.35, 0.35, 0.35]	N/A

Core Parameters#

Name	Type	Description	Default	Range
cloud_capacity	integer	Capacity of the point cloud buffer (should be set to at least the maximum theoretical number of points).	2000000	≥1
post_process_params.circle_nms_dist_threshold	float	A distance threshold of circle NMS.	0.5	≥0.0
post_process_params.iou_nms_search_distance_2d	float	A maximum distance value to search the nearest objects.	10.0	≥0.0
post_process_params.iou_nms_threshold	float	A threshold value of NMS using IoU score.	0.1	≥0.0 ≤1.0
densification_params.world_frame_id	string	A name of frame id where world coordinates system is defined with respect to.	map	N/A
densification_params.num_past_frames	integer	A number of past frames to be considered as same input frame.	1	≥0
diagnostics.max_allowed_processing_time_ms	float	A threshold value for the allowed processing time. If the processing time exceeds this value, it will be considered a warning state. [ms]	200.0	≥0.0
diagnostics.max_acceptable_consecutive_delay_ms	float	A threshold value for the error state. If the duration since the last processing timestamp, which ended within max_allowed_processing_time_ms, exceeds this value, it will be considered an error state. [ms]	1000.0	≥0.0
diagnostics.validation_callback_interval_ms	float	An interval value for a timer callback that checks whether the current state meets the max_acceptable_consecutive_delay_ms condition. [ms]	100.0	≥1.0

Detection Class Remapper#

Name	Type	Description	Default	Range
allow_remapping_by_area_matrix	array	Whether to allow remapping of classes. Row is the original class, column is the class to remap to. The order of the 8x8 matrix classes comes from ObjectClassification msg.	[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]	N/A
min_area_matrix	array	Minimum area for specific class to consider class remapping. [m^2]	[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 12.1, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]	N/A
max_area_matrix	array	Maximum area for specific class to consider class remapping. [m^2]	[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.0, 0.0, 999.999, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 999.999, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 999.999, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]	N/A

Launch Parameters#

These are given by the launch file and are not part of any parameter file.

Name	Type	Default Value	Description
`model_path`	string	`$(var data_path)/lidar_centerpoint/base`	directory containing the model artifacts; relative manifest entries resolve against it
`build_only`	bool	`false`	shutdown the node after TensorRT engine file is built
`logger_name`	string	`lidar_centerpoint`	logger name used for the node logs and debug topics

The `build_only` option#

The autoware_lidar_centerpoint node has build_only option to build the TensorRT engine file from the ONNX file. Although it is preferred to move all the ROS parameters in .param.yaml file in Autoware Universe, the build_only option is not moved to the .param.yaml file for now, because it may be used as a flag to execute the build as a pre-task. You can execute with the following command:

ros2 launch autoware_lidar_centerpoint lidar_centerpoint.launch.xml model_path:=/home/autoware/autoware_data/ml_models/lidar_centerpoint/base build_only:=true

model_path points at the variant folder (base/tiny/sigma/short_range) of the model bundle; the engine files are written there.

Assumptions / Known limits#

The object.existence_probability is stored the value of classification confidence of a DNN, not probability.

Trained Models#

Trained models are hosted on Hugging Face at AutowareFoundation/lidar_centerpoint and downloaded to ~/autoware_data/ml_models/lidar_centerpoint/ by the ansible artifacts role (pinned to tag v4.1).

The bundle holds one self-contained folder per variant; the perception launcher (lidar_dnn_detector.launch.xml) selects the folder from the model name, and lidar_centerpoint.launch.xml takes the folder directly as model_path:

lidar_centerpoint/
├── base/         # model_name: centerpoint
├── tiny/         # model_name: centerpoint_tiny
├── sigma/        # model_name: centerpoint_sigma
└── short_range/  # model_name: centerpoint_short_range

Every variant folder contains the same file set: pts_voxel_encoder.onnx, pts_backbone_neck_head.onnx, ml_package.param.yaml and detection_class_remapper.param.yaml. TensorRT engine files are built locally into the variant folder on first launch (or via build_only:=true).

ml_package.param.yaml declares the bundle version. The node reads major version 4 from minor version 1 upward and aborts otherwise, so a version error means the bundle has to be refreshed with the ansible artifacts role of the autoware repository. A minor bump can add manifest entries the node requires.

Centerpoint was trained in nuScenes (~28k lidar frames) [8] and TIER IV's internal database (~11k lidar frames) for 60 epochs. Centerpoint tiny was trained in Argoverse 2 (~110k lidar frames) [9] and TIER IV's internal database (~11k lidar frames) for 20 epochs.

Training CenterPoint Model and Deploying to the Autoware#

Overview#

This guide provides instructions on training a CenterPoint model using the mmdetection3d repository and seamlessly deploying it within Autoware.

Installation#

Install prerequisites#

Step 1. Download and install Miniconda from the official website.

Step 2. Create a conda virtual environment and activate it

conda create --name train-centerpoint python=3.8 -y
conda activate train-centerpoint

Step 3. Install PyTorch

Please ensure you have PyTorch installed, and compatible with CUDA 11.6, as it is a requirement for current Autoware.

conda install pytorch==1.13.1 torchvision==0.14.1 pytorch-cuda=11.6 -c pytorch -c nvidia

Install mmdetection3d#

Step 1. Install MMEngine, MMCV, and MMDetection using MIM

pip install -U openmim
mim install mmengine
mim install 'mmcv>=2.0.0rc4'
mim install 'mmdet>=3.0.0rc5, <3.3.0'

Step 2. Install mmdetection3d forked repository

Introduced several valuable enhancements in our fork of the mmdetection3d repository. Notably, we've made the PointPillar z voxel feature input optional to maintain compatibility with the original paper. In addition, we've integrated a PyTorch to ONNX converter and a T4 format reader for added functionality.

git clone https://github.com/autowarefoundation/mmdetection3d.git
cd mmdetection3d
pip install -v -e .

Use Training Repository with Docker#

Alternatively, you can use Docker to run the mmdetection3d repository. We provide a Dockerfile to build a Docker image with the mmdetection3d repository and its dependencies.

Clone fork of the mmdetection3d repository

git clone https://github.com/autowarefoundation/mmdetection3d.git

Build the Docker image by running the following command:

cd mmdetection3d
docker build -t mmdetection3d -f docker/Dockerfile .

Run the Docker container:

docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmdetection3d/data mmdetection3d

Preparing NuScenes dataset for training#

Step 1. Download the NuScenes dataset from the official website and extract the dataset to a folder of your choice.

Note: The NuScenes dataset is large and requires significant disk space. Ensure you have enough storage available before proceeding.

Step 2. Create a symbolic link to the dataset folder

ln -s /path/to/nuscenes/dataset/ /path/to/mmdetection3d/data/nuscenes/

Step 3. Prepare the NuScenes data by running:

cd mmdetection3d
python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes

Training CenterPoint with NuScenes Dataset#

Prepare the config file#

The configuration file that illustrates how to train the CenterPoint model with the NuScenes dataset is located at mmdetection3d/projects/AutowareCenterPoint/configs. This configuration file is a derived version of this centerpoint configuration file from mmdetection3D. In this custom configuration, the use_voxel_center_z parameter is set as False to deactivate the z coordinate of the voxel center, aligning with the original paper's specifications and making the model compatible with Autoware. Additionally, the filter size is set as [32, 32].

The CenterPoint model can be tailored to your specific requirements by modifying various parameters within the configuration file. This includes adjustments related to preprocessing operations, training, testing, model architecture, dataset, optimizer, learning rate scheduler, and more.

Start training#

python tools/train.py projects/AutowareCenterPoint/configs/centerpoint_custom.py --work-dir ./work_dirs/centerpoint_custom

Evaluation of the trained model#

For evaluation purposes, we have included a sample dataset captured from the vehicle which consists of the following LiDAR sensors: 1 x Velodyne VLS128, 4 x Velodyne VLP16, and 1 x Robosense RS Bpearl. This dataset comprises 600 LiDAR frames and encompasses 5 distinct classes, 6905 cars, 3951 pedestrians, 75 cyclists, 162 buses, and 326 trucks 3D annotation. In the sample dataset, frames are annotated as 2 frames for each second. You can employ this dataset for a wide range of purposes, including training, evaluation, and fine-tuning of models. It is organized in the T4 format.

Download the sample dataset#

wget https://autoware-files.s3.us-west-2.amazonaws.com/dataset/lidar_detection_sample_dataset.tar.gz
#Extract the dataset to a folder of your choice
tar -xvf lidar_detection_sample_dataset.tar.gz
#Create a symbolic link to the dataset folder
ln -s /PATH/TO/DATASET/ /PATH/TO/mmdetection3d/data/tier4_dataset/

Prepare dataset and evaluate trained model#

Create .pkl files for training, evaluation, and testing.

The dataset was formatted according to T4Dataset specifications, with 'sample_dataset' designated as one of its versions.

python tools/create_data.py T4Dataset --root-path data/sample_dataset/ --out-dir data/sample_dataset/ --extra-tag T4Dataset --version sample_dataset --annotation-hz 2

Run evaluation

python tools/test.py projects/AutowareCenterPoint/configs/centerpoint_custom_test.py /PATH/OF/THE/CHECKPOINT  --task lidar_det

Evaluation results could be relatively low because of the e to variations in sensor modalities between the sample dataset and the training dataset. The model's training parameters are originally tailored to the NuScenes dataset, which employs a single lidar sensor positioned atop the vehicle. In contrast, the provided sample dataset comprises concatenated point clouds positioned at the base link location of the vehicle.

Deploying CenterPoint model to Autoware#

Convert CenterPoint PyTorch model to ONNX Format#

The autoware_lidar_centerpoint implementation requires two ONNX models as input the voxel encoder and the backbone-neck-head of the CenterPoint model, other aspects of the network, such as preprocessing operations, are implemented externally. Under the fork of the mmdetection3d repository, we have included a script that converts the CenterPoint model to Autoware compatible ONNX format. You can find it in mmdetection3d/projects/AutowareCenterPoint file.

python projects/AutowareCenterPoint/centerpoint_onnx_converter.py --cfg projects/AutowareCenterPoint/configs/centerpoint_custom.py --ckpt work_dirs/centerpoint_custom/YOUR_BEST_MODEL.pth --work-dir ./work_dirs/onnx_models

Create the ml_package file for the custom model#

Rename the exported ONNX files to pts_voxel_encoder.onnx and pts_backbone_neck_head.onnx, and create a ml_package.param.yaml next to them. Set the model parameters like point_cloud_range, point_feature_size, voxel_size, etc. according to the training config file. The ML Model Parameters table lists every entry with its default; version has to be declared as well, or the node aborts at startup.

Launch the lidar_centerpoint node#

cd /YOUR/AUTOWARE/PATH/Autoware
source install/setup.bash
ros2 launch autoware_lidar_centerpoint lidar_centerpoint.launch.xml model_path:=/PATH/TO/MODEL_FOLDER/

model_path is used as given: the directory must contain the ONNX files, ml_package.param.yaml and detection_class_remapper.param.yaml.

Launch the lidar_short_range_centerpoint node#

It also provides short_range detections using CenterPoint, served from the short_range/ folder of the model bundle:

cd /YOUR/AUTOWARE/PATH/Autoware
source install/setup.bash
ros2 launch autoware_lidar_centerpoint lidar_centerpoint.launch.xml model_path:=$HOME/autoware_data/ml_models/lidar_centerpoint/short_range

Changelog#

v4.1 (2026/08)#

ml_package.param.yaml gained version and model_params.yaw_norm_thresholds, the latter moved out of centerpoint_common.param.yaml because it is indexed by the predicted class. deploy_metadata.yaml is gone. Bundles of v4.0 and earlier are rejected at startup.

v4.0 (2026/07)#

The model bundle moved to per-variant folders (base/tiny/sigma/short_range) with uniform file names, hosted on Hugging Face. sigma and short_range weights are published for the first time. Earlier revisions remain available as tags on the same repository (v3.0 = flat layout). The v1 and v0 files below are not distributed any more.

v1 (2022/07/06)#

Name	Files	Description
`centerpoint`	`pts_voxel_encoder` `pts_backbone_neck_head`	There is a single change due to the limitation in the implementation of this package. `num_filters=[32, 32]` of `PillarFeatureNet`
`centerpoint_tiny`	`pts_voxel_encoder` `pts_backbone_neck_head`	The same model as `default` of `v0`.

These changes are compared with this configuration.

v0 (2021/12/03)#

Name	Files	Description
`default`	`pts_voxel_encoder` `pts_backbone_neck_head`	There are two changes from the original CenterPoint architecture. `num_filters=[32]` of `PillarFeatureNet` and `ds_layer_strides=[2, 2, 2]` of `RPN`

(Optional) Error detection and handling#

(Optional) Performance characterization#

References/External links#

[1] Yin, Tianwei, Xingyi Zhou, and Philipp Krähenbühl. "Center-based 3d object detection and tracking." arXiv preprint arXiv:2006.11275 (2020).

[2] Lang, Alex H., et al. "PointPillars: Fast encoders for object detection from point clouds." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[3] https://github.com/tianweiy/CenterPoint

[4] https://github.com/open-mmlab/mmdetection3d

[5] https://github.com/open-mmlab/OpenPCDet

[6] https://github.com/yukkysaito/autoware_perception

[7] https://github.com/NVIDIA-AI-IOT/CUDA-PointPillars

[8] https://www.nuscenes.org/nuscenes

[9] https://www.argoverse.org/av2.html

(Optional) Future extensions / Unimplemented parts#

Acknowledgment: deepen.ai's 3D Annotation Tools Contribution#

Special thanks to Deepen AI for providing their 3D Annotation tools, which have been instrumental in creating our sample dataset.

Legal Notice#

The nuScenes dataset is released publicly for non-commercial use under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License. Additional Terms of Use can be found at https://www.nuscenes.org/terms-of-use. To inquire about a commercial license please contact nuscenes@motional.com.