Skip to content



lidar_centerpoint is a package for detecting dynamic 3D objects.

Inner-workings / Algorithms#

In this implementation, CenterPoint [1] uses a PointPillars-based [2] network to inference with TensorRT.

We trained the models using

Inputs / Outputs#


Name Type Description
~/input/pointcloud sensor_msgs::msg::PointCloud2 input pointcloud


Name Type Description
~/output/objects autoware_perception_msgs::msg::DetectedObjects detected objects
debug/cyclic_time_ms tier4_debug_msgs::msg::Float64Stamped cyclic time (msg)
debug/processing_time_ms tier4_debug_msgs::msg::Float64Stamped processing time (ms)


Core Parameters#

Name Type Default Value Description
encoder_onnx_path string "" path to VoxelFeatureEncoder ONNX file
encoder_engine_path string "" path to VoxelFeatureEncoder TensorRT Engine file
head_onnx_path string "" path to DetectionHead ONNX file
head_engine_path string "" path to DetectionHead TensorRT Engine file
build_only bool false shutdown the node after TensorRT engine file is built
trt_precision string fp16 TensorRT inference precision: fp32 or fp16
post_process_params.score_threshold double 0.4 detected objects with score less than threshold are ignored
post_process_params.yaw_norm_thresholds list[double] [0.3, 0.3, 0.3, 0.3, 0.0] An array of distance threshold values of norm of yaw [rad].
post_process_params.iou_nms_target_class_names list[string] - target classes for IoU-based Non Maximum Suppression
post_process_params.iou_nms_search_distance_2d double - If two objects are farther than the value, NMS isn't applied.
post_process_params.iou_nms_threshold double - IoU threshold for the IoU-based Non Maximum Suppression
post_process_params.has_twist boolean false Indicates whether the model outputs twist value.
densification_params.world_frame_id string map the world frame id to fuse multi-frame pointcloud
densification_params.num_past_frames int 1 the number of past frames to fuse with the current frame

The build_only option#

The lidar_centerpoint node has build_only option to build the TensorRT engine file from the ONNX file. Although it is preferred to move all the ROS parameters in .param.yaml file in Autoware Universe, the build_only option is not moved to the .param.yaml file for now, because it may be used as a flag to execute the build as a pre-task. You can execute with the following command:

ros2 launch lidar_centerpoint lidar_centerpoint.launch.xml model_name:=centerpoint_tiny model_path:=/home/autoware/autoware_data/lidar_centerpoint model_param_path:=$(ros2 pkg prefix lidar_centerpoint --share)/config/centerpoint_tiny.param.yaml build_only:=true

Assumptions / Known limits#

  • The object.existence_probability is stored the value of classification confidence of a DNN, not probability.

Trained Models#

You can download the onnx format of trained models by clicking on the links below.

Centerpoint was trained in nuScenes (~28k lidar frames) [8] and TIER IV's internal database (~11k lidar frames) for 60 epochs. Centerpoint tiny was trained in Argoverse 2 (~110k lidar frames) [9] and TIER IV's internal database (~11k lidar frames) for 20 epochs.

Training CenterPoint Model and Deploying to the Autoware#


This guide provides instructions on training a CenterPoint model using the mmdetection3d repository and seamlessly deploying it within Autoware.


Install prerequisites#

Step 1. Download and install Miniconda from the official website.

Step 2. Create a conda virtual environment and activate it

conda create --name train-centerpoint python=3.8 -y
conda activate train-centerpoint

Step 3. Install PyTorch

Please ensure you have PyTorch installed, and compatible with CUDA 11.6, as it is a requirement for current Autoware.

conda install pytorch==1.13.1 torchvision==0.14.1 pytorch-cuda=11.6 -c pytorch -c nvidia

Install mmdetection3d#

Step 1. Install MMEngine, MMCV, and MMDetection using MIM

pip install -U openmim
mim install mmengine
mim install 'mmcv>=2.0.0rc4'
mim install 'mmdet>=3.0.0rc5, <3.3.0'

Step 2. Install mmdetection3d forked repository

Introduced several valuable enhancements in our fork of the mmdetection3d repository. Notably, we've made the PointPillar z voxel feature input optional to maintain compatibility with the original paper. In addition, we've integrated a PyTorch to ONNX converter and a T4 format reader for added functionality.

git clone
cd mmdetection3d
pip install -v -e .

Use Training Repository with Docker#

Alternatively, you can use Docker to run the mmdetection3d repository. We provide a Dockerfile to build a Docker image with the mmdetection3d repository and its dependencies.

Clone fork of the mmdetection3d repository

git clone

Build the Docker image by running the following command:

cd mmdetection3d
docker build -t mmdetection3d -f docker/Dockerfile .

Run the Docker container:

docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmdetection3d/data mmdetection3d

Preparing NuScenes dataset for training#

Step 1. Download the NuScenes dataset from the official website and extract the dataset to a folder of your choice.

Note: The NuScenes dataset is large and requires significant disk space. Ensure you have enough storage available before proceeding.

Step 2. Create a symbolic link to the dataset folder

ln -s /path/to/nuscenes/dataset/ /path/to/mmdetection3d/data/nuscenes/

Step 3. Prepare the NuScenes data by running:

cd mmdetection3d
python tools/ nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes

Training CenterPoint with NuScenes Dataset#

Prepare the config file#

The configuration file that illustrates how to train the CenterPoint model with the NuScenes dataset is located at mmdetection3d/projects/AutowareCenterPoint/configs. This configuration file is a derived version of this centerpoint configuration file from mmdetection3D. In this custom configuration, the use_voxel_center_z parameter is set as False to deactivate the z coordinate of the voxel center, aligning with the original paper's specifications and making the model compatible with Autoware. Additionally, the filter size is set as [32, 32].

The CenterPoint model can be tailored to your specific requirements by modifying various parameters within the configuration file. This includes adjustments related to preprocessing operations, training, testing, model architecture, dataset, optimizer, learning rate scheduler, and more.

Start training#

python tools/ projects/AutowareCenterPoint/configs/ --work-dir ./work_dirs/centerpoint_custom

Evaluation of the trained model#

For evaluation purposes, we have included a sample dataset captured from the vehicle which consists of the following LiDAR sensors: 1 x Velodyne VLS128, 4 x Velodyne VLP16, and 1 x Robosense RS Bpearl. This dataset comprises 600 LiDAR frames and encompasses 5 distinct classes, 6905 cars, 3951 pedestrians, 75 cyclists, 162 buses, and 326 trucks 3D annotation. In the sample dataset, frames are annotated as 2 frames for each second. You can employ this dataset for a wide range of purposes, including training, evaluation, and fine-tuning of models. It is organized in the T4 format.

Download the sample dataset#
#Extract the dataset to a folder of your choice
tar -xvf lidar_detection_sample_dataset.tar.gz
#Create a symbolic link to the dataset folder
ln -s /PATH/TO/DATASET/ /PATH/TO/mmdetection3d/data/tier4_dataset/
Prepare dataset and evaluate trained model#

Create .pkl files for training, evaluation, and testing.

The dataset was formatted according to T4Dataset specifications, with 'sample_dataset' designated as one of its versions.

python tools/ T4Dataset --root-path data/sample_dataset/ --out-dir data/sample_dataset/ --extra-tag T4Dataset --version sample_dataset --annotation-hz 2

Run evaluation

python tools/ projects/AutowareCenterPoint/configs/ /PATH/OF/THE/CHECKPOINT  --task lidar_det

Evaluation results could be relatively low because of the e to variations in sensor modalities between the sample dataset and the training dataset. The model's training parameters are originally tailored to the NuScenes dataset, which employs a single lidar sensor positioned atop the vehicle. In contrast, the provided sample dataset comprises concatenated point clouds positioned at the base link location of the vehicle.

Deploying CenterPoint model to Autoware#

Convert CenterPoint PyTorch model to ONNX Format#

The lidar_centerpoint implementation requires two ONNX models as input the voxel encoder and the backbone-neck-head of the CenterPoint model, other aspects of the network, such as preprocessing operations, are implemented externally. Under the fork of the mmdetection3d repository, we have included a script that converts the CenterPoint model to Autoware compatible ONNX format. You can find it in mmdetection3d/projects/AutowareCenterPoint file.

python projects/AutowareCenterPoint/ --cfg projects/AutowareCenterPoint/configs/ --ckpt work_dirs/centerpoint_custom/YOUR_BEST_MODEL.pth --work-dir ./work_dirs/onnx_models

Create the config file for the custom model#

Create a new config file named centerpoint_custom.param.yaml under the config file directory of the lidar_centerpoint node. Sets the parameters of the config file like point_cloud_range, point_feature_size, voxel_size, etc. according to the training config file.

    class_names: ["CAR", "TRUCK", "BUS", "BICYCLE", "PEDESTRIAN"]
    point_feature_size: 4
    max_voxel_size: 40000
    point_cloud_range: [-51.2, -51.2, -3.0, 51.2, 51.2, 5.0]
    voxel_size: [0.2, 0.2, 8.0]
    downsample_factor: 1
    encoder_in_feature_size: 9
    # post-process params
    circle_nms_dist_threshold: 0.5
    iou_nms_target_class_names: ["CAR"]
    iou_nms_search_distance_2d: 10.0
    iou_nms_threshold: 0.1
    yaw_norm_thresholds: [0.3, 0.3, 0.3, 0.3, 0.0]

Launch the lidar_centerpoint node#

source install/setup.bash
ros2 launch lidar_centerpoint lidar_centerpoint.launch.xml  model_name:=centerpoint_custom  model_path:=/PATH/TO/ONNX/FILE/


v1 (2022/07/06)#

Name URLs Description
centerpoint pts_voxel_encoder
There is a single change due to the limitation in the implementation of this package. num_filters=[32, 32] of PillarFeatureNet
centerpoint_tiny pts_voxel_encoder
The same model as default of v0.

These changes are compared with this configuration.

v0 (2021/12/03)#

Name URLs Description
default pts_voxel_encoder
There are two changes from the original CenterPoint architecture. num_filters=[32] of PillarFeatureNet and ds_layer_strides=[2, 2, 2] of RPN

(Optional) Error detection and handling#

(Optional) Performance characterization#

[1] Yin, Tianwei, Xingyi Zhou, and Philipp Krähenbühl. "Center-based 3d object detection and tracking." arXiv preprint arXiv:2006.11275 (2020).

[2] Lang, Alex H., et al. "PointPillars: Fast encoders for object detection from point clouds." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.








(Optional) Future extensions / Unimplemented parts#

Acknowledgment:'s 3D Annotation Tools Contribution#

Special thanks to Deepen AI for providing their 3D Annotation tools, which have been instrumental in creating our sample dataset.

The nuScenes dataset is released publicly for non-commercial use under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License. Additional Terms of Use can be found at To inquire about a commercial license please contact