Skip to content

Evaluation and Testing#

The evaluation for each model can be found on the GitHub repo of the PoV.

System Evaluation Results#

The section shows the benchmark results of the VisionPilot model on different hardware environment as the references. There are two procedures to conduct the benchmark:

Two sets of computation configurations are used to benchmark the pipile:

  • X86-based Computer: link
  • ARM-based Computer: link

Hardware Spec#

  • CPU: Intel Xeon E-2278GE (16 cores)
  • GPU: NVIDIA Quadro RTX 5000.
  • Memory: 64GB LPDDR5/16G on RTX 5000
  • Driver: Driver Version: 580.82.07 & CUDA Version: 13.0
  • ROS: ROS Jazzy & Zenoh
  • Runtime: TensorRT
  • OS: Ubuntu 24.04.3

link to 3510 (discontinued)

link to AL30 ( Autonomous Driving Solutions)

Benchmark Result#

  • Zenoh:

    Model CPU Utilization GPU Utilization Peak Memory Usage Frame Rate
    SceneSeg 13% 65% 20G 58
    DomainSeg 14% 62% 21G 58
    Scene3D 15% 69% 19G 57
    EgoSpace
    • SceneSeg
    Current FPS: 58
--- Per-frame Timing (microseconds) ---
* Total processing time: 17229 us
* Preprocessing time: 1200 us
* Inference time: 15829 us      
* Output time: 199 us
    • DomainSeg
    Current FPS: 58
--- Per-frame Timing (microseconds) ---
* Total processing time: 16625 us
* Preprocessing time: 1203 us
* Inference time: 15250 us
* Output time: 171 us
    • Scene3D
    Current FPS: 57
--- Per-frame Timing (microseconds) ---
* Total processing time: 17484 us
* Preprocessing time: 204 us
* Inference time: 16817 us
* Output time: 462 us

  • ROS 2:

    Model CPU Utilization GPU Utilization Peak Memory Usage Frame Rate
    SceneSeg 17% 72% 6G 60
    DomainSeg 16% 68% 6G 60
    Scene3D 16% 67% 5.9G 60
    EgoSpace
    • SceneSeg
    * Current FPS: 60
--- Per-frame Timing (microseconds) --- 
* Total processing time: 16272 us
* Preprocessing time: 178 us
* Inference time: 15713 us
* Output time: 381 us
    • DomainSeg
    * Current FPS: 60.00
--- Per-frame Timing (microseconds) --- 
* Total processing time: 15393 us
* Preprocessing time: 253 us
* Inference time: 14672 us
* Output time: 467 us
--------------------------

ARM processors and nVidia AGX Orin#

Hardware spec:#

  • CPU: 12-core ARM Cortex-A78AE CPU at 2.2GHz.
  • GPU: NVIDIA Ampere GPU with 2048 CUDA Cores.
  • Memory: 64GB LPDDR5. The system and GPU memories are shared.
  • Driver: The NVIDIA JetPack 6.0 (Ubuntu 22.04 LTS based) was used.
  • ROS: ROS Humble with Autoware recommended Cyclone DDS settings.
  • Runtime: ONNX runtime 1.19.0 or TensorRT

link to nVidia Jetson Orin AGX

Benchmark results:#

Model CPU Utilization GPU Utilization Peak Memory Usage Frame Rate
SceneSeg
(ONNX runtime)		 91% ~ 99% 99% 45G
including network model (~30G) + other process (15G)		 8
SceneSeg
(TensorRT runtime - FP16)		 57 ~ 66 % 74 % 0.8 % (~0.50 GB) 29.12
DomainSeg
(TensorRT runtime - FP16)		 56 ~ 60 % 88 % 0.8 % (~0.50 GB) 29.85
Scene3D
(TensorRT runtime - FP16)		 53 ~ 56 % 82 % 0.6 % (~0.38 GB) 29.90
SceneSeg
(TensorRT runtime - FP32)		 42 ~ 49 % 99 % 0.6 % (~0.38 GB) 17.10
DomainSeg
(TensorRT runtime - FP32)		 43 ~ 47 % 99 % 0.6 % (~0.38 GB) 17.07
Scene3D
(TensorRT runtime - FP32)		 44 ~ 46 % 99 % 0.6 % (~0.38 GB) 17.03

link to the instructions and complete results.