Modules

A tour of VisionPilot/modules/ - what each component does, what it depends on, and where its own README lives.

Vision Pilot is a CMake project made of independent modules under VisionPilot/modules/, linked together by the application entry point in VisionPilot/app/vision_pilot.cpp.

Sensing

Camera interface (V4L2)

A C++ wrapper that captures frames from a V4L2 device (/dev/video0, …) and hands them downstream as OpenCV cv::Mat. It uses cv::VideoCapture with the CAP_V4L2 backend, guards frame and statistics access with a mutex, lets you request a target FPS and codec, and tracks frames captured, dropped and errored.

Configured by source.v4l2_device and source.v4l2_fps.

Vehicle interface (CAN)

Reads ego speed from the vehicle CAN bus and writes control commands back, using the CAN database in config/vehicle.dbc.

Image preprocessing

Normalisation and geometric preparation of frames before inference, including the homography mapping from config/H.yaml.

Engine

The inference layer over ONNX Runtime. It selects the execution provider (cuda or cpu), targets a specific GPU, manages the TensorRT engine cache directory and workspace budget, and loads either fp32 or quantized int8 weights.

The first run with cuda is slow because TensorRT engines are built and written to engine.cache_dir; subsequent runs reuse them.

Models

Integration of the three Autoware Foundation models - pre- and post-processing, tensor layouts and output decoding.

Safety Guardian

Fusion

Reconciles the perception branch and the end-to-end branch into a single scene interpretation. Where the two disagree, the conservative reading is taken forward.

Planning

Produces the longitudinal command (from the fused lead-object distance, ego speed and the applicable speed limit) and the lateral command (tracking the fused ego path, using the wheelbase L). This is where ACC, AEB, FCW, LKAS, LDW and ISA behaviours live.

Visualization

The largest module, and the one with the most detailed README in the repository. Three capabilities:

HUD rendering

visualization::render_frame() draws the driver-facing overlay on each frame - object boxes, the closest in-path object with its distance, the fused path corridor, lane departure state and the speed limit - and manages the local OpenCV window. Controlled by visualization_on.

Occupancy BEV window (optional)

Built only with -DENABLE_OCCUPANCY=ON. A second window named Occupancy appears beside the HUD showing a heuristic 3D bird’s-eye view of the scene:

• light-grey ground plane with a 0–150 m grid, and a green fused-path corridor
• AutoSpeed objects as extruded boxes - blue for traffic, red for the CIPO, white for ego

Controls: left-drag orbit, right-drag pan, wheel zoom, R resets the camera.

WebRTC streaming (optional)

Streams the rendered HUD to any browser on the network, which is how you monitor a headless vehicle PC. Enabled with webrtc_on = true.

The pipeline is appsrc => queue => videoconvert => vp8enc => rtpvp8pay => webrtcbin in GStreamer, with a libsoup HTTP server that serves a self-contained HTML5 client at / and handles SDP/ICE signalling over a WebSocket at /ws. No browser-side dependencies, no external signalling server.

# with webrtc_on = true, webrtc_port = 8080
# then open:
http://127.0.0.1:8080/

The local OpenCV preview is disabled while WebRTC is active.

Dependencies: GStreamer (base, bad plugins), libsoup2.4, json-glib, OpenCV - all covered by the prerequisites.

Logging

Streams per-frame data directly into a Rerun .rrd recording, which you can scrub through afterwards:

rrd_on  = true
rrd_log = visionpilot.rrd

rerun visionpilot.rrd

Middleware interfaces

Built with -DENABLE_ROS2_INTERFACE=ON.

• camera_ros2_interface - subscribes to sensor_msgs/Image on source.input_camera_topic.
• vehicle_ros2_interface - subscribes to the speed topic and publishes steering and acceleration commands.

This is the path the CARLA bridge uses.