T4 Dataset¶
With t4.enabled=true each run is also written as a T4 dataset, TIER IV's
nuScenes-style format, which t4-devkit
reads (t4_devkit.T4Devkit) and checks (t4sanity). Its sensors are the
driver policy's -- the LiDARs and cameras driver.lidars and driver.cameras
list (External Driver Interface) -- and a key frame is
a policy step (driver.policy_timestep_s): the ego's pose, the LiDARs' sweeps
as one LIDAR_CONCAT cloud in the ego's frame, a frame of every camera, and
the box of every vehicle and pedestrian near the ego, which CARLA knows
exactly.
pip install 'autoware-carla-scenario[t4]'
uv run scenario scenario=cut_in/left ego.entity=carla_driver t4.enabled=true \
'driver.lidars=[{logical_id: lidar_top, position_z: 2.2}]'
Where the data comes from:
- An ego driven by a driver policy (
ego.entity=carla_driver) feeds the dataset what it sends the policy: the sweeps, already in the rig frame, and the camera frames, already JPEG (driver.image_quality). The dataset renders and encodes nothing of its own. A sweep the policy gets can be from a tick before its step (a LiDAR sweeps once a policy step); the key frame is then that tick's, with its ego pose and boxes, as with the recorder's own LiDARs. A sweep from before the recording started, or older than the ticks the recorder keeps, is dropped (logged at debug level). A camera frame carries the tick it was rendered on too (#951). Without a LiDAR, the oldest camera frame's tick is the key frame. With one, a camera frame rendered on another tick than the sweep (a camera faster than the LiDAR, or on the other ticks of its interval) still goes on the step's key frame, at most a policy step off: the dataset stamps a sample's data with the sample's time, and such a camera's frames would otherwise never be written. A step whose every fed item is dropped -- a camera slower than the policy hands it the frame written already -- is no key frame: every sample has sensor data. The entity attaches its LiDARs only for the policy'srenderer_data, sodriver.lidarswithdriver.send_renderer_data: falseis refused before the run starts: the dataset would have no point cloud (#2208). - Any other ego (Autoware, the autopilot, ROS 2) gets the same sensors put on it by the recorder, at the same rate; the recorder's writer encodes their frames.
The dataset is written by autoware-carla-scenario-t4, a Rust writer with a
PyO3 binding (autoware_carla_scenario_t4), installed
with the t4 extra. A run asking for a dataset without it is refused before
anything starts (exit code 2), saying how to install it. Recording is best
effort, like the trajectory log: an error is logged and
the dataset stops, it never fails the scenario.
Settings¶
| Parameter | Type | Default | Description |
|---|---|---|---|
t4.enabled |
bool | false |
Write each run as {ScenarioName}.t4/ |
t4.max_distance_m |
float or null | 100.0 |
Annotate the vehicles and pedestrians within this distance of the ego, a positive, finite distance (another value is refused); null annotates all of them. An actor parked out of the world (a hidden spawn, a trajectory entity before its start) is never annotated; revealed, it is annotated from the frame it appears |
t4.workers |
int | 4 |
Threads writing the files, counting the boxes' points and encoding the recorder's own camera frames; a positive integer (true or 2.5, even 2.0, is refused rather than read as a count) |
driver.lidars, driver.cameras |
list | The sensors and their mounts | |
driver.policy_timestep_s |
float | 0.1 |
Seconds from one key frame to the next |
driver.image_quality |
int | 90 |
The JPEG quality of the camera frames |
driver.rear_axle_offset_m |
float or null | null |
Where the ego's base_link is (null: from the vehicle's wheels, else its box), unless the ego puts it elsewhere: a ROS 2 ego's rear_axle_offset_m, an Autoware ego's base_link_offset_m. The run's trajectory log (its rear_axle record, so its T4Scenes) takes the same offset, by the same rule, so the two put base_link in one place |
A dataset needs at least one LiDAR or camera; without a LiDAR the first camera's frames are the key frames. A batch takes the first case's settings, as it does its traffic backend and its ODD -- a SLURM batch too, whichever task runs a case (SLURM).
Performance¶
The dataset is written on every key frame of every run, so the run's own thread does as little as it can:
- the CARLA states -- the ego's and every road user's -- go to the writer as one array, and the writer turns them into poses and boxes in the map frame;
- a sweep is copied once, into a buffer the writer reuses from frame to frame; nothing of it is touched on the run's thread;
- the writer's threads write the sweeps, count each box's points (on a grid over the boxes, so a point is tested against the few boxes around it), write the JPEG frames and encode the recorder's own camera frames, while the run goes on; at most a bounded number of files wait for them.
A key frame with two 60,000-point sweeps, a JPEG frame and 50 road users costs the run's thread about 0.3 ms.
Layout¶
{ScenarioName}.t4/ the dataset (its id is the directory's name)
0/ its one version
annotation/*.json the tables
data/LIDAR_CONCAT/00000.pcd.bin
data/CAMERA_FRONT_WIDE_120FOV/00000.jpg
map/lanelet2_map.osm the Lanelet2 map the run was on, when it had one
map/map_projector_info.yaml
status.json
| Table | What it holds |
|---|---|
scene, log |
One: the run's file stem, the scenario class, the ego's blueprint, the map |
sample |
One per key frame, linked frame to frame |
ego_pose |
The ego's pose at each key frame, with its twist and acceleration in its own frame |
sensor, calibrated_sensor |
LIDAR_CONCAT at the ego's origin, and each camera (its logical id upper-cased) with its mount and pinhole (or its lens, see Frames) |
sample_data |
Each sweep (.pcd.bin) and camera frame (.jpg) |
instance, sample_annotation |
Each road user, and its box at each key frame, with its velocity and num_lidar_pts |
category |
car, truck, bus, motorcycle, bicycle, pedestrian (from the blueprint's base_type), with a null index |
visibility |
One level, unavailable: the boxes are ground truth, their occlusion is not measured |
vehicle_state |
The ego's pedals and steering (in %), its gear and speed, at each key frame |
attribute, map |
No attributes; the map record names map/lanelet2_map.osm |
A sweep is T4's default layout: float32 (x, y, z, intensity, ring = -1)
per point, every LiDAR's points of the key frame together, in the ego's frame.
The intensity is on T4's 0-255 scale, as Autoware's point clouds and the
T4Scenes' sweeps: CARLA's [0, 1] times 255
(t4.INTENSITY_SCALE), applied once as the sweep is written, whether the
recorder's own LiDAR took it or a driver policy's ego fed it.
Timestamps are Unix microseconds: the wall-clock time the run's clock
started, plus the simulated time since, so key frames are exactly one policy
step apart.
Frames¶
- The map frame is Autoware's: the Lanelet2 map's (CARLA's world frame
with
yflipped, plus the map's MGRS and elevation offsets; see Trajectory Log). Without a Lanelet2 map loaded it is CARLA's withyflipped. - The ego frame (
base_link) is at its rear axle, x forward, y left, z up: the driver policy's rig frame.LIDAR_CONCAT's points are in it. - A camera's frame is the optical one (x right, y down, z forward); its
intrinsics are the pinhole of its
fovand resolution, with no distortion. A camera with alensof its own (F-theta) is written as OpenCV's rational model fitted to the lens: its focal lengths and principal point, andcamera_distortion[k1, k2, p1, p2, k3, k4, k5, k6], T4 having no F-theta model. The fit is logged; for Cosmos3's front wide camera it is within 0.05 px out to the image's corners. t4-devkit v0.8.0's own projection (view_points) applies the distortion before dividing by depth, so project with OpenCV (cv2.projectPoints). A lens OpenCV cannot hold -- one skewed in its rows (linear_cde'se!= 0), or one no rational model fits -- is refused before the run starts, as a config error naming the camera (t4: the lens of camera '<logical_id>' cannot be written to a T4 dataset: ...), rather than leaving every run without a dataset. - A box is its centre and orientation in the map frame, and its size
(width, length, height); velocities are in the map frame too. A pedestrian's velocity is measured from where it stood at the key frame before (one walking on CARLA's walker controller reports 0); every other road user's is the one CARLA reports.
Reading it¶
from t4_devkit import T4Devkit
from t4_devkit.dataclass import LidarPointCloud
t4 = T4Devkit("outputs/2026-10-22/10-00-00/CutInScenario.t4")
sample = t4.sample[0]
path, boxes, _ = t4.get_sample_data(sample.data["LIDAR_CONCAT"]) # boxes in base_link
points = LidarPointCloud.from_file(path).points # (4, N): x, y, z, intensity
t4sanity passes every check of severity ERROR. It warns of two things a
simulated run does not have: the ROS bag (input_bag/, STR005) and the point
cloud map (map/pointcloud_map.pcd, STR009). The tests read every dataset they
write back with t4-devkit v0.8.0, its latest release, and run its sanity
checker on it (test_t4_dataset.py). t4-devkit supports Python 3.10-3.12, and
its sanity checker needs colorama, which it does not declare.
The writer¶
autoware_carla_scenario_t4.T4Writer writes any scene, not only a CARLA run's:
import numpy as np
from autoware_carla_scenario_t4 import T4Writer
w = T4Writer("MyScene.t4/0", scene_name="MyScene")
w.add_sensor("LIDAR_CONCAT", "lidar", (0.0, 0.0, 0.0), (1.0, 0.0, 0.0, 0.0))
w.begin_frame(1_700_000_000_000_000, (0.0, 0.0, 0.0), (1.0, 0.0, 0.0, 0.0))
w.add_annotation(7, "car", (10.0, 0.0, 0.75), (1.8, 4.0, 1.5), (1.0, 0.0, 0.0, 0.0))
w.add_lidar("LIDAR_CONCAT", np.zeros((100, 4), dtype=np.float32))
w.end_frame()
print(w.finish())
Poses are right-handed and quaternions (w, x, y, z); begin_frame_carla,
register_actor and add_carla_boxes take CARLA's states instead. Tokens are
derived from token_seed (the scene name by default), so the same frames
written twice are the same dataset. Its sources, its own tests and its
releases are hakuturu583/autoware_carla_scenario_t4.