Skip to content

Metrics

A run reads two kinds of quantity off the world, under keys the framework fixes:

Reads Example An ODD maps onto it Watched by
Measure (Scenario measures) A condition the ego drives in: what the scenario sets up, while it drives vehicle_ahead_gap_m Yes (measure: in a binding file) Coverage; a sweep draws a scenario's parameters from it
Metric (this page) How the ego drove: an outcome, computed once the run is over collision_at_fault No MetricCondition

An ODD describes the conditions a system is built to operate in, whatever it then does there. A metric reads what it did, so it has no place in one: were an ODD to map onto offroad, the ego's own driving would decide whether a run is inside the domain it is tested in, and an ODD excluding offroad would report a run that left the road as outside the ODD rather than as failed. scenario_measure("offroad"), measure: offroad in a binding file and register_measure("offroad", ...) are all refused.

The built-in metrics

They are the per-timestep driving-quality metrics of NVIDIA's AlpaSim (src/eval/src/eval/scorers), under AlpaSim's own names so results read the same in both, computed from CARLA's actors and map (autoware_carla_scenario.metrics, BUILT_IN_METRICS). An event reads 1 while it holds and 0 while it does not; the collisions, offroad and wrong_lane (Metric.boolean) are yes/no events, which the trajectory log writes as true and false (and the Rerun recording draws as states); every metric of the ego reads nothing without an ego, and those of the road nothing without a map (importance_sampling_weight, the run's weight, reads it with an ego or without). A metric reads one frame; AlpaSim folds a run's frames into one value with the aggregation in the last column (Metric.aggregation).

Key Unit Reads AlpaSim folds with
collision_any The ego's bounding box overlaps another vehicle's or a pedestrian's max
collision_front ... and the other overlaps the ego's front edge max
collision_rear ... its rear edge, and not its front max
collision_lateral ... neither edge: a side swipe max
collision_at_fault A front or a lateral collision max
offroad A corner of the ego's box is on no driving lane max
offroad_or_collision Off the road, or any collision max
offroad_or_collision_at_fault Off the road, or a collision at fault max
wrong_lane The ego heads more than 120 degrees away from the driving lane under its centre (and, where junction lanes are under it, from every one of them); nothing off the road max
min_distance_to_obstacle_m m Box to box, to the nearest other vehicle or pedestrian; 0 touching, nothing when the ego is alone min
min_distance_to_lane_boundary_m m From the ego's box to the edge of the driving lanes around it (lanes beside its own count while their near edge is within 6 m of its centre); 0 off the road min
importance_sampling_weight ratio Not AlpaSim's: the run's importance-sampling weight so far, the product of NADE's p/q and (1-p)/(1-q) (AdversarialTrafficAction); 1 without NADE. Weight a batch of runs by its last value to estimate the naturalistic rate of what they count last
lateral_reversals count Not AlpaSim's: how many times so far the ego's drift across its lane turned back by 0.1 m or more (Weaving) last
lateral_wobble_m m Not AlpaSim's: how far the ego strayed across its lane about its course over the last 4 s, RMS (Weaving) max
steering_reversals count Not AlpaSim's: how many times so far the steering applied to the ego turned back by 0.05 of full lock or more (Weaving) last
turn_indicator_changes count Not AlpaSim's: how many times so far the ego's turn indicators changed, between off, left, right and hazard (Weaving) last

Where AlpaSim's geometry is not CARLA's, the reading follows CARLA: AlpaSim asks whether the union of the lane polygons around the ego covers its box, CARLA whether a point is on a driving lane, so offroad asks it of the box's corners; the lane boundary takes the lanes beside the ego's as parallel strips as wide as CARLA says they are there, an approximation through a junction.

CARLA's point query (get_waypoint(point, project_to_road=False, lane_type=Driving)) misses a band about 0.2 m wide inside a driving lane, next to its boundary with the lane beside it (#1997). Where it finds no lane, the point is projected to the nearest driving lane instead, and is on the road when it lies within that lane's half width, or the half width of the driving lane to its left or right, plus 0.1 m, across from the lane's waypoint (within 0.3 m along it; the lane is followed onto the lanes before or after it when the point is further along) and within 2.5 m of its height. So a lane change reads on the road throughout, and min_distance_to_lane_boundary_m and wrong_lane read the lane in the band too. A point beyond the outer lane's edge -- on a shoulder, a sidewalk, off the map -- is further out than that, and stays off. The tolerances cut both ways: a point up to 0.1 m outside the outer lane's edge, or up to about 0.3 m past the end of a lane that does not continue (a dead end), reads on the road. What remains is the lanes' own gaps: a corner further over the end of a road where its lane does not continue, or between a road's lane and a junction's connecting lane that is narrower there.

In a junction, the connecting lanes overlap each other and the ends of the roads around it, and the lane CARLA finds under the ego may be another approach's, at any angle to its own. So where it heads more than 120 degrees away from that lane, wrong_lane also looks at every driving lane in a junction whose centre line passes within 1 m of the ego's centre (and within the lane's half width plus 0.1 m), and reads 1 only when it heads more than 120 degrees away from all of them. This runs wherever the ego is, so it also covers the end of a road that a junction's lanes reach over; on a road away from any junction lane, the lane found is the only one asked. The junction lanes are sampled every metre, once per world: in a run, once; in a replay (evaluate_metrics), once per call.

The price is a blind spot: the ego driving the wrong way along a junction's connecting lane reads 0 wherever another junction lane heading the ego's way passes within that metre of its centre -- where an opposing connector overlaps its own. Replaying the ego backwards along every connecting lane of the fixture map (430 lanes, every 0.5 m), 13.3% of the frames read 0; driven the right way along their centre lines, 2 frames of one lane read 1.

A road user whose box reads no extent (a box of no area: a point or a segment) holds only the points on it: it meets the ego when it is on or inside the ego's box, and is measured to as a point, not as covering everything around it.

Boxes are those of vehicles (vehicle.*) and pedestrians (walker.*); props are not road users. collision_* reads the boxes, not CARLA's collision sensor (CollisionCondition): it sees a collision as AlpaSim does, and tells the front from the rear.

The boxes are compared on the ground, as AlpaSim's are, on the ego's level of road. A road user whose box is more than 1 m above or below the ego's (each box from its centre's height, the actor's origin plus the box's offset, less and plus its half height) is on a bridge over the ego or in an underpass beneath it: it is in no collision with the ego, and min_distance_to_obstacle_m measures to it across the height between the boxes too -- a car on a bridge 8 m over the ego, their boxes 1.6 m tall, is 6.4 m from it, not 0. The CommonRoad metrics leave it out. The metre lets a road user just up or down a slope from the ego, whose box is level here, still meet it. The lanes -- offroad, wrong_lane, min_distance_to_lane_boundary_m and the weaving drift -- are asked for at the height of the ego, as OffRoadFractionCondition asks at the entity's, so that where roads cross at different heights the lane is the one the ego drives on, not the road under or over it.

Weaving

Three metrics read how the ego weaves from side to side, and a fourth how often its turn indicators changed. Unlike the others they read the run so far, not one frame: the replay of the run's recorded frames (How metrics are computed) hands the ego's motion to autoware_carla_scenario.metrics every frame, in order (track_ego_motion), and they read what it has gathered. The live run does none of this work: it tracks only how far the ego has got (ego_progress(), which tells a run out of time apart), which the replay leaves as it is.

The ego's drift is how far it has moved across its lanes since the run began: its velocity across the lane under its centre, summed over time. The lane's heading is taken anew every frame, so following a bend is not a drift, and moving to the next lane is a drift of a lane's width with no jump at the lane line. Off the driving lanes the drift holds until the ego is back on one.

  • lateral_reversals counts the times the drift turned back by 0.1 m or more: it went right, then came back left by 0.1 m or more. This is the reversal count of driver-workload studies (Markkula and Engström, 2006). A lane change is one drift, however long, and adds nothing; a weave adds two a cycle. Under 0.1 m it is the noise of holding a lane.
  • lateral_wobble_m is how far the drift strayed about its course over the last 4 s: the RMS about the straight line that fits it best there. A weave of 0.25 Hz or faster turns back within the window and reads nearly in full (about 92% at 0.5 Hz); a lane change, slower, is mostly the course, and adds about a tenth of the lane's width while it lasts. It reads nothing before the third frame.
  • In a junction, whose lanes overlap, the drift holds, as it does off the driving lanes; a lane the other way is crossed the ego's way.
  • steering_reversals counts the times the steering applied to the ego (get_control().steer, 1 the vehicle's full lock) turned back by 0.05 or more, about 3.5 degrees at the wheels of a car that locks at 70. It sees the controller's weaving before the vehicle shows it. Closed loop, a driver policy's controller moves its steer by a few hundredths while it holds a lane or a bend, which the gap leaves out; sawing through a turn is a tenth or more.

turn_indicator_changes counts the changes of the indicators the ego shows (its CARLA light state: off, left, right, hazard), from the first frame on: a turn signalled and cancelled is two. It is counted apart from the steering, so an indicator that flickers through a turn and a steering that saws there each read on their own count. A vehicle's light state holds while an indicator is commanded, so its blinking is not a change.

The counts are of the run so far, so a longer run reads more: compare runs of one scenario, or fail on a count, not on a rate:

self.register_fail_condition(
    MetricCondition(LATERAL_REVERSALS, ComparisonRule.GREATER_THAN, 6, label="weaving")
)
self.register_fail_condition(
    MetricCondition(LATERAL_WOBBLE_M, ComparisonRule.GREATER_THAN, 0.5, label="weaving")
)

CommonRoad metrics

Fourteen more read how critical the frame is, with TU Munich's CommonRoad tools: CriMe's criticality measures and CommonRoad-Reach's reachable set (COMMONROAD_METRICS). Each is read on the ego against the road users around it, every one going on as it goes -- its velocity and acceleration held, a braking one stopping -- over a 3 s horizon; the CriMe measures against other road users are the most critical over them, as CriMe folds them. CommonRoad's scenario is planar, so the road users on another level of road than the ego's -- more than 1 m above or below its box, and higher or lower than a 10 % grade climbs over the distance between them, so a lead on the same road up a slope stays in -- are left out of it, as they are of the collisions.

Key Unit Reads Folds with
crime_ttc_s s Until the ego hits a road user ahead in its lane, both holding their acceleration until they stop (CriMe TTC); inf when they never meet, however far off min
crime_thw_s s Until the ego reaches where the road user ahead is now (CriMe THW) min
crime_hw_m m From the ego's front to the rear of the road user ahead, along the lane (CriMe HW) min
crime_ttb_s s Left before the ego must brake at full to avoid a collision (CriMe TTB) min
crime_ttce_s s Until the ego and a road user come closest (CriMe TTCE) min
crime_dce_m m Between the ego and a road user when they come closest; 0 when they would collide (CriMe DCE) min
crime_a_long_req_mps2 m/s^2 The longitudinal acceleration the ego needs not to hit the road user ahead, negative to brake: the least deceleration that avoids it, counting a contact within the prediction horizon (horizon_s, 3 s) after the ego has stopped (CriMe ALongReq); -inf when braking alone cannot avoid a contact within it min
crime_a_lat_req_mps2 m/s^2 The lateral acceleration the ego needs to steer clear of a road user (CriMe ALatReq) max
crime_btn ratio The deceleration the ego needs not to hit the road user ahead over the most it can brake (11.5 m/s^2); above 1 braking alone no longer avoids it, inf when no braking avoids a contact within the prediction horizon (horizon_s, 3 s) (CriMe BTN) max
crime_stn ratio The lateral acceleration the ego needs to steer clear of a road user over the most it can steer (8 m/s^2); above 1 steering alone no longer avoids it (CriMe STN) max
crime_ttr_s s How long the ego can go on as it goes before no evasive manoeuvre -- braking, kick-down, steering left or right, each simulated -- avoids the collision (CriMe TTR); at or under WTTR; inf when it hits no one, 0 when none does even now. Evaluated every 0.5 s by default min
crime_wttr_s s How long the ego can go on as it goes before no way it can go is free of collisions to the end of the horizon, by CommonRoad-Reach's reachable set (CriMe WTTR); inf when it hits no one, 0 when no way is left even now. Every road user is predicted in a straight line along its velocity, so on a bend it can read inf with a lead closing fast (see the straight-line predictions below) min
reach_drivable_area_m2 m^2 The road the ego can still reach without a collision at the end of the horizon (CommonRoad-Reach); 0 when every way it can go meets a road user: a collision it can no longer avoid min
reach_drivable_area_sum_m2 m^2 The same, summed over the horizon's steps (CriMe's DA) min

A measure with no one to measure from (no road user ahead for TTC) reads nothing, and so does every one while the ego is on no lanelet. A run failed on one is a safety_margin failure. They are watched like any other metric:

from autoware_carla_scenario import ComparisonRule, MetricCondition
from autoware_carla_scenario.metrics import CRIME_TTC_S, REACH_DRIVABLE_AREA_M2

self.register_fail_condition(
    MetricCondition(CRIME_TTC_S, ComparisonRule.LESS_THAN, 1.5, label="TTC under 1.5 s")
)
self.register_fail_condition(
    MetricCondition(
        REACH_DRIVABLE_AREA_M2,
        ComparisonRule.LESS_THAN_OR_EQUAL,
        0.0,
        label="no way out",
    )
)

CommonRoad's packages cannot share the framework's environment: commonroad-io pins protobuf 3.20, where the framework's gRPC code needs 4 or later, and CommonRoad-Reach builds for CPython 3.10 and 3.11 only. They run in worker processes of their own, the commonroad_metrics project at the root of the repository (commonroad_metrics/README.md), spoken to in JSON lines, on the frames a run recorded every 0.1 s of simulated time (How metrics are computed), the frames shared among several workers at once (four, or fewer on a machine with fewer cores). The workers read the world's map as CommonRoad: roadgen (>= 0.42.0, the map extra) converts the world's OpenDRIVE once per machine, cached under ~/.cache/autoware_carla_scenario/commonroad, and stay up for the next run on the map. roadgen validates the map before it writes it and has no option to skip that: a map it refuses fails the run with "roadgen could not convert ... to CommonRoad". The nishishinjuku fixture map (data/nishishinjuku_carla.xodr) is one (#2065): roadgen reports 520 issues on it, nearly all of them in the OpenDRIVE itself -- road links one way only, lane links and junction connections joining lanes that do not meet (as far as 66 m apart, where a junction names a connecting road that does not touch its incoming road), and reference lines of linked roads up to 6.6 m apart -- and 18 lanes whose ends are 2 to 7 cm narrower or wider than their boundaries. Until the map is regenerated, no CommonRoad metric can be judged on it. A worker evaluates each frame moved to the ego and turned so the lane under it runs along +x: CriMe projects speeds onto a lane with the cosine of the lane's own heading, which is the speed along the lane only there. CriMe 0.4 stores a lane's headings in [0, 2pi) and interpolates between them as plain numbers, so on a bend, where the headings either side of the ego straddle 0, it read a heading as much as 2pi off -- TTC inf and ALongReq 0 with a lead closing fast; the worker interpolates them unwrapped, for every CriMe measure that reads a lane's heading. CriMe 0.4 also drops the sign of the other road user's velocity and acceleration along the lane, reading a braking lead as speeding up and a car coming the other way in the ego's lane as going its way; the worker reads TTC and ALongReq -- and so BTN, and ALatReq and STN, which take TTC -- with their sign, the ego's velocity too. A road user braking to a stop then stops and stays there, as the worker predicts every road user, rather than reversing as CriMe's constant acceleration would carry it on: TTC is when the ego reaches it, each holding its acceleration until it stops, so a car creeping to a halt ahead of a stopped ego is no conflict. ALongReq is the least constant deceleration that keeps the ego off the road user ahead in that model. It is CriMe's (the deceleration that matches the other's speed as the ego reaches it) where the gap closes on a road user going the ego's way that does not stop first; it departs from CriMe for an opening gap, which needs 0 (CriMe reads the relative velocity squared over twice the gap) or, for a lead braking to a stop, the deceleration that stops the ego behind where it stops; and for a car coming the other way, the ego must stop far enough back that the car, going on as it goes until it stops, does not reach it within the horizon after the ego has stopped: with the ego at 10 m/s and a car 40 m ahead coming at 10 m/s, ALongReq is -15 m/s^2 (stopped 3.3 m on after 0.67 s, the car 3 s further off). A contact counts while the ego is moving and within the prediction horizon (horizon_s) after it has stopped: a pedestrian crossing the lane a little towards the ego, or a car rolling back, that would reach a standing ego only after the horizon needs no more than the ego stopping short of it. When no braking avoids a contact within the horizon -- a car coming at the ego that does not stop in time -- ALongReq is -inf and BTN inf. TTC has no horizon, as CriMe's: it is when they would meet, however far off. A road user behind the ego gives no ALongReq. CriMe reads a road user whose box overlaps the ego's -- a negative bumper-to-bumper headway, a collision -- as one behind (HW, THW and TTC inf, ALongReq and BTN 0); the worker reads one overlapping it with its centre ahead of the ego's, or level with it, as touching: HW and THW 0 and, while the ego closes on it, TTC 0, ALongReq -inf, BTN inf, and ALatReq and STN inf, so a condition such as crime_ttc_s < 1 holds on the frames of a collision. One whose centre is behind the ego's, or one beside the ego whose box does not touch it (CriMe counts a road user sharing any lanelet with the ego as in its lane, a car straddling a lane line or the ego changing lanes), still reads HW inf. The lanelets the worker keeps are those within radius_m of the ego anywhere along their centre line, and always those it is on, so a long straight lanelet stays in from its middle. The reachable set (the drivable area, WTTR) is that of the ego's own box: CommonRoad-Reach's discs and the radius it inflates the other road users by are derived from the ego's length and width, not its default vehicle's; a reversing ego reaches backwards. On a frame of ten road users the eight CriMe measures from TTC to ALatReq take about 95 ms together, BTN and STN 170 ms more, the drivable area about 155 ms (over 2 s instead of 3, 45 ms), and WTTR about 110 ms on a frame with a collision ahead, next to nothing on one without. TTR simulates the ego's manoeuvres: about 2 s a frame with a collision ahead.

The worker predicts every road user -- the ego's own trajectory too, though not its reachable set -- in a straight line along its velocity, each holding its acceleration until it stops; no prediction follows its lane. TTC, HW, THW and ALongReq (and BTN, ALatReq and STN) measure along the lane and follow a bend, but what is read from where the predictions meet does not: WTTR, the drivable area, TTCE and DCE. On a bend the two straight paths part sideways before the road users would touch: with the ego at 15 m/s 30 m behind a lead at 5 m/s round a 300 m bend, TTC is 2.5 s but WTTR reads inf and DCE 0.7 m, where on a straight road WTTR is 2.2 s and DCE 0. A WTTR of inf on a curved road is not evidence that nothing closes in; watch TTC there.

A slow metric is evaluated less often than every period_s (0.1 s): at its own period, metric_periods_s, by default 0.5 s for crime_ttr_s (DEFAULT_METRIC_PERIODS_S). A frame is asked only the metrics due on it, and a metric condition counts each NG sample until the next (the last until the run's end), so max_ng_seconds is still simulated time. Set the periods of any CommonRoad metric; the dictionary given replaces the one set, so start from it to keep the others':

from autoware_carla_scenario.commonroad import (
    commonroad_settings,
    configure_commonroad_metrics,
)

configure_commonroad_metrics(
    metric_periods_s={**commonroad_settings().metric_periods_s, "crime_wttr_s": 0.3},
)

A CommonRoad metric is computed only when a metric condition reads it, so a heavy one costs nothing until a scenario judges on it. The heavy ones (HEAVY_METRICS, crime_ttr_s today) can also be turned off for a run that should not wait on them, a quick iteration on a scenario that judges on TTR: a metric condition on one is then not judged -- the run neither fails nor passes on it, and a warning says so. They are on by default; turn them off with metrics.heavy=false, $AUTOWARE_CARLA_SCENARIO_HEAVY_METRICS=0, or configure_commonroad_metrics(heavy_metrics=False):

uv run scenario scenario=intersection_passing/straight metrics.heavy=false
AUTOWARE_CARLA_SCENARIO_HEAVY_METRICS=0 uv run scenario scenario=intersection_passing/straight

The scenario command (a single run, a batch, a sweep, and the SLURM tasks alike) decides in this order:

  1. metrics.heavy when a config sets it to true or false (the first case's of a batch that does): it overrides the environment variable. Besides true and false, it takes their spellings as the run's switches do ("false", "off", "no", 0 are false); anything else (ture, 2) is refused, naming metrics.heavy, in any case of a batch before the first one runs;
  2. otherwise $AUTOWARE_CARLA_SCENARIO_HEAVY_METRICS: off when it is 0, false, off or no;
  3. otherwise on.

A metrics block that is absent or null sets nothing; one that is not a mapping (metrics: false, 0, '', a list) is refused, naming metrics and the value, rather than read as unset.

The shipped config.yaml leaves metrics.heavy null, so the environment variable decides unless the command line or a scenario's config says otherwise. The command leaves your variable as it is and writes what it decided (1 or 0) to $AUTOWARE_CARLA_SCENARIO_HEAVY_METRICS_DECIDED (HEAVY_METRICS_DECIDED_ENV), which CommonRoadSettings reads before $AUTOWARE_CARLA_SCENARIO_HEAVY_METRICS: it carries the decision to the runs' own processes and to the SLURM jobs (launcher.env_passthrough's AUTOWARE_* passes it; a SLURM task runs with it rather than decide again from its own cases, which are only some of the batch's). Set by the command, not by you: the command never reads it to decide, and overwrites it each time it decides. A batch decides once, for all its cases; the cases of a Hydra --multirun (scenario=a,b) each decide for themselves, from the variable as you set it, not as an earlier case's metrics.heavy decided.

The workers are started with, in order, the command in $AUTOWARE_CARLA_SCENARIO_COMMONROAD_WORKER, commonroad-metrics-worker on the PATH, or, in a checkout, uv run --project commonroad_metrics commonroad-metrics-worker (which builds its environment the first time). A run judged on a CommonRoad metric whose workers cannot be started fails with CommonRoadWorkerError rather than pass on a metric that read nothing. So does a run whose workers do not know a metric it judges on (a commonroad-metrics-worker on the PATH older than the framework: its init answer lists the metrics it knows), and a run on which a metric fails in every frame evaluated: a fail condition on it would otherwise pass without having judged anything. A metric that fails only in some frames reads nothing in those, with a warning, as one with no one to measure from does. "Every frame" counts each frame a worker was asked: one in which the ego is on no lanelet reads nothing without failing, which the worker's answer does not tell apart from a frame with no one to measure from, so a metric that fails in every frame on the map but reads nothing off it is not caught. A worker that fails during a run (exits, answers out of protocol -- an answer that leaves a metric out, or calls it unknown, or gives a value that is not a number or null, included -- or does not answer in time) fails that run the same way, and takes the pool down with it: the next run starts fresh workers rather than reuse a dead one, as does a run that finds a pooled worker has exited. When one of the workers cannot be started, those that did are stopped.

The metrics are evaluated once the run is over, outside the run's simulated-time limits, so a worker is given wall-clock time instead: one that has not answered a frame (all the metrics due on it) within request_timeout_s, 120 s by default, is killed and the run fails with CommonRoadWorkerError, rather than hold up the batch after it forever. That is many times what the slowest metric takes on a frame (TTR, about 2 s). A worker has start_timeout_s, 1800 s by default, to start and read the map: the first uv run builds its environment meanwhile. Either can be None to wait as long as it takes.

configure_commonroad_metrics() (in autoware_carla_scenario.commonroad) changes the period (period_s, and metric_periods_s for each metric), the number of workers (workers, at least 1), the time step (dt), the horizon (horizon_s), the radius (radius_m), CommonRoad-Reach's settings, the command and the timeouts (request_timeout_s, start_timeout_s). The periods, dt, horizon_s, radius_m and the timeouts (unless None) must be positive, finite numbers -- a bool or a string is refused --, heavy_metrics is a bool (a NumPy one too; None is the default) -- not the string "false", which would leave the heavy metrics on --, cache_dir is None or a path (a str is taken as one), metric_periods_s is None (the default periods) or a mapping, reach_settings is None (no overrides) or a mapping of sections to mappings ({"reachable_set": {"size_grid": 0.3}}) whose section and key names are strings and whose values JSON can carry (finite numbers; not NumPy's float32), and command is None (find the worker) or a non-empty tuple or list of non-empty strings (not a str): a bad value raises ValueError, naming the field, when the settings are built, not later in a run. A section or key CommonRoad-Reach's configuration does not have (planning.size_grid for reachable_set.size_grid) is refused by the worker as it starts, before the first frame. A timeout far past what poll() waits at once (about 25 days) is waited in pieces.

AlpaSim's other metrics need more than the world, and are not metrics here:

AlpaSim metric Needs
progress, progress_rel, dist_to_gt_trajectory, dist_to_gt_location, gt_dist_traveled_m The logged ground-truth trajectory of the ego
min_ade@*, plan_deviation, open_loop_collision The driver's planned trajectories
safety_monitor_triggered The driver's safety monitor
img_is_black A camera image
dist_traveled_m, duration_frac_20s, avg_dist_between_incidents The run, not a frame

Judging a run on a metric

MetricCondition(metric, rule, value, tolerance, *, max_ng_seconds, label) is a fail condition judged once the run is over. A frame is NG when the metric satisfies the comparison; the rule defaults to GREATER_THAN_OR_EQUAL and the value to 1, so an event metric on its own reads "it happens". The run fails when its NG frames add up to more than max_ng_seconds of simulated time -- with the default 0, at its first NG frame -- at the time they do. "More than" means by over 1e-9 s, the tolerance the other time conditions use: the clock is a running sum of ticks, so a metric NG for exactly max_ng_seconds can add up to a hair over it, and that holds. Each NG frame counts until the next frame evaluated, and the last until the run's end, its last recorded frame: a metric evaluated every tick reads its last sample then, so that one counts nothing, while one sampled at a coarser period (crime_ttr_s, the CommonRoad metrics) counts its last sample for the time left to the end. A frame the metric reads nothing in is not NG.

from autoware_carla_scenario import ComparisonRule, MetricCondition
from autoware_carla_scenario.metrics import (
    COLLISION_AT_FAULT,
    CRIME_TTC_S,
    MIN_DISTANCE_TO_OBSTACLE_M,
    OFFROAD,
)


class MyScenario(BaseScenario):
    def setup(self) -> None:
        ...
        self.register_fail_condition(
            MetricCondition(COLLISION_AT_FAULT, label="collision at fault")
        )
        # Off the road for more than a second, in all.
        self.register_fail_condition(
            MetricCondition(OFFROAD, max_ng_seconds=1.0, label="off the road")
        )
        self.register_fail_condition(
            MetricCondition(
                MIN_DISTANCE_TO_OBSTACLE_M,
                ComparisonRule.LESS_THAN,
                0.5,
                label="closer than half a metre",
            )
        )
        self.register_fail_condition(
            MetricCondition(
                CRIME_TTC_S,
                ComparisonRule.LESS_THAN,
                1.5,
                max_ng_seconds=0.3,
                label="TTC under 1.5 s",
            )
        )

Judged after the run, a metric condition neither ends a run early nor composes: PersistentCondition, NotCondition, AndCondition, OrCondition and StickyCondition refuse one (max_ng_seconds is how long it must hold), and so does register_pass_condition. A run that passed, or failed later than the metric did, fails on the metric: the earliest wins, its failure records the condition, when and what it found. Every metric condition's status in the result says what it found: ng_seconds, when it first went NG (first_ng_at) and its most critical value (worst; null when it is not finite, a CriMe BTN of infinity say: the result files are strict JSON, which has no NaN or Infinity, and any value of a result that is not finite is written as null). In the Scenario Editor it is the Metric card, under World, with its Max NG seconds.

A run a metric condition fails is a failure of what the metric reads (Why a run failed): a collision metric collision, offroad and min_distance_to_lane_boundary_m lane_keeping, wrong_lane traffic_rule, min_distance_to_obstacle_m and the CommonRoad metrics safety_margin, and the weaving metrics and turn_indicator_changes stability. offroad_or_collision and its at-fault twin read two things and say neither, nor does the importance-sampling weight: register a condition on them with its category.

How metrics are computed

A run with a metric condition records its road users every tick -- each vehicle's and walker's place, velocity, acceleration and box, the ego's steering and light state, the importance-sampling weight (autoware_carla_scenario.metric_batch.FrameRecorder) -- and nothing more: the simulation never waits on a metric. When the run is over, while its CARLA map is still loaded, the frames are replayed: AlpaSim's metrics and the run's own every tick, in order, by the code that reads them off a live world, on a stand-in world answering with the recorded actors and the run's map; the CommonRoad metrics every 0.1 s, by the workers at once. A metric that raises in a frame reads nothing there, the first exception logged with a warning; one that raises in every frame evaluated (a map whose get_waypoint raises, say) fails the run with MetricEvaluationError (autoware_carla_scenario.metric_batch), as a CommonRoad metric failing in every frame does with CommonRoadWorkerError (below): a fail condition on it would otherwise pass without having judged anything. Then each metric condition judges its metric's series. The metrics, and each metric condition's NG frames, go into the run's trajectory log (metric and condition records) and so into its Rerun recording (Metrics in the recording).

To read an AlpaSim metric off a live world yourself -- in a probe, a test -- read_metric(key, world) returns its value in the frame, or None; a CommonRoad metric reads nothing there, nor do the ones that read the run so far (Weaving, and importance_sampling_weight), which only the replay of the recorded frames gathers. evaluate_metrics(frames, carla_map, keys) computes any of them over recorded frames, on carla_map, whichever map the run's MapManager holds.

Off the road as a fail condition

offroad is a measurement, and a strict one: it reads 1 as soon as any one corner of the ego's box is on no driving lane. A corner over a gap between a road's lane and a junction's connecting lane, or a small offset between the OpenDRIVE lanes and where the vehicle drives, counts though the vehicle is in its lane, so MetricCondition(OFFROAD, ...) fails such runs (unless its max_ng_seconds lets them through). To judge a run by how much of the box is off instead, use OffRoadFractionCondition(fraction, entity_name, *, label), a condition rather than a metric:

from autoware_carla_scenario import OffRoadFractionCondition

self.register_fail_condition(OffRoadFractionCondition(0.3, label="offroad"))

Every tick it samples a 5 x 3 grid of points over the entity's bounding box (the ego's unless entity_name names another): 5 evenly spaced along its length and 3 across its width, the box's edges and corners included. A point is off when no driving lane is under it, as the metrics ask it: when get_waypoint(point, project_to_road=False, lane_type=carla.LaneType.Driving) finds no lane there, and the point is not within the half width (plus 0.1 m) of the lane it projects to or one beside it either (above). It is asked at the height of the entity, so that where roads cross at different heights the lane is the one it is on. It fails once at least fraction (in (0, 1]) of the 15 points are off. The fraction is of points, not of area: one corner off is 1/15, the front edge 3/15, one long side 5/15. fraction=1/15 fails on any one point, the way the metric does on a corner; 0.3 fails at 5 points, so it tolerates a corner or the front edge over a lane gap and fails once a whole long side of the box is off. In the Scenario Editor it is the Off the road card, under Entity.

A scenario or a Python ODD may import any name of autoware_carla_scenario.metrics -- the keys, Metric, BUILT_IN_METRICS, is_metric() and read_metric() -- and is checked against them like any other (Static Check). A Metric is frozen: assigning to one of its fields fails the check, as it fails the run. A scenario may build its own (Metric("my_metric", read)); the check calls its read with a world and refuses one that returns anything but a float or an Optional[float].